How do I know if an AI hiring tool is screening out qualified candidates?

A weak shortlist or missed credible candidates is a signal to investigate, not proof that the tool rejected qualified people. Review the screen's role-level recommendations, then trace what your company did with them: human reviews, overrides, interviews, rejections, and hires. Add evidence that the assessment measures something connected to performance in the job. A vendor-wide result cannot answer those three questions for the role.

What should an employer ask an AI hiring vendor before renewing a screening tool?

Ask for the model and version used, the operating threshold, the role-level applicant counts and recommendation rates, and the validity evidence the vendor believes applies to the job. Your company must add its own disposition record showing how recommendations affected human review, interviews, rejections, overrides, and hires. Renewal should be based on the complete role-level record, not the vendor's portfolio result alone.

Can a hiring vendor's validation study prove the tool works for my job?

Not automatically. An external study can support the employer's record when it establishes validity, addresses fairness and other material variables, and covers jobs with substantially similar major work behaviors. The employer still has to document why that evidence applies to the role, applicant population, setting, and current use. A broad or pooled vendor study does not establish that fit by itself.

Your Hiring Vendor's Report May Be Answering the Wrong Question

Your critical-role shortlist keeps missing credible candidates. The automated screen can stop a qualified applicant before anyone on your team reviews their experience. Yet the vendor's report says the hiring system looks acceptable.

That report may be accurate. It may also be answering the wrong question.

You are deciding whether to renew or approve the screen used for one job. The report combines results from many positions, applicant pools, employers, and model configurations. It answers whether the portfolio looks acceptable in the aggregate. You need to know whether this screen should keep deciding who reaches human review for this role.

Renewal is where the evidence gap becomes your problem. Once the company authorizes the screen, it becomes part of the operating architecture for the role. It shapes who can produce interview evidence, who remains invisible, and what future hiring data the company will learn from.

If you are accountable for whether hiring produces the capability the business needs, the first question is simple. What evidence supports this screen for this job?

The report may be right at the wrong level

A vendor-wide result can help an employer monitor a portfolio. It can show broad recommendation patterns and flag areas that deserve attention.

A job-level approval asks a narrower question. The answer depends on what the model recommended for that position, what the employer did with those recommendations, and whether the procedure measures something that matters to performance in the role.

Aggregation changes what the report can show. A pooled result that does not trigger the chosen adverse-impact screen can still absorb a materially different pattern in one position. The size of the combined dataset may make the report look more authoritative while moving it farther from the decision in front of you.

This is the distinction the approval process has to preserve. Portfolio evidence describes the portfolio. A critical-role screen needs evidence that reaches the role.

One study shows what the average can conceal

Rishi Bommasani and his coauthors analyzed 4,197,168 applications to 1,746 positions at 156 employers using one vendor's game-based assessments. The records covered December 2018 through December 2022, and the racial analysis used self-reported data. The models produced recommendations that employers could use when deciding who advanced.

The researchers reported that the pooled vendor-wide racial-group analysis did not trigger the study's 0.80 adverse-impact screen.

The result changed when the researchers examined jobs separately. The position-level analysis was limited to jobs with at least 30 applicants who self-reported race. After a Benjamini-Hochberg multiple-testing correction, 82 of 772 eligible positions in the Black-applicant analysis and 47 of 883 eligible positions in the Asian-applicant analysis met the study's criteria (Bommasani et al., 2026).

The pooled result and the position findings came from the same dataset. The pooled finding did not become false. It answered a broader question and concealed conditions inside particular positions.

The paper supports a narrow conclusion. A pooled result that did not trigger the study's chosen adverse-impact screen can coexist with position-specific patterns that warrant inspection. Employer intent, legal liability, downstream hiring decisions, and predictive validity remain outside that conclusion.

For an operating executive, that boundary matters. The study gives you a reason to inspect the job. It does not give you a verdict on the screen.

A recommendation is only the first record

A screen produces a recommendation. Your company decides what that recommendation does.

The distinction disappears when the approval packet stops at the vendor's report. A “do not recommend” result may end an application automatically, send it to a lower-priority queue, trigger another review, or be overridden by a hiring manager. Those paths create different employer outcomes even when the model output is identical.

The employer therefore needs a disposition record. It should show which applicants received human review, which recommendations were overridden, who reached an interview, and who was rejected or hired. Without that record, the company knows what the tool produced but not how the hiring process used it.

The role still needs a second connection. A recommendation rate describes who the model recommended. It does not establish who advanced or whether the score identifies people who can perform the work.

Federal selection-procedure guidance reflects that distinction. 29 C.F.R. § 1607.7 permits employers to use external criterion-related validity evidence under stated conditions. The available studies must establish validity, and the jobs must involve substantially the same major work behaviors, supported by appropriate job analyses. The provision also requires attention to fairness evidence and other material variables that could affect validity. The evidence need not originate in the individual position, but the employer must document why it applies to the role and current use. Users remain responsible for compliance.

A vendor study can support the record when its evidence fits the job and the way the procedure is used. Fit has to be shown.

Match the approval record to the decision

Before the company renews or approves the screen, require one role-level packet with three separate records.

Recommendation record. Show the model and version, operating threshold, time period, applicant counts, and recommendation rates for the position under review.
Disposition record. Show what the employer did after each recommendation, including human review, overrides, interviews, rejections, and hires.
Performance-related evidence. Identify what the procedure is expected to measure or predict, the criterion-related validity evidence connecting it to that outcome, and why the supporting evidence applies to the role and current use.

A complete packet makes the basis for the decision visible. It does not by itself clear the screen. Approve or renew only when the evidence supports the intended use and relevant risks have been addressed. If the records remain incomplete or contradictory, narrow automated use, add independent human review, collect additional evidence, or defer renewal.

A small sample limits what the company can conclude. It should narrow confidence, not invite the broad report to fill the gap.

Keep the vendor-wide report for portfolio monitoring. Use role-level evidence for the role-level decision.

The vendor can supply the system, analysis, and supporting studies. Your company owns the decision to let that screen keep shaping access to a critical role.

Before a reassuring report closes the discussion, check the question it actually answered.

Frequently asked questions about AI hiring screens

© 2026 Lauren Carter. All rights reserved. This publication and its contents including analysis, frameworks, and editorial content are proprietary. Reproduction, adaptation, or redistribution in any form requires prior written permission from Lauren Carter.

Subscribe to Lauren's Executive Briefing