When AI Interviews, Humans also Win, Or do They?

September 4, 2025 Carsten Krause 0 Comments 18 min read

What CHROs Must Learn from a 2025 70,000-Person Study in Human–AI Recruiting

By Carsten Krause, September 3, 2025

A Turning Point in HR Strategy?

Chief Human Resource Officers (CHROs) have long wrestled with the tension between efficiency and human judgment in recruitment. Job interviews consume enormous recruiter time, introduce biases, and are difficult to scale. At the same time, organizations worry that automation could erode candidate experience or compromise decision quality. A new large-scale field experiment of the University of Chicago and Erasmus University with Globlal Recruiting firm PSG provides decisive evidence that both fears and assumptions deserve to be revisited.

Researchers at the University of Chicago and Erasmus University ran a randomized experiment with more than 70,000 applicants for entry-level customer service roles in the Philippines. Applicants were randomly assigned to three groups: a human recruiter interview, an AI voice agent interview, or a choice between the two. In every case, final hiring decisions were made by human recruiters. This design separates the interview process itself from evaluation, allowing us to see what happens when AI augments the front line of recruiting.

The results were clear.

Applicants interviewed by AI voice agents received

12 percent more job offers,
were 18 percent more likely to start,
and had 17 percent higher 30-day retention

compared to those interviewed by human recruiters.

For CHROs, this is not just about efficiency, but it is evidence of a structural shift toward human–AI collaboration that produces measurably better workforce outcomes.

The technology shift in the recruiting process

Every organizations and departments are feeling the impact of disruptive AI technology. In my book HI + AI = ECI for Elevated Collaborative Intelligence I cover multiple use cases of how organizations are leveraging AI to re-engineer existing processes and supplement their products and services with data and AI enabled value-add services.

The people sector is no difference here.

There are multiple use cases spanning the full scope of HR releated processes that can be complemented, automated and enhanced with AI capabilities.

The sometimes dreaded Applicant tracking systems are an early example of HR driven automation in the talent acquisition process where machine learning and AI are reviewing resumes scoring it on relevance of key words, job types, duration etc. to weed out the right candidates to take to the next stage in the hiring process.

Of course there are also internal use cases such as talent management, talent development and other HR processes that can take advantage of this process. At one of my recent clients the people team is experimenting with an AI leadership coach and assistant. Effectively leaders can talk to or message their AI leadership coach about any topic related to talent development, performance discussions and any topic that might come into mind with core information specific to the company for context.

For this to work and specifically for the interview study we are discussing in this article there are certain aspects to to be covered from an ECI frameworks perspective:

The ECI formula is designed to optimize the following input factors (aka leading indicators):

HI (Human Intelligence) plus AI (Artificial Intelligence) x T (the technology multiplier or roadblock) – R (addressing risk, regulations, cost, trust in AI etc.)

From a Human Intelligence perspective we need to consider which are the aspects of the inputs that humans address best such as intuition, empathy, supposedly building a personal connection with candidates. This is especially the case with jobs where the talent pool is scarcier and long term relationships wih candidates are important for potential future roles.

From an Artificial Intelligence and Technology perspective there are pre-requisites to make this scenario work:

The AI voice recruiter is leveraging multi modal processing of voice recognition and voice technology that is pre-trained on english or other languages to cover the key questions to ask during the interview.

For this to work a cloud based data intelligence architecture with language and voice modalities enabled needs to be in place to retrieve questions, interpret voice input from the candidates in near realtime just like a sales voice AI agent would have to do to achieve a fluent conversational experience similar to natural human to human communication.

There is a deep learning component where it is benificial to leverage a cloud technology provider that already optimized for natural language processing scenarios like this to accomodate for different candidates with different voices, accents, lower quality connections and microphone equipment while still recognizing spoken words during the interview.

Since the amount of interviewees in this case exceeded 70,000 the technology layer needs to be set to scale with potential increased loads in high demand periods and at the same time cost optimized to not use very expensive models.

The vocal information from the interviewee needs to be recorded and stored for later retrieval as part of the candidate file which wold be post-processed to validated top candidates for the human-in-the-loop review by the hiring manager or team.

From a risk perspective the data set that the voice AI agent got trained on needs to minimize bias and more importantly hallucinations to stick to the interview script. It also needs to account for different accents and potential slower responses of non-native speakers just to name a few points.

High-level technology topology (Microsoft Azure example)

Inbound calls or web voice sessions land in Azure Communication Services, flow through a low-latency speech → LLM → speech loop, and then into your recruiting workflow. Transcripts, audio, and metadata land in secure storage and databases; recruiters review and decide in your ATS with APIs exposed via API Management. Everything is locked down with private endpoints, Key Vault, and Managed Identity, monitored in Azure Monitor.

Components by layer

Channel & ingress

Azure Communication Services (ACS): PSTN/SIP/WebRTC calling, Call Automation, recording, compliant storage handoff.
Azure Front Door + Web Application Firewall: global entry for web/app traffic and webhook callbacks.
Azure Application Gateway (WAF) inside VNet: L7 routing to internal services when you don’t want public exposure.

Real-time voice pipeline (the “conversation loop”)

Azure Cognitive Services Speech to Text (STT): real-time streaming speech recognition; use Custom Speech for domain terms and accents.
Azure OpenAI Service (LLM): conversation state, question flow, eligibility logic, and tool calling. Use GPT-4.1/4o or small task models for classification; Prompt Flow (Azure AI Studio) to manage prompts, tools, evals.
Azure Cognitive Services Text to Speech (Neural TTS): low-latency synthesis; use Custom Neural Voice only with documented consent and gating.
Optional: Azure AI Content Safety in the loop for input/output filtering, PII redaction, and toxicity policies.

Orchestration & runtime

Azure Container Apps or Azure Kubernetes Service (AKS): hosts the “Voice Agent” microservice and turn-manager; gRPC/WebSocket for low latency.
Dapr sidecars (on Container Apps/AKS): service discovery, pub/sub, bindings to queues and stores.
Azure Functions (Durable Functions) for out-of-band workflows: send follow-up SMS/email, schedule retakes, trigger recruiter tasks.
Azure Service Bus (queues/topics) or Event Hubs: decouple real-time loop from downstream processing and recruiter review workloads.
Azure API Management: externalizes clean REST endpoints for ATS/CRM integrations and web apps; handles auth, quotas, and versioning.

Knowledge, retrieval, and guardrails

Azure AI Search (Cognitive Search) with vector + keyword indexing: job descriptions, company policies, location rules, scripts.
Azure OpenAI “On Your Data” or custom RAG layer: retrieval-augmented prompts grounded on indexed content.
Azure Policy + Azure AI Content Safety: enforce “announce-AI” script, disallow off-topic responses, filter sensitive content inline.

Data & storage

Azure Storage (Blob) / ADLS Gen2: raw call audio, diarized audio chunks, JSON turn logs, transcripts, model traces; lifecycle policies to move to cool/archive.
Azure Cosmos DB (Core API) or Azure SQL Database: applicant/session state, interview outcomes, feature flags, and experiment arms.
Azure Cache for Redis: session state for sub-second turn latency.
Azure Files or Blob Static Website: serve static interview guides and assets to internal tools.

Integration with ATS/HRIS

Azure Logic Apps: connectors for Workday, SAP SuccessFactors, Greenhouse, iCIMS, etc.; retry policies, idempotency keys.
API Management facade over ATS callbacks, with request validation and schema enforcement.
Power BI or Fabric for executive dashboards over the lakehouse (ADLS Gen2).

Security, identity, privacy, and governance

Microsoft Entra ID (Azure AD): SSO, Conditional Access, MFA; app registrations for services.
Managed Identities: no secrets in code; services authenticate to other Azure resources.
Azure Key Vault with customer-managed keys (CMK): encrypt Storage, Cosmos/SQL, and Azure OpenAI; store connection strings for non-Azure endpoints.
Virtual Network, Private Endpoints, Private DNS Zones: keep Speech, OpenAI, Storage, Cosmos/SQL off the public internet; egress via NAT Gateway.
Azure Firewall or third-party NGFW; network security groups; threat intel mode.
Microsoft Defender for Cloud + Defender for APIs: posture management, workload protections, secret scanning.
Microsoft Purview: data map, lineage, access policies; DLP on transcripts; subject access request workflows.
Optional Confidential Computing: DCasv5 VMs/AKS node pools for sensitive inference or transcript processing.

Observability, reliability, cost

Azure Monitor, Log Analytics, Application Insights: end-to-end tracing across Speech/LLM turns, prompt IDs, latency histograms, word error rate, and ASR/TTS token usage.
Azure Load Testing for conversational throughput; chaos testing on queues.
Autoscaling: Container Apps KEDA triggers on concurrent calls; scale Speech/OpenAI quotas per region.
Cost Management + budgets; split meters by app, region, environment.

Recommended data contracts and artifacts

Turn log schema: timestamp, channel, text, confidence, energy, interruptions, guardrail flags, prompt version, RAG doc IDs.
Interview summary object: eligibility flags, required topics covered count, risk indicators, recruiter checklist deltas.
Feature store (optional): store transcript-derived features (vocabulary richness, syntactic complexity, exchange count) for analytics and model evaluation—mirrors the study’s features on page 24 and Table E.4

Why Recruiters Got It Wrong

Before the experiment, the recruitment firm surveyed its own professionals, asking them to forecast outcomes of AI-led interviews. Most predicted worse results: lower quality, fewer offers, weaker retention. The field data showed the exact opposite.

What explains this disconnect?

Most recruiters assumed that AI would struggle with rapport, nuance, and flexibility.

In reality, the AI’s structured consistency actually produced more comprehensive interviews. The transcripts revealed that AI-led interviews covered more job-relevant topics on average (6.8 compared to 5.5 in human-led interviews) and were more likely to be classified as comprehensive conversations that opened and closed organically.

AI and the human recruiters were given the same template to cover on average 8 questions plus introduction and wrap up. As it turns out AI was a lot more consistent compared to its human counterparts.

To be clear this experiment did not replace all human contact, but had AI (a voice AI assistant type large language model) do the heavy lifting, parsing the information for a human to review and make the final decision.

The AI agent also elicited higher-value linguistic features. Applicants spoke with greater vocabulary richness, syntactic complexity, and conversational depth—features that strongly predicted job offers in human-led interviews. At the same time, AI reduced low-signal behaviors such as backchannel cues or filler questions. Recruiters, when later reviewing transcripts and recordings, rated AI-interviewed applicants more positively.

This dynamic demonstrates an important principle: human evaluators, given better inputs, can make better decisions. AI did not replace recruiters—it gave them richer, more structured candidate information to work with.

Candidates Choices Surprised Everyone

The study also tested applicant choice. When given the option, 78 percent of candidates chose the AI voice agent over a human recruiter. This finding directly challenges the assumption that candidates demand human contact in early stages.

On the other hand there were also drop outs where interviewees decided to stop the interview or declined the interview, because they did not want to speak with an AI agent.

A Strategy for Reduction in Bias.

Not surprisingly the AI that was specifically trained not to be biased achieved more balanced results vs human interviewers related to gender, orientation and ethnic groups. Survey data showed that applicants viewed AI as fair, efficient, and even preferable. Net Promoter Scores—the industry’s key metric for candidate satisfaction—were statistically indistinguishable between AI and human interviews. Importantly, reports of gender-based discrimination were nearly halved in AI-led interviews compared to human ones. While some candidates rated AI interactions as less “natural,” many appreciated the reduced stress and greater sense of impartiality.

Interestingly, applicants with lower standardized test scores were more likely to choose AI. While this raises questions about self-sorting, it also suggests that AI can expand access for candidates who might otherwise feel disadvantaged in subjective human interactions.

Recruiter Behavior Under the Microscope

The experiment also explored how recruiters responded to AI-led interviews. Across more than 130 recruiters, 69 percent extended more offers after reviewing AI-led interviews than after human-led ones. Recruiter comments were more positive as well: 31 percent of AI-reviewed transcripts were rated positively compared to 24 percent of human-led interviews.

Yet subtle behavioral shifts emerged. Recruiters placed slightly less weight on AI-generated interview scores and more on standardized language tests. This signals a cautious trust dynamic: while recruiters valued AI’s structure, they leaned more heavily on independent measures when interpreting AI-led interviews. Far from undermining outcomes, this balanced weighting contributed to stronger hiring results.

Economics of AI Recruiting

Beyond candidate and recruiter behavior, the study analyzed operational implications. AI voice agents accelerated initial scheduling—interviews could happen within hours rather than days. However, review times lengthened slightly, as recruiters needed to evaluate unfamiliar transcripts rather than their own direct impressions. The net effect was a small increase in overall hiring process time, though offset by higher retention and match quality.

The cost analysis of the study showed that in most wage environments, AI-led interviews become cost-effective after a few thousand interviews. In high-wage markets, the break-even point was under 2,500 interviews. Given that major outsourcing firms process tens of thousands of applications annually, the economics strongly favor adoption.

Elevated Collaborative Intelligence in Action

As mentioned earlier these findings align precisely with the HI + AI = ECI^TM framework I outline in my book HI + AI = ECI: Elevated Collaborative Intelligence^TM. ECI is built on the recognition that humans and AI bring distinct, complementary strengths.

In this experiment, AI demonstrated its capability to manage structured, high-volume, and consistency-driven tasks—interviewing thousands of applicants with uniform quality. Human recruiters then exercised judgment, empathy, and discretion in making final decisions. Together, outcomes improved not by marginal percentages but by double-digit gains in hiring, onboarding, and retention.

This is the very definition of elevated collaborative intelligence. AI did not displace human recruiters. It elevated them by shifting their role from repetitive interviewing toward strategic evaluation and decision-making. The recruiters who once feared erosion of their craft were, in fact, empowered. Their decisions were better, their applicant pools more qualified, and their own work more focused on where human value truly lies.

Strategic Guidance for CHROs

For CHROs, the implications are urgent and actionable.

First, AI interviewing is not a threat to candidate experience. Satisfaction, fairness, and even reports of reduced discrimination suggest that properly designed AI systems can improve employer brand.

Second, AI enhances retention, a metric where most HR leaders struggle, especially in high-turnover sectors like customer service. Third, cost dynamics increasingly favor adoption, especially in high-wage regions or large-scale recruiting contexts.

The lesson is clear: AI voice agents should not be seen as a replacement for HR professionals but as force multipliers. Firms that embrace this model can free recruiters to focus on culture, empathy, and leadership, while AI ensures rigor, consistency, and efficiency in the first screening layer.

The CDO TIMES Bottom Line

The 70,000-person experiment on AI-led interviews is not a future scenario; it is hard data from a global recruitment process. Applicants were more likely to receive offers, start jobs, and stay employed when interviewed by AI agents, all while human recruiters retained final control. Candidate satisfaction did not fall, and in some respects improved.

For technology leaders supporting talent teams and human resource departments and CHROs, this another empirical evidence yet that the future of HR is not human versus AI—it is human with AI. The formula is simple: HI + AI = ECI. Elevated Collaborative Intelligence is not theory. It is happening today in the interview rooms of global recruiting firms. The organizations that act on this evidence will build faster, fairer, and more resilient talent pipelines. Those that hesitate risk being left with processes that are slower, less fair, and less effective.

Sources

Brian Jabarian and Luca Henkel. Voice AI in Firms: A Natural Field Experiment on Automated Job Interviews. SSRN, August 18, 2025. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5395709
Carsten Krause, The AI Ready Leader – HI + AI = ECI^TM Elevated Collaborative Intelligenc for AI Powered Transformation: A Blueprint for Executive Leaders to Master Humain-AI Synergy through the ECI Framework, September 15 2025
Data and charts throughout drawn directly from the study (pp. 1–40)

Love this article? Embrace the full potential and become an esteemed full access member, experiencing the exhilaration of unlimited access to captivating articles, exclusive non-public content, empowering hands-on guides, and transformative training material. Unleash your true potential today!

Order the AI + HI = ECI book by Carsten Krause today! at cdotimes.com/book

Subscribe on LinkedIn: Digital Insider

Become a paid subscriber for unlimited access, exclusive content, no ads: CDO TIMES

Do You Need Help?

Consider bringing on a fractional CIO, CISO, CDO or CAIO from CDO TIMES Leadership as a Service. The expertise of CDO TIMES becomes indispensable for organizations striving to stay ahead in the digital transformation journey. Here are some compelling reasons to engage their experts:

Deep Expertise: CDO TIMES has a team of experts with deep expertise in the field of Cybersecurity, Digital, Data and AI and its integration into business processes. This knowledge ensures that your organization can leverage digital and AI in the most optimal and innovative ways.
Strategic Insight: Not only can the CDO TIMES team help develop a Digital & AI strategy, but they can also provide insights into how this strategy fits into your overall business model and objectives. They understand that every business is unique, and so should be its Digital & AI strategy.
Future-Proofing: With CDO TIMES, organizations can ensure they are future-proofed against rapid technological changes. Our experts stay abreast of the latest AI, Data and digital advancements and can guide your organization to adapt and evolve as the technology does.
Risk Management: Implementing a Digital & AI strategy is not without its risks. The CDO TIMES can help identify potential pitfalls and develop mitigation strategies, helping you avoid costly mistakes and ensuring a smooth transition with fractional CISO services.
Competitive Advantage: Finally, by hiring CDO TIMES experts, you are investing in a competitive advantage. Their expertise can help you speed up your innovation processes, bring products to market faster, and stay ahead of your competitors.

By employing the expertise of CDO TIMES, organizations can navigate the complexities of digital innovation with greater confidence and foresight, setting themselves up for success in the rapidly evolving digital economy. The future is digital, and with CDO TIMES, you’ll be well-equipped to lead in this new frontier.

Do you need help with your digital transformation initiatives? We provide fractional CAIO, CDO, CISO and CIO services, do a Preliminary ECI and Tech Navigator Assessment and we will help you drive results and deliver winning digital and AI strategies for you!

Subscribe now for free and never miss out on digital insights delivered right to your inbox!