The Human Backstop: AI Agents Rely on Contractors for Complex Tasks
Startups are patching in human workers to handle phone calls and scheduling errors that frontier models still struggle to execute reliably.
By Chloe Martin ·
Artificial intelligence agents marketed to handle complex digital tasks frequently fail when they must navigate phone systems or make specific reservations, forcing companies to insert human workers into the workflow. While large language models from OpenAI and Meta excel at browsing and data retrieval, they often stumble on real-world interactions that require nuanced judgment or verbal negotiation. This gap has created a hybrid model where software handles the initial research, and human contractors finish the job.
Marissa Mayer, former CEO of Yahoo and current head of Dazzle AI, described the current limitations of top-tier models. She noted that while the software can generate a strong list of home renovation services, it may fail when booking the actual appointment. Mayer said her company checks AI suggestions to ensure the service is reliable before presenting it to the user. This process acknowledges that current models lack the common sense to distinguish between similar items, such as window glass and tabletop glass, a confusion that led one user to ask the agent if a table top would fit in a window.
The correction in that case happened within seconds because Dazzle flagged the error and passed it to a human. Mayer stated that the human resolved the issue quickly, whereas the agent would have consumed a large number of computing tokens or failed to find a solution. She described the market as being in a transitionary period where humans are necessary to bridge the gap between digital suggestions and physical execution.
Duckbill, a startup founded by Meghan Joyce, has built an application programming interface specifically for this purpose. Joyce, who previously worked at Uber, explained that her company breaks tasks down into small, manageable pieces for contract workers. These workers, who are often 1099 employees or contact center labor, handle the specific actions that the AI cannot complete. Joyce compared the model to ride-hailing services, where tasks are delegated to people who can perform them consistently.
Shivani Poddar, CEO of Wajo AI, is implementing a similar approach. Her platform uses human workers to handle phone calls and CAPTCHA tests that block automated bots. Poddar argued that collaborative AI is necessary to retain users. She warned that people will abandon the technology if agents cannot complete tasks without human assistance.
The use of human contractors has raised privacy questions among users and employees. Meta employees previously expressed concerns about internal tests that used humans to make phone calls for the company's Muse agent. A Meta spokesperson stated at the time that this testing was core to the product development process. Meta currently does not use humans to complete user requests for Muse.
To address privacy concerns, Dazzle and Duckbill conduct background checks on their contractors. Mayer said Dazzle screens for discretion and good judgment, qualities she associated with high-quality personal assistants. Joyce noted that Duckbill performs full background checks on its U.S.-based contractors, treating them similarly to drivers for delivery and ride-share apps. Both companies aim to ensure that the human intervention feels like a seamless extension of the service rather than an intrusion.
The industry is moving toward a model where AI handles the informational load while humans handle the transactional load. This division of labor allows companies to promise end-to-end service completion even when the underlying technology cannot yet perform every step independently. The reliance on human oversight is becoming a standard feature of consumer-facing AI products.
Source: CNBC
