When should CIOs fix, scale back or stop AI? – TechTarget
Consultant Gleb Tsipursky explains when to redesign AI workflows, narrow their role and measure their real value.
An AI assistant being tested by an insurance company encountered an unexpected problem: Its interface sometimes failed to display lengthy responses containing policy language.
For insurance adjusters, that was more than an inconvenience. They needed to verify the precise wording of policies when evaluating claims or preparing customer correspondence. If the assistant couldn’t reliably display that information, even an accurate AI-generated answer might not be usable.
That was the challenge Gleb Tsipursky encountered while consulting with the insurer. Tsipursky, a behavioral scientist with a Ph.D. and CEO of Disaster Avoidance Experts, discusses how he addressed that issue, why another project called for a more limited role for generative AI and what CIOs need to know when deciding where AI belongs in their operations.
Editor’s note: This interview was edited for length and clarity.
You encountered a problem while helping an insurance company test an AI assistant for its adjusters. What happened, and how did you decide what needed to change?
Gleb Tsipursky: In one property insurance project, we developed assistance for adjusters who needed to find relevant policy language, summarize it and prepare communications grounded in the policy documents. During testing, a platform update created an unpredictable failure: long answers containing extensive policy references sometimes disappeared from the interface.
I didn’t tell the adjusters to trust a broken experience because the underlying language model was impressive. I tested a different interface and moved the policy-advice workflow into Microsoft Teams, where it handled those longer responses more dependably.
I also kept policy summarization and the more exacting task of pulling verbatim policy quotations for letters in separate assistants. Exact quotation had different formatting and matching requirements, so a single catchall assistant was a poorer design. We tested prompts against distinct claim scenarios and preserved adjusters’ responsibility for the underlying coverage judgment.
That episode matters because it’s easy to misdiagnose an AI problem. The model may generate a plausible answer, yet the interface, source retrieval or review process can make the whole workflow unusable. The answer was to change the workflow and narrow each tool’s job, rather than count a successful model response as a successful business outcome. I don’t have verified before-and-after financial numbers for that particular redesign, and I wouldn’t describe it as a production system that was shut down.
You made a different decision about how much AI to use in another project for the same insurer. What was the reasoning?
Tsipursky: In another project for the same insurer, we designed a staged approach to flagging potentially problematic comments on forms for staff review. Straightforward conditions and keyword-based checks could flag obvious issues first; more flexible AI interpretation was reserved for nuanced comments that simple rules could miss. Staff remained responsible for evaluating the flagged cases.
This was a conscious decision to use less generative AI than a fully AI-driven design would have used. The projected benefit was lower complexity and more transparent handling of routine cases, while reserving AI for the part of the problem where natural-language understanding might justify its costs.
“Comparing AI with an intentionally clumsy manual process produces a good demonstration and a bad investment decision.”
Gleb Tsipursky
behavioral scientist and CEO of Disaster Avoidance Experts
How should CIOs compare AI-assisted work with conventional approaches to determine whether they’re actually getting better results?
Tsipursky: I would treat the decision as a controlled business experiment, beginning with a clearly defined unit of work. For an insurer, that might be one claim file reviewed to an acceptable standard. For a service center, one customer issue resolved correctly. For a CIO’s own team, one incident investigated and documented.
The baseline must be the best reasonable existing workflow, including any straightforward process improvement or conventional automation. Comparing AI with an intentionally clumsy manual process produces a good demonstration and a bad investment decision.
Where possible, randomly assign comparable cases to AI-assisted and conventional workflows, stratifying by complexity and the employee’s experience. A cross-over design, in which similarly skilled staff rotate between methods on matched cases, can reduce the risk that an unusually fast team makes one method look better. Train the AI group sufficiently before measuring results but count that training as part of deployment costs.
Use independent quality checks where the reviewer can avoid knowing which method produced the output. Keep the pilot long enough to include difficult cases, exceptions and the second or third week after the novelty wears off.
What costs do organizations tend to overlook when evaluating whether AI is worthwhile?
Tsipursky: CIOs should calculate cost per accepted business outcome rather than cost per AI-generated output. I would organize the measurement into three stages.
Before the result, count the time employees spend assembling context, cleaning inputs, finding documents and writing prompts. During production, count model and application charges, drafting, source checking, fact checking, corrections and human approvals. After delivery, count exceptions, customer complaints, downstream corrections, audits and the time other departments spend repairing weak material. Then add the fixed costs of implementation, training, security, monitoring, integration and future updates.
The most common accounting mistake is to attribute a time saving to one employee while ignoring costs transferred to others. Imagine that a staff member saves 12 minutes preparing a case summary. The recipient now spends seven minutes checking citations, four minutes correcting omissions and three minutes resolving an escalation. The company has spent 14 minutes downstream to save 12 upstream. Across 100 cases a week, that is more than three hours of extra labor before the model bill. The numbers here are illustrative, but the measurement problem is very real.
The economic test also needs to distinguish capacity from cash. If an employee saves five hours a week, the organization has gained potential capacity, but it hasn’t automatically reduced payroll expense or generated five hours’ worth of profit. Ask what those hours actually enable: more cases handled without extra hires, fewer outside-contractor hours, shorter queues, better retention, additional revenue or higher-quality service.
Who should decide when an AI application needs to be scaled back, redesigned or discontinued?
Tsipursky: Before deployment, every substantial AI use case needs an accountable business-process owner, a technical owner and explicit pause-and-retire rules. The business owner should have authority to stop or redesign the workflow when it fails to deliver the agreed business outcome. The CIO’s organization should control technical permissions, reliability, integration, security and reversibility. Frontline employees must have a straightforward way to flag failures and pause a risky action without being punished for speaking up.
I would separate an emergency pause from a routine investment decision. A suspected disclosure of sensitive data, a repeatable serious error or an autonomous action outside its permitted scope calls for an immediate technical stop. A workflow that’s safe but merely underwhelming calls for a scheduled business review: keep it, improve the prompt or retrieval, switch to a cheaper model, automate with rules, reduce autonomy or retire it.
Retirement is a normal part of portfolio management. CIOs should expect to celebrate a well-reasoned shutdown, not hide it.
“The lesson for managers is that responsible nonuse is a legitimate outcome of competent AI experimentation.”
Gleb Tsipursky
behavioral scientist and CEO of Disaster Avoidance Experts
When leadership wants employees to adopt AI, how can organizations avoid rewarding usage instead of results?
Tsipursky: Make a simple change to the management question. Stop asking, “How much AI are you using?” Ask, “Which customer, quality, efficiency or revenue problem did you improve, and what evidence shows that you improved it?” Usage data has a role in diagnosing whether employees have access and are trying the tool. It should not become the performance target.
I recommend that teams present both an AI success and an evidence-backed decision to avoid, simplify or stop AI. Give employees credit for discovering that a rules-based workflow beats a chatbot, or that human judgment belongs at the final approval step. Reward someone who finds an invented legal citation or a misleading claims summary before it goes to a customer. Those people have protected the business. The lesson for managers is that responsible nonuse is a legitimate outcome of competent AI experimentation.
What’s the biggest misunderstanding CIOs have about turning AI adoption into meaningful changes in how their organizations work?
Tsipursky: The biggest mistake is thinking that deploying AI means transforming work. CIOs know how to roll out conventional enterprise software: procure a platform, configure it centrally, write procedures, train employees and require everyone to follow the new process. That works for standardized systems such as an expense platform or a CRM.
Generative AI creates a different opportunity. A claims adjuster, lawyer, salesperson, analyst or administrative employee can increasingly describe their own recurring task in ordinary language, refine a prompt and build a useful reusable assistant without waiting for a traditional development project. Much of the value therefore depends on the initiative and domain knowledge of the people closest to the work.
I typically ask employees two questions: “Which part of your job do you most dislike doing?” and “What important work do you never have enough time to do?” Start with the first category so employees can spend more time on the second.
So, I would ask CIOs to report a different set of indicators: how many recurring workflows were redesigned, what happened to their net time and quality, how much exception and verification work they created, how many employees can now maintain or improve the tools and which underperforming tools were deliberately retired.
James Alan Miller is a veteran technology editor and writer and lead editor for CIO News at TechTarget. He directs coverage of enterprise technology strategy, AI, software, data, infrastructure and the decisions shaping how CIOs manage increasingly complex IT environments.
YOU MUST AGREE TO OUR TERMS OF USE AND PRIVACY POLICY.
Already a subscriber? Sign in with your email above.
Get the free newsletter read by industry experts
AI represents a powerful new tool for cybersecurity professionals, but the technology is not without risk. Discover what CISOs need to know when deciding how to use AI.
Keep up with the story. Subscribe to the free TechTarget CIO newsletter
Subscribe to TechTarget for top news, trends & analysis
Get the free newsletter read by industry experts
AI represents a powerful new tool for cybersecurity professionals, but the technology is not without risk. Discover what CISOs need to know when deciding how to use AI.
The free newsletter covering the top industry headlines
source
This is a newsfeed from leading technology publications. No additional editorial review has been performed before posting.


