9 famous analytics and AI disasters – CIO
Insights from data and machine learning algorithms can be invaluable, but mistakes can cost you reputation, revenue, or even lives. These high-profile analytics and AI blunders illustrate what can go wrong.
In 2017, The Economist declared that data, rather than oil, had become the world’s most valuable resource. The refrain has been repeated ever since. Organizations across every industry have been and continue to invest heavily in data and analytics. But like oil, data and analytics have their dark side.
According to CIO’s State of the CIO 2023 report, 34% of IT leaders say that data and business analytics will drive the most IT investment at their organization this year. And 26% of IT leaders say machine learning/artificial intelligence will drive the most IT investment. Insights gained from analytics and actions driven by machine learning algorithms can give organizations a competitive advantage, but mistakes can be costly in terms of reputation, revenue, or even lives.
Understanding your data and what it’s telling you is important, but it’s also important to understand your tools, know your data, and keep your organization’s values firmly in mind.
Here are a handful of high-profile analytics and AI blunders from the past decade to illustrate what can go wrong.
In August 2023, newspaper chain Gannett announced that it would pause the use of an AI tool called LedeAI after several dispatches written by the AI went viral for being repetitive, poorly written, and lacking key details.
CNN pointed to one example, preserved by the Internet Archive’s Wayback Machine, which opened with, “The Worthington Christian [[WINNING_TEAM_MASCOT]] defeated the Westerville North [[LOSING_TEAM_MASCOT]] 2-1 in an Ohio boys soccer game on Saturday.”
CNN found similar stories written by LedeAI in other local Gannett papers, including the Louisville Courier Journal, AZ Central, Florida Today, and the Milwaukee Journal Sentinel.
After the stories were roundly mocked on social media, Gannett opted to pause the use of LedeAI in all local markets that had been using the service.
In a statement to CNN, LedeAI CEO Jay Allred expressed regret and promised an around-the-clock effort to correct the problems.
In August 2023, tutoring company iTutorGroup agreed to pay $365,000 to settle a suit brought by the US Equal Employment Opportunity Commission (EEOC). The federal agency said the company, which provides remote tutoring services to students in China, used AI-powered recruiting software that automatically rejected female applicants ages 55 and older and male applicants ages 60 and older.
The EEOC said more than 200 qualified applicants had been automatically rejected by the software.
“Age discrimination is unjust and unlawful,” EEOC Chair Charlotte A. Burrows said in a statement. “Even when technology automates the discrimination, the employer is still responsible.”
iTutorGroup denied any wrongdoing but did decide to settle the suit. As part of the settlement and consent decree, it agreed to adopt new anti-discrimination policies.
Advances made in 2023 by large language models (LLMs) have stoked widespread interest in the transformative potential of generative AI across nearly every industry. OpenAI’s ChatGPT has been at the center of this surge in interest, foreshadowing how generative AI holds the power to disrupt the nature of work in nearly every corner of business.
But the technology still has ways to go before it can reliably take over most business processes, as attorney Steven A. Schwartz learned when he found himself in hot water with US District Judge P. Kevin Castel in 2023 after using ChatGPT to research precedents in a suit against Colombian airline Avianca.
Schwartz, an attorney with Levidow, Levidow & Oberman, used the OpenAI generative AI chatbot to find prior cases to support a case filed by Avianca employee Roberto Mata for injuries he sustained in 2019. The only problem? At least six of the cases submitted in the brief did not exist. In a document filed in May, Judge Castel noted the cases submitted by Schwartz included false names and docket numbers, along with bogus internal citations and quotes. Schwartz’s partner, Peter LoDuca, was Mata’s lawyer of record and signed the brief, putting himself in jeopardy as well.
In an affidavit, Schwartz told the court that it was the first time he had used ChatGPT as a legal research source and he was “unaware of the possibility that its content could be false.” He admitted that he had not confirmed the sources provided by the AI chatbot. He also said that he “greatly regrets having utilized generative artificial intelligence to supplement the legal research performed herein and will never do so in the future without absolute verification of its authenticity.”
In June 2023, Judge Castel imposed a $5,000 fine on Schwartz and LoDuca. In a separate ruling in June, Judge Castel dismissed Mata’s lawsuit against Avianca.
Since the COVID-19 pandemic began, numerous organizations have sought to apply machine learning (ML) algorithms to help hospitals diagnose or triage patients faster. But according to the UK’s Turing Institute, a national center for data science and AI, the predictive tools made little to no difference.
MIT Technology Review has chronicled a number of failures, most of which stem from errors in the way the tools were trained or tested. The use of mislabeled data or data from unknown sources was a common culprit.
Derek Driggs, a machine learning researcher at the University of Cambridge, together with his colleagues, published a paper in Nature Machine Intelligence that explored the use of deep learning models for diagnosing the virus. The paper determined the technique not fit for clinical use. For example, Driggs’ group found that their own model was flawed because it was trained on a data set that included scans of patients that were lying down while scanned and patients that were standing up. The patients who were lying down were much more likely to be seriously ill, so the algorithm learned to identify COVID risk based on the position of the person in the scan.
A similar example includes an algorithm trained with a data set that included scans of the chests of healthy children. The algorithm learned to identify children, not high-risk patients.
In November 2021, online real estate marketplace Zillow told shareholders it would wind down its Zillow Offers operations and cut 25% of the company’s workforce — about 2,000 employees — over the next several quarters. The home-flipping unit’s woes were the result of the error rate in the machine learning algorithm it used to predict home prices.
Zillow Offers was a program through which the company made cash offers on properties based on a “Zestimate” of home values derived from a machine learning algorithm. The idea was to renovate the properties and flip them quickly. But a Zillow spokesperson told CNN that the algorithm had a median error rate of 1.9%, and the error rate could be much higher, as much as 6.9%, for off-market homes.
CNN reported that Zillow bought 27,000 homes through Zillow Offers since its launch in April 2018 but sold only 17,000 through the end of September 2021. Black swan events like the COVID-19 pandemic and a home renovation labor shortage contributed to the algorithm’s accuracy troubles.
Zillow said the algorithm had led it to unintentionally purchase homes at higher prices than its current estimates of future selling prices, resulting in a $304 million inventory write-down in Q3 2021.
In a conference call with investors following the announcement, Zillow co-founder and CEO Rich Barton said it might be possible to tweak the algorithm, but ultimately it was too risky.
In 2019, a study published in Science revealed that a healthcare prediction algorithm, used by hospitals and insurance companies throughout the US to identify patients to in need of “high-risk care management” programs, was far less likely to single out Black patients.
High-risk care management programs provide trained nursing staff and primary-care monitoring to chronically ill patients in an effort to prevent serious complications. But the algorithm was much more likely to recommend white patients for these programs than Black patients.
The study found that the algorithm used healthcare spending as a proxy for determining an individual’s healthcare need. But according to Scientific American, the healthcare costs of sicker Black patients were on par with the costs of healthier white people, which meant they received lower risk scores even when their need was greater.
The study’s researchers suggested that a few factors may have contributed. First, people of color are more likely to have lower incomes, which, even when insured, may make them less likely to access medical care. Implicit bias may also cause people of color to receive lower-quality care.
While the study did not name the algorithm or the developer, the researchers told Scientific American they were working with the developer to address the situation.
In March 2016, Microsoft learned that using Twitter interactions as training data for machine learning algorithms can have dismaying results.
Microsoft released Tay, an AI chatbot, on the social media platform. The company described it as an experiment in “conversational understanding.” The idea was the chatbot would assume the persona of a teen girl and interact with individuals via Twitter using a combination of machine learning and natural language processing. Microsoft seeded it with anonymized public data and some material pre-written by comedians, then set it loose to learn and evolve from its interactions on the social network.
Within 16 hours, the chatbot posted more than 95,000 tweets, and those tweets rapidly turned overtly racist, misogynist, and anti-Semitic. Microsoft quickly suspended the service for adjustments and ultimately pulled the plug.
“We are deeply sorry for the unintended offensive and hurtful tweets from Tay, which do not represent who we are or what we stand for, nor how we designed Tay,” Peter Lee, corporate vice president, Microsoft Research & Incubations (then corporate vice president of Microsoft Healthcare), wrote in a post on Microsoft’s official blog following the incident.
Lee noted that Tay’s predecessor, Xiaoice, released by Microsoft in China in 2014, had successfully had conversations with more than 40 million people in the two years prior to Tay’s release. What Microsoft didn’t take into account was that a group of Twitter users would immediately begin tweeting racist and misogynist comments to Tay. The bot quickly learned from that material and incorporated it into its own tweets.
“Although we had prepared for many types of abuses of the system, we had made a critical oversight for this specific attack. As a result, Tay tweeted wildly inappropriate and reprehensible words and images,” Lee wrote.
Like many large companies, Amazon is hungry for tools that can help its HR function screen applications for the best candidates. In 2014, Amazon started working on AI-powered recruiting software to do just that. There was only one problem: The system vastly preferred male candidates. In 2018, Reuters broke the news that Amazon had scrapped the project.
Amazon’s system gave candidates star ratings from 1 to 5. But the machine learning models at the heart of the system were trained on 10 years’ worth of resumes submitted to Amazon — most of them from men. As a result of that training data, the system started penalizing phrases in the resume that included the word “women’s” and even downgraded candidates from all-women colleges.
At the time, Amazon said the tool was never used by Amazon recruiters to evaluate candidates.
The company tried to edit the tool to make it neutral, but ultimately decided it could not guarantee it would not learn some other discriminatory way of sorting candidates and ended the project.
In 2012, an analytics project by retail titan Target showcased how much companies can learn about customers from their data. According to the New York Times, in 2002 Target’s marketing department started wondering how it could determine whether customers are pregnant. That line of inquiry led to a predictive analytics project that would famously lead the retailer to inadvertently reveal to a teenage girl’s family that she was pregnant. That, in turn, would lead to all manner of articles and marketing blogs citing the incident as part of advice for avoiding the “creepy factor.”
Target’s marketing department wanted to identify pregnant individuals because there are certain periods in life — pregnancy foremost among them — when people are most likely to radically change their buying habits. If Target could reach out to customers in that period, it could, for instance, cultivate new behaviors in those customers, getting them to turn to Target for groceries or clothing or other goods.
Like all other big retailers, Target had been collecting data on its customers via shopper codes, credit cards, surveys, and more. It mashed that data up with demographic data and third-party data it purchased. Crunching all that data enabled Target’s analytics team to determine that there were about 25 products sold by Target that could be analyzed together to generate a “pregnancy prediction” score. The marketing department could then target high-scoring customers with coupons and marketing messages.
Additional research would reveal that studying customers’ reproductive status could feel creepy to some of those customers. According to the Times, the company didn’t back away from its targeted marketing, but did start mixing in ads for things they knew pregnant women wouldn’t buy — including ads for lawn mowers next to ads for diapers — to make the ad mix feel random to the customer.
Thor Olavsrud covers data analytics, business intelligence, and data science for CIO.com. He resides in New York.
Sponsored Links
This article was autogenerated from a news feed from CDO TIMES selected high quality news and research sources. There was no editorial review conducted beyond that by CDO TIMES staff. Need help with any of the topics in our articles? Schedule your free CDO TIMES Tech Navigator call today to stay ahead of the curve and gain insider advantages to propel your business!
In 2017, The Economist declared that data, rather than oil, had become the world’s most valuable resource. The refrain has been repeated ever since. Organizations across every industry have been and continue to invest heavily in data and analytics. But like oil, data and analytics have their dark side.
According to CIO’s State of the CIO 2023 report, 34% of IT leaders say that data and business analytics will drive the most IT investment at their organization this year. And 26% of IT leaders say machine learning/artificial intelligence will drive the most IT investment. Insights gained from analytics and actions driven by machine learning algorithms can give organizations a competitive advantage, but mistakes can be costly in terms of reputation, revenue, or even lives.
Understanding your data and what it’s telling you is important, but it’s also important to understand your tools, know your data, and keep your organization’s values firmly in mind.
Here are a handful of high-profile analytics and AI blunders from the past decade to illustrate what can go wrong.
In August 2023, newspaper chain Gannett announced that it would pause the use of an AI tool called LedeAI after several dispatches written by the AI went viral for being repetitive, poorly written, and lacking key details.
CNN pointed to one example, preserved by the Internet Archive’s Wayback Machine, which opened with, “The Worthington Christian [[WINNING_TEAM_MASCOT]] defeated the Westerville North [[LOSING_TEAM_MASCOT]] 2-1 in an Ohio boys soccer game on Saturday.”
CNN found similar stories written by LedeAI in other local Gannett papers, including the Louisville Courier Journal, AZ Central, Florida Today, and the Milwaukee Journal Sentinel.
After the stories were roundly mocked on social media, Gannett opted to pause the use of LedeAI in all local markets that had been using the service.
In a statement to CNN, LedeAI CEO Jay Allred expressed regret and promised an around-the-clock effort to correct the problems.
In August 2023, tutoring company iTutorGroup agreed to pay $365,000 to settle a suit brought by the US Equal Employment Opportunity Commission (EEOC). The federal agency said the company, which provides remote tutoring services to students in China, used AI-powered recruiting software that automatically rejected female applicants ages 55 and older and male applicants ages 60 and older.
The EEOC said more than 200 qualified applicants had been automatically rejected by the software.
“Age discrimination is unjust and unlawful,” EEOC Chair Charlotte A. Burrows said in a statement. “Even when technology automates the discrimination, the employer is still responsible.”
iTutorGroup denied any wrongdoing but did decide to settle the suit. As part of the settlement and consent decree, it agreed to adopt new anti-discrimination policies.
Advances made in 2023 by large language models (LLMs) have stoked widespread interest in the transformative potential of generative AI across nearly every industry. OpenAI’s ChatGPT has been at the center of this surge in interest, foreshadowing how generative AI holds the power to disrupt the nature of work in nearly every corner of business.
But the technology still has ways to go before it can reliably take over most business processes, as attorney Steven A. Schwartz learned when he found himself in hot water with US District Judge P. Kevin Castel in 2023 after using ChatGPT to research precedents in a suit against Colombian airline Avianca.
Schwartz, an attorney with Levidow, Levidow & Oberman, used the OpenAI generative AI chatbot to find prior cases to support a case filed by Avianca employee Roberto Mata for injuries he sustained in 2019. The only problem? At least six of the cases submitted in the brief did not exist. In a document filed in May, Judge Castel noted the cases submitted by Schwartz included false names and docket numbers, along with bogus internal citations and quotes. Schwartz’s partner, Peter LoDuca, was Mata’s lawyer of record and signed the brief, putting himself in jeopardy as well.
In an affidavit, Schwartz told the court that it was the first time he had used ChatGPT as a legal research source and he was “unaware of the possibility that its content could be false.” He admitted that he had not confirmed the sources provided by the AI chatbot. He also said that he “greatly regrets having utilized generative artificial intelligence to supplement the legal research performed herein and will never do so in the future without absolute verification of its authenticity.”
In June 2023, Judge Castel imposed a $5,000 fine on Schwartz and LoDuca. In a separate ruling in June, Judge Castel dismissed Mata’s lawsuit against Avianca.
Since the COVID-19 pandemic began, numerous organizations have sought to apply machine learning (ML) algorithms to help hospitals diagnose or triage patients faster. But according to the UK’s Turing Institute, a national center for data science and AI, the predictive tools made little to no difference.
MIT Technology Review has chronicled a number of failures, most of which stem from errors in the way the tools were trained or tested. The use of mislabeled data or data from unknown sources was a common culprit.
Derek Driggs, a machine learning researcher at the University of Cambridge, together with his colleagues, published a paper in Nature Machine Intelligence that explored the use of deep learning models for diagnosing the virus. The paper determined the technique not fit for clinical use. For example, Driggs’ group found that their own model was flawed because it was trained on a data set that included scans of patients that were lying down while scanned and patients that were standing up. The patients who were lying down were much more likely to be seriously ill, so the algorithm learned to identify COVID risk based on the position of the person in the scan.
A similar example includes an algorithm trained with a data set that included scans of the chests of healthy children. The algorithm learned to identify children, not high-risk patients.
In November 2021, online real estate marketplace Zillow told shareholders it would wind down its Zillow Offers operations and cut 25% of the company’s workforce — about 2,000 employees — over the next several quarters. The home-flipping unit’s woes were the result of the error rate in the machine learning algorithm it used to predict home prices.
Zillow Offers was a program through which the company made cash offers on properties based on a “Zestimate” of home values derived from a machine learning algorithm. The idea was to renovate the properties and flip them quickly. But a Zillow spokesperson told CNN that the algorithm had a median error rate of 1.9%, and the error rate could be much higher, as much as 6.9%, for off-market homes.
CNN reported that Zillow bought 27,000 homes through Zillow Offers since its launch in April 2018 but sold only 17,000 through the end of September 2021. Black swan events like the COVID-19 pandemic and a home renovation labor shortage contributed to the algorithm’s accuracy troubles.
Zillow said the algorithm had led it to unintentionally purchase homes at higher prices than its current estimates of future selling prices, resulting in a $304 million inventory write-down in Q3 2021.
In a conference call with investors following the announcement, Zillow co-founder and CEO Rich Barton said it might be possible to tweak the algorithm, but ultimately it was too risky.
In 2019, a study published in Science revealed that a healthcare prediction algorithm, used by hospitals and insurance companies throughout the US to identify patients to in need of “high-risk care management” programs, was far less likely to single out Black patients.
High-risk care management programs provide trained nursing staff and primary-care monitoring to chronically ill patients in an effort to prevent serious complications. But the algorithm was much more likely to recommend white patients for these programs than Black patients.
The study found that the algorithm used healthcare spending as a proxy for determining an individual’s healthcare need. But according to Scientific American, the healthcare costs of sicker Black patients were on par with the costs of healthier white people, which meant they received lower risk scores even when their need was greater.
The study’s researchers suggested that a few factors may have contributed. First, people of color are more likely to have lower incomes, which, even when insured, may make them less likely to access medical care. Implicit bias may also cause people of color to receive lower-quality care.
While the study did not name the algorithm or the developer, the researchers told Scientific American they were working with the developer to address the situation.
In March 2016, Microsoft learned that using Twitter interactions as training data for machine learning algorithms can have dismaying results.
Microsoft released Tay, an AI chatbot, on the social media platform. The company described it as an experiment in “conversational understanding.” The idea was the chatbot would assume the persona of a teen girl and interact with individuals via Twitter using a combination of machine learning and natural language processing. Microsoft seeded it with anonymized public data and some material pre-written by comedians, then set it loose to learn and evolve from its interactions on the social network.
Within 16 hours, the chatbot posted more than 95,000 tweets, and those tweets rapidly turned overtly racist, misogynist, and anti-Semitic. Microsoft quickly suspended the service for adjustments and ultimately pulled the plug.
“We are deeply sorry for the unintended offensive and hurtful tweets from Tay, which do not represent who we are or what we stand for, nor how we designed Tay,” Peter Lee, corporate vice president, Microsoft Research & Incubations (then corporate vice president of Microsoft Healthcare), wrote in a post on Microsoft’s official blog following the incident.
Lee noted that Tay’s predecessor, Xiaoice, released by Microsoft in China in 2014, had successfully had conversations with more than 40 million people in the two years prior to Tay’s release. What Microsoft didn’t take into account was that a group of Twitter users would immediately begin tweeting racist and misogynist comments to Tay. The bot quickly learned from that material and incorporated it into its own tweets.
“Although we had prepared for many types of abuses of the system, we had made a critical oversight for this specific attack. As a result, Tay tweeted wildly inappropriate and reprehensible words and images,” Lee wrote.
Like many large companies, Amazon is hungry for tools that can help its HR function screen applications for the best candidates. In 2014, Amazon started working on AI-powered recruiting software to do just that. There was only one problem: The system vastly preferred male candidates. In 2018, Reuters broke the news that Amazon had scrapped the project.
Amazon’s system gave candidates star ratings from 1 to 5. But the machine learning models at the heart of the system were trained on 10 years’ worth of resumes submitted to Amazon — most of them from men. As a result of that training data, the system started penalizing phrases in the resume that included the word “women’s” and even downgraded candidates from all-women colleges.
At the time, Amazon said the tool was never used by Amazon recruiters to evaluate candidates.
The company tried to edit the tool to make it neutral, but ultimately decided it could not guarantee it would not learn some other discriminatory way of sorting candidates and ended the project.
In 2012, an analytics project by retail titan Target showcased how much companies can learn about customers from their data. According to the New York Times, in 2002 Target’s marketing department started wondering how it could determine whether customers are pregnant. That line of inquiry led to a predictive analytics project that would famously lead the retailer to inadvertently reveal to a teenage girl’s family that she was pregnant. That, in turn, would lead to all manner of articles and marketing blogs citing the incident as part of advice for avoiding the “creepy factor.”
Target’s marketing department wanted to identify pregnant individuals because there are certain periods in life — pregnancy foremost among them — when people are most likely to radically change their buying habits. If Target could reach out to customers in that period, it could, for instance, cultivate new behaviors in those customers, getting them to turn to Target for groceries or clothing or other goods.
Like all other big retailers, Target had been collecting data on its customers via shopper codes, credit cards, surveys, and more. It mashed that data up with demographic data and third-party data it purchased. Crunching all that data enabled Target’s analytics team to determine that there were about 25 products sold by Target that could be analyzed together to generate a “pregnancy prediction” score. The marketing department could then target high-scoring customers with coupons and marketing messages.
Additional research would reveal that studying customers’ reproductive status could feel creepy to some of those customers. According to the Times, the company didn’t back away from its targeted marketing, but did start mixing in ads for things they knew pregnant women wouldn’t buy — including ads for lawn mowers next to ads for diapers — to make the ad mix feel random to the customer.
Thor Olavsrud covers data analytics, business intelligence, and data science for CIO.com. He resides in New York.
Sponsored Links
This article was autogenerated from a news feed from CDO TIMES selected high quality news and research sources. There was no editorial review conducted beyond that by CDO TIMES staff. Need help with any of the topics in our articles? Schedule your free CDO TIMES Tech Navigator call today to stay ahead of the curve and gain insider advantages to propel your business!
