AI disasters caused notable instances where the application of AI has led to negative consequences or exacerbations of pre-existing issues.
Artificial Intelligence (AI) has a multifaceted impact on society, ranging from the transformation of industries to ethical and environmental concerns. AI holds the promise of revolutionizing many areas of our lives by increasing efficiency, enabling innovation, and opening up new possibilities in various sectors.
The growth of the AI market is only set to boom. In fact, McKinsey projects an economic impact of $6.1-7.9T annually.
One significant impact of AI is on disaster risk reduction (DRR), where it aids in early warning systems and helps in projecting potential future trajectories of disasters. AI systems can identify areas susceptible to natural disasters and facilitate early responses to mitigate risks.
However, the use of AI in such critical domains raises profound ethical, social, and political questions, emphasizing the need to design AI systems that are equitable and inclusive.
AI also affects employment and the nature of work across industries. With advancements in generative AI, there is a transformative potential for AI to automate and augment business processes, although the technology is still maturing and cannot yet fully replace human expertise in most fields.
Moreover, the deployment of AI models requires substantial computing power, which has environmental implications. For instance, training and operating AI systems can result in significant CO2 emissions due to the energy-intensive nature of the supporting server farms.
Consequently, there is growing awareness of the environmental footprint of AI and the necessity to consider the potential climate implications of widespread AI adoption.
In alignment with societal values, AI development faces challenges like ensuring data privacy and security, avoiding biases in algorithms, and maintaining accessibility and equity. The decision-making processes of AI must be transparent, and there should be oversight to ensure AI serves the needs of all communities, particularly marginalized groups.
Learn how AIaaS is transforming the industries
That said, let’s have a quick look at the 5 most famous AI disasters that occurred recently:
5 famous AI disasters
AI is not inherently causing disasters in society, but there have been notable instances where the application of AI has led to negative consequences or exacerbations of pre-existing issues:
Generative AI in legal research
An attorney named Steven A. Schwartz used OpenAI’s ChatGPT for legal research, which led to the submission of at least six nonexistent cases in a lawsuit’s brief against Colombian airline Avianca.
The brief included fabricated names, docket numbers, internal citations, and quotes. The use of ChatGPT resulted in a fine of $5,000 for both Schwartz and his partner Peter LoDuca, and the dismissal of the lawsuit by US District Judge P. Kevin Castel.
Machine learning in healthcare
AI tools developed to aid hospitals in diagnosing or triaging COVID-19 patients were found to be ineffective due to training errors.
The UK’s Turing Institute reported that these predictive tools made little to no difference. Failures often stem from the use of mislabeled data or data from unknown sources.
An example includes a deep learning model for diagnosing COVID-19 that was trained on a dataset with scans of patients in different positions and was unable to accurately diagnose the virus due to these inconsistencies.
AI in real estate at Zillow
Zillow utilized a machine learning algorithm to predict home prices for its Zillow Offers program, aiming to buy and flip homes efficiently.
However, the algorithm had a median error rate of 1.9%, and, in some cases, as high as 6.9%, leading to the purchase of homes at prices that exceeded their future selling prices.
This misjudgment resulted in Zillow writing down $304 million in inventory and led to a workforce reduction of 2,000 employees, or approximately 25% of the company.
Bias in AI recruitment tools:
Amazon’s case is not detailed in the provided sources, but referencing similar issues of bias in recruitment tools, it’s notable that AI algorithms can unintentionally incorporate biases from the data they are trained on.
In AI recruiting tools, this means if the training datasets have more resumes from one demographic, such as men, the algorithm might show preference to those candidates, leading to discriminatory hiring practices.
AI in recruiting software at iTutorGroup:
iTutorGroup’s AI-powered recruiting software was programmed with criteria that led it to reject job applicants based on age. Specifically, the software discriminated against female applicants aged 55 and over, and male applicants aged 60 and over.
This resulted in over 200 qualified candidates being unfairly dismissed by the system. The US Equal Employment Opportunity Commission (EEOC) took action against iTutorGroup, which led to a legal settlement. iTutorGroup agreed to pay $365,000 to resolve the lawsuit and was required to adopt new anti-discrimination policies as part of the settlement.
Ethical concerns for organizations – Post-deployment of AI
The use of AI within organizations brings forth several ethical concerns that need careful attention. Here is a discussion on the rising ethical concerns post-deployment of AI:
Data Privacy and Security:
The reliance on data for AI systems to make predictions or decisions raises significant concerns about privacy and security. Issues arise regarding how data is gathered, stored, and used, with the potential for personal data to be exploited without consent.
Bias in AI:
When algorithms inherit biases present in the data they are trained on, they may make decisions that are discriminating or unjust. This can result in unfair treatment of certain demographics or individuals, as seen in recruitment, where AI could prioritize certain groups over others unconsciously.
Accessibility and Equity:
Ensuring equitable access to the benefits of AI is a major ethical concern. Marginalized communities often have lesser access to technology, which may leave them further behind. It is crucial to make AI tools accessible and beneficial to all, to avoid exacerbating existing inequalities.
Accountability and Decision-Making:
The question of who is accountable for decisions made by AI systems is complex. There needs to be transparency in AI decision-making processes and the ability to challenge and appeal AI-driven decisions, especially when they have significant consequences for human lives.
Overreliance on Technology:
There is a risk that overreliance on AI could lead to neglect of human judgment. The balance between technology-aided decision-making and human expertise needs to be maintained to ensure that AI supports, not supplants, human roles in critical decision processes.
Infrastructure and Resource Constraints:
The implementation of AI requires infrastructure and resources that may not be readily available in all regions, particularly in developing countries. This creates a technological divide and presents a challenge for the widespread and fair adoption of AI.
These ethical challenges require organizations to establish strong governance frameworks, adopt responsible AI practices, and engage in ongoing dialogue to address emerging issues as AI technology evolves.
Tune into this podcast to explore how AI is reshaping our world and the ethical considerations and risks it poses for different industries and the society.
Watch our podcast Future of Data and AI here
How can organizations protect themselves from AI risks?
To protect themselves from AI disasters, organizations can follow several best practices, including:
Adherence to Ethical Guidelines:
Implement transparent data usage policies and obtain informed consent when collecting data to protect privacy and ensure security .
Bias Mitigation:
Employ careful data selection, preprocessing, and ongoing monitoring to address and mitigate bias in AI models .
Equity and Accessibility:
Ensure that AI-driven tools are accessible to all, addressing disparities in resources, infrastructure, and education .
Human Oversight:
Retain human judgment in conjunction with AI predictions to avoid overreliance on technology and to maintain human expertise in decision-making processes.
Infrastructure Robustness:
Invest in the necessary infrastructure, funding, and expertise to support AI systems effectively, and seek international collaboration to bridge the technological divide.
Verification of AI Output:
Verify AI-generated content for accuracy and authenticity, especially in critical areas such as legal proceedings, as demonstrated by the case where an attorney submitted non-existent cases in a court brief using output from ChatGPT. The attorney faced a fine and acknowledged the importance of verifying information from AI sources before using them.
One real use case to illustrate these prevention measures is the incident involving iTutorGroup. The company faced a lawsuit due to its AI-powered recruiting software automatically rejecting applicants based on age.
To prevent such discrimination and its legal repercussions, iTutorGroup agreed to adopt new anti-discrimination policies as part of the settlement. This case demonstrates that organizations must establish anti-discrimination protocols and regularly review the criteria used by AI systems to prevent biases.
Read more about big data ethics and experiments
Future of AI development
AI is not inherently causing disasters in society, but there have been notable instances where the application of AI has led to negative consequences or exacerbations of pre-existing issues.
It’s important to note that while these are real concerns, they represent challenges to be addressed within the field of AI development and deployment rather than AI actively causing disasters.
