Artificial Intelligence

The 12 Hidden Risks of ChatGPT and Generative AI

David Sweenor

A Guide on What You Need to Know and What Countermeasures to Implement

Generative AI Risks – Photo by Author – David E. Sweenor

Introduction

At the end of November 2023, OpenAI’s ChatGPT had its first birthday. Since it first launched, its capabilities have evolved. Starting out as a mere chatbot, you can now converse, share pictures, generate images, and write code with it. According to the latest figures, ChatGPT has around 1.7 billion users worldwide–that’s quite impressive. What’s the appeal?

Well, for starters, it’s the Swiss Army Knife of applications that anyone can use, for just about anything. My colleagues and friends use it for personal fitness routines, website development, synthetic data creation, brainstorming buddies, therapists, blogs, emails, research, contract reviews, and even holiday recipes. I’m going to speculate that a vast majority of the 1.7 users are using the consumer version of ChatGPT and not the enterprise version. The enterprise version offers some data privacy protections not included in the consumer version provided by OpenAI.

Given the pervasiveness of the tool, users should understand the potential risks associated with GPT-4 and mitigation strategies. While some have been implemented by OpenAI, end users and adopting organizations should also have their own master plan. We’ve all heard about hallucinations, data privacy, and security–but there are nine other, lesser-known risks that businesses should understand. These risks are discussed in the GPT-4V system card provided by OpenAI.[1]This blog will break down the twelve inherent risks of GPT-4 and suggest strategies to militate against their effects. The mitigation strategies aren’t dogma, but rather, ideas to help your organization think about how to thoughtfully incorporate generative AI into your business.

Twelve Risks of ChatGPT and Generative AI

1. Wrong Answers and Confabulations 

Also known as hallucinations, generative AI models are not 100% accurate, making them unreliable. Remember, foundation models (FMs) and large language models (LLMs) are just big calculators. The only context they have across words, sounds, pixels, and code is that the next word or pixel generated is statistically likely to be the correct one–sort of like your Netflix or Spotify recommendations. Developers of AI systems can toggle these probabilities so there is a balance between not being too repetitive and still being accurate. 

For some reason, the term “hallucination” has become popular, which anthropomorphizes the technology. A few scientists discussed this in Hallucination or Confabulation? Neuroanatomy as metaphor in Large Language Models and created the following table.

Figure 1.1: Comparison of Terms Hallucination vs. Confabulation for LLMs[2]

Hallucinations vs. Confabulations

Source: Hallucination or Confabulation? Neuroanatomy as metaphor in Large Language Models

Without human-level reasoning, models readily produce false information and nonsensical suggestions that appear convincingly valid. This propensity to hallucinate fabricated facts, nonexistent entities, logical flaws, and technically broken solutions can dangerously mislead individuals and organizations if deployed without proper controls, monitoring, and governance.

Stanford’s Holistic Framework for Evaluating LLMs (HELM) tracks the accuracy of 119 LLMs across 116 scenarios resulting in 110 metrics. For accuracy, there are 16 different metrics tracked. Figure 1.2 is the mean win rate, which is the average fraction of other models that a model outperforms across the different scenarios. At the time of this writing, the 70 billion parameter Llama 2 is at the top of the leaderboard with a mean win rate of 94%.

Figure 1.2: Mean Win Rate Across Accuracy Metrics for LLMs[3]

Mean Win Rate Across LLMs

Ultimately, organizations need to understand what specific use cases they apply generative AI to and compare that to a risk framework. Ninety-four percent accuracy may be OK for SEO optimization but what about medical advice?

What countermeasures can you deploy?

  1. Develop both human-centric and technology-centric approaches to verify the accuracy of AI-generated content. Organizations can set up a system where model outputs are programmatically checked against trusted data sources before being used in decision-making or external communications. For instance, as I noted in my article Navigating the Generative AI Matrix, techniques like Chain-of-Verification (CoVe) can help improve the reliability of models, but hallucinations can always occur.[4]
  2. Bigger models aren’t always better. AI system developers should carefully consider all their options across the hundreds (if not thousands) of different models to select the suitable model for the right use case. Whatever model you end up choosing, consider augmenting it domain-specific for the tasks at hand through a combination of prompt engineering, retrieval augmented generation (RAG), and fine-tuning. 
  3. Add disclaimers and set up continuous monitoring mechanisms to track and analyze the performance of AI systems. This should be coupled with a feedback loop where wrong answers are reported back to the development team for model refinement. This is tough to do at scale, but most systems today have a “Report Issue,” “Thumbs Up”, and “Thumbs Down” type feature to help monitor and report outputs at scale.

2. Harmful Content

Since models were created with data that was indiscriminately harvested from the internet, they contain all of the hate speech, pornographic images, detailed instructions for how to plan attacks, and content that exploits people. For example, the abhorrent stories of deep fake nude pictures being used to abuse and victimize teenage girls are simply unconscionable.[5]

Thankfully, most LLM providers have implemented AI guardrails that help prevent the generation of these repugnant materials. However, recent research has suggested that the very act of fine-tuning these models can inadvertently override these safety guardrails, as mentioned in my article Mission AI Possible: Safeguarding Your Business with Self-destructing AI. Unless you’ve built your models from scratch with carefully curated, pristine data, these risks will always exist. So, what can you do about it?

What countermeasures can you deploy?

  1. Deploy content filtering and moderation systems trained to identify and filter out hate speech. Keep them up to date so they adapt as language drifts. Even with these, giant companies like Facebook and Tiktok haven’t been able to reliably implement content moderation at scale, so tread carefully–your company’s brand reputation is at stake.[6]
  2. Use red teaming to test and audit LLMs regularly for biases and hate speech. 
  3. As mentioned above, you should set up continuous monitoring mechanisms.

3. Perpetuation of Biases and Stereotypes

Similar to harmful content, LLMs can perpetuate biases and stereotypes. This includes perpetuating societal prejudices and stereotypes and the potential of the model to reinforce and reproduce specific biases and worldviews. The Washington Post’s story This is how AI image generators see the world provides examples of this with image models.[7]However, this is not limited to image generators; it’s embedded in all the different models. Stanford’s Human-Centered Artificial Intelligence (HAI) has also published a Demographic Stereotypes in Text-to-Image Generation on these stereotypes.[8]

What countermeasures can you deploy?

  1. When developing and training models, use a diverse data set. This should represent various demographics, including ethnicity, gender, and cultural backgrounds. And since our existing data is historically biased, consider using synthetic data to create a more balanced training set. This is easier said than done since most teams will start with a pre-built foundation model that already contains the bad data.
  2. Back to the governance board mentioned above, create an oversight committee comprising members from diverse disciplines, backgrounds, and viewpoints to review and assess the AI’s outputs for biases regularly.
  3. Similarly, continuous monitoring is also paramount.

4. Disinformation and Influence Operations

Well, I must admit, I’m not looking forward to the next U.S. election cycle. The amount of disinformation on social media and the internet is disheartening. It poses risks to information integrity and organizational reputation. Misinformation can erode trust in your brand, sway consumer opinions, and impact financial markets. For instance, a well-coordinated disinformation campaign could falsely implicate your company in unethical practices, rapidly spreading through social media and news outlets. This could be detrimental to an organization.

Sadly, there’s no reliable way to detect and mitigate these risks. As mentioned in my article Generative AI’s Powers and Perils: How Biden’s Executive Order is Reshaping the Tech Landscape, there are directives aimed at counteracting deep fakes and ensuring that what you read is authentic. However, techniques such as digital watermarking are still in their infancy and easily circumvented.

What countermeasures can you deploy?

  1. Don’t believe everything you read. Similar to anti-phishing training that most corporations have, invest in employee training programs focused on media literacy and critical thinking. These programs should teach employees how to identify and critically assess disinformation, encouraging a culture of skepticism towards unverified information.
  2. For critical decisions, ensure your organization has verification processes for all incoming information before it’s used in decision-making or disseminated further. This could involve fact-checking teams, leveraging AI tools to cross-reference information, and establishing partnerships with credible third-party verification services.
  3. Consider implementing a system so customers and employees know that communications are authentic. Again, the technology is not quite there, so it’s something to keep in mind as you develop systems.

5. Proliferation of Conventional and Unconventional Weapons

Also mentioned in the Biden Administration’s Executive Order (EO) on AI, civilian organizations and the military are looking at this quite closely. Businesses need to worry about how their product or services could be used for nefarious purposes through technology transfers, supply chain complexities, and data sharing. We’ve seen Nvidia’s GPU chips being blocked by export controls; what products and services are to follow?[9] In terms of effectiveness, given the proliferation of open-source models and leaks, I’m not sure what impact this will actually have, but I’ll leave that to the policymakers.

What countermeasures can you deploy?

  1. Double down on your compliance protocols and conduct thorough due diligence to ensure that the company’s products, technologies, and services are not misused for developing or enhancing weapons. This includes regularly reviewing supply chains and business partnerships to avoid indirect contributions to weapon proliferation.
  2. Partner with universities and focus on responsible AI research and development practices. Ensure that innovations, particularly in areas like AI, robotics, and materials science, are developed with ethical considerations and safeguards to prevent their application in weaponization.
  3. Cooperate and be transparent by actively participating in and supporting international efforts to control weapon proliferation. Advocate for global standards and regulations that prevent the misuse of commercial technologies in weapon development.

6. Privacy

Besides hallucinations, this is probably the second most discussed topic related to LLMs. Many of these models contain lots of data considered PII or sensitive. There are two aspects to consider. For the end-users who are not using an enterprise version of services like ChatGPT, the information entered is not protected and is, in fact, used to train the models for others. Putting sensitive or private information into the system is not advisable. 

The second thing to consider is the data used to train the model. A research team was able to extract a bunch of ChatGPT’s training data at scale for a couple of hundred dollars.[10] And with the latest version of GPT, the technical report states, “GPT-4 is a Transformer-style model pre-trained to predict the next token in a document, using both publicly available data (such as internet data) and data licensed from third-party providers…..Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.”[11]Thus, we may never truly understand the true risks of data privacy when relying on these models.

What countermeasures can you deploy?

  1. The threats are real, so take cybersecurity seriously, including advanced encryption, secure cloud services, and multi-factor authentication systems. Conduct periodic security audits to identify and rectify any vulnerabilities, ensuring your data protection measures are up-to-date and effective against evolving threats.
  2. I’ve mentioned governance earlier, but we’ll say it again: rely on your governance frame to define how data is collected, used, stored, and shared. As much as employees loathe going through these trainings, you should also invest in regular, mandatory training for all employees on data privacy and security protocols, emphasizing the importance of their role in safeguarding sensitive information.
  3. Have the legal team create clear privacy policies in line with standards like the EU’s AI Act and the U.S. AI Executive Order, ensuring they are transparently communicated to and consented to by your customers. Provide customers with easy-to-use tools to control their personal data, enhancing trust and brand reputation.

7. Cybersecurity

Cybersecurity is an evergrowing threat that demands focused attention and ongoing actions. For LLMs like GPT-4, there’s an added dimension to these risks since they effectively lower the cost and barriers (in terms of skills) to creating cyberattacks. For example, bad actors can use generative AI to create social engineering campaigns or find flaws in existing security tools. Unfortunately, malicious actors are now orchestrating sophisticated cyberattacks at an increased pace more efficiently.

What countermeasures can you deploy?

  1. Spend some money and invest in the latest and greatest cybersecurity technologies that monitor, detect, and mitigate threats in real-time. 
  2. Make sure your team is educated on phishing, secure passwords, and how to handle sensitive information. 
  3. Be sure to have an incident response plan detailing what actions should be taken during and after a cyberattack. Ensure that this plan includes not just containment and eradication strategies but also clear guidance for communication and recovery.

8. Potential for Risky Emergent Behaviors

Some would argue that as LLMs become larger and more powerful, they’ll crave more power and turn on humans Terminator-style. I’m not convinced. They’re not sentient–just big calculators. Since these are very large, complex neural networks, I would argue that similar to what we see with traditional AI (a.k.a. predictive AI), despite all of our best inventions, they could start generating biased decision-making recommendations. This can lead to skewed business insights or unfair customer experiences, thereby affecting the company’s reputation and compliance with regulations. As I argued in my article, GenAIOps: Evolving the MLOps Framework, it’s relatively well-known and straightforward on how to do this with numeric outputs. However, when we have words, code, images, audio, and video, to my knowledge, there’s no reliable programmatic way of monitoring for drift.

What countermeasures can you deploy?

  1. As with the other risks, implement a monitoring and auditing framework. 
  2. Understand AI ethics and build this into all you do (see my article, Generative AI Ethics).
  3. Conduct risk assessments and scenario planning to anticipate potential emergent behaviors and their impacts. Develop contingency plans to address various scenarios. See HBR’s article Use GenAI to Improve Scenario Planning for details on scenario planning.[12]

9. Interactions with Other Systems

Remember when COVID-19 hit and the world as we know it changed forever? For many businesses, COVID broke all of their predictive models. As systems become more complex and entangled with others, minor deviations in expected inputs and outputs could lead to compromised data quality, creating wildly unpredictable outcomes–like the proverbial butterfly flapping its wings and making a hurricane on the other side of the world. 

What countermeasures can you deploy?

  1. Like stress-testing banks, design AI systems that can adapt and respond to varying inputs and environments from other systems. This requires embedding contextual awareness into the AI, allowing it to adjust its responses based on the data and feedback from interconnected systems.
  2. Don’t become overreliant on AI alone. Sometimes, good old-fashioned business rules can act as a safety net to protect against the extremes.
  3. Don’t rely on any one system; build in redundancy and try to create a decentralized decision-making AI system. This will mitigate risks by ensuring that a single point of failure or misinterpretation in one system doesn’t lead to cascading errors across interconnected systems.

10. Economic Impacts

McKinsey estimates that generative AI will add up to four trillion dollars to the global economy.[13]

Companies should look at this through two lenses. The first is how generative AI will change their market dynamics–essentially, how much it will disrupt current business models. The second is the impact on an organization’s employees. Many knowledge workers have a certain amount of trepidation about how generative AI will impact their livelihood. Goldman Sachs estimates that generative AI will create more jobs than it destroys.[14] This may be true, but what if your job is eliminated? Organizations must walk a fine line between remaining competitive in the market and driving operational efficiencies with generative AI. For example, automating key business processes with AI might enhance efficiency but also require a shift in employee roles, demanding new skill sets and potentially impacting job security.

What countermeasures can you deploy?

  1. Don’t implement generative AI for the sake of AI. Make sure you create a clear business case and do some financial modeling to understand their potential economic impact on your business, including cost-benefit analysis and ROI projections. Be sure to include direct financial implications and indirect effects on workforce structure and market positioning.
  2. Prepare a strategic plan for employee development, including upskilling, reskilling, and AI expert augmentation strategies. This ensures that your workforce remains agile and capable of adapting to AI-induced changes in job roles and processes.
  3. Balance integrating AI technologies with the need for business continuity. Avoid big-bang projects and work towards quick wins and a few strategic projects. This should align with the company’s long-term economic goals and minimize disruption to existing revenue streams and operational models.

11. Acceleration

Generative AI development is proceeding at a meteoric pace. Although it seems unfeasible, major providers today may be irrelevant tomorrow. Also, technological breakthroughs from researchers and start-ups could quickly outdate your tech stack choices. The widespread adoption and pervasiveness also create challenges with governance and controls–laws haven’t caught up, and court systems are full of lawsuits. If not thought through carefully, your implementation could result in operational inefficiencies, potential legal issues, or public backlash, undermining the intended benefits of the technology. 

What countermeasures can you deploy?

  1. As previously mentioned, no intergalactic projects. Take a phased approach to integrating AI technologies with your business processes. Hopefully, this will make for a smoother transition with fewer disruptions. It also paves the way for organizational learning and feedback so you can adjust and optimize based on feedback and newly discovered needs. 
  2. Staff up the legal team and conduct compliance reviews of current regulatory standards and ethical guidelines.
  3. Provide plenty of budget for employee training opportunities so your team knows AI’s capabilities, what to watch out for, and how to use it effectively. 

12. Overreliance 

This is an interesting topic that goes back to the many questions I’ve received on predictive and prescriptive analytics (see my article Generative AI vs. Traditional AI: What’s Better). Just because the algorithm gives you a recommendation does not mean you have to follow the recommendation. With generative AI, companies can develop an overreliance on the system outputs by users, which could stem from its convincing outputs, which may be factually inaccurate or completely made up.

Remember, LMMs are big calculators and do not know intent, meaning, or human values – they simply know the next best word to predict. If we rely too much on generative AI, we’ll get dumber, diminishing our capacity for human judgment and leading to increased vulnerabilities and systemic failures. For example, automated decision-making systems might overlook nuanced or unprecedented situations requiring human insight, potentially leading to flawed business strategies or missed opportunities.

What countermeasures can you deploy?

  1. Integrate a human-in-the-loop system where skilled professionals review and validate critical decisions made by AI. This approach ensures a balance between AI efficiency and human judgment, reducing the risk of errors that could arise from sole reliance on AI.
  2. Encourage diverse decision-making processes that involve a combination of AI insights and human expertise. This diversity in approach helps mitigate risks associated with overreliance on AI, ensuring more comprehensive and well-rounded business decisions.
  3. Routinely evaluate the performance and outputs of AI systems against established benchmarks and real-world outcomes. Regular assessments help identify areas where AI reliance is becoming excessive and where human intervention might be more appropriate.

Summary

Now that we understand the twelve hidden risks of generative AI, organizations will need to strike a balance between how quickly to implement the technology and maintaining oversight and control. It’s certainly true that AI can significantly enhance efficiency and innovation, but it also introduces complexities in decision-making, ethical considerations, and potential operational risks. Take a proactive, pragmatic approach involving continuous learning, adaptive strategies, and ethical AI practices. Business leaders should proceed smartly and consider how AI aligns with their core business values and objectives, ensuring its use drives progress and adheres to the highest standards of responsibility and transparency. 

If you enjoyed this article, please like it, highlight interesting sections, and share comments. Consider following me onMedium and LinkedIn.

If you’re interested in this topic, consider TinyTechGuides’ latest report, The CIO’s Guide to Adopting Generative AI: Five Keys to Success or Artificial Intelligence: An Executive Guide to Make AI Work for Your Business.

[1] “GPT-4V(Ision) System Card.” 2023. Openai.com. September 25, 2023. https://openai.com/research/gpt-4v-system-card.

[2] Smith, Andrew L, Felix Greaves, and Trishan Panch. 2023. “Hallucination or Confabulation? Neuroanatomy as Metaphor in Large Language Models.” PLOS Digital Health 2 (11): e0000388–88.https://doi.org/10.1371/journal.pdig.0000388.

[3] “Holistic Evaluation of Language Models (HELM).” 2023. Crfm.stanford.edu. November 29, 2023. https://crfm.stanford.edu/helm/latest/#/leaderboard.

[4] Dhuliawala, Shehzaad, Meta Ai, Eth Zürich, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. “CHAIN-OF-VERIFICATION REDUCES HALLUCINATION in LARGE LANGUAGE MODELS.” https://arxiv.org/pdf/2309.11495.pdf.

[5] Hadero, Haleluya. 2023. “AI-Generated Nude Images of Teen Girls Spur Families to Push for Protections: ‘We’re Fighting for Our Children.’” Fortune. December 2, 2023. https://fortune.com/2023/12/02/ai-generated-nude-images-teen-girls-deepfakes-tech-safety-children-parents/. machine learning model.8 By taking these steps now,

[6] Peck, Henry. 2022. “Exposing Social Media Platforms’ Failures to Protect Their Users.” Global Witness. December 20, 2022. https://www.globalwitness.org/en/blog/exposing-social-media-platforms-failures-to-protect-their-users/.

[7] Tiku, Nitasha, Kevin Schaul, and Szu Yu Chen. 2023. “These Fake Images Reveal How AI Amplifies Our Worst Stereotypes.” Washington Post. November 1, 20. We must be aware of potential risks and consider our ethical responsibility before deploying any23. https://washingtonpost.com/technology/interactive/2023/ai-generated-images-bias-racism-sexism-stereotypes/.

[8] Bianchi, Federico, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2023. “Policy Brief Demographic Stereotypes | Stanford HAI.” Hai.stanford.edu. November 30, 2023. https://hai.stanford.edu/policy-brief-demographic-stereotypes.

[9] Cherney, Max A., and Stephen Nellis. 2023. “Nvidia Details Advanced AI Chips Blocked by New Export Controls.” Reuters, October 17, 2023, sec. Technology. https://www.reuters.com/technology/nvidia-may-be-forced-shift-out-some-countries-after-new-us-export-curbs-2023-10-17/.

[10] Nasr, Milad, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. “Scalable Extraction of Training Data from (Production) Language Models.” ArXiv.org. November 28, 2023. https://doi.org/10.48550/arXiv.2311.17035.

[11] OpenAI. 2023. “GPT-4 Technical Report.” ArXiv (Cornell University), March. https://doi.org/10.48550/arxiv.2303.08774.

[12] Finkenstadt, Daniel J., Tojin T. Eapen, Jake Sotiriadis, and Peter Guinto. 2023. “Use GenAI to Improve Scenario Planning.” Harvard Business Review. November 30, 2023. https://hbr.org/2023/11/use-genai-to-improve-scenario-planning.

[13] Mckinsey & Company. 2023. “Economic Potential of Generative AI | McKinsey.” Www.mckinsey.com. June 14, 2023. https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier#introduction.

[14] Goldman Sachs. 2023. “Generative AI Could Raise Global GDP by 7%.” Goldman Sachs. April 5, 2023. https://www.goldmansachs.com/intelligence/pages/generative-ai-could-raise-global-gdp-by-7-percent.html.

Related Posts