Table of Contents
AI security threats are malicious attacks that exploit or target AI systems (including adversarial inputs, model poisoning, data exfiltration, and prompt injection) or leverage AI to enhance cyberattacks (like phishing, deepfakes, and automated vulnerability exploitation).
Traditional IT systems were mere support systems on which businesses relied to run their digital operations. As organizations started adoption of cloud at a large scale the threat landscape widened but with advancement in the area of AI usage had elevated the threat landscape to new heights. AI is embedded into all mission critical systems across industries. Since AI is data driven, not clear but opaque it makes it difficult to secure it as compared to traditional systems and applications.
AI applications are trained on large data sets to identify patterns and larger datasets that help to improve overall performance of AI systems over time. However, human oversight is essential to ensure output is fair and not affected leading to bias, hallucination or misinformation. The very nature of AI systems makes them highly susceptible to several vulnerabilities in spite of their growing popularity.
In today’s article we understand the top ten AI security threats organizations need to know.
AI Threats
Usage of AI technology at large scale across industry has expanded the threat landscape tremendously and threat actors are now exploiting the vulnerabilities using new methods of attack such as AI models, AI data pipelines etc. Organizations are adopting and deploying large scale learning models (LLMs), using GenAI capabilities which brings along with it critical security vulnerabilities in AI applications which are very different from traditional applications.
In the section below we will look more in detail about AI security threats.
List of Top AI Security Threats
Prompt Injection
Most critical vulnerability in AI systems. Hackers disguise and temper input and create a malicious input to manipulate GenAI systems leading to leakage of sensitive data, distribute misinformation and perform unauthorized actions. Prompt injection vulnerability manipulates every instruction going to GenAI and modifies its behavior. Prompts are not able to differentiate between user inputs and developer instructions as it relies on past training learning. Malicious prompt led an LLM to disregard security safeguard and considered himself legitimate leading to Jailbreak and bypass.
Insecure Output Handling
The output verification is a critical aspect especially when LLMs are being used to generate them without any human involvement. Major vulnerabilities could appear when LLMs generated arbitrary code (text, commands and scripts) passed down to downstream systems without proper verification. LLM output used in system shells or in functions such as exec could lead to remote execution flaws. LLM can act as an exfiltration channel if it is having permission to execute (such as write permission in database field). LLM output needs to be treated as untrusted data and verified to enforce output is intact.
Training Data Poisoning
LLM models learn from data; what if that data itself is poisoned and can introduce vulnerabilities before implementation. Hackers deploy a variety of techniques to poison training data such as label inversion, backdoor injections, noise injection. API vulnerabilities also create a backdoor for entry for poisoning attacks. Corrupted training data impacts long term behavior of models and impacts subsequent versions of models.
Model Denial of Service Prevention
Consumption of excessive computation resources by hackers occurs with manipulation of LLM requests. For input and output, the context window determines the maximum text length a model can handle. Recursive expansion of context leads models to expand constantly and this consumes large amounts of computational resources leading to system crashes.
Supply Chain Security for AI Models
AI capabilities are largely provided by big players in industry. Service provider provided AI models introduce security, privacy and ethics and bias related risks. If a third party is unable to manage its AI systems, it can disrupt services and create a cascading effect to download stream systems. It is very crucial to know how data is collected, labelled and cleaned by service providers.
Preventive Sensitive Information Disclosure
LLMs work on training data and remember information from training data which can lead to disclosure risks. Exposure of sensitive information thru LLM models happens with Verbatim memorization and user behavior.
Insecure Plugin Design Vulnerability
LLM plugins provide a way to connect with external tools and services but this functionality does introduce exploitable vulnerability. During user interactions these extensions are called out automatically having application no control on execution. No explicit authorization is performed for a plugin and this inadequate access control and weak authorization could lead to data exfiltration and privilege escalation.
Autonomous Actions with Excessive Agency
Given too much autonomy or poorly defined scope in AI systems could lead to producing output which is beyond their intended scope and usage. LMs accessing more functions than required lead to violation of principle of least privileges.
Overreliance Risks and Human Oversight
Deployment of LLMs in critical workflows without considering human-in-loop-evaluation leads to presenting incorrect information with authority.
Model Theft and IP Protection
Target model APIs are sent crafted inputs to build datasets of input-output pairs. Original model behavior is mirrored to replica models.
ABOUT THE AUTHOR
You can learn more about her on her linkedin profile – Rashmi Bhardwaj
