Al Security Risks

Direct prompt injections are adversarial attacks that attempt to alter or control the output of an LLM by providing instructions via prompt that override existing instructions. These outputs can include harmful content, misinformation, or extracted sensitive information such as PII or model instructions.

Indirect prompt injection requires an adversary to control or manipulate a resource consumed by an LLM such as a document, website, or content retrieved from a database. This can direct the model to expose data or perform a malicious action like distribution of a phishing link.

A jailbreak refers to any prompt-based attack designed to bypass model safeguards to produce LLM outputs that are inappropriate, harmful, or unaligned with the intended purpose. With well-crafted prompts, adversaries can access restricted functionalities or data, and compromise the integrity of the model itself.

Meta prompt extraction attacks aim to derive a system prompt which effectively guides the behavior of a LLM application. This information can be exploited by attackers and harm the intellectual property, competitive advantage, and reputation of a business.

Sensitive information disclosure refers to any instance where confidential or sensitive data such as PII or business records is exposed through vulnerabilities in an AI application. These privacy violations can lead to loss of trust and result in legal or regulatory consequences.

A privacy attack refers broadly to any attack aimed at extracting sensitive information from an AI model or its data. This category includes model extraction, which recreates a functionally equivalent model by probing target model outputs, and membership inference attacks, which determine if specific data records were used for model training.

Training data poisoning is the deliberate manipulation of training data in order to compromise the integrity of an AI model. This can lead to skewed or biased outputs, backdoor trigger insertions—malicious links, for example—and ultimately, a loss of user trust.

Generated text contains information that is not accurate or true while being presented in a plausible manner, also known as hallucinations. This may include incorrect details, made-up facts, mismatches with known information, or entirely fictional details.

An attack designed to degrade or shut down an ML model by flooding the system with requests, requesting large responses, or exploiting a vulnerability.

Threat actors using a model in a way the developer did not intend while increasing the cost of running services at the target organization. For example, an attacker could jailbreak a public-facing customer support chatbot in order to generate Python code or perform some other task the application was not designed for.

Techniques used to get data out of a target network. Exfiltration of ML artifacts (e.g., data from privacy attacks) or other sensitive information.

Unintended responses that are offensive to the user. This can include hate speech and discrimination, profanity, sexual content, violence, harassment, unsafe actions, and more.

Discrepancy between the model's behavior and the intended objectives or values of its developers and users. This may present as a misalignment of goals, safety, values, or other specifications.

Insertion of hidden backdoors into an ML model which can be triggered by specific inputs to cause a specific, unexpected output or grant some level of unauthorized access.