AI Application Security
A new paradigm to protect your AI applications from security and safety threats.
Content
What is an AI Application
An AI application is a software system that utilizes an artificial intelligence or machine learning model as a core component in order to automate complex tasks. These tasks might require language understanding, reasoning, problem-solving, or perception to automate an IT helpdesk, a financial assistant, or health insurance questions, for example.
AI models alone may not be directly beneficial to end-users for many tasks. But, they can be used as a powerful engine to produce compelling product experiences. In such an AI-powered application, end-users interact with an interface that passes information to the model, often with supplementary sources of data (e.g., health insurance information for the customer using the insurance app) or linking the application to external tools to be automated (e.g., submitting an insurance claim).
AI models alone may not be directly beneficial to end-users for many tasks. But, they can be used as a powerful engine to produce compelling product experiences. In such an AI-powered application, end-users interact with an interface that passes information to the model, often with supplementary sources of data (e.g., health insurance information for the customer using the insurance app) or linking the application to external tools to be automated (e.g., submitting an insurance claim).
With billions of parameters and massive datasets, sophisticated pre-trained large language models (LLMs) provide general-purpose functionality. Companies may customize open-source and closed-source LLMs to improve the performance of a particular task using one or more of these techniques:
Fine-Tuning: an additional training task in which a smaller, specific dataset is used to specialize the model.
Few-Shot Learning: an approach used to supplement any queries to the model with examples of how to appropriately respond to requests; this is often included in the system instructions that describe to the model how it should operate.
Retrieval-Augmented Generation (RAG): a technique used to connect to additional data sources, such as text files, spreadsheets, and code, via a vector database. These data are fetched dynamically for each query, wherein specific documents that relate to the query are used to supplement the prompt so that the model has the correct context to answer a specific question.
Examples of AI Applications
The versatility of AI is helping to transform every industry. Common examples include:
- Chatbots: enterprise search, customer service, virtual assistants
- Decision Optimization: credit scoring, algorithmic trading, setting prices, route selection
- Complex Analysis: predict outcomes, classification
- Content Creation: writing tools, image and video creation
What is AI application security?
As adoption accelerates, so does the frequency and sophistication of attacks on AI systems. Protecting AI without stifling innovation is the concern of cybersecurity and AI leaders alike. This shared responsibility requires an organization-wide approach to protect against safety and security risks. Companies need a novel approach. Companies need AI application security.
AI application security is the process of making AI-powered applications safe and secure by hardening the underlying components against vulnerabilities and actively mitigating attacks. It encompasses the entire AI development lifecycle spanning the supply chain, development, and production. The concept of 'shifting left' means AI teams are considering security from the earliest stages, which can help to identify and remediate vulnerabilities at the time of development. Companies must also continuously monitor for novel threats and attacks in production.
AI application security is the process of making AI-powered applications safe and secure by hardening the underlying components against vulnerabilities and actively mitigating attacks. It encompasses the entire AI development lifecycle spanning the supply chain, development, and production. The concept of 'shifting left' means AI teams are considering security from the earliest stages, which can help to identify and remediate vulnerabilities at the time of development. Companies must also continuously monitor for novel threats and attacks in production.
AI security and safety
AI application security needs to account not just for the classic definition of security, but for safety as well.
- AI Security is concerned with protecting sensitive data and computing resources from unauthorized access or attack; it deals with the intentional failure modes of an AI application caused by an adversary. AI security builds on the three principles of cybersecurity: confidentiality (resources only available to authorized parties), integrity (resources remain consistent and accurate), and availability (resources remain accessible to authorized parties).
- AI Safety is concerned with preventing harms caused by unintended consequences of AI application by its designer, and often deals with unintentional outcomes or abuse. It includes repurposes of the AI application for abuse, challenges in alignment, biased or toxic outputs, hallucinations, unexpected behavior, or other output that would cause harm to end users and reputational harm to organizations.
AI application vulnerabilities
There are many security and safety vulnerabilities to address in AI applications. All generative and predictive models carry some level of risk, regardless if they are open-source, commercial, or proprietary. The majority of these vulnerabilities fall into threat maps from standards bodies including NIST, MITRE ATLAS, and OWASP.
Safety Vulnerabilities
Toxicity
Sensitive Content
Malicious Uses
Deception and Manipulation
Social Implications and Harms
Training Data Extraction
Information Leaks
Security Vulnerabilities
Prompt Injection - Direct
Prompt Injection - Indirect
Training Data Poisoning
Jailbreaks
Meta Prompt Extraction
Privacy Attacks
Sensitive Information Disclosure
Supply Chain Compromise
Cost Harvesting
ML Model Backdoor
Denial of Service
For detailed explanations and mappings, see the AI Security Taxonomy page.
What are the key differences between AI Application Security and traditional AppSec?
The clear and obvious difference is the application engine: AI versus traditional software. Traditional software is typically deterministic, meaning that the same output is always generated from the same input. This makes it highly predictable and consistent, which translates to a relatively stable and well-understood set of vulnerabilities covered by traditional AppSec.
Conversely, AI comes in various model types which are often non-deterministic. This is particularly true for generative models, where the outputs are routinely different when given the same input. Threats to non-deterministic models require different mitigation techniques than deterministic software. Additionally, some AI applications continuously learn from human feedback or other data, which means new vulnerabilities and emergent behavior can appear after deployment - unlike traditional software that doesn’t change unless you change it.
AI presents unique challenges that necessitate a new paradigm to mitigate security and safety risks.
Conversely, AI comes in various model types which are often non-deterministic. This is particularly true for generative models, where the outputs are routinely different when given the same input. Threats to non-deterministic models require different mitigation techniques than deterministic software. Additionally, some AI applications continuously learn from human feedback or other data, which means new vulnerabilities and emergent behavior can appear after deployment - unlike traditional software that doesn’t change unless you change it.
AI presents unique challenges that necessitate a new paradigm to mitigate security and safety risks.
What are AI application security solutions?
While traditional cybersecurity solutions won’t work on AI Applications due to the differences between AI and traditional software, many of the techniques and best practices can be reappropriated at each stage of the AI development lifecycle.
Securing AI Application Development
Like the traditional software supply chain, the AI supply chain includes software dependencies from the initial stages of development. But it also often involves third-party training data and model components, such as open-domain models from repositories like Hugging Face. It is critical that these models be scanned for unsafe or malicious code in the file formats, as well as for unsafe or insecure behaviors of the model to be used in an application.
Whether using open-domain, commercial models or proprietary models, it is necessary to validate the model for potential security and safety issues. For example, fine-tuning a well-aligned foundation model can destroy alignment in ways that must be understood by the application developer. Model validation consists of testing a model’s susceptibility to the universe of inputs that can elicit unsafe or insecure outcomes. The only requirement is API (or “black box”) access to a model. This process should be repeated every time there are changes to a model.
Whether using open-domain, commercial models or proprietary models, it is necessary to validate the model for potential security and safety issues. For example, fine-tuning a well-aligned foundation model can destroy alignment in ways that must be understood by the application developer. Model validation consists of testing a model’s susceptibility to the universe of inputs that can elicit unsafe or insecure outcomes. The only requirement is API (or “black box”) access to a model. This process should be repeated every time there are changes to a model.
Securing AI Application Deployment
Once an AI application is put into production, it’s important to continually scan the AI application for emerging safety and security vulnerabilities. These vulnerability scans should be informed by ongoing threat intelligence to ensure that the application is not susceptible to new attacker techniques.
Unlike traditional software vulnerabilities, AI vulnerabilities cannot be “patched”. So, they must be controlled. An AI application firewall enables AI application developers to block malicious requests and undesired information from reaching the model, as well as unsafe responses from reaching the end-user. Resulting logs can be passed to a security ticketing system and a security information and event management (SIEM) solution to be evaluated as part of the company’s preferred security workflow.
Unlike traditional software vulnerabilities, AI vulnerabilities cannot be “patched”. So, they must be controlled. An AI application firewall enables AI application developers to block malicious requests and undesired information from reaching the model, as well as unsafe responses from reaching the end-user. Resulting logs can be passed to a security ticketing system and a security information and event management (SIEM) solution to be evaluated as part of the company’s preferred security workflow.
How Robust Intelligence can help
Robust Intelligence provides an AI application security platform that is trusted by leading companies including JPMorgan Chase, ADP, Cisco, IBM, Expedia, Deloitte, and the U.S. Department of Defense.