AI Risk Management

The new paradigm to manage AI risk.

What is AI risk?

AI risk management has become a best practice among enterprises in order to measure and mitigate risk to meet organizational requirements. To fully understand this, we must first define AI risk.

AI risk can be expressed as a probabilistic expectation:

AI risk = (likelihood of an AI model error or exploit) x (its potential effect)

This definition is instructive because it provides a recipe for action. AI errors and vulnerabilities are many, occur frequently, and vary across model tasks and modalities. Examples include arbitrary code execution, data poisoning, prompt injection, model extraction, hallucinations, data drift, unexpected behavior, bias predictions, and toxic output.

The effects of a model error depend largely on the use case. They can be financial, legal, or reputational. More importantly, they can have devastating consequences for users, such as leaked personal information, denied healthcare coverage, or declined loan approval decisions.

How is AI risk different from traditional software risk?

AI models are not just a function of the underlying “code”; they are fundamentally also a function of the data. In traditional software functions, one defines rigorous abstractions rooted in procedural software logic in order to transform an input into a given output. Yet the very advantage of AI is that it can “learn” the logic from a corpus of data, rather than the user explicitly defining it. Because the user cannot directly observe this logic, this makes the burden of testing much harder on the user. Simply testing on a few datapoints and edge cases is no longer enough to assert “correctness”. The user must now rigorously test the performance of the model with evaluation datasets, and assert that the model can also make correct predictions on a potentially infinite set of possible data distributions.

A corollary is that the metrics used to evaluate the performance of a AI model are different from those used to test software. This is because AI models are approximate learners on noisy data; in most practical settings, it is impossible for models to be 100% correct on the entire data distribution. For example, accuracy is a common metric for evaluating the performance of a classification model, whereas software is often tested for functionality and usability. In addition, simply testing top-level metrics on a dataset is not enough. There are many other dimensions in which AI models can fail.

AI models also come in a wide array of modalities/tasks, ranging from binary classification on tabular data to language-based responses from generative AI. Each modality and tasks come with their own unique failure modes and testing challenges in the dimensions mentioned above. Moreover, since AI involves automating critical decisions, there are test that we would like to run that pertain to other aspects of integrity of the system. This includes testing against bias and fairness, as well as abuses to sensitive information that may be encoded in the model and data.

AI is presents unique challenges that necessitate a new paradigm to mitigate risk.

What is AI risk management?

AI risk management refers to a suite of tools and practices deployed to proactively protect organizations and end users from the distinctive risks of AI. It involves measuring the risks, and implementing solutions to minimize them. As per the definition of AI risk, it can be minimized by reducing the likelihood of a failure or by reducing the impact severity. Although it may resemble some software engineering best practices, AI risk management is entirely different.

According to NIST, as defined in their AI Risk Management Framework:
AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.

Why do companies need AI risk management?


Security Risk

Security risk refers to AI system vulnerabilities that may be exploited maliciously or inadvertently added by end users to generate an unintended output. This can include attacks on the model, data, or underlying software. Proprietary, commercial, and open-source models may be susceptible to a variety of threats including supply chain risk, data poisoning, prompt injection, PII extraction, and model theft.

Ethical Risk

Ethical risk stems from model behavior that violates norms, laws, regulations, or other governance standards. This may be present in the training data or be the result of production data over time. Examples include bias predictions, toxic outputs, exclusiveness, and prejudice responses.

Operational Risk

Operational risk is the result of deviations in model predictions. This can be the result of data drift, hallucination, corrupt data, corner case inputs, and broken data pipelines. Silent model failures are especially difficult to detect, as they don’t crash the model outright and have subtle effects on downstream metrics.

AI risk management as an industry standard

AI risk is the subject of existing and forthcoming AI standards and regulation. The landscape is comprised of voluntary frameworks, guidelines, and legislation. Many companies are adopting AI risk management now in order to get ahead of anticipated legislation and additional regulation.

The most notable examples include:
The U.S. Congress recognized the need for guidance in AI risk management and mandated NIST (National Institute of Standards and Technology) to create a framework to do just that. In January, 2023 they released the first version of the NIST AI Risk Management Framework.
The Biden-Harris Administration secured voluntary commitments from 15 large AI companies to adhere to specific AI risk management measures. This agreement serves to accomplish two goals: increase the public’s confidence in AI services developed by “big tech” and serve as a benchmark for what enterprises should require from their own AI systems, including the vendors in their AI stack.
The EU AI Act is a first of its kind legislative initiative aimed at regulating AI on a risk-based approach. The text is very prescriptive for high-risk AI providers, notably in risk management, data governance, technical documentation, and record keeping.

Automating AI risk management

Manual evaluation can be slow and expensive. Large enterprises spend millions of dollars to manually discover errors in their AI and engineer mitigations. Using tools to automate AI risk management can turn passive frameworks into active practices. This automation helps to proactively measure and mitigate risk, giving companies the confidence to deploy AI at scale. There are many benefits, including:

Real-time validation

It should be expected that all models will receive malicious inputs and generate undesired outputs; generative models are especially vulnerable since outputs are passed to users immediately. Real-time validation of inputs and outputs is an essential form of automation that is required to protect production models.

Comprehensive testing

The non-deterministic nature of AI models make it difficult to predict failures. Automated, comprehensive testing is far superior to manual, ad hoc testing of models and data. By running hundreds of tests and algorithmically-generated red teaming attacks, you can ensure your models are rigorously validated. This also helps enforce a uniform risk standard across the company rather than rely on the subjectivity of any individual or team.

Resource efficiency

Manual validation of models is slow and expensive. It can take upwards of 80 hours to test each model and map the results to an AI risk management framework, as opposed to a fraction of that with automated validation. This process needs to be repeated each time a new version of a model is released, which happens frequently with commercial and open-source models. An automated process also removes the validation burden from data scientists, who are typically not experts in governance and compliance.

How Robust Intelligence can help

Robust Intelligence provides a platform for end-to-end AI risk management that is trusted by leading companies including JPMorgan Chase, Cisco, Expedia, Deloitte, PwC, and the U.S. Department of Defense.