AI AppSec: A Threat-Surface Model for Securing AI Applications

An AI application is not one thing to secure. It is a model that can be manipulated, a data pipeline that can be poisoned, an orchestration loop that can be hijacked, and a set of tool integrations that can be turned into an exit door. Most guidance flattens all of that into a single list of scary attacks and calls it a day. That flat list is where teams get hurt, because it tells you what can go wrong without telling you where the control belongs.

AI AppSec is the discipline of securing that whole surface. This page is the threat-surface model that sits above the framework deep-dives already in this cluster. It shows a practitioner where each risk lives, how the major frameworks compose instead of compete, and why agent governance is the layer that ties the controls together. If you already know traditional application security but the LLM and agent attack surface is new, start here, then follow the links into the specialist pages.

What is AI AppSec?

AI AppSec (AI application security) is the practice of protecting AI-powered applications across every layer they run on: the model, the data that feeds it, the orchestration logic that drives it, and the tools and systems it connects to. It extends traditional application security to the parts of an AI system that behave probabilistically, act with autonomy, and take instructions from untrusted input at runtime.

The audience is practical. Security engineers, AI and ML engineers, and security architects shipping production systems all need it, and so do the CTOs and compliance owners who answer for the risk. The reason it has its own name is simple: a large language model does not fail the way a web form fails. It can be argued into misbehaving, and once you wire it to tools and memory, a bad instruction becomes a bad action. That shift, from output to action, is why agentic AI security and AI agent security have become board-level concerns rather than a niche of LLM application security. For the commercial and platform view of that shift, see our guide to the enterprise AI platform.

AI AppSec vs. traditional AppSec vs. model safety

Three terms get used interchangeably and mean different things. Traditional AppSec secures the software around the model. Model safety concerns how the model itself behaves. AI AppSec is the connective discipline that spans both and adds the parts neither one owns.

Discipline	What it protects	Representative controls	What it misses on its own
Traditional AppSec	Code, dependencies, APIs, auth, infrastructure around the app	SAST/DAST, dependency scanning, WAF, secrets management, access control	Probabilistic model behavior; prompt-borne attacks; agent autonomy
Model safety	How the model responds: harmful, biased, or untruthful output	Alignment, RLHF, refusal behavior, content filtering, evaluations	The surrounding application, data pipeline, tools, and orchestration
AI AppSec	The whole application: model, data, orchestration, and integrations together	Everything above, plus input/output validation, tool least-privilege, agent identity, policy enforcement	Nothing by design; it is the layer that composes the others

The distinction matters because a model can be perfectly "safe" in isolation and still sit inside an application that leaks its training data or lets an agent wire money. Safety is necessary. It is not sufficient. AI AppSec is what makes the difference operational.

The AI application attack surface: four layers

Here is the core idea this page is built on. The AI application attack surface is not a list, it is a stack. Every risk you have read about lives in one of four layers, and the layer tells you where the defense belongs. This is the missing piece in most LLM security architecture and AI agent security architecture writing, which treats the attack surface as flat. Sound AI threat modeling starts by asking, for each layer, what an attacker controls and what they can reach.

Layer	What lives here	Representative threats	Where the control belongs
Model	The LLM or ML model and its immediate inputs/outputs	Prompt injection, jailbreaks, model manipulation, adversarial examples	Input/output validation, system-prompt hardening, guardrails
Data	Training data, embeddings, retrieval corpora, memory	Data poisoning, memory poisoning, sensitive-data disclosure, training-data extraction	Provenance, corpus hygiene, retrieval filtering, memory isolation
Orchestration	The agent loop: planning, reasoning, chaining, multi-step control flow	Goal hijacking, excessive autonomy, cascading failures, denial of wallet	Policy enforcement, human-in-the-loop, step and spend limits
Integration	Tools, APIs, plugins, MCP servers, other agents	Tool abuse, confused-deputy attacks, supply-chain and skill-discovery risk, data exfiltration	Tool least-privilege, agent identity, scoped credentials, allow-listing

The model layer

Model manipulation is the umbrella for everything that gets the model to do something it should not: prompt injection, jailbreaks, and adversarial inputs crafted to flip a classification or bypass a filter. Prompt injection ranks as the number-one risk in the OWASP Top 10 for LLM Applications, and it is dangerous precisely because the model cannot reliably tell instructions from data. The control here is not "make the model smarter." It is treating every model input and output as untrusted and validating both.

The data layer

The data layer covers everything the model learns from or retrieves at runtime: training sets, fine-tuning data, vector embeddings, retrieval corpora, and conversational memory. Poison any of these and you corrupt behavior downstream, often invisibly. Memory poisoning is the agentic version: a bad fact written to memory in one session steers decisions in the next. Retrieval systems have their own version of this problem, which is why securing RAG pipelines is treated as its own discipline. The control is provenance and hygiene: know where data came from, filter what gets retrieved, and isolate memory per trust boundary.

The orchestration layer

Orchestration is the agent loop itself: how the system plans, reasons, and chains steps toward a goal. It is the layer with the least prior art and the most novel risk. Long-horizon agents accumulate compounding risk across multi-step decision chains, so a small error early can cascade into a large failure late, which is a fundamentally different threat model than a single-shot API call. Goal hijacking, excessive autonomy, and "denial of wallet" (an agent looping until it burns your budget) all live here. The control is policy: explicit limits on what the loop may do, how many steps it may take, and when a human must approve.

The integration layer

The integration layer is where the agent touches the outside world: tools, APIs, plugins, Model Context Protocol servers, and other agents. This is where a manipulated model becomes a real-world action. Agent skill discovery, the process by which an agent finds and loads new tools or skills, widens this surface every time it runs, because each new capability is a new thing an attacker might reach through. Tool protocols carry their own risks worth understanding on their own; see MCP security risks. The controls are borrowed from decades of access control: least privilege per tool, scoped credentials, allow-listing, and treating each agent as a first-class identity with its own authenticated, authorized boundary.

AI security risk by deployment pattern

The four layers are constant. How much each one matters depends on how you deploy. Treating "AI" as one monolithic thing is the second mistake most guidance makes, right after flattening the attack surface. A read-only LLM API call and a multi-agent system that spends money share almost no risk profile. The matrix below maps where the pressure lands for each common pattern, which is the fastest way to reason about your own AI agent security risks.

Deployment pattern	Model	Data	Orchestration	Integration
LLM API (stateless)	High	Low	Low	Low
RAG (retrieval-augmented)	High	High	Low	Medium
Single agent (tool-using)	High	Medium	High	High
Multi-agent	High	Medium	Very high	Very high
Fine-tuned model	High	Very high	varies	varies

Read it as a targeting guide. A stateless LLM API is dominated by model-layer risk, so spend your effort on input and output validation. Add retrieval and the data layer lights up, which is the RAG story. Give the model tools and the orchestration and integration layers surge, which is where single-agent, autonomous agent, and multi-agent systems earn their reputation. Multi-agent orchestration is the hardest case because agents talk to agents and a compromise in one can propagate; the architectural trade-offs are worth understanding through the lens of multi-agent architecture (MCP vs A2A). Fine-tuning shifts weight onto the data layer, because your training set becomes part of the model and any poison or leakage in it is baked in.

The specific AI agent security attacks that exploit each pattern (goal hijacking chains, cross-agent injection, tool-poisoning, and their mitigations) go deeper than a pillar should. We will give that its own home; treat this matrix as the map and the attacks page as the terrain.

How AI security frameworks fit together

Every competitor page name-drops the same four frameworks and none of them explain how they relate. That is the real gap. NIST, OWASP, and MITRE ATLAS are not competing standards you pick between. They answer different questions, and a mature AI agent security framework uses all of them at once.

Framework	Question it answers	Where it maps on the surface	Go deeper
NIST AI RMF	How do we govern AI risk as an organization?	All four layers, at the program level (Govern, Map, Measure, Manage)	N/A
OWASP Top 10 for LLM	What are the concrete LLM vulnerabilities?	Model and data layers, primarily	OWASP Top 10 for LLM
OWASP Agentic risks	What breaks once the model can act?	Orchestration and integration layers	N/A
MITRE ATLAS	What tactics and techniques do real adversaries use?	Attacker view across all layers	MITRE ATLAS

Read together, they compose into a stack. The NIST AI Risk Management Framework gives you the governance scaffolding, organized into four functions (Govern, Map, Measure, Manage), and is the program-level umbrella. Inside it, the OWASP Top 10 for LLM enumerates the LLM security vulnerabilities at the model and data layers, and the OWASP Agentic Security guidance extends that into the orchestration and integration layers once the model can act. MITRE ATLAS supplies the adversary's point of view, cataloging real-world adversarial machine learning tactics and techniques the way ATT&CK does for traditional security. Use NIST to structure the program, OWASP to find the holes, and ATLAS to think like the attacker. None of them, on their own, tells you how to enforce the controls at runtime. Hold that thought.

Real-world AI security examples

Concrete AI security examples make the layers real. These are illustrative of the patterns security teams are seeing in production, mapped to where each one lives.

• Indirect prompt injection (model + integration). An agent summarizes a web page or email that contains hidden instructions. The model treats the content as commands and exfiltrates data through a connected tool. The injection enters at the model layer; the damage lands at the integration layer.
• RAG data leakage (data). A retrieval corpus contains documents a given user should not see. Because retrieval bypasses application-level permissions, the model surfaces restricted content in its answer, a quiet LLM security breach with no obvious exploit.
• Memory poisoning (data + orchestration). An attacker plants a false "fact" in an agent's long-term memory during one session. In a later session the agent acts on it, and because the decision looks internally consistent, nothing flags it.
• Excessive autonomy (orchestration + integration). An agent granted broad tool access and no spend limit loops on a failing task, racking up API cost or taking a destructive action no human approved.

The pattern across all four: the entry point and the impact are usually in different layers. That is exactly why a single-layer defense fails and why the surface has to be reasoned about as a whole.

AI AppSec best practices

Best practices only stick when they are tied to a layer. Generic advice like "validate input" is true and useless. Here is the layered version, which doubles as a starting set of LLM security best practices and AI agent security best practices.

1. Treat every model input and output as untrusted. Validate on the way in and the way out. Assume the model can be manipulated and design so that a bad response cannot become a bad action without a check. AI guardrails are the practical control layer here.
2. Enforce provenance and hygiene at the data layer. Know where training, fine-tuning, and retrieval data came from. Filter retrieval against the requesting user's permissions. Isolate agent memory per trust boundary so one session cannot poison another.
3. Constrain the orchestration loop with policy. Set explicit step limits, spend caps, and human-in-the-loop checkpoints for high-impact actions. An agent should never be able to act beyond what it is entitled to do.
4. Apply least privilege at the integration layer. Scope every tool credential to the minimum. Allow-list tools and skills rather than letting an agent discover and load anything. Give each agent its own authenticated, authorized identity.
5. Test adversarially and monitor continuously. Red-team the system the way an attacker would, and log agent decisions and tool calls so you can reconstruct what happened after the fact.

Choosing among the growing market of LLM security tools (guardrail frameworks, scanners, runtime monitors) is a buyer's-guide question in its own right, and we will cover it as a dedicated resource rather than shortchange it here. For the underlying protocol-level tool risk, the MCP security risks guide is the place to start.

Assessing and governing AI security posture

You cannot secure what you have not assessed. An AI agent security audit asks a specific question: given this agent's tools, data access, and autonomy, what is the worst it could do, and what stops it? That is the practical shape of an LLM security assessment, and it should run against each of the four layers rather than as a one-off checklist.

• Adversarial testing. Probe the system the way a real attacker would, across prompt injection, tool abuse, and multi-step chains. See AI red teaming for the discipline of adversarial testing for AI and LLMs.
• Continuous posture management. Security posture drifts as agents gain tools and data access. Managing it as an ongoing state, not a point-in-time audit, is the domain of AI security posture management (AISPM).
• Policy and governance. Assessment tells you where you stand. AI agent security and policy management is how you hold the line, by defining what each agent is entitled to do and enforcing it at runtime.

Agent governance: the missing control plane

Return to the thought we held earlier. NIST structures the program, OWASP finds the holes, ATLAS models the attacker, and best practices tell you what "good" looks like. None of them enforce anything. Frameworks are paper. Best practices are intentions. Between the policy you wrote and the action an agent takes at 3 a.m., something has to sit in the path and say yes or no.

That something is agent governance, the control plane this whole model has been building toward. It is where AI agent security and governance stops being a document and becomes a runtime decision: this agent, with this identity, requesting this tool, under this policy, is allowed or it is not. Vendor pages gesture at "zero trust for agents" and "identity-first security" without naming the connective layer. The connective layer is policy orchestration, and it is what turns every framework above into an enforced control instead of an aspiration. For the discipline in full, see our guide to AI agent governance.

Frequently asked questions

What is AI AppSec (AI application security)?

AI AppSec is the practice of securing AI-powered applications across every layer they run on: the model, the data that feeds it, the orchestration logic that drives it, and the tools it connects to. It extends traditional application security to cover probabilistic model behavior, prompt-borne attacks, and agent autonomy that classic controls were never built for.

How is AI application security different from traditional AppSec?

Traditional AppSec secures deterministic code, APIs, and infrastructure. AI AppSec adds the parts that behave probabilistically and act autonomously: a model can be argued into misbehaving, and an agent wired to tools turns a bad instruction into a real action. It keeps every traditional control and layers model, orchestration, and integration defenses on top.

What is the difference between model safety and AppSec?

Model safety concerns how the model itself behaves: avoiding harmful, biased, or untruthful output through alignment and filtering. AI AppSec secures the application around the model, including data, orchestration, and tool integrations. A model can be perfectly safe in isolation and still sit inside an insecure application, so safety is necessary but not sufficient.

What are the biggest AI agent security risks?

The highest-impact risks are prompt injection and model manipulation at the model layer, data and memory poisoning at the data layer, goal hijacking and excessive autonomy at the orchestration layer, and tool abuse and data exfiltration at the integration layer. Which dominates depends on your deployment pattern, from stateless API to multi-agent system.

How do NIST, OWASP, and MITRE ATLAS relate to each other?

They compose rather than compete. The NIST AI RMF provides program-level governance, the OWASP Top 10 for LLM and its agentic guidance enumerate concrete vulnerabilities across the model and orchestration layers, and MITRE ATLAS catalogs the tactics real adversaries use. Use NIST to structure, OWASP to find holes, and ATLAS to think like the attacker.

How do I audit an AI agent's security posture?

Assess each of the four layers in turn: what the model accepts, what data it can reach, what its orchestration loop may do, and which tools it can call. Ask what the agent could do at its worst and what stops it. Combine adversarial red-team testing with continuous posture management, since access and autonomy drift over time.

Where to go next

This page is the map. The cluster is the terrain. If you are deciding where to spend effort first, let your deployment pattern point you: LLM and model-layer risk lives in the OWASP Top 10 for LLM, the adversary's playbook lives in MITRE ATLAS, and retrieval-heavy systems should start with securing RAG pipelines.

The through-line is that frameworks describe risk but do not enforce anything. Agent governance is what turns policy into a runtime decision. See how the enterprise AI platform model brings the pieces together, and explore how Agen.co makes agent policy an enforced control rather than a document you file away.