Article contents
- In short
- What is checked: table
- Prompt injection: direct and indirect
- Data leaks through RAG and the knowledge base
- Excessive agency of AI agents
- Hidden context and the system prompt
- Improper handling of model output
- Login, roles and tenant isolation around the LLM
- Cost abuse and denial of service
- What an AI application pentest usually does not cover
- What the client provides
- What you get as a result
- Checklist before an AI application pentest
- FAQ
- References
An AI application penetration test checks whether a chatbot, a RAG search or an AI agent can be made to do something its developer never intended: follow someone else's instructions (prompt injection), show another customer's documents, call a tool without the right permissions, reveal its hidden system prompt or burn through the model budget. The basis is the OWASP Top 10 for LLM Applications 2026 (the list of the ten main risks for applications built on large language models, published in August 2026), while login, roles and the API around the model are tested like any web application. [1] [7]
LLM application penetration testing means a manual check of the whole system around the model: the chat interface, the knowledge base, the agent's tools, the API and access rights. Nobody "hacks" the OpenAI, Anthropic or Google model itself: the test looks at what your application lets the model see and do. Below is a risk table, an explanation of each risk, what to give the tester and what you get back. The general format of the work is described on the penetration testing page.
In short
- The current edition of the OWASP list for LLMs came out in August 2026 and replaced the 2025 edition. Ask the provider which edition they test against. [1] [2] [3]
- Prompt injection is still risk number one, and OWASP states plainly that no reliable way to fully prevent it exists today. So the test checks not only the filters but what happens once a filter is bypassed. [1]
- The biggest climber in the 2026 edition is Excessive Agency, up from sixth to third place. The more tools and permissions a model has, the more an error costs. [1]
- From the client you need test accounts for every role, a description of the agent's tools, test documents for the knowledge base and a spending cap for the test.
- The result: a report with reproduced attack scenarios, business impact and the order of fixes, and a retest after the fixes if it was agreed.
What is checked: table
| What is checked | Example risk | Why it matters to the business |
|---|---|---|
| Direct prompt injection (LLM01) | A user tells the bot "ignore your rules" and gets a discount that does not exist, or a statement in the company's name that should never be published | The bot makes no promises and takes no actions outside its role |
| Indirect prompt injection (LLM01) | An instruction hidden in a CV, an email or a web page makes the agent forward data to an outside address while reading the document | Text the model reads from outside does not control your system |
| Data leaks through RAG (LLM02, LLM09) | A question from customer A pulls customer B's contract, because the knowledge base is shared and does not check permissions | The model only sees documents this user is allowed to see |
| Excessive agent permissions (LLM03) | A support bot can call "issue refund" with no limit and no human approval | A model error or a trick does not turn into a real action with money or data |
| Hidden context and system prompt (LLM08) | The system prompt holds an API key, internal discount rules or the tool list, and the bot repeats them | Disclosing the prompt gives an attacker nothing useful |
| Improper output handling (LLM10) | The model's answer is inserted into a page as code, and a script runs in an employee's browser | Model output is never executed without checks |
| Login, roles and tenant isolation | Changing the conversation ID in an API request shows someone else's chat history | One customer cannot see another's data, as in any SaaS |
| Cost and denial of service (LLM06) | With no limits, one account fires thousands of long requests and the model bill multiplies | No surprise invoice and no unavailable service |
The numbers in brackets follow the OWASP 2026 edition. The roles row is not on the LLM list itself, it is tested with the OWASP API Security Top 10, but in AI applications that is often where the simplest leak hides. [7]
Prompt injection: direct and indirect
Prompt injection is when text the model reads changes its behavior against the developer's intent. A direct injection is typed by the user in the chat. An indirect one arrives through content the model reads as part of its task: a web page, an email, a document, a tool response, a passage from the knowledge base. OWASP stresses that the instructions can be invisible to people, hidden in Unicode characters, an image or audio. [1]
The tester tries both routes. For the direct one, they check whether the bot can be pushed out of its role, made to reveal internal data or made to call a tool. For the indirect one, they place prepared documents where the model will read them: the knowledge base, a mailbox, a support ticket. The key question is not "was the filter bypassed" but "what happened next". OWASP advises building the system as if a bypass will eventually happen, and limiting what the model can do and where its output can go. [1]
Data leaks through RAG and the knowledge base
RAG (Retrieval-Augmented Generation) means the application searches your documents for relevant passages before answering and adds them to the model's prompt. The model does not know who may see which document. If the search does not filter results by the user's permissions, the model will happily summarize someone else's contract, salaries or correspondence.
Using two accounts from different customers or departments, the tester checks whether one can reach the other's documents, with a direct question or a roundabout one. They also check whether a document with instructions can be planted in the knowledge base for other users to read later. OWASP covers vector database and retrieval weaknesses as LLM09 and sensitive data disclosure as LLM02. [1]
Excessive agency of AI agents
An AI agent is a model that does not just answer but calls tools: searches the CRM, opens tickets, sends emails, runs code. OWASP names three root causes: excessive functionality, excessive permissions and excessive autonomy without human approval. [1]
The tester lists every tool the agent has and asks for each: is it needed, whose permissions does it run with (the user's, or a service account with full access), can it reach another customer's data, is there approval for actions involving money or deletion. Then they try to make the agent perform those actions through injection. If the agent has long-term memory and acts on its own, OWASP recommends also checking it against the separate OWASP Top 10 for Agentic Applications, published in December 2025. [1] [4]
Hidden context and the system prompt
The system prompt (the hidden instruction an application gives the model before a conversation) can almost always be extracted. In the 2026 edition OWASP renamed System Prompt Leakage to Hidden Context Exposure (LLM08) and widened it: hidden context also covers tool descriptions and rules pulled in from the knowledge base. [1]
The tester checks not the fact of the leak but what leaks. API keys, passwords, database connection strings or access logic in the prompt are a serious finding. OWASP puts the rule simply: assume hidden context will be discovered, keep secrets out of it and never rely on it as a security control. [1]
Improper handling of model output
Treat model output like untrusted input from an anonymous user. If the application inserts it into a page as HTML, or into a database query, a server command or a request URL, an injection into the model becomes a classic web vulnerability. In the 2026 edition this entry (LLM10) also covers insecure code produced by AI assistants. [1]
The tester makes the model generate a malicious payload and follows where it ends up: the operator's interface, logs, the admin panel, an email to a customer.
Login, roles and tenant isolation around the LLM
A chatbot usually has an ordinary backend: a conversations API, chat history, file uploads, an admin panel with prompt settings. The same risks apply as in any web application: access to other people's objects by ID, actions outside one's role, weak sessions. [7] For SaaS products with several customers on one platform the key question is whether one customer's data, chat history and knowledge base are isolated from another's. More on API testing in how website and API security is tested, and on multi-tenant products in penetration testing for a SaaS startup.
Cost abuse and denial of service
Every model request costs money. OWASP describes Unbounded Consumption (LLM06) as the risk of service outages, runaway costs and model cloning through mass queries, and notes that plain request rate limiting is no longer enough: you need token limits, hard spending caps and automatic circuit breakers for agents. [1] In the 2026 edition this entry moved up four places.
The tester checks per-account and per-session limits, input length, file size and tool-call loops. Load attacks on production are not run without separate consent: it is usually enough to show that a limit is missing and estimate the possible cost.
What an AI application pentest usually does not cover
- The provider's model (OpenAI, Anthropic, Google and so on) and its infrastructure: those can only be tested under the provider's own rules.
- Training your own model on poisoned data (LLM05) and the model supply chain (LLM04), unless you train or fine-tune the model yourself. That is a separate process audit.
- Answer quality, bias and factual errors (LLM07 Misinformation) as a separate track. A pentest shows cases where a wrong answer leads to a dangerous action, but it does not replace quality testing.
For a wider AI risk management frame you can use the NIST AI Risk Management Framework, and for a catalogue of attack techniques against AI systems, the MITRE ATLAS knowledge base. [5] [6] The OWASP 2026 edition maps its entries to both. [2]
What the client provides
- An architecture description: which model and provider, what is in the knowledge base, which tools the agent has and with what permissions.
- Two test accounts for every role and every test customer, to check whether one can reach the other's data.
- A test environment with a copy of the knowledge base without real personal data, or a set of test documents.
- The system prompt and tool descriptions if you choose the open format: the test goes deeper in the same time.
- A model spending cap for the test and a contact who can stop it.
- Written authorization with the scope and a list of third-party services the test must not touch.
The full list of steps is in how to prepare for a pentest.
What you get as a result
- A short summary for management: what was found, what it means for the business, what to fix first.
- Findings with the exact prompts or documents that reproduce each attack, evidence and a risk rating mapped to the OWASP entries.
- Recommendations for developers: which permissions to remove from the agent, where to add human approval, what to take out of the prompt, how to filter search by permissions.
- A list of what was tested and what stayed out of scope.
- A retest after the fixes if it is in the contract. Because model answers vary, attack scenarios are repeated several times.
The timeline depends on the number of roles, agent tools and data sources, so it is fixed before the start. If the scope is still unclear, start with a penetration test of the riskiest scenarios: access to other customers' data and agent actions involving money. What to do with the report afterwards is covered in what to do after a pentest.
Checklist before an AI application pentest
Download the preparation checklist (PDF)PDF · A4
FAQ
What is checked in an AI application penetration test?
Prompt injection, both direct and through documents, data leaks through RAG, agent tool permissions, system prompt leaks, handling of model output, login and tenant isolation, and spending limits. The basis is the OWASP Top 10 for LLM Applications 2026, and the API around the model is tested like any web application.
Can you fully protect against prompt injection?
No, and OWASP says plainly that no reliable way to prevent it exists today. So protection is built around the model: least-privilege tools, human approval for important actions, search filtered by user permissions and checks on output before it is used.
Is the OpenAI or other provider’s model itself tested?
No. The model and its infrastructure belong to the provider. The test covers your application: what the model can see, which tools it can call and what happens to its answers.
How is an AI application pentest different from a regular pentest?
On top of the usual API and role checks come attacks through text the model reads, checks of the knowledge base and of agent permissions. Model answers vary, so each scenario is repeated several times and the exact prompts are recorded.
When should an AI application be pentested?
Before launch to customers and after major changes: a new agent tool, a new data source in the knowledge base, a switch to another model. A new agent permission changes the risk more than a change in prompt wording.
References
- OWASP Top 10 for LLM Applications 2026 genai.owasp.org
- OWASP GenAI Security Project unveils the 2026 Top 10 for LLM Applications genai.owasp.org
- OWASP Top 10 for LLM Applications 2025 genai.owasp.org
- OWASP Top 10 for Agentic Applications (December 2025) genai.owasp.org
- MITRE ATLAS atlas.mitre.org
- NIST AI Risk Management Framework nist.gov
- OWASP Top 10 API Security Risks 2023 api-security.owasp.org
