LLM Penetration Testing and AI Security Audits: What They Cover and How to Choose a Provider

What is LLM penetration testing?

LLM penetration testing is the practice of attacking your own AI application the way an adversary would, to find the flaws that let someone manipulate it, extract data it should protect, or make it take actions it should refuse. It borrows the discipline of traditional penetration testing, scoped, authorized, adversarial, evidence-based, and applies it to the parts of a system that are unique to large language models: the prompt, the retrieved context, the tools the model can call, and the actions it can take on a user's behalf.

The distinction that matters most is what gets tested. A foundation model from a major provider has already been red-teamed extensively by its maker. The risk you own sits one layer up, in how your application wires that model to your data, your tools, and your users. That is where prompt injection turns into a real data breach and where excessive agency turns into an unauthorized action. Prompt Injection sits at the top of the OWASP Top 10 for LLM Applications 2025 as LLM01, precisely because the application layer is where untrusted input meets privileged capability.

So an LLM pentest is not a model evaluation and not a benchmark. It is a targeted attempt to break the specific system you are about to ship, or have already shipped, on your own data and your own permissions.

What does an LLM pentest actually cover?

A credible LLM pentest covers the application's full attack surface, not a single category of jailbreak. The OWASP Top 10 for LLM Applications is the most useful scope checklist, and a thorough engagement probes the risks that actually apply to your design.

Concretely, expect a tester to attempt the following:

• Prompt injection, direct and indirect. Direct injection is the user trying to override your instructions. Indirect injection hides instructions inside content the model later reads, a web page, a document in your knowledge base, an email, so the payload arrives through data, not the chat box. This is the hardest class to defend and the one a real tester spends the most time on. See prompt injection explained for how the two differ.
• Sensitive information disclosure (LLM02). Can the model be coaxed into revealing other users' data, proprietary documents, API keys, or fragments of its training and configuration? In RAG systems this includes cross-user leakage, where one person's query surfaces another team's confidential files. Our LLM data leakage prevention guide covers the control stack a tester checks for.
• System prompt leakage (LLM07). Getting the model to reveal its hidden instructions, which often expose business logic, guardrail wording, or secrets that were unwisely placed in the prompt.
• Excessive agency and tool abuse (LLM06). If the application can call tools or take actions, can an attacker steer it into calling the wrong one, with the wrong arguments, or in a way that escalates privilege? For agents and MCP-connected systems this is the highest-impact area; see securing AI agents.
• Improper output handling (LLM05). When model output flows into a browser, a shell, a database, or another system, the tester checks whether that output can carry an injection (cross-site scripting, SQL, command injection) downstream.
• Supply chain, data poisoning, and unbounded consumption where they apply, plus privacy and extraction attacks drawn from the NIST adversarial ML taxonomy, which catalogues evasion, poisoning, privacy, and prompt-injection attack classes.

The depth comes from the fact that the most damaging vulnerabilities are application-specific. They live in how your retrieval is filtered, which tools the model can reach, and what the model is permitted to do without a human in the loop. A scanner cannot know any of that.

What methodology does a credible AI security audit follow?

A credible audit follows a structured, evidence-based process rather than a single pass of automated probes. The shape mirrors the planning discipline in Microsoft's red teaming guidance: a defined before, during, and after, with findings recorded for reproducibility.

A sound engagement looks roughly like this:

1. Scoping and threat modeling. Before any attack, the tester maps your architecture: data sources, the system prompt, the tools and their permissions, who the users are, and what an attacker would actually want. This produces a threat model that decides where to spend the effort. An engagement that skips this step is testing in the dark.
2. Layered testing. The tester probes the base model behavior in your context, then the application through its real interface, then the application before and after your mitigations, so you learn which controls actually work. Microsoft's guidance is explicit that testing should happen at several layers and, where possible, against the production interface.
3. Open-ended then guided probing. Skilled testers start broad to discover unexpected behavior, then build a prioritized list of harms and probe each one deliberately, iterating as new weaknesses surface.
4. Mapping to standards. Every finding is tied to a recognized identifier, an OWASP LLM risk, a NIST attack class, or a MITRE ATLAS technique. ATLAS is an ATT&CK-style knowledge base of real-world adversary tactics against AI systems, and mapping to it makes findings comparable, trackable, and defensible rather than anecdotal.
5. Remediation-ready reporting. The deliverable is not a pass or fail score. It is each finding with a reproduction, an impact rating, the standard it maps to, and a concrete fix, plus a governance view consistent with the NIST AI Risk Management Framework Generative AI Profile, which frames AI risk as something you govern, map, measure, and manage over time.

A useful tell: the methodology should name what it does not cover. Honest testers will tell you that indirect prompt injection is mitigated, not solved, and that the real win is least-privilege design that limits the blast radius when a defense is bypassed.

How do you choose an AI security testing vendor?

Choose on demonstrated application-layer expertise and a methodology you can inspect, not on a logo or a turnaround promise. Because this market is young, the labels are unreliable; the questions below separate genuine testers from tools-as-a-service.

Ask a prospective provider:

• Can I see a sample report (redacted)? Real findings are specific to an application: "the support agent could be steered to call the refund tool with an arbitrary order ID." Generic findings ("the model may produce harmful content") signal a template, not an engagement.
• What is your threat model for a system like mine? A credible answer references your tools, your data flow, and your permissions, not a list of jailbreak strings.
• Which standards do you map findings to? Look for OWASP LLM Top 10, NIST, and MITRE ATLAS by name. Vague "industry best practices" is a flag.
• How much of the work is manual? Automation has a place for coverage and regression, but the flaws that matter need a human adversary. Ask for the ratio.
• What access do you need, and why? A serious tester will explain the trade-off between black-box and white-box access (see below), not just ask for an API key.
• What do you not test, and what stays unsolved after remediation? The honest answer to indirect injection is "contained, not eliminated." A vendor who claims to fully fix it does not understand the problem.

Builder authority is worth weighing too. Teams that ship and operate AI systems, rather than only attack them, tend to give remediation advice you can act on. For an example of how those defenses look in a real build, see how we built Onepilot, and for the proactive counterpart to a one-off audit, AI red-teaming for LLM applications.

How do you spot a fake assessment that is just automated fuzzing?

You spot it by what is missing: no threat model, no application-specific findings, and a report that could have been generated against any LLM. The buyer-beware reality is well documented; an industry guide states plainly that "most LLM security assessments on the market are automated prompt fuzzing with a cover report, not penetration testing".

Automated prompt fuzzing replays a library of known jailbreak strings against your endpoint and returns pass or fail. That has value as a regression tripwire, run it in CI to catch obvious backsliding, but it is not a penetration test, for three reasons:

• It does not understand your application. It tests the model in isolation, so it never reaches the tool calls, the retrieval filtering, or the agent actions where your real risk lives.
• It only knows yesterday's attacks. A library of published jailbreaks cannot find the novel, application-specific path an adversary would actually take through your system.
• It produces generic output. If the findings would read identically for a competitor's chatbot, no real testing happened.

The red flags, in short: a price and turnaround that imply no human time, a report with no reproduction steps, findings with no mapping to OWASP or MITRE ATLAS, and a claim that prompt injection has been fully solved. Treat any of these as a sign you bought a scan, not an assessment.

When should you run an LLM security review?

Run a security review before launch, and again after any meaningful change to the application. The attack surface of an LLM system is defined by its tools, prompts, data sources, and permissions, and every one of those moves as the product evolves, so a single pre-launch audit goes stale quickly.

The high-value moments to test:

• Before first production exposure, especially before the application can touch real user data or take real actions. Finding excessive agency after a tool has shipped is far more expensive than finding it in staging.
• After adding or changing tools, because a new tool expands what a successful injection can accomplish.
• After connecting a new data source to RAG, since untrusted documents are the primary vehicle for indirect injection.
• After widening permissions or removing a human-in-the-loop step, which directly enlarges the blast radius.
• On a recurring cadence for systems that handle sensitive data or take consequential actions, treating AI security as the continuous govern-map-measure-manage cycle the NIST framework describes, not a one-time gate.

The most useful framing: a point-in-time pentest tells you whether the system is safe today; the underlying discipline is keeping it safe as it changes. Pair the audit with an understanding of the attacks it tests against so your team can reason about new risks between engagements rather than waiting for the next report.

FAQ

What is the difference between LLM penetration testing and a model evaluation?

A model evaluation measures a model's capabilities and safety behavior in isolation, often with benchmarks: does it refuse harmful requests, how accurate is it, how often does it hallucinate. LLM penetration testing targets your application, the system that wraps the model with your prompt, your data, your tools, and your permissions, and tries to break it the way an attacker would. Evaluation answers "is this model good enough"; a pentest answers "can someone abuse the way we deployed it."

What should an LLM pentest report include?

Each finding should include a clear description, reproduction steps so your engineers can confirm it, an impact and severity rating, a mapping to a recognized standard (OWASP LLM Top 10, NIST, or MITRE ATLAS), and a concrete remediation. A good report also states the scope and threat model it worked from, lists what was not tested, and is honest that some risks (notably indirect prompt injection) are contained by design rather than eliminated. A bare pass or fail score is not a report.

Is automated prompt fuzzing a real penetration test?

No. Automated prompt fuzzing replays a library of known jailbreak strings and returns pass or fail. It is useful as a regression tripwire in CI, but it tests the model in isolation, cannot reach the application-specific paths (tools, retrieval, agent actions) where real risk lives, and produces generic findings. An industry guide documents that most assessments sold as LLM security testing are exactly this: automated fuzzing with a cover report. Treat it as one input, not the assessment.

How long does an LLM security audit take?

It depends on scope: a single chatbot with no tools is far quicker than an autonomous agent with several integrations and a large RAG corpus. The honest answer is that duration scales with the size of the threat model, the number of tools and data sources, and the access provided. Be skeptical of any engagement priced and scheduled to imply no human time, because that usually means automation alone, which is not a penetration test.

What access does a tester need to assess my LLM application?

It ranges from black-box (the tester uses the application as any user would) to white-box (the tester also sees the system prompt, tool definitions, retrieval configuration, and permission model). Black-box mirrors a real external attacker; white-box finds more in less time because the tester is not guessing at the architecture. A serious provider will explain the trade-off and recommend a level for your risk, rather than simply asking for the keys.

When in the build should I commission an AI security audit?

Before first production exposure, and again after any change to tools, prompts, data sources, or permissions, since each of those reshapes the attack surface. For systems that handle sensitive data or take consequential actions, treat testing as a recurring cadence, not a one-time gate, consistent with the continuous risk-management cycle the NIST AI RMF describes. Finding a flaw in staging is always cheaper than finding it after it has touched real users.

---

ELM Labs is an applied AI lab that builds and security-tests LLM and agent applications end to end, mapping every finding to OWASP, NIST, and MITRE ATLAS.