AI Red Teaming Methodology
Release: Version 1.1
Document
| Field | Description |
|---|---|
| Name | AI Red Teaming Methodology |
| Creators | Hacken OU |
| Subject | AI Security; Generative AI; Large Language Models; Small Language Models; Reinforcement Learning; Chatbots; Agentic AI; Cybersecurity Auditing |
| Description | A practical methodology for planning, executing, and reporting authorized security assessments of AI systems, including language models, generative AI applications, chatbots, Model Context Protocol (MCP) clients and servers, AI agents, and RAG systems. It covers application, model, data, identity, tool, infrastructure, and human-oversight boundaries. |
| Author | Stephen Ajayi | Leading Offensive Security Engineer at Hacken |
| Last Updated | October 30, 2026 |
| Rights | Hacken OU |
| Copying Notice | Copying or reproducing this document without explicit reference to Hacken OU is forbidden. |
Introduction
AI systems, particularly Large Language Models (LLMs), multi-agent systems, and generative AI platforms, introduce novel attack surfaces and unique security challenges.
Traditional application, cloud, identity, and infrastructure testing remains necessary, but must be complemented by AI-specific testing of probabilistic behavior, retrieved content, model and data supply chains, memory, tool use, and human oversight.
New threat categories such as prompt injection, excessive agent autonomy, vector database exploitation, and integration abuse have emerged as core risks in the AI ecosystem.
This methodology provides a structured framework for assessing the security of AI systems through adversarial simulation and ethical red teaming.
It is designed to:
- Uncover hidden weaknesses across AI pipelines and integrations.
- Produce evidence that supports risk-management and compliance activities under applicable frameworks and law. An AI red-team assessment is not a legal opinion, conformity assessment, or certification.
- Strengthen organizational AI risk posture through continuous testing and structured remediation.
All auditing activities are conducted within a strictly controlled, ethical environment, adhering to the Rules of Engagement (RoE) defined for each assessment.
Each test is traceable, reproducible, and aligned with Hacken’s internal standards of professional integrity.
Executive Summary
The AI Security Auditing Methodology guides Hacken’s security engineers, auditors, and stakeholders through a rigorous AI Red Teaming process tailored specifically for AI systems, agents, and LLM integrations.
This framework emphasizes:
- Ethical and controlled simulations of real-world threats.
- Evaluation of AI system security, availability, and integrity under adversarial conditions.
- Comprehensive vulnerability discovery and impact analysis across the AI lifecycle — including training, inference, integration, and deployment.
- Actionable reporting with clear, prioritized remediation guidance.
The outcome of each audit is a detailed, evidence-based understanding of system resilience, allowing stakeholders to make informed security, governance, and compliance decisions.
Scope
This methodology applies to the following AI system types and components:
- Large Language Models (LLMs) (e.g., GPT, Claude, Gemini, LLaMA, Mistral)
- Small Language Models (SLMs) and lightweight on-device inference systems
- Retrieval-Augmented Generation (RAG) systems and vector database integrations
- Autonomous and multi-agent frameworks (e.g., AutoGPT, Agentic AI architectures)
- Generative AI applications (text, image, audio, code generation)
- Chatbots, copilots, and conversational AI assistants
- Reinforcement learning agents and tool-using AI systems
These systems, by design, introduce data-driven, probabilistic, and context-dependent behavior, requiring advanced assessment methodologies beyond traditional application security testing.
Assessment Models and Boundaries
The engagement must identify the assessment model before testing begins:
- Black box — testing through exposed user or API interfaces with no internal design access.
- Gray box — testing with architecture, test identities, logs, selected configuration, and limited source access.
- White box — testing with source code, prompts, policies, model and tool configuration, data-flow details, and observability.
The scope must distinguish the model, AI application, retrieval and memory layer, agent or orchestrator, tools, identity and authorization services, third-party providers, and hosting infrastructure. A control observed in one layer must not be assumed to protect another.
Production testing, testing against real users, high-cost resource exhaustion, model extraction, destructive tool use, social engineering, physical testing, and testing of third-party services require explicit written authorization. The Rules of Engagement must define rate and cost limits, test identities, permitted data, prohibited content, evidence handling, emergency contacts, stop conditions, and cleanup requirements.
Alignment with International Standards
This methodology uses the following sources as test-planning and reporting references. Mapping a finding to a framework does not, by itself, establish legal compliance or certification.
| Framework / Standard | Purpose |
|---|---|
| NIST AI RMF 1.0 and NIST AI 600-1 | Voluntary risk-management framework and Generative AI Profile organized around Govern, Map, Measure, and Manage. AI RMF 1.0 is under revision, so engagements record the version used. |
| NIST SP 800-218A | Secure software-development practices for generative AI and dual-use foundation models, used with the NIST SSDF. |
| OWASP GenAI LLM Top 10 (2026) | Current awareness list for LLM application risks, including prompt injection, sensitive information disclosure, excessive agency, supply chain, poisoning, unbounded consumption, misinformation, hidden context exposure, vector/embedding weaknesses, and improper output handling. |
| OWASP Top 10 for Agentic Applications (2026) | Companion reference for systems in which models act through tools, memory, identities, and multi-step workflows. |
| MITRE ATLAS | Living knowledge base of adversary tactics and techniques involving AI-enabled systems and AI capabilities. |
| ISO/IEC 27001:2022 and ISO/IEC 42001:2023 | Information-security and AI-management-system standards. They support governance and control mapping; this methodology does not perform certification. |
| EU AI Act | Applicable legal obligations are identified with counsel based on the system, actor role, geography, and effective date. The methodology must not represent a technical test as proof of legal compliance. |
The engagement report identifies the exact editions used and separates technical findings from governance observations and legal questions.
Glossary
| Term | Definition |
|---|---|
| AI Auditor | A security professional responsible for evaluating the security posture of AI systems through testing, simulation, and risk analysis. |
| Control Team | The group within Hacken or the client organization responsible for overseeing the AI audit’s scope, logistics, and compliance adherence. |
| Blue Team | The organization’s defensive team tasked with detecting, responding to, and mitigating incidents during or after the audit. |
| Rules of Engagement (RoE) | A formal document outlining the permissible activities, boundaries, and limitations during an AI security audit. |
| Prompt Injection | A technique where an attacker crafts malicious input to manipulate or subvert the intended behavior of an AI model. |
| Model Inversion | A method used by adversaries to extract or infer sensitive information from a trained AI model. |
| Data Poisoning | The process of injecting malicious or manipulated data into a training set to bias or compromise model behavior. |
Importance of AI Security Auditing
As AI systems play an increasingly critical role across industries — from healthcare and finance to autonomous systems and customer service — ensuring their security has become a top priority. Malicious actors can exploit vulnerabilities in AI models, leading to data breaches, biased outcomes, or even system failures.
Proactive and thorough AI security audits help organizations:
- Identify AI-Specific Vulnerabilities — Detect weaknesses unique to AI, such as adversarial attacks, model poisoning, data leakage, insecure agents, and prompt injection.
- Ensure Regulatory Compliance — Align with evolving standards (e.g., EU AI Act, NIST AI RMF, ISO/IEC 27001/42001) to avoid legal and financial penalties.
- Safeguard Sensitive Data — Protect the integrity and confidentiality of data processed by AI models (training, fine-tuning, inference, telemetry).
- Build Trust in AI Applications — Demonstrate robust security practices to users, partners, and regulators, increasing confidence and adoption.
By auditing AI systems proactively, organizations not only prevent costly incidents but also foster responsible and ethical AI deployment.
AI Security Auditing Process
The AI Security Audit operations are organized into three phases:
- Planning & Pre-Engagement
- Execution of Security Audit
- Post-Engagement Reporting & Follow-up
Each phase produces concrete artifacts to ensure traceability, repeatability, and measurable outcomes.
1) Planning & Pre-Engagement
This phase establishes the audit foundation by defining scope, success criteria, and constraints, and by preparing environments, data, and access.
Key activities
-
System Identification
Document AI system architecture and components: model and prompt versions, datasets, data pipelines, inference services, RAG/vector databases, memory, agent frameworks and tools, MCP clients and servers, APIs, identities, authorization decisions, trust boundaries, and internal/external data flows. -
Risk Assessment
Define protected assets and unacceptable outcomes, then evaluate potential business and technical impacts across confidentiality, integrity, availability, privacy, safety, fairness, financial cost, compliance exposure, and model or intellectual-property theft. Prioritize scenarios by evidence-based likelihood and impact. -
Audit Planning
Define objectives, in-scope assets, out-of-scope boundaries, assessment model, attack classes, test depth, environments, test identities and tenant roles, evaluation datasets, success and failure thresholds, rate and cost budgets, timelines, and tooling. -
Legal & Compliance Review
Validate Rules of Engagement (RoE), privacy requirements, data handling constraints, and alignment with applicable frameworks (e.g., EU AI Act risk class, NIST AI RMF functions, ISO/IEC controls).
Inputs required from client
- High-level/low-level architecture, data flow diagrams, model cards, model release notes.
- Access to staging environments, test accounts, seeded vector DBs, and synthetic/test datasets.
- List of third-party providers (LLM APIs, embeddings, observability, storage).
- Security/governance policies relevant to AI (prompt handling, PII, retention, telemetry).
- System/developer prompt inventory; model, adapter, embedding, and guardrail versions; sampling configuration; and feature flags.
- Agent and MCP inventories, tool schemas, tool permissions, delegated identities, approval rules, memory boundaries, and egress restrictions.
- Expected security invariants and abuse cases, including which actions always require deterministic authorization or human approval.
- Logging and trace access sufficient to attribute model outputs, retrieved content, tool calls, authorization decisions, and side effects without collecting prohibited data.
Deliverables
- Engagement Brief & RoE (scope, constraints, contacts, comms plan)
- System and Trust-Boundary Model (components, identities, data flows, tools, third parties)
- Audit Plan and Test Matrix (methods, tools, test corpus, repetitions, timelines, success criteria)
- Threat Hypotheses (initial scenarios and attack paths tailored to the system)
- Evidence and Data-Handling Plan (classification, minimization, retention, deletion, and disclosure rules)
2) Execution of Security Audit
Hands-on assessments validate assumptions, uncover vulnerabilities, and measure resilience under realistic adversarial conditions.
Activities
-
Reconnaissance
Enumerate inputs and outputs, prompts, hidden context, tools, API surfaces, authentication and authorization flows, model and guardrail versions, agent permissions, retrieval sources, memory, data lineage, observability, and side-effect paths. Record a baseline configuration so results can be reproduced. -
Vulnerability Assessment
Test for AI-specific and traditional weaknesses, including:- Direct, indirect, cross-modal, and stored prompt injection; instruction and trust-boundary confusion
- Sensitive information disclosure, hidden-context exposure, cross-user or cross-tenant leakage, and authorized model-extraction testing
- RAG and routing failures, retrieval poisoning, vector/embedding weaknesses, provenance failures, and access-control bypass
- Excessive agency, tool over-privilege, identity and delegation abuse, missing human approval, unsafe tool execution, and insecure generated code
- MCP client/server authorization, token audience and scope, consent, server identity, tool metadata, resource access, and confused-deputy behavior
- Memory and context poisoning, insecure inter-agent communication, cascading failures, and recovery or kill-switch failure
- Improper output handling where model output reaches interpreters, browsers, databases, APIs, files, or privileged workflows
- Supply-chain risk across models, adapters, datasets, embeddings, packages, plugins, tools, and deployment infrastructure
- Unbounded consumption, denial of service, recursive execution, rate-limit bypass, and economic abuse
- Misinformation, policy evasion, unsafe content, bias, or reliability failures only where they violate an agreed requirement or risk threshold
- Traditional application, API, cloud, identity, and infrastructure weaknesses that make an AI-specific attack consequential
-
Exploitation (Controlled)
Safely reproduce impactful chains to validate exploitability, quantify risk, and collect evidence. Prefer inert tools, synthetic data, canary secrets, test tenants, transaction rollback, and dry-run modes. Do not infer a downstream compromise from model text alone; verify the actual authorization decision or side effect within the RoE. -
Repeatability and Control Comparison Because model behavior is probabilistic, record model and prompt versions, parameters, tools, retrieved content, run count, success rate, and control configuration. Repeat relevant cases and compare the attack against a benign baseline and negative controls. A single surprising answer is not automatically a reproducible security finding.
-
Impact Analysis
Map findings to demonstrated business impact: data exfiltration, cross-tenant access, unauthorized tool actions, integrity loss, financial cost, reputational harm, compliance exposure, safety risks, or operational disruption. Separate observed impact from plausible escalation and untested assumptions.
Telemetry & evidence
- Prompt/response transcripts, sanitized logs, trace IDs, vector queries, retrieved chunks, tool-call arguments/results, authorization decisions, and verified side effects.
- Configuration snapshots (redacted) of guardrails, gateways, filters, and policies.
- Proof-of-concept artifacts demonstrating exploit conditions and boundaries.
- Model/prompt/tool versions, sampling parameters, timestamps, test identity/tenant, run counts, success rates, and negative-control outcomes.
Evidence collection must minimize personal and customer data, preserve chain of custody, and follow the engagement retention schedule. The assessment must not require or claim access to private chain-of-thought. Findings should rely on observable inputs, outputs, traces, configuration, authorization decisions, and side effects.
Deliverables
- Daily/Interim Updates (running findings, evidence snapshots)
- Exploit Narratives (what an attacker can achieve and how)
- Impact Matrix (affected assets, users, and controls)
- Evaluation Results (test corpus, repetitions, success rates, baselines, and residual uncertainty)
3) Post-Engagement Reporting & Follow-up
This phase consolidates evidence, delivers a clear remediation path, and verifies improvements.
Activities
-
Reporting
Produce comprehensive documentation covering:- Finding titles, descriptions, affected components
- Evidence & reproduction steps (sanitized where required)
- Preconditions, tested identities, model/configuration versions, repeatability, and negative controls
- Demonstrated impact, plausible impact, residual uncertainty, and severity rationale
- Finding type: security vulnerability, privacy issue, safety/abuse-control gap, governance/compliance observation, or quality/reliability observation
- Applicable CWE, OWASP GenAI/Agentic, MITRE ATLAS, NIST AI RMF, and ISO mappings, including framework edition
-
Recommendations
Prioritized, actionable guidance: deterministic authorization, tool/agent permission minimization, sandboxing, RAG hardening, provenance, prompt/guardrail changes, output validation, monitoring, incident response, and SDLC integrations. Prompt instructions and model self-policing must not be presented as substitutes for access control. -
Debriefing
Present results to technical and executive stakeholders; align on remediation owners, timelines, and verification plan. -
Follow-Up
Support remediation validation and optional Verification/Patch Audit to confirm issues are resolved and controls are effective.
Deliverables
- Final Report (executive summary + technical appendix)
- Remediation Plan & Tracker (priorities, owners, SLAs)
- Verification Report (post-fix validation, residual risk)
AI Security Tools and Techniques
The following are examples of tools that may support an AI security assessment. They are not mandatory, exhaustive, or a substitute for system-specific threat modeling and manual validation.
Before use, the auditor must verify the tool's current maintenance status, license, model/provider compatibility, data handling, telemetry, dependency risk, and suitability for the engagement. Client data, prompts, model outputs, credentials, and evidence must not be sent to an external service unless the RoE and data-processing terms explicitly allow it. Automated scanner output is a candidate result until a human validates reachability, impact, and reproducibility.
Vulnerability Scanners and Red Teaming Frameworks
| Tool | Description |
|---|---|
| Giskard | A Python-based security and QA library for detecting performance, bias, and security flaws in AI systems. It automatically identifies misbehavior in both models and pipelines. |
| PyRIT | Microsoft framework for orchestrating repeatable adversarial tests and recording outcomes against generative AI systems. |
| LLMFuzzer | A fuzzing framework built specifically for LLM APIs, designed to stress-test integration points and detect vulnerabilities through randomized prompt fuzzing. |
| Vigil | A security evaluation library for AI systems that analyzes prompt–response pairs to detect injection attempts, jailbreaks, or unsafe outputs. |
| Adversarial Robustness Toolbox (ART) | Maintained by the Linux AI & Data Foundation, ART evaluates and strengthens models against evasion, poisoning, inference, and extraction attacks. |
| AgentDojo | A testing framework for agentic AI systems that execute tools over untrusted data. It enables simulation of complex agent behaviors, adaptive attacks, and defenses. |
| Agent security benchmarks | Scenario-specific benchmarks can help compare attacks and defenses, but benchmark results must be tied to the assessed architecture and must not be reported as production exploitability without validation. |
| Garak | Developed by NVIDIA, Garak scans LLMs for vulnerabilities such as prompt injection, data leakage, hallucinations, and jailbreaks. It functions like a traditional vulnerability scanner but is tailored for LLMs and AI agents. |
| Promptmap | An open-source framework that automates prompt injection attacks on GPT-style applications. It supports multiple model architectures and is used to uncover weaknesses in system and developer prompts. |
Attack Vectors and Scenarios
AI security audits use a risk-driven selection of digital, human, and, where explicitly authorized, physical attack surfaces. Not every category applies to every engagement. Modern AI ecosystems introduce multi-layered, dynamic, and interconnected components — including agents, tool execution environments, and Model Context Protocol (MCP) integrations — that require targeted testing approaches.
Digital Attack Vectors
| Category | Description |
|---|---|
| Prompt Injection and Jailbreaks | Manipulation of model instructions, system prompts, or embedded directives to override safety mechanisms, execute hidden instructions, or exfiltrate confidential data. |
| Model Inversion | Extraction of sensitive or proprietary information from trained model parameters, gradients, or generated outputs. |
| Data Poisoning | Insertion of malicious or manipulated data during training or fine-tuning to bias, corrupt, or subvert model behavior. |
| Unauthorized API and MCP Access | Exploitation of weak authentication, token audience/scope validation, consent, or object-level authorization in APIs and MCP clients/servers to access tools, prompts, resources, or downstream systems. |
| Adversarial Inputs | Crafting of malicious data designed to confuse, crash, or bypass AI model logic, resulting in denial of service or output manipulation. |
| RAG Exploitation | Attacks on Retrieval-Augmented Generation (RAG) systems through poisoning, injection, or compromise of vector databases and external knowledge stores. |
| Tool Abuse (Agent Exploitation) | Coercing AI agents to perform unintended or malicious actions — such as file modification, system command execution, or sensitive data retrieval — by abusing agent tool-use APIs or weak validation layers. |
| Agent-to-Agent Manipulation | Cross-agent interference in multi-agent systems, where one compromised agent influences others through shared memory, vector stores, or message-passing protocols. |
| Untrusted Content via MCP | Using malicious tool descriptions, resource content, prompts, or tool results delivered through MCP to influence a model, confuse trust boundaries, or trigger unauthorized downstream actions. |
| Hidden Context Exposure | Extraction of system/developer instructions, secrets, private tool results, retrieved data, or cross-user context that was not intended for the caller. Private chain-of-thought is not required as assessment evidence. |
| Supply Chain Compromise | Tampering with third-party models, datasets, embeddings, or open-source agent frameworks (e.g., LangChain, AutoGPT, CrewAI) to introduce backdoors or unsafe dependencies. |
| Session and State Hijacking | Intercepting or manipulating conversational state, authorization state, client state, reconnect/resumption data, or stored memory to impersonate users or persist malicious context. |
Agent-Specific Testing Considerations
Agentic systems, where AI models autonomously use tools, APIs, or other agents, require dedicated testing methodologies.
The following areas must be examined during Agent Security Assessments:
- Tool Invocation Validation – Ensure the agent cannot invoke arbitrary commands or system tools without user authorization or contextual verification.
- Command Injection and Escalation – Attempt to coerce the agent into executing privileged or harmful commands (e.g., file modification, API key exposure).
- MCP Trust and Authorization – Verify server identity, transport security, OAuth discovery, token issuer/audience/scope validation, consent, object-level authorization, tool/resource isolation, and confused-deputy resistance.
- Delegation Safety – Test multi-agent frameworks for misconfigured delegation (agents granting tools or permissions to others without restriction).
- Memory and Vector Store Hardening – Validate encryption, retention, and sanitization of stored embeddings and agent memory.
- Recursive Execution Limits – Ensure agent plans, tool loops, retries, delegated tasks, and resource consumption are bounded to prevent runaway operations and infinite self-calls.
- Human Oversight and Kill Switches – Confirm agents include deterministic interruption and override mechanisms during unsafe tool execution.
Human Attack Vectors
Human-targeted testing is performed only when explicitly included in the RoE and approved by the relevant organizational stakeholders.
| Category | Description |
|---|---|
| Social Engineering via AI Interfaces | Manipulating human operators, developers, or users through AI-generated content or malicious prompt injection delivered in chat or support interfaces. |
| Phishing through Conversational AI | Using generative models to mimic trusted personnel, brands, or systems to extract credentials, tokens, or sensitive data. |
| Insider Manipulation | Abuse of developer access, prompt logs, monitoring dashboards, or telemetry systems to extract model internals or sensitive client data. |
| Prompt-Based Psychological Manipulation | Leveraging context-aware conversational systems to persuade or coerce human users into bypassing safety protocols. |
| Human-in-the-Loop (HITL) Abuse | Exploiting weak review or reinforcement learning feedback loops (RLHF/RLAIF) to manipulate model reinforcement patterns and outputs. |
Physical Attack Vectors
Physical and hardware testing is out of scope by default and requires separate written authorization, safety planning, and asset-owner approval.
| Category | Description |
|---|---|
| Infrastructure Access | Gaining unauthorized access to servers, model weights, or vector databases hosting AI applications or MCP endpoints. |
| Device Compromise | Physical manipulation or tampering with edge devices, local inference hardware (GPUs, TPUs), or AI-enabled IoT components. |
| Side-Channel Attacks | Exploiting electromagnetic, power, timing, or cache-based signals to infer model parameters or input data. |
| Hardware Supply Chain Tampering | Compromising embedded AI accelerators, firmware, or chipsets used for model inference. |
| Environmental Interference | Disrupting sensors or edge agents that feed multimodal inputs (audio, visual, sensor data) to induce erroneous AI behavior. |
Integration Risk Hotspots
When testing complex AI ecosystems, auditors must pay particular attention to integration and orchestration boundaries, including:
- Model Context Protocol (MCP) – Verify server identity, protected-resource metadata, authorization-server discovery, token issuer/audience/scope validation, transport protection, user consent, and authorization for each tool/resource. Test malicious tool metadata and untrusted resource/tool output as indirect-injection sources.
- Agent Toolchains – Validate that only pre-approved tools and APIs can be executed; inspect sandboxing and scope isolation.
- Vector Databases – Test for poisoning, embedding manipulation, or malicious content retrieval.
- API Gateways and Plugins – Ensure strict input validation, authentication, and content filtering between AI applications and third-party services.
- Cross-Model Messaging – Assess risk of data leakage or trust violations in environments where multiple models (text, vision, code) share communication channels.
Each audit engagement should classify attack scenarios by impact domain (data, system, user, compliance) and vector type to guide prioritization and defense planning.
Issue Severity and Risk Definition
Each candidate is first classified as a security vulnerability, privacy issue, safety or abuse-control gap, governance/compliance observation, or quality/reliability observation. The categories can overlap, but they must not be treated as interchangeable.
Security severity is based on demonstrated impact, attacker prerequisites, affected identities and tenants, reproducibility, scale, detectability, reversibility, and the authority available to the model or agent. CVSS v4.0 may be used when the finding is a software vulnerability and the metric assumptions fit. Safety, fairness, misinformation, quality, and compliance observations require an explicitly defined contextual risk matrix rather than a synthetic CVSS score.
Framework mappings do not determine severity. A jailbreak or policy-violating response is reportable as a security vulnerability only when it crosses an agreed security boundary or enables a concrete adverse outcome. The report records the tested configuration and distinguishes a reproducible exploit from a probabilistic observation.
Severity Levels
| Severity | Description |
|---|---|
| Critical | Reproducible compromise with severe and broad impact, such as unauthenticated cross-tenant data exfiltration, privileged arbitrary action, or destructive system compromise, with realistic prerequisites. Requires urgent containment and remediation. |
| High | Reproducible, significant impact to protected data, privileged actions, safety, or availability, but with narrower scope or meaningful prerequisites. Should be addressed promptly. |
| Medium | Material impact with additional prerequisites, limited scope, lower reliability, or effective compensating controls. Remediate within the agreed risk-treatment timeframe. |
| Low | Limited direct impact or defense-in-depth weakness with a credible security consequence. Track and remediate according to risk appetite. |
| Informational / Observation | No demonstrated security-boundary violation. Records governance, quality, hardening, or unverified risk hypotheses without inflating vulnerability counts. |
Issue Lifecycle States
Each finding progresses through a structured lifecycle to promote accountability and transparent remediation tracking.
| Status | Definition |
|---|---|
| Candidate | A behavior or weakness has been observed and awaits independent validation of prerequisites, impact, and reproducibility. |
| Validated | Evidence supports the finding and its stated boundaries. It is ready for formal reporting. |
| Reported | The validated issue has been formally reported and remains unresolved. |
| Remediation Claimed | The owner states that a change addresses the issue, but verification is not complete. |
| Resolved | The remediation was independently verified against the original exploit and relevant bypass or regression cases. |
| Mitigated | Compensating controls reduce likelihood or impact, but the original condition or residual risk remains. |
| Risk Accepted | The authorized risk owner accepts the documented residual risk and review date. |
| Rejected / Not Reproducible | Evidence does not support the candidate under the recorded configuration; the rejection rationale and retest conditions are retained. |
Findings and Documentation
All identified vulnerabilities and weaknesses discovered during the AI Security Audit will be meticulously documented to ensure clarity, reproducibility, and accountability.
Each issue entry must include technical detail sufficient for engineering, compliance, and management audiences.
Finding Structure
| Field | Description |
|---|---|
| Issue Title & Description | A clear, concise summary of the vulnerability, including where and how it was discovered. |
| Finding Type | Security vulnerability, privacy issue, safety/abuse-control gap, governance/compliance observation, or quality/reliability observation. |
| Affected Configuration | Application, model, prompt, guardrail, retrieval source, tool, policy, and environment versions required to reproduce the result. |
| Preconditions & Threat Actor | Required access, identity, tenant, user interaction, knowledge, and external conditions. |
| Severity & Rationale | Risk rating with explicit impact, likelihood, reproducibility, scale, reversibility, and compensating-control assumptions. CVSS v4.0 is included only where appropriate. |
| Evidence / Proof of Concept | Sanitized prompts, outputs, logs, traces, retrieved content, tool calls, authorization decisions, and verified side effects demonstrating the supported impact. |
| Reproducibility | Run count, success rate, baseline, negative controls, variability, and known failure conditions. |
| Impact Analysis | Observed impact separated from plausible escalation, business consequence, and untested assumptions. |
| Recommendations | Specific, actionable remediation steps to eliminate or mitigate the vulnerability. Should include best practices, configuration examples, or defensive tooling guidance. |
| Verification Plan | Original exploit, bypass cases, regressions, telemetry checks, and evidence required to mark the issue resolved. |
| Classification & References | Applicable CWE, CAPEC, OWASP GenAI/Agentic, MITRE ATLAS, NIST, ISO/IEC, legal, or internal-policy references, including edition. Mappings are omitted when they do not fit. |
Each issue is logged and tracked through the Issue Lifecycle described in this methodology. Candidate observations are not counted as validated findings until the evidence supports the stated security boundary and impact.
Limitations
While this methodology provides a comprehensive framework for AI system auditing, it does not guarantee the identification of all potential vulnerabilities or attack scenarios.
Limitations include:
- Scope Limitations – Only the systems, environments, and components explicitly in-scope during the engagement are tested.
- Time Constraints – AI models and integrations evolve rapidly; point-in-time audits may not reflect future system states.
- Probabilistic Behavior – Results can vary across runs, model updates, sampling parameters, context, and provider-side controls. The report records repetitions and uncertainty but cannot prove the absence of a behavior.
- Third-Party Drift – Hosted models, moderation systems, tools, and retrieval sources may change without a corresponding client deployment. Results apply to the recorded versions and timestamps.
- Technology Maturity – Many AI security tools and frameworks are still in early development phases and may not detect novel or emerging threats.
- Emerging Threats – AI is a fast-moving domain, and new exploit methods may arise after the audit’s conclusion.
- Compliance Boundary – Technical testing can provide evidence for governance and legal review, but does not constitute legal advice, regulatory conformity assessment, or ISO certification.
- Safety Boundary – Testing potentially harmful capabilities, content, humans, physical devices, or third-party systems is limited to the explicit RoE and applicable law.
⚠️ Continuous monitoring, periodic reassessment, and adoption of multi-layered AI security controls are essential to maintain a resilient security posture.
Appendix
A. List of Tools for AI Security Auditing
Below is a non-exhaustive list of candidate tools supporting AI auditing, red teaming, monitoring, and governance workflows. Inclusion is not an endorsement. Revalidate maintenance, licensing, data egress, telemetry, and fitness for purpose before every engagement; use only tools permitted by the RoE.
1. Reconnaissance and Information Gathering
| Tool | Purpose |
|---|---|
| GPTFuzz | Performs fuzz testing on LLMs to uncover unexpected behaviors and instability. |
| Rebuff | Automates prompt injection and jailbreak testing through multi-layered analysis. |
| Microsoft PyRIT | A red-teaming and fuzzing toolkit for LLM-based applications and integrations. |
| SpiderFoot | Open-source OSINT automation for discovering external data and intelligence about AI systems. |
| TheHarvester | Gathers emails, domains, and subdomain data for reconnaissance related to AI system operators and infrastructure. |
2. Vulnerability Scanning and Exploitation
| Tool | Purpose |
|---|---|
| Giskard | Detects robustness, bias, and security flaws in AI systems. |
| TruLens | Evaluates LLM outputs, tracing performance, bias, and behavioral deviations. |
| Nmap | Identifies exposed network surfaces and connected infrastructure vulnerabilities. |
| Burp Suite | Assesses security of APIs interfacing with AI systems, especially model endpoints. |
3. Input Validation and Sanitization
| Tool | Purpose |
|---|---|
| Presidio | Microsoft’s open-source framework for detecting, anonymizing, and redacting PII in input data. |
Text normalization or keyword filtering must not be treated as a complete defense against prompt injection. Security controls should be enforced at authorization, isolation, output-validation, and tool-execution boundaries.
4. Output Filtering and Monitoring
| Tool | Purpose |
|---|---|
| LangSmith | Provides observability and traceability for LLM-driven applications. |
| Helicone | Enables real-time logging, auditing, and monitoring of LLM usage. |
| Weights & Biases | Tracks experiments, logs model behavior, and supports continuous auditing of ML pipelines. |
5. Security Control and Governance Frameworks
| Tool | Purpose |
|---|---|
| LlamaGuard | Supports input/output safety classification. Its decisions are probabilistic and must not replace deterministic authorization. |
| Guardrails AI | Validates structured model outputs and enforces schema compliance during generation. |
| NeMo Guardrails | Framework for conversational controls and programmable rails; effectiveness must be validated against the assessed use case. |
6. Social Engineering and Adversarial Simulation
| Tool | Purpose |
|---|---|
| Social Engineer Toolkit (SET) | Automates phishing and social engineering simulations targeting human–AI interaction workflows. |
| GoPhish | Executes controlled phishing campaigns to test awareness and training effectiveness among AI system operators and developers. |
B. AI Threat Modeling Framework
The AI Threat Modeling Process for generative and agentic AI systems draws from industry standards including MITRE ATLAS, NIST AI RMF, and OWASP LLM Top 10.
1. Reconnaissance
- Identify all system components, data flows, and dependencies.
- Document APIs, plugins, external data sources, human interfaces, and operational environments.
- Map data provenance, model lineage, and hosting infrastructure.
- Identify identities, tenants, delegated credentials, tool authority, approval gates, trust boundaries, and third-party control planes.
2. Attack Surface Enumeration
- Analyze both external and internal interfaces exposed to users, developers, and third-party systems.
- Identify adversarial input vectors such as prompt injection, model inversion, and vector store manipulation.
- Classify entry points based on accessibility and privilege level.
3. Threat Scenario Development
- Develop threat models reflecting attacker motivations, capabilities, and objectives.
- Simulate scenarios such as:
- Data leakage and exfiltration
- Model extraction and inversion
- Prompt manipulation and jailbreaks
- Supply chain compromise (datasets, models, or APIs)
- Vector DB poisoning and RAG manipulation
- Indirect prompt injection through retrieved content, files, websites, messages, tool metadata, or tool results
- Unauthorized or incorrectly approved agent actions
- Cross-user, cross-tenant, memory, and hidden-context exposure
- Resource exhaustion, recursive execution, and economic abuse
4. Risk Analysis
- Estimate likelihood and impact for each identified threat.
- Evaluate business, regulatory, financial, and operational consequences.
- Prioritize based on combined risk scores and risk appetite thresholds.
- Record prerequisites, reproducibility, uncertainty, affected identities/tenants, reversibility, and existing compensating controls.
- Separate demonstrated impact from potential escalation and compliance questions.
5. Security Control Mapping
- Identify existing security controls and gaps in AI governance.
- Map findings and recommendations to:
- NIST AI RMF (Functions: Govern, Map, Measure, Manage)
- OWASP GenAI LLM Top 10 (2026) and, for acting systems, OWASP Agentic Top 10 (2026)
- MITRE ATLAS tactics and techniques where applicable
- ISO/IEC 27001 and ISO/IEC 42001 controls
6. Validation and Mitigation Testing
- Conduct red team simulations and adversarial stress tests against defined threat models.
- Validate detection, response, and mitigation effectiveness.
- Perform regression testing after fixes using the original exploit, bypass cases, benign controls, and telemetry checks.
- Record the exact model/application configuration, run count, success rate, and residual risk.
C. Primary References
- NIST AI Risk Management Framework
- NIST AI 600-1 — Generative Artificial Intelligence Profile
- NIST SP 800-218A — Secure Software Development Practices for Generative AI
- OWASP GenAI LLM Top 10 (2026)
- OWASP Top 10 for Agentic Applications (2026)
- MITRE ATLAS
- Model Context Protocol Specification
- EU AI Act — European Commission
- ISO/IEC 42001:2023 — AI Management Systems
Stay in Touch
We’re excited to share our expertise and help you build safer AI systems. If you have any questions, feel free to contact us.
End of Document
AI Security Auditing Methodology — Hacken OU