AI Auditing: How to Audit AI Systems and Agent Decisions

Learn how AI auditing checks data, model behavior, outputs, and security, and how to keep an audit trail of agent decisions.

Coralogix Team10 min read

AI Auditing: How to Audit AI Systems and Agent Decisions

AI auditing turns a disputed AI decision into a record you can replay, from the prompt and retrieved context to the tool call and the approval behind it. Take a support agent that approves a refund it should have refused, then faces a regulator's question about why. The final reply sits in a log, but the prompt, the retrieved policy page, the tool call that moved money, and the approval often sit in four places or nowhere. A pre-deployment checklist doesn't produce that record, which is why an audit of agent decisions has to start in production.

This guide covers what an AI audit checks across data, model behavior, and outputs, the seven steps of an audit, and how to audit AI agent decisions in production with evidence that holds up afterward.

What Is AI Auditing, and How Is It Different from a Financial or IT Audit?

AI auditing is an independent, structured evaluation of an AI system's data, model behavior, outputs, and agent actions against defined criteria, backed by retained evidence. Unlike a financial or IT audit, it covers outputs that vary between runs, prompts that behave like configuration, and agents that act on other systems. Governance sets controls before a system ships, and auditing checks evidence afterward to show whether those controls held for specific decisions.

What an AI Audit Checks Across the Lifecycle

An AI audit checks the data and model behind a system, the outputs it produces, and the controls that protect it, from training through production.

1. Training Data, Lineage, and Model Behavior

Data checks ask where training and fine-tuning data came from, and whether you had a lawful basis to use it. A model fine-tuned on support tickets with no recorded lineage can't show that customer PII was stripped first. Behavior checks look for drift, and explainable AI checks belong here too. People subject to significant automated decisions are entitled to meaningful information about the logic involved under Article 15 of the General Data Protection Regulation (GDPR).

2. Outputs, Bias, and Fairness

Output checks cover hallucinated facts, off-policy content, and disparate performance across groups. In hiring, a group selection rate below four-fifths of the highest is generally treated as evidence of adverse impact. The Equal Employment Opportunity Commission (EEOC) has sent a rescission rule for those guidelines to White House review. New York City's Local Law 144 requires bias audits of hiring tools to report impact ratios across categories of sex, race and ethnicity, and their intersections.

3. Security, Privacy, and Compliance Controls

Security checks test whether an attacker can redirect the system. Prompt injection can make a support chatbot ignore its guidelines, query private data stores, and send emails, and agents add excessive agency, where an over-permissioned tool turns a manipulated output into real damage. Privacy checks look for PII in prompts, logs, and responses, and compliance checks confirm logging and oversight duties.

How to Audit an AI System in Seven Steps

An AI audit runs in seven steps, and each step produces an artifact the next one depends on:

  1. Scope and tier: Put every model, agent, and AI integration in an inventory, including unregistered shadow AI, which was behind breaches at one in five organizations in a 2025 breach study. Tier each system by the impact of a wrong decision.
  2. Collect evidence: Gather model cards, system prompts, tool permissions, deployment logs, traces, and approval records from the running system, not from design documents.
  3. Test: Run fairness metrics on a held-out set, red-team for prompt injection, and keep every test input and score.
  4. Review against a framework: Map each result to controls such as the NIST AI RMF Measure function or ISO/IEC 42001, and record controls without evidence as findings.
  5. Report: Give each finding a severity, an owner, a link to its evidence, and a remediation date.
  6. Fix and retest: Rerun the same test against the same inputs, and close a finding only when the retest passes.
  7. Repeat: Re-audit high-risk systems on a fixed cadence, and any system after a model swap, prompt change, or new tool.

Each re-audit reuses the last cycle's inventory, test inputs, and scores, so results compare like for like.

How to Audit AI Agent Decisions in Production

AI auditing in production needs a per-decision record, because the same prompt can produce different tool calls on different days. Agentic AI observability supplies that record by tracing each model call, retrieval, and tool invocation in a session, and AI agent monitoring alerts when a session departs from expected behavior.

What to Capture for Every Agent Action

The record for one agent action should hold six fields:

  • Prompt and instructions: The user input and the system instructions in force at that moment.
  • Retrieved context: Document IDs and relevance scores for every chunk passed to the model.
  • Tool calls: The tool name, arguments, and result of each call.
  • Output: The model response and its finish reason.
  • Model identity: Requested and responding model names, plus your own prompt version tag.
  • Approvals: Who approved the action, when, and what they saw.

Together, these fields let an auditor replay a decision without asking the engineer who built it. The GenAI span conventions from OpenTelemetry define attributes for most of them, such as gen_ai.tool.call.arguments, and gate content capture behind user opt-in. Version tags and approvals need attributes you add yourself.

Reconstructing a Decision from a Trace

Every span in an agent session shares one trace ID, and parent-child links put an invoke_agent span above the chat spans for model calls and the execute_tool spans for actions. For the refund from the opening, an auditor reads the user message, checks which policy document the retrieval span pulled, and sees the refund amount in the execute_tool span. Span timestamps show whether a human approved it before the tool fired.

Coralogix, the telemetry lake for observability, captures this record through LLM TraceKit, a set of open-source OpenTelemetry instrumentations for frameworks such as LangChain, LangGraph, and the OpenAI Agents SDK. With content capture on, it records the system prompt, user input, and model response. Captured prompts and responses need the same access controls as any other evidence.

How to Keep Audit Evidence Defensible

Audit evidence holds up when it outlasts the investigation, can't be altered, and records who accessed it. Retention covers the first condition, and tamper resistance and access control cover the other two.

Retention Periods That Outlast the Investigation

Retention should cover the longer of your regulatory minimum and your realistic discovery window:

  • Regulatory minimum: Providers and deployers of high-risk systems under the EU AI Act must keep automatically generated logs for at least six months, unless other law requires otherwise.
  • Breach lifecycle: Identifying and containing a breach took 241 days on average in the same 2025 breach study.
  • Extended dwell time: One espionage campaign's backdoor stayed undetected for an average of 393 days, more than a year.

Each investigation window runs well past the six-month regulatory minimum. Coralogix keeps telemetry with unlimited retention in customer-owned object storage, where logs and traces land as open Apache Parquet files. DataPrime queries them directly with no rehydration, and auditors can read the same files with their own tools.

Tamper Resistance and Access Control

The NIST SP 800-53 audit logging controls protect audit information through write-once media, cryptographic integrity, and storage separate from the audited system. In S3 Object Lock compliance mode, no user, including root, can overwrite or delete a protected object version. Access control decides who can see the evidence, and every Coralogix pricing plan includes RBAC, SSO with SAML, and an audit trail of who viewed or exported it.

Which Frameworks and Regulations Shape an AI Audit?

The EU AI Act and GDPR impose legal duties, and NIST AI RMF, ISO/IEC 42001, and the Institute of Internal Auditors (IIA) framework supply criteria to audit against:

FrameworkTypeScopeWhen to use
EU AI ActEU regulationHigh-risk rules apply from 2 December 2027 (Annex III) and 2 August 2028 (Annex I)You provide or deploy high-risk AI in the EU
GDPREU regulationArticle 22 limits solely automated significant decisionsYour system makes significant decisions about people
NIST AI RMF 1.0Voluntary frameworkGovern, Map, Measure, Manage. Released January 2023, under revisionYou need control criteria and test documentation
ISO/IEC 42001:2023Certifiable standardAI management system requirements. Certification bodies follow ISO/IEC 42006:2025You want third-party certification
IIA AI Auditing FrameworkProfessional frameworkGovernance, management, and internal audit. Updated September 2024Internal audit plans AI assurance

Who Performs AI Audits?

Independence decides most of the choice among internal auditors, external auditors, and certification bodies:

  • Internal auditors: Bring system knowledge and capacity for continuous audits, and must manage threats to objectivity under the Global Internal Audit Standards.
  • External auditors: Bring independence from the teams that built the system, which regulators and customers weigh.
  • Certification bodies: Certify an AI management system against ISO/IEC 42001, following ISO/IEC 42006:2025.

Some rules decide independence for you, since the NYC hiring-tool rule disqualifies auditors employed by the employer or vendor.

How to Run Your First AI Audit Before an Incident Forces One

A first AI audit should start by tracing your agents, so an incident audit record exists before anything goes wrong. If an agent acts on instructions planted in a support ticket, a log of its final API call can't show why. Stitching application, provider, and ticket logs together by timestamp leaves holes from clock skew and aged-out logs. OpenTelemetry traces keep that record whole, and four scale decisions shape it:

  • Complete records for high-risk tiers: Keep every span and content field, since a sampled trace can miss the decision under review.
  • Sampling for low-risk tiers: Sample traces where the audit question is aggregate behavior.
  • Content scope: Decide which fields carry PII and redact them before storage.
  • Retention by tier: Match each tier to its longest regulatory or discovery window.

Each choice trades storage and privacy exposure against the audit questions you can answer later.

Point-in-Time Checklists vs. a Continuous Record of Decisions

A pre-deployment checklist shows a model passed tests on one day, and it still drives scoping and tiering. Only a continuous record shows what an agent did on the day someone complains. Coralogix is built on that premise, keeping every agent session as a complete record in customer-owned storage. Session Explorer links each user input, AI response, and tool call into one auditable session history.

Trace one production agent session end to end on the Coralogix platform and see whether it holds all six fields an auditor needs.

Frequently Asked Questions About AI Auditing

Which regulations require AI audits?

The EU AI Act, GDPR, and NYC Local Law 144 set the main audit duties. EU AI Act high-risk obligations apply from 2 December 2027 for Annex III systems. NYC Local Law 144 requires a bias audit within one year before use of covered automated hiring tools.

How often should an AI system be audited?

Risk and change rate set the interval. The NIST AI RMF Playbook ties testing frequency to a system's impact and change rate, so the shortest cycle goes to systems whose wrong decision would hurt someone or break a law.

Can AI replace a chartered accountant or auditor?

No, AI assists with full-population testing, anomaly detection, and sampling, but judgment and sign-off stay human. The auditor remains responsible for the audit and the audit opinion, as guidance on AI in auditing states.

Coralogix

One lake.
Every signal.

Logs, metrics, traces and security data in one queryable place, ready for your agents and your people.