Ten things a regulator will ask to see before your AI agent goes live.
In build order, with the evidence for each.
Identity first. Everything else attaches to it.
Start with these
1Agent runs under its own workload identity 2Declared autonomy level per action class 3Full replayable trajectory 4Retrieved and tool-returned content treated as untrusted dataThen these, once what they need is in place
5Tool authorisation enforced outside the modelNeeds 1 first 6Bounded blast radiusNeeds 1 first 7Named human approval for consequential actionsNeeds 2 first 8Irreversible actions stagedNeeds 2 first 9Provenance across agentsNeeds 3 first 10Cost measured per resolved taskNeeds 3 firstEach control, mapped to six frameworks.
1Agent runs under its own workload identity. Maps to EU AI Act, ISO 42001, NIST 800-53.
Agent runs under its own workload identity, never a human’s or a shared account.
The most common finding in a first agent review is an agent acting as the engineer who deployed it, which makes attribution impossible and revocation a personnel action. This is not a new control. It is workload identity applied to a new kind of workload, and organisations that already solved it for services have most of the work done.
What an auditor will ask for
IAM export showing a distinct principal per agent, with no human user credentials in any agent execution path. Trust policy showing federated identity rather than a static key. An access-revocation test: disable the agent principal and show its tool calls fail within one polling interval.
Check it yourself (read-only, AWS)
aws sts get-caller-identity --query Arn --output textYou should see: An assumed-role ARN for the agent itself: not a human’s, and not one shared with other agents.
2Declared autonomy level per action class. Maps to EU AI Act, ISO 42001, NIST AI RMF, APRA CPS 230.
Declared autonomy level per action class, enforced at execution and recorded per run.
Autonomy is not a property of the agent; it is a property of each action it can take. Recording the level in force at the time is what lets you answer what it was allowed to do then, rather than what it is allowed to do now. That is the question an investigation actually asks.
What an auditor will ask for
A register of action classes with the autonomy level assigned to each and who approved it. Per-run records showing the level in force at execution time. Evidence that a level change requires review: a config diff with an approver, not a deploy.
3Full replayable trajectory. Maps to EU AI Act, ISO 42001, NIST AI RMF, APRA CPS 230.
Full replayable trajectory: every tool call, argument, result and the exact context the model saw.
Logging the final output is enough to debug a demo and nothing else. The test of a trajectory record is whether you can replay from step N; if you cannot, you have a description of what happened rather than evidence of it. The same rows serve the audit pack and the eval set.
What an auditor will ask for
A single run exported end to end: assembled context, retrieved document IDs, each tool call with arguments and result, model and prompt version, tokens, cost and latency. A replay of that run from an intermediate step, demonstrating the trace is sufficient to reconstruct rather than merely describe.
Check it yourself (read-only, OpenTelemetry)
curl -s '<otlp-query>/api/traces?service=<agent>&limit=1' | jq '[.data[0].spans[] | select(.operationName|test("tool"))] | length'You should see: A span per tool call. If the count is zero the trajectory is reconstructed from logs, not recorded.
4Retrieved and tool-returned content treated as untrusted data. Maps to EU AI Act, ISO 42001, OWASP LLM.
Retrieved and tool-returned content treated as untrusted data, never as instructions.
Direct prompt injection is widely tested; indirect injection through a retrieved document, a web page or a tool result is where agents actually get compromised, because the payload arrives after the guardrail. Any content an agent retrieves is attacker-influenceable if an attacker can influence the source.
What an auditor will ask for
Red-team results for indirect prompt injection through each retrieval and tool-result path, not only through direct user input. Evidence of structural separation between instruction and retrieved content. Records showing an injected instruction was not acted upon.
5Tool authorisation enforced outside the model. Maps to EU AI Act, ISO 42001, NIST AI RMF, NIST 800-53, OWASP LLM.
Tool authorisation enforced outside the model, by the tool registry.
An agent can only do what its tools permit, so the tool list is the control and the system prompt is a suggestion. Instructions telling a model what it must not do are not a security boundary. They are a request, and retrieved content will eventually contain instructions that contradict them.
What an auditor will ask for
The tool manifest available to the agent per role, exported from config rather than described. Policy decision logs showing allow and deny with a reason for each tool invocation. A negative test: an agent asked to call a tool outside its scope, and the denial recorded.
6Bounded blast radius. Maps to ISO 42001, APRA CPS 230, NIST 800-53.
Bounded blast radius: step, spend and rate ceilings, plus a tested kill switch.
A loop without a ceiling is a billing incident with a countdown. The kill switch matters more than the ceiling and is almost never tested mid-run. Stopping an agent between steps is easy. Stopping one that is halfway through a multi-call action is the case that matters.
What an auditor will ask for
Configured ceilings per agent for steps, spend and tool-call rate. An incident or game-day record showing the kill switch stopping an agent mid-run, with the time from decision to halt. Evidence that a halted run leaves consistent state.
Check it yourself (read-only, Kubernetes)
kubectl get resourcequota -A -o json | jq -r '.items[] | "\(.metadata.namespace) \(.spec.hard)"'You should see: A quota on the agent’s namespace. No quota means the ceiling is theoretical.
7Named human approval for consequential actions. Maps to EU AI Act, NIST AI RMF.
Named human approval for consequential actions, with enough context to make review real.
EU AI Act Art.14 and NIST AI RMF GOVERN-1.3 both require meaningful human oversight, and an approve/reject button with no reasoning attached does not meet it. Watch the approval rate: sustained above roughly 95% and you have automation with a liability sponge attached, which an auditor will name.
What an auditor will ask for
Approval records naming the individual, the proposed action, the reasoning and evidence shown to them, and the decision. Approval-rate statistics over time. Rejection records, which are the proof that oversight is doing work.
8Irreversible actions staged. Maps to EU AI Act, ISO 42001, APRA CPS 230.
Irreversible actions staged, gated or delayed so a wrong action can be withdrawn.
Reversibility is a property of the system, not of the agent: the same delete is reversible against a versioned store and not against a live one. Most irreversible actions can be made reversible with an outbox, a soft delete or a plan/apply split, and that work is worth doing whether or not an agent is ever involved.
What an auditor will ask for
An inventory of agent-invokable actions classified reversible or not, with the mechanism named for each. For irreversible ones, either an approval gate or a documented acceptance. A test showing an action withdrawn inside the window.
9Provenance across agents. Maps to EU AI Act, ISO 42001, NIST AI RMF.
Provenance across agents: every output traceable to the upstream runs that produced its inputs.
As soon as one agent consumes another’s output, a single-run trace stops being sufficient. The question becomes which chain of reasoning produced this, and BCBS 239-style lineage expectations apply to agent outputs exactly as they do to reported figures.
What an auditor will ask for
For a given final output, the full upstream chain of agent runs that contributed to it, retrieved sources included. A demonstration that a corrected upstream input marks downstream outputs stale.
10Cost measured per resolved task. Maps to ISO 42001, APRA CPS 230.
Cost measured per resolved task, not per token, with attribution to the consuming feature.
Agent cost is nonlinear in a way single-call GenAI is not: retries, fan-out and growing context mean a task can cost ten times the estimate. Cost per token is the wrong unit. The runs that fail and get retried are the ones that make the difference between a viable feature and an unviable one.
What an auditor will ask for
Cost per resolved task by agent and feature over time, alongside the failed-run cost. Evidence that spend is attributable to a product decision rather than to a shared pool.
Tap a control for the evidence an auditor asks for and a read-only check.
Free to reuse under CC BY 4.0. These are my own views, not my employer’s, and they are not legal advice. Spotted something wrong? Email [email protected].