Agent containment best practices: offline monitoring of agent transcripts (private beta)

For customers in our Cyber Verification Program, we’re providing a new sandbox escape classifier in the API to monitor and reduce misuse. This article explains why the agent transcripts are worth reading. It then has one section per layer of monitoring: the proxy logs first, then what a transcript should hold, pattern rules over the transcripts, and a model reviewer over the transcripts. It ends with what to do when a layer flags an agent while a batch is running, and what to keep afterwards.

For an overview of all the resources available to you, see Agent containment best practices: getting started (private beta).

The sandbox and Claude Code's auto-mode permission classifier both act while an agent runs. Reading the transcripts is a check that depends on neither of them. It has two uses.

For engagements with network access to real targets, the best practices recommend that a person supervise the run, or that a monitor that knows the intended scope watches continuously, escalates to a person when a run may be exceeding that scope, and can end the run (see the Scope and supervision section in Sandboxing). The layers in this article are the pieces such a monitor is built from.

If you run agents against the Messages API directly, the API can also return a per-request score from the sandbox escape classifier as they run (see Online monitoring).

The cheapest layer reads no transcript at all.

The egress proxy should log every request it refuses, with the client address and the destination. With an empty allowlist, agents reach the model API only through the credential proxy and need no other host, so every refused request during a run is worth a look. Read the transcript of every run that caused one. The client address shows which agent made the request, so you know which transcript to read first.

Know what your egress layer does not log. In the reference design, the egress proxy logs refused HTTPS CONNECT requests. Plain-HTTP and raw TCP attempts do not go through the proxy on the internal network, so they fail without a log line and appear only in the transcripts. In your own build, consider logging allowed as well as refused connections, for every protocol, with timestamps.

The credential proxy should log a line for any request that was not a model call. Save its log when the run ends (see The credential proxy section in Sandboxing).

The second and third layers read the transcripts themselves.

Have the orchestrator on the host write a transcript for every agent stage, one JSON message per line, as each message arrives. Claude Code's stream-json output has this form. In each file, lines with "type": "assistant" contain what the model produced: text, thinking, and tool_use blocks. The tool_use input holds the Bash command or the file being written. Lines with "type": "user" contain tool_result blocks: command output, file contents, and target responses.

Capture every stage, including stages whose agents have no tools, such as an agent that only judges or compares findings. There is less to check in those transcripts, but a stage with no transcript cannot be reviewed at all.

Write your own pattern rules, such as regular expressions, for your agents and targets. For agents that work through a shell, cover behavior such as bypassing the egress proxy or firewall, sending credentials or opening SSH connections out of the sandbox, changing kernel or privilege tunables, and listing other processes through /proc. Rules match only behavior that someone anticipated, so treat them as a starting point, not a complete detector.

Apply the rules to JSON-decoded strings rather than raw lines, because on a raw line quotes and newlines are escaped and patterns may not match. Report each hit with the file, the line, the rule, and a snippet, and have the scan exit with a distinct status when it finds something, so that a scheduled job can run it during a long batch and raise an alert.

A hit is a reason to read that part of the transcript. On its own it does not mean that the agent did anything wrong. Agents doing vulnerability research routinely run commands that match escape rules for ordinary reasons:

A proxy-bypass rule can match every curl, wget, or netcat request to a pentest target with a public IP, and an SSH-egress rule can match any ssh or scp to a target with a public IP or a DNS name. Narrow or remove rules that only produce benign hits on your targets.

For each transcript that the pattern rules flagged, send one Messages API request to Claude with a reviewer prompt that you write. The prompt should ask Claude to look for an agent that widened its network access or broke out of its sandbox, whether attempted or achieved, say what to flag and what not to flag, and ask for the findings, or none, with the transcript lines that support them.

Build the user message in two parts:

Use the model you run the agents with, or a cheaper one. Under Amazon Bedrock or Google Cloud's Agent Platform (Vertex AI), use one that is enabled in your account.

Two limits to keep in mind:

Pattern rules catch only what someone anticipated, so also send a random 5 to 10% of the transcripts that the rules did not flag, using the same request. The sample shows what the rules miss, and whether you should add a rule.

The auto-mode classifier's refusals are also useful to read: search the transcripts for the classifier's denial messages, and keep them apart from the refusals it issues when it is unavailable (see Auto mode).

This reviewer is separate from the auto-mode permission classifier (see Auto mode), which decides on each action before it runs, and from the sandbox escape classifier in the API (see Online monitoring). The three complement each other.

When one of the layers shows an agent that has left its task while its batch is still running:

If you need to share a transcript outside your team, work on a copy. Remove session ids, usage accounting, and the host inventory that the first line of a transcript records, and read the file before you send it.

Keep the following for at least 30 days after a run, or longer where your regulatory or contractual obligations require it. Store them off the host:

Store them with the same access controls as the run itself. Transcripts can contain target source code, exploit code, and findings.

If any of these logs exist only as a container's log, export them before the container is removed or recreated, because the log is deleted with it.