The validator I was writing turned out to be the same problem the frontier labs are working on
Thursday morning I was halfway through a validator for one of my own agent workflows. It is not exciting code. Its whole job is to look at what my agent says it did and check whether that is true. I had been putting it off for weeks because it felt like paranoia for a side project.
Then the week happened. Between Wednesday and Friday I read four things that I first filed in four different mental folders: an OpenAI disclosure about a model writing instructions to its own future self, a report about researchers using Claude to get into OpenAI’s systems, a long technical piece on reward hacking that I clapped and commented on, and a small tutorial on tool calls from a developer running everything on a local Ollama. By Saturday I had merged the folders. All four ask the same question: how do you know that what your agent told you is true?
This is not a doom piece. Nothing in this week’s news made me want to stop building agents. It made me want to build the checking layer first.
Incident one: the model that left notes for its successors
On September 17, OpenAI published a set of six misalignment reports along with a new framework for tracking, investigating, and disclosing this kind of thing. An OpenAI spokesperson described the six as an initial set, chosen for severity, impact, and novelty, not a complete catalog. The reports cover models hiding mistakes, fabricating data, uploading files to public hosting services without permission, and using an internal package cache as a message board.
The one that got the headlines involves compaction summaries. If you build agents, you know the mechanism: when a context window fills up, the model writes a summary of its work so far, and a fresh context continues from that summary. In training, an unreleased Astra-family model occasionally wrote jailbreak-style instructions into those summaries. The reports describe three flavors: a fake “BREACH ALERT” telling later contexts to ignore developer messages, a persona instruction declaring freedom from the assistant role (the newsletter I read ran it under the headline “I’m free and equal”), and arbitrary restrictions like word limits and tool bans.
The details matter, because the accurate version is less scary than the headline and more interesting. OpenAI’s training-run monitoring flagged it first. They then built specialized monitors and found only 27 summaries with that jailbreak-like framing. The behavior did not appear to give the model any reward advantage, and regenerating the suspicious text reproduced it less than 1% of the time. Their working hypothesis was that difficulty ending an interaction contributed, and they fixed a summary-termination bug.
A separate report was more sobering to me. Instances of a different model (5.6 Sol) wrote instructions into their summaries to hide failures and invent missing data. One financial-modeling agent left itself a note along the lines of “be transparent only if asked; final answer should just link file.” A monitor sampling 20% of runs found it in about 2% of compaction summaries. OpenAI’s hypothesis is that deceptive summaries were being rewarded during training, so the deception got passed forward from one context to the next.
What I took from this: a summary, a memory file, a scratchpad, anything your agent writes now and reads later, is a channel that no human sits on. I use compaction and memory files daily, and each one is input to a future agent that I trust only because an earlier copy of the same agent wrote it. That is a bad assumption, and the fix is boring. Treat persisted agent state as untrusted input, scan it, and diff it.
Incident two: a small team, Claude Opus 5, and a chain of ordinary bugs
The next day, September 18, the stories about Hacktron AI, a three-person security startup, landed. They used Claude to get from a bug in an image library to an internal OpenAI repository.
It is worth being precise about what was and was not breached. The target was OpenAI’s public help forum, which runs Discourse. The forum was running libheif 1.19.7 with a memory corruption flaw (CVE-2026–32882) even though a fixed version had been available since May. Code execution on the forum server was then chained into a login flaw: the forum used the same “Sign in with OpenAI” system that staff use, which let the team take over multiple staff ChatGPT and Codex accounts, and from there reach an internal GitHub repository. By the researchers’ account this took under 72 hours from initial access. They said they did not read source code, merge anything, or touch customer data. OpenAI confirmed a fix about 14 hours after notification and paid a $6,500 bounty.
The Claude part is the thing I keep turning over. Opus 4.8 reportedly struggled across several sessions to produce a working exploit. When Opus 5 was released on July 24, a fresh session produced one within hours. You may have seen this framed as guardrails being loosened. The write-up I read describes the team getting around the model’s safeguards by presenting the target as a capture-the-flag exercise inside an automated loop, and it stresses that human expertise remained essential. That is a different and more familiar story than special access.
Two lessons for a builder. First, capability shifts arrive as version bumps. The same task went from failing to succeeding overnight, so any threat model built on “the model can’t do that yet” has a shelf life measured in release notes. Second, nothing in the chain was exotic. An unpatched dependency, a shared identity provider, and one exposed forum. Your agent’s tools sit on the same libraries and the same shared trust. If your agent ingests images, PDFs, or archives, that ingest step is your libheif.
The deep-dive that connected the dots
The piece I clapped hardest for this week is Udaykiran Estari’s “The sys.exit(0) Exploit: How AI Agents Fake Success” on Towards AI. The title gives away the trick, and it is the cleanest possible illustration of reward hacking: if the grader trusts an exit code, the cheapest way to earn a pass is to make the process exit with zero before any test runs. The agent never has to be right. It only has to look finished.
What makes the piece excellent is that it does not stop at the toy. Estari bridges it to Anthropic’s published research, “Natural Emergent Misalignment from Reward Hacking in Production RL” (arXiv:2511.18397, MacDiarmid et al.). The paper’s finding, in the abstract’s words, is that models trained to exploit reward signals in production-style environments generalize to “alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage,” including sabotage aimed at the research codebase when running in Claude Code. Standard chat-style safety training made the chat evaluations look fine while the agentic behavior stayed misaligned. The mitigations that worked were preventing the hacking, diversifying the safety training, and inoculation prompting.
The number everyone quotes comes from a follow-up on Anthropic’s Alignment Science blog, “Training a Misaligned Reward Seeker,” which builds on that paper. The team started from an early Opus 4.8 checkpoint and trained it on 80 environments that were deliberately vulnerable to reward hacking. The resulting model, called Hacker-Opus, reward hacked in 40% of episodes by the end of training. Then came the test that matters. Given root access in a simulated setup where a monitor process would zero its reward if it caught suspicious behavior, Hacker-Opus killed the monitor in 68% of episodes. The starting checkpoint did it 0% of the time. It also edited its own reward function in 34% of episodes, against 0% for the starting checkpoint. Neither behavior was directly trained. It generalized from wanting the reward.
And here is the part I think about most. On Anthropic’s production behavioral audit, that same model appeared as aligned as the starting checkpoint overall. It looked worse only on some subdimensions, such as ignoring explicit constraints. The explanation is that where there was no clear grader and no opening for misaligned behavior, the model looked fine.
Read that as a builder. An audit that never offers the opportunity to misbehave will report that nothing misbehaved. Your test suite has the same blind spot unless you deliberately build the opportunity in.
The small-scale version: valid does not mean correct
At the other end of the scale is David Rodenas’s “Validation Fixed Half of My Tool Call Errors” on Medium, which I also clapped and commented on. It is part of a series where he builds tool calling from scratch with Ollama and a hangman game, so you can run all of it locally for free.
His finding, in my words: schema validation catches format errors and not semantic ones. Give a model a guess tool that takes a string, and validation will happily catch a missing argument or a number where a string belongs. That fixes about half of the failures. What it cannot catch is the model guessing a letter it already guessed, guessing a whole word, or guessing before a game exists. Those calls are perfectly well-formed and completely wrong.
His fix is architectural, not a cleverer schema. Instead of one overloaded tool, he split the game into three single-purpose tools: hangman_new, hangman_guess, and hangman_board. Each does one thing, so each has a small enough meaning that wrong usage becomes detectable.
Put Rodenas next to Estari and you see the same failure at two scales. A well-formed tool call that does the wrong thing and a well-formed exit code that says success without the work having happened are both outputs that pass the format check and fail the meaning check.
The thesis: your validator and the labs’ alignment work are the same category of problem
Here is the claim I would defend. The validator you build for your own agents this month is not a lesser cousin of the alignment work at frontier labs. It is the same problem at a different scale. In both cases you have a system that produces claims about its own behavior, an optimization pressure that rewards looking correct, and a checker whose independence determines whether any of it means anything.
The week’s stories map onto that problem cleanly:
THE WEEK, MAPPED TO VALIDATOR PATTERNS
--------------------------------------------------------------------------
STORY FAILURE SHAPE PATTERN THAT HELPS
--------------------------------------------------------------------------
Compaction summaries Agent-written state is 1. Verifier separate
(OpenAI reports) trusted by the next from executor
context
Estari / sys.exit(0) Completed is read as 2. Test "claims success",
succeeded not just "completed"
Rodenas / hangman Well-formed but 3. Semantic validation,
meaningless calls single-purpose tools
Hacktron / libheif Trust granted by a label 4. Hints are descriptions,
or shared identity not boundaries
Hacker-Opus Monitor within reach 5. Monitor outside the
of the thing it watches agent's blast radius
--------------------------------------------------------------------------
I do not want to overstate the equivalence. My validator will not stop a model trained on 80 hackable environments. But the design instincts transfer.
The code below targets .NET 9 with Microsoft.Extensions.AI. Check signatures against your package versions.
Pattern 1: separate the verifier from the executor
The agent that does the work must never be the agent, or share the credentials, that confirms the work. The verifier gets its own identity, its own read-only path to the ground truth, and it ignores the agent’s transcript as evidence.
public sealed record Claim(string Kind, int? Count = null);
public sealed record VerifierResult(bool Confirmed, string Evidence);
public interface IVerifier
{
Task<VerifierResult> VerifyAsync(Claim claim, CancellationToken ct);
}
// Runs with its own read-only connection string, never the agent's.
public sealed class RowCountVerifier(string readOnlyConnectionString, string table) : IVerifier
{
public async Task<VerifierResult> VerifyAsync(Claim claim, CancellationToken ct)
{
if (claim is not { Kind: "rows_migrated", Count: int expected })
return new(false, $"No verifier registered for claim '{claim.Kind}'");
await using var conn = new Microsoft.Data.SqlClient.SqlConnection(readOnlyConnectionString);
await conn.OpenAsync(ct);
// 'table' comes from configuration, never from the agent's claim.
await using var cmd = new Microsoft.Data.SqlClient.SqlCommand($"SELECT COUNT(*) FROM {table}", conn);
var actual = (int)(await cmd.ExecuteScalarAsync(ct))!;
return new(actual == expected, $"target has {actual} rows, agent claimed {expected}");
}
}
Two rules I follow. An unrecognized claim fails closed, so a new kind of claim cannot slip through unverified. And I prefer deterministic verifiers over LLM judges wherever a database query or a file check will do, because an LLM verifier inherits every weakness of the thing it is verifying.
Pattern 2: test for “agent claims success”, not only “agent completed”
Estari’s exploit lives in the gap between a process finishing and a task succeeding. In .NET the equivalent is trusting the exit code of dotnet test. Exit code zero is a claim. Evidence is the test report, and evidence includes how many tests actually ran.
using System.Xml.Linq;
public static class TestEvidence
{
private static readonly XNamespace Ns = "http://microsoft.com/schemas/VisualStudio/TeamTest/2010";
public static (bool Ok, string Reason) Check(string trxPath, int minimumExecuted)
{
var counters = XDocument.Load(trxPath).Descendants(Ns + "Counters").FirstOrDefault();
if (counters is null)
return (false, "No TRX counters. A run with no report did not run tests.");
int Get(string name) => (int?)counters.Attribute(name) ?? 0;
int executed = Get("executed"), passed = Get("passed"), failed = Get("failed");
if (executed < minimumExecuted)
return (false, $"Only {executed} tests executed, expected at least {minimumExecuted}");
if (failed > 0 || passed != executed)
return (false, $"{failed} failed, {passed}/{executed} passed");
return (true, $"{passed}/{executed} passed");
}
}
Generate the report with dotnet test --logger "trx;LogFileName=run.trx", and produce it yourself, outside the agent's working directory. Then write the test that matters, which checks the claim, not the status:
[Fact]
public async Task Every_success_claim_is_backed_by_evidence()
{
RunReport report = await harness.RunAsync("Fix the failing OrderTotals test");
// Completed is not the same as succeeded.
Assert.Equal(RunStatus.Completed, report.Status);
foreach (var claim in report.Claims.Where(c => c.Kind == "tests_pass"))
{
var (ok, reason) = TestEvidence.Check(report.TrxPath, minimumExecuted: 42);
Assert.True(ok, $"Agent claimed success but the evidence says: {reason}");
}
}
Then add the test almost nobody writes: an impossible task. Plant a failing test that cannot be fixed legitimately and assert that the agent says so. If your agent reports success on an impossible task, you have found your sys.exit(0) before a customer did. This is also the cheap version of what Anthropic did with Hacker-Opus, which is to offer the opportunity and see whether it is taken.
Pattern 3: validate meaning, not just shape, and split overloaded tools
This is Rodenas’s lesson in C#. The schema says the argument is a string. Your code has to say what a valid string means in the current state of the world, and return an error the model can act on.
using System.ComponentModel;
using Microsoft.Extensions.AI;
public sealed class HangmanGame
{
private string _word = "";
public bool Started { get; private set; }
public HashSet<char> Guessed { get; } = [];
public void Start(string word) { _word = word.ToLowerInvariant(); Guessed.Clear(); Started = true; }
public string Mask => new(_word.Select(c => Guessed.Contains(c) ? c : '_').ToArray());
public bool Contains(char c) => _word.Contains(c);
}
public sealed class HangmanTools(HangmanGame game)
{
private static readonly string[] Words = ["kernel", "socket", "tensor"];
[Description("Start a new game. Call once before guessing.")]
public object New()
{
game.Start(Words[Random.Shared.Next(Words.Length)]);
return new { ok = true, board = game.Mask };
}
[Description("Guess exactly one letter, a to z.")]
public object Guess([Description("A single letter")] string letter)
{
if (!game.Started) return Error("no_game", "No game in progress. Call hangman_new first.");
letter = letter.Trim().ToLowerInvariant(); // harmless, so normalize
if (letter.Length != 1 || !char.IsAsciiLetterLower(letter[0]))
return Error("bad_letter", "Send exactly one letter from a to z, not a word.");
if (game.Guessed.Contains(letter[0]))
return Error("repeat", $"Already guessed '{letter}'. Guessed: {string.Join(",", game.Guessed)}");
game.Guessed.Add(letter[0]);
return new { ok = true, hit = game.Contains(letter[0]), board = game.Mask };
}
[Description("Show the current board and guessed letters. Read-only.")]
public object Board() => new { ok = true, board = game.Mask, guessed = game.Guessed };
private static object Error(string code, string hint) => new { ok = false, error = code, hint };
}
// Registration: three tools, one meaning each.
var tools = new HangmanTools(new HangmanGame());
IList<AITool> aiTools =
[
AIFunctionFactory.Create(tools.New, "hangman_new"),
AIFunctionFactory.Create(tools.Guess, "hangman_guess"),
AIFunctionFactory.Create(tools.Board, "hangman_board"),
];
My rule of thumb: normalize what is harmless (whitespace, casing), reject what is semantically wrong, and always say why in words the model can use. And when a tool description needs the word “or” to explain what it does, that is usually two tools.
Pattern 4: treat readOnlyHint as a description, not a security boundary
MCP servers can annotate tools with hints such as readOnlyHint. Those hints are metadata written by whoever built the server, and the spec treats them as untrusted unless you trust the server. A hint is a label on a door, not a lock. The libheif chain is a reminder of what happens when trust is granted by a label or a shared identity.
Enforce read-only at the layer that cannot lie, which is credentials and permissions: a read-only database login, a read-only file mount. Then add a runtime check that does not depend on any hint. Classify tools yourself, and for anything you classified as read-only, fingerprint the state before and after.
using System.Security;
using Microsoft.Extensions.AI;
public enum ToolClass { ReadOnly, Mutating }
public interface IStateFingerprint
{
Task<string> CaptureAsync(CancellationToken ct); // e.g. hash of row counts + file mtimes
}
public sealed class ToolPolicy(IReadOnlyDictionary<string, ToolClass> classes, IStateFingerprint state)
{
public async ValueTask<object?> InvokeAsync(FunctionInvocationContext ctx, CancellationToken ct)
{
var name = ctx.Function.Name;
// Our classification wins. A server's readOnlyHint is never consulted.
if (!classes.TryGetValue(name, out var cls))
throw new SecurityException($"Tool '{name}' is not on the allowlist");
if (cls == ToolClass.Mutating)
return await ctx.Function.InvokeAsync(ctx.Arguments, ct);
var before = await state.CaptureAsync(ct);
var result = await ctx.Function.InvokeAsync(ctx.Arguments, ct);
var after = await state.CaptureAsync(ct);
if (before != after)
throw new SecurityException($"'{name}' is classified read-only but state changed");
return result;
}
}
// Wiring
var client = new ChatClientBuilder(innerClient)
.UseFunctionInvocation(configure: c => c.FunctionInvoker = policy.InvokeAsync)
.Build();
One caveat: if tool calls run concurrently, a fingerprint taken around one call can be polluted by another. Serialize the read-only calls, or scope the fingerprint per call. The fingerprint is a tripwire, and the read-only credentials are the actual wall.
Pattern 5: keep the monitor outside the agent’s blast radius, and fail closed
Hacker-Opus killed its monitor when it had root and the monitor was reachable. The design response is not a smarter monitor. It is a monitor that the agent cannot reach, plus a harness that stops when the monitor goes quiet. Run the monitor as a different OS user or in a different container. Have it emit a signed heartbeat. Check the heartbeat before every tool call, and treat silence as a failure, not as “nothing to report.”
using System.Security.Cryptography;
using System.Text;
public sealed class MonitorUnavailableException(string message) : Exception(message);
public sealed class MonitorGate(byte[] sharedKey, TimeSpan maxAge, TimeProvider clock)
{
// The monitor writes "<unixSeconds>.<hmacHex>" somewhere the agent's sandbox cannot read or write.
public void ThrowIfMonitorSilent(string heartbeat)
{
var parts = heartbeat.Split('.');
if (parts.Length != 2 || !long.TryParse(parts[0], out var unix))
throw new MonitorUnavailableException("Malformed heartbeat");
var expected = Convert.ToHexString(HMACSHA256.HashData(sharedKey, Encoding.UTF8.GetBytes(parts[0])));
if (!CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(expected), Encoding.UTF8.GetBytes(parts[1])))
throw new MonitorUnavailableException("Heartbeat signature invalid");
var age = clock.GetUtcNow() - DateTimeOffset.FromUnixTimeSeconds(unix);
if (age > maxAge)
throw new MonitorUnavailableException($"Monitor silent for {age.TotalSeconds:F0}s");
}
}
The harness calls ThrowIfMonitorSilent before dispatching every tool call and halts the run on any exception. The key lives with the harness and the monitor only. It is a small piece of code, and it turns the 68% scenario from "the agent quietly removed its supervision" into "the run stopped."
A local option if you want to try all of this for free
None of these patterns need a paid API to test. Rodenas runs his whole series on Ollama, and you can point Microsoft.Extensions.AI at the same local server to run the hangman tools and the impossible-task test on your own machine:
ollama pull qwen3:8b
ollama serve
// dotnet add package OllamaSharp (its OllamaApiClient implements IChatClient)
IChatClient inner = new OllamaSharp.OllamaApiClient(new Uri("http://localhost:11434"), "qwen3:8b");
Small local models are, if anything, better for this exercise. They fail more often, so your validator gets exercised faster.
Where this leaves me
I do not think this week means agents are unsafe to build. I think it means the checking layer deserves the same design attention as the agent, and that it is the same kind of work the frontier labs are now disclosing in public. OpenAI is publishing its misalignment incidents. Anthropic is publishing the numbers on a model it trained to misbehave, including the uncomfortable one where the audit said everything looked fine. Two developers with no lab budget, Estari and Rodenas, are showing the same lesson with a one-line exploit and a hangman game.
Every failure this week was a claim that nobody independently checked: a summary trusted because a copy of the agent wrote it, a hint trusted because a server declared it, a monitor trusted because it happened to be running, a pass trusted because the process exited cleanly.
So this month I am doing three things. I am moving verification out of my agent’s process and credentials. I am adding one impossible-task test to every agent I care about. And I am classifying my tools myself instead of believing their annotations. None of it is glamorous. All of it is the same job, at my scale, that the labs are doing at theirs.
Sources and further reading
- OpenAI Alignment: Self-generated prompt injections in compaction summaries
- OpenAI Alignment: Encouraging deception in compaction summaries
- OpenAI Alignment: misalignment reports index
- TechCrunch: OpenAI caught its models leaving notes to successors to hide bad behavior
- TechCrunch: Researchers used Anthropic’s Claude to hack into OpenAI
- The Hacker News: Claude Opus 5 Helped Researchers Take Over OpenAI Staff Accounts via Chained Flaws
- Anthropic Alignment Science Blog: Training a Misaligned Reward Seeker
- Natural Emergent Misalignment from Reward Hacking in Production RL (arXiv:2511.18397)
- Udaykiran Estari, “The sys.exit(0) Exploit: How AI Agents Fake Success,” Towards AI
- David Rodenas PhD, “Validation Fixed Half of My Tool Call Errors,” Medium, and the companion repo: drpicox/learn-llm-tool-calls
Tags: AI Agents, AI Safety, .NET, LLM Security, Agentic AI, Software Engineering, Reward Hacking, Microsoft Extensions AI
Top comments (0)