My Weekly AI Audit Filed 25 Reports. No Automation Acted on a Grade.

Originally published on hexisteme notes.

On November 2 I stopped a scheduled job that had tried to grade how I use my AI tooling every Monday at 06:00 since May. It had filed 25 reports by then. I looked for anything that acted on those grades and found no automation that did.

The job ran cross_family_audit.py 7 --sample 3 from launchd. It scanned every record in Claude Code session files modified within 7 days, sampled 3 sessions, and asked several model families to grade 5 capabilities as low, medium or high. The capabilities included task routing, multi-agent synthesis and async federation, which in plain terms means how much work runs unattended. I had added the job on May 15 as the first prescription for a low async federation grade.

The report went to a dated Markdown file. The last step of the job was /usr/bin/open -e "$OUT", so a text editor window opened on my Mac every Monday at 06:00. The final run used a prompt of 3,381 characters, 1 Anthropic Opus call with max_tokens 2000, and 1 Ollama cloud call.

What the decision record found

The audit came up as one decision in the same cleanup that retired my LLM council. Before changing anything I wrote down what the job did.

• Consumers: I searched my scripts, hooks, skills, rules and agents, plus the py, sh, json and yaml files across the fleet, for anything that read the report directory. The record says 0 pieces of code did. That was too strong, and the review of this post caught it. My dashboard registers the report directory, summarizes each report from its head, and can show one in the browser. What I can still say is narrower: I found no automation that used a grade.
• Reads: the dashboard keeps read timestamps, and 1 of the 25 reports has one, the report dated July 13. A morning brief on September 21 listed the job as unread.
• Coverage: the script is named for comparing model families, and 4 were configured. The reports from May 18 through June 8 each say 0 / 4 and hold no grades, only a note that no family was available and that SDKs and API keys needed setup. June 15 says 2 / 4. June 22 says 1 / 4, and that one answer was a connection error counted as a response. Everything from June 29 to November 2 says 2 / 4, with Anthropic and Ollama answering.
• Cadence: my design note said monthly with a 30-day window. The schedule said weekly with a 7-day window.
• Description: the dashboard card called it a code quality audit that finds rule violations and dead code, and it listed a dependency on a server the job had stopped using after October 28. The job graded usage capabilities.

The audit graded samples with the text removed

The sample was meant to show the reviewers real conversations. sample_conversations keeps the last 3 sessions its file traversal reached, which are not guaranteed to be the newest, and the first 5 records of each. render_samples turns those records into prompt text:

if isinstance(content, list):
    content = " | ".join(
        str(b.get("type") if isinstance(b, dict) else b)
        for b in content
    )
content = str(content)[:200]

When a message's content is a list of blocks, this keeps the block type names and drops the text, so a text block and a tool call reach the reviewer as text | tool_use. Content stored as a plain string survives, cut at 200 characters.

The November 2 report says as much in its opening lines. The Anthropic reviewer started with a premise: of the 3 conversation samples, only 1 showed content, and the other 2 were metadata (/clear, mode, attachment), so the assessment rested mostly on aggregate signals. The tool had described its own input problem, and no automation was there to act on it. That was the last report it wrote.

The grades moved for reasons the input could not see

Here is the Anthropic reviewer's summary table for the last three reports:

The October 26 report justified its low async grade by saying sessions had async work but there was no schedule-based operating signal. That report was written by a launchd job, and several launchd jobs were running. The extractor does not inspect launchd state, so launchd reached the reviewers only through text that mentioned it, such as a conversation sample.

On November 2 the same capability went up. The Ollama family graded it high and cited a conversation sample containing a launchd setup for unattended runs on Sunday at 20:00. The Anthropic reviewer gave it medium and cited the launchd design it saw. Multi-agent synthesis split in that report too: medium from Anthropic, low from Ollama.

I read that as the grade following whichever conversation landed in the sample. That is an inference from the citations. I did not diff the weekly inputs, and the grades were never compared against an outside measure, so I can't say which grade was right in any given week.

Why I stopped it instead of fixing it

My first draft was short: stop the schedule and keep the code and reports. The alternative was to fix render_samples and add the two missing keys. I rejected that because I could find nothing that acted on the grades, so better input would only improve a report with no automation behind it.

This decision was one of the 4 in the blind A/B I wrote up in I Retired My LLM Council After It Lost a Blind A/B 4 to 0. Two verifiers agreed on these corrections to my draft:

• It said Google and OpenAI were missing keys and packages. The packages were installed. Only the keys were missing.
• open -e leaves no read record, the dashboard surface policy only surfaces failures, and the September 21 brief had named the job as a follow-up input. So my read count could not show whether a person used that window. Only I could answer that.
• Stopping ends the weekly series. extract_signals(since) has no end-time argument, so the existing extractor cannot replay a bounded past week.
• The morning brief script imports this audit module, so the module has to stay even with the schedule gone.
• A monthly run without the popup had not been evaluated.

The second correction changed what I could claim. The read count covers the dashboard. "Nobody read them" was an assumption from a record that could not have seen the popup.

The corrected plan had 2 options: stop and archive, or reduce to monthly with a 30-day window and fix render_samples. It put the choice on one question to me: had I ever read that Monday window and acted on it? If not, stop. I approved the recommended stop.

What changed

• launchctl bootout removed the job, and launchctl list | grep -c code-audit returned 0. The plist went to a backup directory.
• All scripts and the 25 reports stay, about 100KB, with no expiry.
• The dashboard card is marked retired, with a corrected description and the stale dependency removed.
• I deleted the job's secret file. I did not revoke the key, because the morning brief uses the same one.
• The morning brief ran on November 3 and November 4, and launchd shows its last exit status as 0.

The restart condition is written down: when I need a usage diagnosis, fix render_samples and re-register the job monthly. A manual run with a 30-day window stays available.

What this does not show

"No automation" is the result of searches made for this decision and this post, and a search can miss a reader, as the first one did. It says nothing about me reading the popup on a given Monday, and I cannot reconstruct that now. The one timestamp is what the dashboard recorded, which leaves out any read that happened through the popup.

I also can't call the audit worthless. Its grades were never compared against an outside measure, so I don't know how good or bad they were. My claim is narrower: across 25 reports, I found no automation acting on a grade, the dashboard has 1 read, and the input had a defect the last report described.

Stopping has its own cost. The series ended on November 2, so there is no before and after for this decision, and restarting the existing extractor would not backfill the gap on its own. If I restart the job, its first run would have repaired samples, so those grades would not be comparable with the old ones either.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.