Every Agent Session Is a Test Run

The short version

Every morning at 05:00, a scheduled job opens the transcripts of everything my AI coding agent did in the last 24 hours. It leaves me a list of proposed edits to the skills it used. Not a summary of what I did — a list, each item citing the exact line of the transcript and the exact line of the file that caused the problem.

I built it on top of xskills, my collection of agent skills. It is called x-skills-daily-reflection. It turned my skill set from “a folder of instructions I wrote once” into “a tool that gets measurably less wrong every week.”

The problem: skills rot silently

In the last post I argued for building your own agent skillset. That argument has a hidden bill, and I named it there: maintenance is on you, forever.

Here is what maintenance looks like. A skill is a markdown file of instructions. When it is wrong, nothing crashes: no stack trace, no failed test, no red build. What happens instead is worse: the agent reads a stale instruction, stumbles, improvises, and succeeds anyway. The work gets done. The only trace is a slightly longer session, a repeated tool call, a correction you typed mid-flow and forgot about two minutes later.

That trace lives in a place nobody looks: the session transcript. Every conversation your agent has is a test run of the skills it used, and every transcript is a test report that gets thrown away.

People solved this for code decades ago. CI runs your tests on every push. What is missing is the equivalent for instructions: a job that treats your own agent sessions as the test suite. Your skill files are the code under test.

The loop in one picture

flowchart TD
    A["yesterday's sessions"] --> B["scan for friction"]
    B --> C["verified proposals, never edits"]
    C --> D{"your five minutes"}
    D -- "accept" --> E["the skill improves"]
    E -. "tomorrow" .-> A
    D -- "defer" --> C

Three pieces make that loop run, and none of them are clever:

automation/daily-reflection/
├── collect-sessions.mjs   # gather last 24h of sessions, scan for friction
├── precheck.sh            # fail closed: skip the run when there is nothing honest to do
└── runbook.md             # the agent's instructions for the headless run

The collector walks every project directory my agent CLI (Crush) has touched recently. It exports the sessions modified in the last 24 hours and runs the x-autoreflection scanner over each one. The scanner extracts mechanical signals from each session: failed commands, repeated tool calls, moments I corrected the agent, skills loaded but never used, questions asked in prose instead of a structured prompt. Each signal keeps a severity, a suspect skill, and quoted evidence.

The precheck runs before the agent is invoked and fails closed. It skips the day when a binary is missing, when skills/ has uncommitted edits, or when no session in the window touched a skill. That git check is the whole philosophy in four lines: every proposal cites a skill by file:line, and uncommitted edits move those lines. A reflection that runs against a moving target produces plausible garbage. Plausible garbage is the one output you cannot afford from a system whose entire job is to be trustworthy.

The runbook is what the headless agent follows. It is plain markdown, but written for an agent that cannot ask questions:

• At most 4 sessions, at most 10 proposals. Caps, not targets. Without a human in the loop, an agent optimizing for “good retro” will find something to say about everything. A quiet day produces a three-line digest. The runbook says: do not invent findings to fill it.
• Verify every signal against the real file. A signal is a lead, not a finding. The agent opens the actual SKILL.md and decides. Keep: the file is wrong. Re-grade: real friction, but not that skill’s fault. Drop: the transcript misled the scanner. A grep exiting 1 because it found nothing is the answer, not a failure. Dropping is recorded, so tomorrow’s run does not re-litigate it.
• Blame the instruction, not the agent. “The SKILL.md did not say” is a gap. “I forgot” is not.
• Every proposal has Signal, Target, Change, and Check. The Check must be a command that runs: a test, a lint rule, a script that exits 0 once the fix is in. A proposal you cannot check is a wish.

And then the rule that defines the whole system, printed twice in the runbook:

Never edit anything under skills/. Propose, then stop. The review decides.

Why the loop does not close itself

The obvious design is a fully closed loop: cron fires, agent reflects, agent edits the skill, agent commits. It would take an afternoon to build, and it would be the wrong system.

Not because agents edit files badly — they are fine at it. Because a self-editing instruction set has no audit trail and no stopping condition. If today’s run misreads a signal and “fixes” a skill into uselessness, tomorrow’s run reflects on sessions shaped by that bad edit. The drift compounds. The precheck already knows this, which is why it refuses to run against a dirty tree. Skip the review, and you are not editing a file; you are editing the audit trail.

So the output is a DIGEST.md with proposals, and the output of my morning coffee is five minutes of checkboxes: accept, defer, drop. Accepted proposals route by size. A one-line edit goes in directly, a cluster goes through x-fix, and a design decision goes to x-plan with the reflection attached. The automation read 40 sessions. It handed me three verified, checkable changes instead of a vague feeling that something was off.

This is the same shape as a deploy pipeline. The pipeline builds, tests, and stages everything automatically — but something with judgement presses the button.

What it found, honestly

The loop is honest about its limits. Mechanical scanning only sees friction that leaves a trace. A wrong-but-successful instruction produces no failed command to flag, which is why the runbook lets the agent log manual findings it noticed by reading. And it costs one small agent run per day — on a quiet day, the precheck skips even that.

Most days are the three-line digest. But the pattern of what surfaces is what you cannot see from inside:

• The same gap twice is a defect. A missing flag in one session is a hypothesis. The same command failing in three sessions is a bug with a patch.
• Loaded but unused skills are a smell. If the agent keeps loading a skill and abandoning it, the description promises the wrong thing or the body answers the wrong question.
• Documentation drift is catchable. A documented command that no longer runs leaves one trace: the agent tries it, fails, and improvises. Hours later, that trace is in a transcript. Without the loop, you find it months later, when you run the command yourself.

Build the minimal version

You do not need Orca, a skill framework, or any of my code. You need a place where your agent’s conversations are stored, a scheduler, and the agent’s headless mode.

# crontab: 5 5 * * *  ~/reflection/run.sh

agent sessions export --since 24h --out /tmp/reflection/sessions/   # collect
agent run --headless --runbook ~/reflection/runbook.md \
  --context /tmp/reflection/sessions/                               # reflect

And a first runbook can be five sentences:

1. Read every transcript in the folder.
2. List every moment the user corrected you, retried a command, or worked around a missing capability.
3. For each, check whether an instruction file could have prevented it.
4. For each that could, write one proposal: which file, what change, what command proves it.
5. Output the list to DIGEST.md. Edit nothing.

The first week it will mostly produce noise, and the noise is informative — it shows you which signals to filter. Mine learned to stop flagging grep’s exit codes after I made “drop the signal, record why” part of the procedure.

The part people get wrong

Two failure modes, both of which I built first.

The loop that must find something. You schedule a daily reflection. For three days it says “nothing to report.” The temptation is to loosen the criteria until something appears. Do not. A reflection that never comes back empty is not reflecting. It is generating content.

Closing the loop to save five minutes. The human review feels like the weakest link, so it is the part everyone automates away first. It is the load-bearing part. Five minutes of checkboxes a day is the price of an instruction set that improves in one direction.

The short version, again

Your agent’s conversations are already a test suite for your workflows — the only one that runs on real work instead of synthetic benchmarks. A cron job, a collector script, and a runbook are enough to stop throwing that test suite away.

The system does not improve itself. It improves your half-hour a week into a targeted, evidenced, checkable five minutes. Every accepted proposal makes the next week a little smoother, and the next digest a little shorter. Most mornings it is three lines. That is the system telling you it worked.