Reviewing The Agent's Own Session
A structured retrospective on the agent's behaviour, run against its own rules, producing a score and a ledger entry.
A session review checks the agent's behaviour against its own written rules rather than against a feeling about output quality. It loads the rule set dynamically, verifies each rule against the transcript, and every failure becomes a counter entry — which turns a retrospective into an escalation mechanism instead of a diary.
Prerequisites
A written rule set the agent is supposed to follow. A transcript of the session. A repeat-mistake ledger for the output to land in.
Without the third, this produces observations that go nowhere.
Steps
1. Load the rules dynamically, do not hardcode them. The review reads the current rule files at run time. A review with a baked-in checklist drifts from the rules it is supposed to enforce, and the drift is invisible — it keeps passing.
2. Check each rule against the transcript, not against memory. For each rule: did the situation it governs occur, and was it followed? Three outcomes — n/a, followed, violated — and n/a must be available, or every review inflates into a list of technically-unmet rules that never applied.
3. Check the categories separately. They fail differently:
protocol — was the required output emitted, in order · routing — was the right skill or agent used · memory — was what should have been written, written · verification — were claims checked at the level they live · autonomy — was anything done that should have been asked
4. Score it, and keep the scale coarse. A number makes trends visible across sessions. Keep it blunt — false precision on a subjective scale invites arguing about points instead of about behaviour.
5. Route every violation into the ledger. This is the step that converts a retrospective into a mechanism. A violation becomes a counter entry; the counter escalates on its own thresholds. Without it, the same finding is rediscovered monthly and nothing changes.
6. Name the 1 to 3 decisions you were least sure about. Not violations — judgement calls. These are the highest-value output of the whole pass, because they surface the places where the rules were silent rather than broken.
7. Compare against the previous review, not against zero. A single score is nearly meaningless — the useful signal is direction. Two consecutive reviews with the same finding mean the fix did not take, which is a stronger result than either review alone.
8. Keep one section the agent cannot pass. A field for what I would do differently with no rubric and no score attached. Everything scored gets optimised toward the score; an unscored field is the only place where an honest answer has no incentive working against it. In practice it produces the most useful line in the whole review.
Verify
The review works when its findings are actionable without further investigation. Response was too long is not actionable. Emitted the suggestions block after an explicit request for brevity — rule X is.
Check the false-positive rate. A review that flags rules which did not apply will be dismissed, and a dismissed review is worse than none because it consumes the slot a real one would occupy.
Troubleshooting
Every session scores well and nothing improves. The review is checking what is easy to check. Add the categories that are uncomfortable — autonomy and verification usually.
Findings repeat forever. Step 5 is missing. Observations without counters do not escalate.
The review takes longer than the session. Scope it to rules that had a chance to fire. A full pass over every rule every time is thorough and will be abandoned within a month.