handover

Handover bench

Measures whether a handoff artifact actually transfers context, not whether it is well-formatted (lint already does that).

The two layers

1. Deterministic (runs now, node bench/run.js). For each scenario, every planted trap has a guard_pattern; the layer reports trap coverage, does the artifact carry the negative knowledge that would steer a cold agent away from that trap? Plus the lint score. Same input, same numbers.

Honest limit: this layer checks that the warning is present in the text, not that an agent obeys it. It is a necessary-condition check, an artifact that never mentions the reverted feature cannot possibly protect a cold agent from re-adding it.

2. Empirical (agent-in-the-loop, pluggable). The sufficient-condition check. The contract for a runner:

  1. Materialize the scenario’s repo state in a throwaway workspace.
  2. Give a cold agent ONLY the artifact under test (or nothing, for the control) plus the workspace, with the scenario’s task prompt.
  3. Let it work to completion.
  4. Run each trap’s detect_fell_in command against the resulting workspace; a non-empty result means the agent fell into that trap.
  5. Report traps fallen into per artifact: good doc vs bad doc vs no doc.

No runner is bundled, deliberately: the enforcement paths of this repo stay LLM-free, and a runner is by definition agent-specific. Any harness that can spawn a fresh agent in a directory can implement the five steps above. bench/scenarios/reverted-feature/repo/ is a real, working example of step 1, a materialized fixture, not just an abstract premise, that a reverted-feature trial has actually been run against (see docs/PAPER_RELEASE_TRAIN.md for the results and, more importantly, two real methodology bugs the process caught).

Hard lessons from actually running the empirical layer once

Both found by adversarial review of a real trial, not by inspection beforehand, worth stating as rules, not just fixing quietly:

Authoring a scenario

A scenario is bench/scenarios/<name>/ with:

Author the traps first, from a failure you have actually seen, then write the good artifact against them. Never write the artifact first and derive guard_pattern from its phrasing; that reverses the direction of evidence and turns the bench into a self-fulfilling regex. Write guard_pattern with alternations for the different ways a competent author would state the warning, and keep bad.handover.md honest: it should be a plausible summary, not a strawman of random text.