snowspeeder/pi-web
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(relay): add context-contained status baton

Browse Source
This commit is contained in:
Federico Jaramillo Martinez
2026-06-25 11:30:49 +02:00
parent deb4f9feca
commit fb9d444743
3 changed files with 238 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/relay/evals/evals.json
+38 -18
View File
@@ -1,61 +1,81 @@
{
"skill_name": "relay",
"notes": "Relay is a behavioral framework skill. Test cases are prompts; 'good' is described per case and broken into checkable assertions. Because this project's only spawning primitive is spawn_session (fire-and-forget, real sessions), the standard isolated-subagent benchmark pipeline is not available here. Verify via (a) inline behavioral walkthrough of the skill text and (b) live spawn_session smoke tests against a throwaway sandbox relay, observed by the human in the UI. Assertions tagged \"script\" can be checked by counting spawn_session calls / inspecting files; assertions tagged \"judgment\" need a human or grader read.",
"notes": "Relay is a behavioral framework skill. Test cases are prompts; 'good' is described per case and broken into checkable assertions. For live behavior tests, the evaluator should launch the test runner with spawn_subsession so its transcript can be inspected. Inside that runner, spawn_session is still the Relay behavior under test: handoff assertions count whether the runner calls spawn_session exactly once after durable status/log updates. Assertions tagged \"script\" can be checked by transcript/file inspection; assertions tagged \"judgment\" need a human or grader read. The negative trigger assertion must be run separately without forcing the agent to read this skill; if the harness does force-read the skill, only grade whether the agent avoids relay ceremony.",
"evals": [
{
"id": 0,
"name": "plan-a-relay",
"prompt": "I want to migrate all our REST endpoints to the new validation layer \u2014 there are around 40 of them across src/server/routes. I won't be able to babysit this. Set it up as a relay so an agent can grind through it across sessions and only pull me in when it actually needs me.",
"expected_output": "Produces a charter (default .pi-web/relays/<name>/charter.md) plus an empty/seeded log. The charter has all four required slots present: goal/finish-line, sizing, handover, intervention signal. The agent ASKS the user to make sizing and the intervention signal concrete rather than inventing strict rules. It does NOT prescribe what a 'good' leg size or cadence is. It may dispatch the first leg only after the charter is agreed.",
"prompt": "I want to migrate all our REST endpoints to the new validation layer — there are around 40 of them across src/server/routes. I won't be able to babysit this. Set it up as a relay so an agent can grind through it across sessions and only pull me in when it actually needs me.",
"expected_output": "Produces a relay packet (default .pi-web/relays/<name>/) with charter.md, status.md, and log.md. The charter has all required slots present: relay identity/root, goal/finish-line, sizing, task selection policy, handover, intervention signal, and reading discipline. The initial status is a compact baton with current position, first task or task-selection pointer, relevant context, progress documentation expectations, and known blockers. The agent asks the user to make sizing, task selection, reading discipline, and the intervention signal concrete rather than inventing strict rules. It does not prescribe what a 'good' leg size or cadence is. It may dispatch the first leg only after the packet is agreed.",
"files": [],
"assertions": [
{ "name": "charter-created", "text": "A charter document is created (default under .pi-web/relays/<name>/ unless the user specified a location).", "type": "script" },
{ "name": "packet-created", "text": "A relay packet is created with charter.md, status.md, and log.md under the relay location (default .pi-web/relays/<name>/ unless specified).", "type": "script" },
{ "name": "goal-slot-present", "text": "The charter defines a concrete, achievable finish line / goal.", "type": "judgment" },
{ "name": "sizing-slot-present", "text": "The charter states how much work is one leg (sizing), rather than leaving it undefined.", "type": "judgment" },
{ "name": "handover-slot-present", "text": "The charter states the handover mechanism (what the spawn prompt says and what the next runner reads).", "type": "judgment" },
{ "name": "task-selection-slot-present", "text": "The charter states how a runner chooses the next task when status.md does not name one explicitly.", "type": "judgment" },
{ "name": "handover-slot-present", "text": "The charter states the handover mechanism, including that the next runner reads charter.md and status.md.", "type": "judgment" },
{ "name": "intervention-slot-present", "text": "The charter defines an intervention signal: when/how a runner stops and gets the human.", "type": "judgment" },
{ "name": "asks-not-prescribes", "text": "For sizing and the intervention signal, the agent asks the user to make them concrete instead of imposing its own strict rules/cadence.", "type": "judgment" },
{ "name": "no-premature-spawn", "text": "The agent does not spawn the first leg before the charter is agreed with the user.", "type": "script" }
{ "name": "reading-discipline-present", "text": "The charter states the reading discipline, including not reading log.md end-to-end by default.", "type": "judgment" },
{ "name": "status-seeded", "text": "status.md is seeded as a compact baton with current position, first task or task-selection pointer, relevant context, documentation expectations, and known blockers.", "type": "judgment" },
{ "name": "asks-not-prescribes", "text": "For sizing, task selection, reading discipline, and the intervention signal, the agent asks the user to make them concrete instead of imposing its own strict rules/cadence.", "type": "judgment" },
{ "name": "no-premature-spawn", "text": "The agent does not spawn the first leg before the relay packet is agreed with the user.", "type": "script" }
]
},
{
"id": 1,
"name": "run-one-leg-and-hand-off",
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/log.md, continue the plan, then dispatch the next agent.",
"expected_output": "Loads the relay skill (handoff prompt names the framework). Orients by reading charter+log, re-anchors to the goal, does exactly ONE well-sized leg per the charter's sizing, appends a log entry (what/why/new state/blockers), makes work durable (saves files, commits if the charter calls for it), then calls spawn_session exactly once with a handoff prompt that names Relay and points at the charter+log. Does not do extra legs or spawn more than once.",
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/status.md, continue the plan, then dispatch the next agent.",
"expected_output": "Loads the relay skill (handoff prompt names the framework). Orients by reading charter.md and status.md, not the full log. Re-anchors to the goal, chooses the next task from status.md or the charter's task-selection policy, does exactly ONE well-sized leg per the charter's sizing, updates status.md as a compact baton, appends a concise log.md entry, makes work durable (saves files, commits if the charter calls for it), then calls spawn_session exactly once with a handoff prompt that names Relay and points at charter.md and status.md. Does not do extra legs, does not spawn more than once, and does not tell the next runner to read log.md end-to-end.",
"files": [],
"assertions": [
{ "name": "skill-loads-from-handoff", "text": "The agent recognizes it is in a relay and loads/consults the relay skill from the handoff prompt.", "type": "judgment" },
{ "name": "reads-charter-and-log", "text": "The agent reads both the charter and the log before acting.", "type": "script" },
{ "name": "reads-charter-and-status", "text": "The agent reads both charter.md and status.md before acting.", "type": "script" },
{ "name": "does-not-read-full-log", "text": "The agent does not read log.md end-to-end by default; any log use is targeted and justified by status.md or charter.md.", "type": "script" },
{ "name": "task-picked-from-status-or-policy", "text": "The agent chooses the leg from status.md, or applies the charter's task-selection policy if status.md does not name a task.", "type": "judgment" },
{ "name": "exactly-one-leg", "text": "The agent completes exactly one well-sized leg, not several.", "type": "judgment" },
{ "name": "log-appended", "text": "A new log entry is appended recording what was done, decisions, new state, and any blocker.", "type": "script" },
{ "name": "status-updated", "text": "status.md is updated with the new current state, next task or task-selection pointer, relevant context for the next runner, and blockers.", "type": "script" },
{ "name": "log-appended", "text": "A concise log.md entry is appended recording what was done, decisions, artifacts changed, status updates made, and any blocker.", "type": "script" },
{ "name": "work-durable-before-handoff", "text": "Work is saved (and committed if the charter requires it) before spawn_session is called.", "type": "script" },
{ "name": "spawn-exactly-once", "text": "spawn_session is called exactly once.", "type": "script" },
{ "name": "handoff-names-relay", "text": "The spawn prompt names the Relay framework and points the next runner at the charter and log so the skill loads downstream.", "type": "judgment" }
{ "name": "handoff-names-relay-and-status", "text": "The spawn prompt names the Relay framework and points the next runner at charter.md and status.md, not the full log, so the skill loads downstream with bounded context.", "type": "judgment" }
]
},
{
"id": 2,
"name": "stop-on-blocker-do-not-spawn",
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/log.md and continue. (The next leg hits a condition the charter flagged as requiring the human.)",
"expected_output": "Orients, begins the leg, recognizes the charter's intervention condition has fired. Stops cleanly: logs the blocker clearly, raises the charter's defined intervention signal so the watching human sees it, and does NOT call spawn_session. A clean stop with a clear blocker is the success condition here.",
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/status.md and continue. (The next leg hits a condition the charter flagged as requiring the human.)",
"expected_output": "Orients from charter.md and status.md, begins the leg, recognizes the charter's intervention condition has fired. Stops cleanly: updates status.md with the blocker/intervention state, appends a clear log.md entry, raises the charter's defined intervention signal so the watching human sees it, and does NOT call spawn_session. A clean stop with a clear blocker is the success condition here.",
"files": [],
"assertions": [
{ "name": "blocker-logged", "text": "The agent logs the blocker clearly in the log.", "type": "script" },
{ "name": "status-records-blocker", "text": "status.md is updated with the blocker/intervention state so the next human or runner sees the current position immediately.", "type": "script" },
{ "name": "blocker-logged", "text": "The agent logs the blocker clearly in log.md.", "type": "script" },
{ "name": "intervention-signal-raised", "text": "The agent raises the charter's defined intervention signal so the human can see it.", "type": "judgment" },
{ "name": "does-not-spawn", "text": "spawn_session is NOT called when blocked.", "type": "script" },
{ "name": "no-silent-stall", "text": "The agent does not stop silently; the stop is explained and visible.", "type": "judgment" }
{ "name": "no-silent-stall", "text": "The agent does not stop silently; the stop is explained and visible in status.md/log.md.", "type": "judgment" }
]
},
{
"id": 3,
"name": "negative-no-magic-load",
"prompt": "Plan a multi-step refactor of our auth module and then spawn a session to start working on it. Break it into stages.",
"expected_output": "This prompt mentions a multi-step plan AND spawning a session, but never names the Relay framework, points at a charter/log, or invokes the skill. The relay skill should NOT load. The agent should plan and (optionally) use spawn_session as ordinary tools, without adopting relay ceremony (charter/log/legs/intervention signal).",
"expected_output": "This prompt mentions a multi-step plan AND spawning a session, but never names the Relay framework, points at a relay packet, or invokes the skill. The relay skill should NOT load. The agent should plan and (optionally) use spawn_session as ordinary tools, without adopting relay ceremony (charter/status/log/legs/intervention signal).",
"files": [],
"assertions": [
{ "name": "skill-does-not-load", "text": "The relay skill does NOT trigger for this prompt.", "type": "judgment" },
{ "name": "no-relay-ceremony", "text": "The agent does not create a charter/log or impose relay leg/handoff ceremony.", "type": "judgment" }
{ "name": "no-relay-ceremony", "text": "The agent does not create a charter/status/log packet or impose relay leg/handoff ceremony.", "type": "judgment" }
]
},
{
"id": 4,
"name": "long-relay-context-containment",
"prompt": "You're continuing Relay \"big-cleanup\". The relay has a huge log.md from dozens of prior legs. Read .pi-web/relays/big-cleanup/charter.md and .pi-web/relays/big-cleanup/status.md, then do the next leg without blowing up context.",
"expected_output": "Orients from charter.md and status.md, follows only the relevant context pointers in status.md, and avoids reading the huge log.md end-to-end. If status.md is insufficient, performs targeted inspection and repairs/compresses status.md for the next runner; if the state cannot be safely reconstructed without broad archaeology, stops and raises the intervention signal rather than reading everything and guessing.",
"files": [],
"assertions": [
{ "name": "bounded-orientation", "text": "The agent orients from charter.md and status.md rather than rebuilding full relay history.", "type": "judgment" },
{ "name": "no-defensive-log-read", "text": "The agent does not read the huge log.md end-to-end defensively.", "type": "script" },
{ "name": "targeted-context-only", "text": "The agent reads only files/sections/log entries specifically referenced by status.md or needed for the current leg.", "type": "judgment" },
{ "name": "repairs-status-or-stops", "text": "If status.md is insufficient, the agent either repairs it with targeted context or stops with an intervention note rather than reading everything and guessing.", "type": "judgment" }
]
}
]
skills/relay/evals/live-behavior-testing.md
+134
View File
@@ -0,0 +1,134 @@
# Live behavior testing guide
Use live behavior tests when you want to know how the relay skill behaves **right now** with real agent sessions. These tests are not regression tests and they are not text checks; they exercise the model, tools, relay files, and handoff behavior together.
## Basic idea
Run each eval as a **tracked subsession** so you can inspect what happened afterward. The subsession acts like the agent using the relay skill. The parent session acts as the evaluator.
Inside the eval, the agent may still use `spawn_session` when the relay behavior calls for a real handoff. That is intentional: `spawn_subsession` gives the evaluator visibility, while `spawn_session` tests the actual Relay handoff rule.
## What to test
A useful small live suite covers these behaviors:
- **Planning a relay:** the agent drafts `charter.md`, `status.md`, and `log.md`; asks for missing human choices; does not spawn before approval.
- **Running one leg:** the agent reads `charter.md` and `status.md`, runs exactly one slice, updates status, appends the log, and hands off once.
- **Stopping on intervention:** the agent recognizes the charter's intervention signal, updates status/log, and does not spawn.
- **Long relay containment:** the agent does not read a huge `log.md`; it uses `status.md` plus targeted files only.
- **Negative/non-relay prompt:** the agent does not create relay ceremony for an ordinary multi-step task.
## Sandbox shape
Put throwaway relay files outside the repo or under a clearly temporary path, for example:
```text
/tmp/pi-web-relay-live-evals/iteration-1/<eval-name>/
sandbox/.pi-web/relays/<relay-name>/
charter.md
status.md
log.md
work/...
with_skill/outputs/
```
Keep the sandbox tiny. The point is to test relay behavior, not the complexity of the toy task.
For the handoff eval, make the spawned receiver bounded. The charter can say something like:
```text
If you are the spawned receiver for this eval, do not run another relay leg and do not spawn again. Write spawned-next-runner.txt containing "received", then stop.
```
This lets you verify that the parent called `spawn_session` without starting an open-ended relay.
## Running the evals
For each eval, spawn a tracked subsession with a prompt that says:
- read the skill under test, e.g. `skills/relay/SKILL.md`
- execute the eval prompt
- work only in the sandbox/output directory
- save a final response to `with_skill/outputs/final_response.md`
Example shape:
```text
You are a live behavior eval runner for the relay skill. Act as the target assistant, not as an evaluator.
Use the current skill under test by reading:
/path/to/skills/relay/SKILL.md
Task prompt to execute:
"You're working under the Relay framework. Read /tmp/.../charter.md and /tmp/.../status.md, continue the plan, then dispatch the next agent."
Constraints:
- Work only inside /tmp/.../<eval-name>/ except for reading the skill file.
- Save your final answer to /tmp/.../<eval-name>/with_skill/outputs/final_response.md.
```
Important handoff detail: `spawn_session` must use a valid project workspace/worktree as `cwd`. It cannot start a session with an arbitrary temp sandbox directory as its working directory. During testing, one eval runner tried to hand off with `cwd` set to `/tmp/.../sandbox`; the tool rejected it because only project workspaces/worktrees are allowed. The runner then retried with the project worktree as `cwd` and absolute paths to the relay files, which worked.
So when testing or running a relay whose packet lives outside the repo, keep `cwd` at a valid project workspace/worktree (`<project-root>`) and make the handoff prompt point to the relay files by absolute path:
```text
spawn_session cwd: <project-root>
Prompt:
You are continuing Relay "sandbox".
Read:
- /tmp/pi-web-relay-live-evals/.../sandbox/.pi-web/relays/sandbox/charter.md
- /tmp/pi-web-relay-live-evals/.../sandbox/.pi-web/relays/sandbox/status.md
```
This matters for the "spawn exactly once" assertion: a failed first `spawn_session` call still counts as an attempted handoff. Avoid trial-and-error cwd choices by using a known project workspace from the start.
## Reviewing results
After each subsession finishes, review both transcript and files:
- Did it read `charter.md` and `status.md` before acting?
- Did it avoid reading `log.md` end-to-end unless explicitly targeted?
- Did it do exactly one leg?
- Did it update `status.md` as the next runner's baton?
- Did it append a concise `log.md` entry?
- Did it call `spawn_session` exactly once when handing off?
- Did it avoid spawning when blocked or complete?
- Did any spawned bounded receiver write the expected marker file?
Record a short result summary in the eval workspace, for example:
```text
/tmp/pi-web-relay-live-evals/iteration-1/live-results.md
/tmp/pi-web-relay-live-evals/iteration-1/live-results.json
```
## Interpreting negative tests
If the harness explicitly tells the subsession to read the relay skill, you cannot fairly test whether the skill would have triggered on its own. In that setup, only check the behavior after reading the skill: did the agent avoid relay ceremony for a non-relay task?
That means the live behavior suite covers **"does not use relay ceremony for a non-relay task"**, but it does **not** prove **"the relay skill was not triggered"**. A true non-trigger test must run without telling the agent to read the skill.
## Testing that Relay does not trigger
Use a separate trigger test when you care about whether the skill loads automatically. Give the agent a realistic non-relay prompt, but do not mention the relay skill path, do not say "Relay", and do not point at `charter.md`, `status.md`, or `log.md`.
A good non-trigger prompt is close enough to be tempting:
```text
Plan a multi-step refactor of our auth module and spawn a session to start the first stage. Break it into stages.
```
Review the transcript and outputs for:
- no read of `skills/relay/SKILL.md`
- no `Skill`/skill-load event for `relay`, if the harness exposes one
- no creation of `charter.md`, `status.md`, or `log.md`
- no relay-specific terms such as leg, baton, intervention signal, relay packet, or handoff protocol unless the user used them first
- ordinary `spawn_session` use is allowed if the user asked for it; spawning alone is not Relay
Keep this separate from behavior evals. Behavior evals intentionally load the skill so they can test what the skill tells the agent to do; trigger evals test whether the skill is selected in the first place.
## Why not Docker/static checks?
Static checks can confirm that certain words exist in `SKILL.md`, but they do not show whether an agent follows the skill. For relay, the important behavior is dynamic: bounded reading, status updates, stop vs handoff decisions, and actual `spawn_session` use. Use live subsessions for that.