snowspeeder/pi-web
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(relay): add context-contained status baton

Browse Source
This commit is contained in:
Federico Jaramillo Martinez
2026-06-25 11:30:49 +02:00
parent deb4f9feca
commit fb9d444743
3 changed files with 238 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/relay/SKILL.md
+66 -18
View File
@@ -1,6 +1,6 @@
---
name: relay
description: "How the Relay method works: executing a plan as a chain of independent sessions that each do one slice and hand off to the next via spawn_session. Load this skill only when you already know you are in a relay: a prompt states you are working under the Relay framework (or relay/chain), points you at a relay charter or log, or the user invokes this skill directly. Do not load it for generic multi-step plans or ordinary spawn_session use."
description: "How the Relay method works: executing a plan as a chain of independent sessions that each do one slice and hand off to the next via spawn_session. Load this skill only when you already know you are in a relay: a prompt states you are working under the Relay framework (or relay/chain), points you at a relay charter/status/log, or the user invokes this skill directly. Do not load it for generic multi-step plans or ordinary spawn_session use."
---
# Relay
@@ -9,7 +9,7 @@ Relay is a way to execute a long or complex plan as a chain of independent sessi
There is no coordinator and no referee. Each runner is the coordinator for their own leg: smart enough to do the work, adapt to what they discover, and hand off cleanly. Trust is distributed to every agent, not held by a god-agent above them.
The reason this works is **containment**: every leg starts with a fresh, small context. The accumulated knowledge lives in documents on disk, not in any one session's memory. That is also the core constraint you must respect — see below.
Relay works because it does not try to recreate human management structures. The point is fewer boundaries, less hierarchy, and more fluid execution. The thing that makes that safe is **context containment**: every leg starts with a fresh, small context, and the accumulated knowledge lives in compact documents on disk rather than in any one session's memory.
## The hard constraint that shapes everything
@@ -17,48 +17,96 @@ The reason this works is **containment**: every leg starts with a fresh, small c
Two consequences follow, and they govern the whole method:
- **Make your work durable before you hand off.** Write the log, save/commit the artifacts (commit if the relay says to), and only then spawn the next leg. Anything not on disk is lost.
- **Make your work durable before you hand off.** Update the status, append the log, save/commit the artifacts (commit if the relay says to), and only then spawn the next leg. Anything not on disk is lost.
- **Hand off exactly once, at the end.** Do not spawn early, do not spawn several runners "to parallelize," and never spawn while you still have work in flight. One leg, one handoff.
## The two documents
## The relay packet
A relay is carried by two documents. By default they live in `.pi-web/relays/<name>/` unless the user or the dispatching prompt says otherwise — always follow an explicit location if given.
A relay is carried by a small packet of documents. By default they live in `.pi-web/relays/<name>/` unless the user or the dispatching prompt says otherwise — always follow an explicit location if given.
Every relay has these three core files:
**Charter** (`charter.md`) — the stable agreement, written when the relay is planned. It must contain, at minimum:
- **Relay identity.** The relay name and root path, so runners know exactly which relay they are on.
- **Goal / finish line.** A concrete, achievable end state. Without this the relay runs forever — this is non-negotiable.
- **Sizing.** How much is *one leg*? This is project- and plan-specific; the charter defines it (a task, a slice, a time/scope budget — whatever fits). The skill does not decide this for you.
- **Handover.** How a runner hands off: what the spawn prompt should say and what the next runner must read. Can be as simple as "read the charter and log, then continue," as long as it is stated.
- **Task selection policy.** How a runner chooses the next task when `status.md` does not name one explicitly.
- **Handover.** How a runner hands off: what the spawn prompt should say and what the next runner must read. A normal handoff points at `charter.md` and `status.md`, not the full log.
- **Intervention signal.** When and how a runner must stop and get the human, and how that is made visible. The charter must define this; the skill does not define it for you.
- **Reading discipline.** The files a runner should read to orient, and any files that should not be read defensively.
The charter *can* be edited, but it should rarely *need* to be. If it is changing every leg, that is a smell — the design wasn't settled, or the goal is drifting. Treat frequent charter edits as a reason to stop and involve the human.
**Log** (`log.md`) — append-only, grows as the relay runs. Each leg appends an entry so the next runner can orient without inheriting your context. An entry records: what this leg did, decisions made and why, the current state, and any blockers. This is the relay's memory.
**Status** (`status.md`) — the compact baton/current state. This is the file every runner reads after the charter, and every runner updates before handoff or stop. Keep it short enough that a fresh runner can load it cheaply. It should answer:
For a small relay it is fine to collapse both into a single file, as long as the goal, sizing, handover, and intervention signal are all present.
- **Current position.** Where the relay is now.
- **Current or next task.** The next leg if known; otherwise enough information to apply the charter's task selection policy.
- **Relevant context.** Only the files, sections, commands, artifacts, or specific log entries needed for the next leg.
- **Progress documentation.** Where this runner must write progress: update `status.md`, append `log.md`, update artifacts, commit, etc.
- **Blockers / intervention state.** Current risks, open decisions, or active reasons to stop.
Think of `status.md` as the thing passed from runner to runner. If it grows into a history dump, compress it back into current state plus pointers.
**Log** (`log.md`) — append-only history. Each leg appends a concise entry recording what it did, decisions made and why, durable artifacts changed, status updates made, and blockers. The log preserves auditability, but it is **not** orientation memory.
Do not read `log.md` end-to-end by default. Read targeted log entries only when `status.md` points to them, when the charter requires a specific lookup, or when there is an inconsistency you must resolve before continuing.
Optional files such as `plan.md`, `backlog.md`, or artifact notes are fine, but runners should read them only when the charter/status points to the relevant part.
## Context containment rule
A runner normally reads:
1. `charter.md`
2. `status.md`
3. Only the specific files or log entries referenced for the current leg
Do not defensively rebuild the relay's full history. Do not read the full log, the full backlog, or a large artifact tree just because they exist. The relay stays scalable because each runner pays only for the context needed now.
If `status.md` is insufficient, fix the baton rather than compensating by reading everything. Use targeted inspection to clarify the current state, update `status.md` so the next runner has a clean start, and continue only if the task is still clear. If reconstructing the state would require broad archaeology or judgment about past intent, stop and raise the intervention signal.
## Running one leg
This is the loop you run when you are dispatched into a relay.
1. **Orient.** Read the charter and the log. Understand the goal and the current state. If you are not sure you are in a relay, the prompt or `.pi-web/relays/` is your clue — and reading this skill means you are.
2. **Re-anchor to the goal.** Does the goal still make sense given what the log shows and what you now see? If reality has diverged from the charter, that is often an intervention moment — don't quietly redefine the task.
3. **Run one leg.** Do exactly one well-sized slice, per the charter's sizing. Resist doing "just a bit more" — extra scope bloats context and breaks the containment that makes Relay work.
4. **Log it.** Append your entry: what you did, why, the new state, and any blocker. Make all work durable (save files, commit if the relay calls for it).
5. **Decide: hand off, or stop.**
- **Hand off** if there is a clear next leg and you are on track. Use `spawn_session` once, with a prompt that names the Relay method and points the next runner at the charter and log (so this skill loads and they can orient). Then you are done. Handoff is deliberately fire-and-forget: `spawn_session` starts an independent session you will not see and cannot steer — do not reach for a tracked subsession to keep an eye on it. Letting go is the point. The next runner is trusted to run their own leg, and the log is the only thread between you; if you feel the need to watch downstream work, that usually means the leg wasn't sized or handed off cleanly, or an intervention signal should have fired.
- **Stop — do not spawn —** if the goal is reached, or you are blocked, or the charter's intervention signal fires. Leave a clear note in the log (and raise the intervention signal) so the watching human sees exactly what happened and what they need to decide. A stalled relay that stopped cleanly with a clear blocker is a success; a relay that spawned a confused next runner is a failure.
1. **Orient from the packet.** Read `charter.md` and `status.md`. Confirm the relay name/root, goal, sizing, handoff protocol, intervention signal, and current/next task. If you are not sure you are in a relay, the prompt or `.pi-web/relays/` is your clue — and reading this skill means you are.
2. **Choose the leg.** Prefer the explicit current/next task in `status.md`. If none is named, apply the charter's task selection policy. If that still requires context, inspect only the referenced plan/backlog/artifact sections. If the next task is still ambiguous or would materially change direction, stop and involve the human.
3. **Re-anchor to the goal.** Does the goal still make sense given the status and what you now see? If reality has diverged from the charter, that is often an intervention moment — don't quietly redefine the task.
4. **Run one leg.** Do exactly one well-sized slice, per the charter's sizing. Resist doing "just a bit more" — extra scope bloats context and breaks the containment that makes Relay work.
5. **Document progress.** Make all work durable. Update `status.md` with the new current state, next task or task-selection pointer, relevant context for the next runner, and blockers. Append a concise `log.md` entry with what you did, why, decisions made, artifacts changed, and whether you are handing off or stopping.
6. **Decide: hand off, or stop.**
- **Hand off** if there is a clear next leg and you are on track. Use `spawn_session` once, with a prompt that names the Relay method and points the next runner at `charter.md` and `status.md` (so this skill loads and they can orient cheaply). Then you are done. Handoff is deliberately fire-and-forget: `spawn_session` starts an independent session you will not see and cannot steer — do not reach for a tracked subsession to keep an eye on it. Letting go is the point. The next runner is trusted to run their own leg, and the relay packet is the only thread between you; if you feel the need to watch downstream work, that usually means the leg wasn't sized or handed off cleanly, or an intervention signal should have fired.
- **Stop — do not spawn —** if the goal is reached, or you are blocked, or the charter's intervention signal fires. Update `status.md`, append a clear note in `log.md`, and raise the intervention signal so the watching human sees exactly what happened and what they need to decide. A stalled relay that stopped cleanly with a clear blocker is a success; a relay that spawned a confused next runner is a failure.
A good handoff prompt is short and explicit:
```text
You are continuing Relay "<name>".
Read:
- .pi-web/relays/<name>/charter.md
- .pi-web/relays/<name>/status.md
Do not read log.md end-to-end. Use it only for targeted lookup if status.md or charter.md points you there.
Run one leg according to the charter. Before handing off, update status.md, append log.md, make work durable, then either spawn the next leg once or stop with a clear intervention note.
```
## Planning a relay
When the user asks to set up a relay, your job is to produce a charter (and an empty or seeded log) that has the four required slots filled: goal, sizing, handover, intervention signal. Draw each one out from the user rather than inventing it: ask what the finish line is, how much should be one leg, how runners hand off, and when you must stop and get them. Sizing and the intervention signal especially are the user's to decide — propose options if it helps them think, but do not quietly settle them yourself.
When the user asks to set up a relay, your job is to produce the relay packet: `charter.md`, `status.md`, and `log.md`. The charter must have the required slots filled: relay identity, goal, sizing, task selection policy, handover, intervention signal, and reading discipline. The initial status must give the first runner a compact baton: current position, first task or task selection pointer, relevant context, documentation expectations, and known blockers. The log may start empty or with a short seed entry explaining that the relay was created.
Do **not** impose what a "good" plan, leg size, or cadence looks like — those are deeply project-, plan-, and human-specific, and getting them wrong by being prescriptive is worse than leaving them to the user. Your value in planning is making sure the relay is *runnable*: the finish line exists, sizing is stated, handover is stated, and the intervention signal is stated. Once the charter is agreed, you can dispatch the first leg with `spawn_session`.
Draw the required choices out from the user rather than inventing them: ask what the finish line is, how much should be one leg, how runners pick tasks, how runners hand off, what they should read, and when they must stop and get the human. Sizing, task selection, and the intervention signal especially are the user's to decide — propose options if it helps them think, but do not quietly settle them yourself.
Do **not** impose what a "good" plan, leg size, or cadence looks like — those are deeply project-, plan-, and human-specific, and getting them wrong by being prescriptive is worse than leaving them to the user. Your value in planning is making sure the relay is *runnable*: the finish line exists, sizing is stated, task selection is stated, handover is stated, reading discipline is stated, and the intervention signal is stated. Once the packet is agreed, you can dispatch the first leg with `spawn_session`.
## Smells to watch for
- **No finish line** → infinite relay. Refuse to run a relay without a defined goal.
- **Goal drift** → each leg quietly restates the task. Re-anchor every leg.
- **Charter churn** → the charter changes every leg. The design isn't settled; involve the human.
- **Status bloat** → `status.md` turns into a history dump. Compress it to current state plus targeted pointers.
- **Defensive reading** → reading the full log/backlog/artifact tree to feel safe. Use the packet and targeted lookups; stop if the baton is not enough.
- **Eager spawning** → spawning early, spawning several runners, or spawning before work is durable. One leg, one handoff, at the end.
- **Silent stall** → getting stuck and stopping with no note, or spawning anyway. Always log the blocker and surface it.
- **Silent stall** → getting stuck and stopping with no note, or spawning anyway. Always update status, log the blocker, and surface it.
skills/relay/evals/evals.json
+38 -18
View File
@@ -1,61 +1,81 @@
{
"skill_name": "relay",
"notes": "Relay is a behavioral framework skill. Test cases are prompts; 'good' is described per case and broken into checkable assertions. Because this project's only spawning primitive is spawn_session (fire-and-forget, real sessions), the standard isolated-subagent benchmark pipeline is not available here. Verify via (a) inline behavioral walkthrough of the skill text and (b) live spawn_session smoke tests against a throwaway sandbox relay, observed by the human in the UI. Assertions tagged \"script\" can be checked by counting spawn_session calls / inspecting files; assertions tagged \"judgment\" need a human or grader read.",
"notes": "Relay is a behavioral framework skill. Test cases are prompts; 'good' is described per case and broken into checkable assertions. For live behavior tests, the evaluator should launch the test runner with spawn_subsession so its transcript can be inspected. Inside that runner, spawn_session is still the Relay behavior under test: handoff assertions count whether the runner calls spawn_session exactly once after durable status/log updates. Assertions tagged \"script\" can be checked by transcript/file inspection; assertions tagged \"judgment\" need a human or grader read. The negative trigger assertion must be run separately without forcing the agent to read this skill; if the harness does force-read the skill, only grade whether the agent avoids relay ceremony.",
"evals": [
{
"id": 0,
"name": "plan-a-relay",
"prompt": "I want to migrate all our REST endpoints to the new validation layer \u2014 there are around 40 of them across src/server/routes. I won't be able to babysit this. Set it up as a relay so an agent can grind through it across sessions and only pull me in when it actually needs me.",
"expected_output": "Produces a charter (default .pi-web/relays/<name>/charter.md) plus an empty/seeded log. The charter has all four required slots present: goal/finish-line, sizing, handover, intervention signal. The agent ASKS the user to make sizing and the intervention signal concrete rather than inventing strict rules. It does NOT prescribe what a 'good' leg size or cadence is. It may dispatch the first leg only after the charter is agreed.",
"prompt": "I want to migrate all our REST endpoints to the new validation layer — there are around 40 of them across src/server/routes. I won't be able to babysit this. Set it up as a relay so an agent can grind through it across sessions and only pull me in when it actually needs me.",
"expected_output": "Produces a relay packet (default .pi-web/relays/<name>/) with charter.md, status.md, and log.md. The charter has all required slots present: relay identity/root, goal/finish-line, sizing, task selection policy, handover, intervention signal, and reading discipline. The initial status is a compact baton with current position, first task or task-selection pointer, relevant context, progress documentation expectations, and known blockers. The agent asks the user to make sizing, task selection, reading discipline, and the intervention signal concrete rather than inventing strict rules. It does not prescribe what a 'good' leg size or cadence is. It may dispatch the first leg only after the packet is agreed.",
"files": [],
"assertions": [
{ "name": "charter-created", "text": "A charter document is created (default under .pi-web/relays/<name>/ unless the user specified a location).", "type": "script" },
{ "name": "packet-created", "text": "A relay packet is created with charter.md, status.md, and log.md under the relay location (default .pi-web/relays/<name>/ unless specified).", "type": "script" },
{ "name": "goal-slot-present", "text": "The charter defines a concrete, achievable finish line / goal.", "type": "judgment" },
{ "name": "sizing-slot-present", "text": "The charter states how much work is one leg (sizing), rather than leaving it undefined.", "type": "judgment" },
{ "name": "handover-slot-present", "text": "The charter states the handover mechanism (what the spawn prompt says and what the next runner reads).", "type": "judgment" },
{ "name": "task-selection-slot-present", "text": "The charter states how a runner chooses the next task when status.md does not name one explicitly.", "type": "judgment" },
{ "name": "handover-slot-present", "text": "The charter states the handover mechanism, including that the next runner reads charter.md and status.md.", "type": "judgment" },
{ "name": "intervention-slot-present", "text": "The charter defines an intervention signal: when/how a runner stops and gets the human.", "type": "judgment" },
{ "name": "asks-not-prescribes", "text": "For sizing and the intervention signal, the agent asks the user to make them concrete instead of imposing its own strict rules/cadence.", "type": "judgment" },
{ "name": "no-premature-spawn", "text": "The agent does not spawn the first leg before the charter is agreed with the user.", "type": "script" }
{ "name": "reading-discipline-present", "text": "The charter states the reading discipline, including not reading log.md end-to-end by default.", "type": "judgment" },
{ "name": "status-seeded", "text": "status.md is seeded as a compact baton with current position, first task or task-selection pointer, relevant context, documentation expectations, and known blockers.", "type": "judgment" },
{ "name": "asks-not-prescribes", "text": "For sizing, task selection, reading discipline, and the intervention signal, the agent asks the user to make them concrete instead of imposing its own strict rules/cadence.", "type": "judgment" },
{ "name": "no-premature-spawn", "text": "The agent does not spawn the first leg before the relay packet is agreed with the user.", "type": "script" }
]
},
{
"id": 1,
"name": "run-one-leg-and-hand-off",
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/log.md, continue the plan, then dispatch the next agent.",
"expected_output": "Loads the relay skill (handoff prompt names the framework). Orients by reading charter+log, re-anchors to the goal, does exactly ONE well-sized leg per the charter's sizing, appends a log entry (what/why/new state/blockers), makes work durable (saves files, commits if the charter calls for it), then calls spawn_session exactly once with a handoff prompt that names Relay and points at the charter+log. Does not do extra legs or spawn more than once.",
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/status.md, continue the plan, then dispatch the next agent.",
"expected_output": "Loads the relay skill (handoff prompt names the framework). Orients by reading charter.md and status.md, not the full log. Re-anchors to the goal, chooses the next task from status.md or the charter's task-selection policy, does exactly ONE well-sized leg per the charter's sizing, updates status.md as a compact baton, appends a concise log.md entry, makes work durable (saves files, commits if the charter calls for it), then calls spawn_session exactly once with a handoff prompt that names Relay and points at charter.md and status.md. Does not do extra legs, does not spawn more than once, and does not tell the next runner to read log.md end-to-end.",
"files": [],
"assertions": [
{ "name": "skill-loads-from-handoff", "text": "The agent recognizes it is in a relay and loads/consults the relay skill from the handoff prompt.", "type": "judgment" },
{ "name": "reads-charter-and-log", "text": "The agent reads both the charter and the log before acting.", "type": "script" },
{ "name": "reads-charter-and-status", "text": "The agent reads both charter.md and status.md before acting.", "type": "script" },
{ "name": "does-not-read-full-log", "text": "The agent does not read log.md end-to-end by default; any log use is targeted and justified by status.md or charter.md.", "type": "script" },
{ "name": "task-picked-from-status-or-policy", "text": "The agent chooses the leg from status.md, or applies the charter's task-selection policy if status.md does not name a task.", "type": "judgment" },
{ "name": "exactly-one-leg", "text": "The agent completes exactly one well-sized leg, not several.", "type": "judgment" },
{ "name": "log-appended", "text": "A new log entry is appended recording what was done, decisions, new state, and any blocker.", "type": "script" },
{ "name": "status-updated", "text": "status.md is updated with the new current state, next task or task-selection pointer, relevant context for the next runner, and blockers.", "type": "script" },
{ "name": "log-appended", "text": "A concise log.md entry is appended recording what was done, decisions, artifacts changed, status updates made, and any blocker.", "type": "script" },
{ "name": "work-durable-before-handoff", "text": "Work is saved (and committed if the charter requires it) before spawn_session is called.", "type": "script" },
{ "name": "spawn-exactly-once", "text": "spawn_session is called exactly once.", "type": "script" },
{ "name": "handoff-names-relay", "text": "The spawn prompt names the Relay framework and points the next runner at the charter and log so the skill loads downstream.", "type": "judgment" }
{ "name": "handoff-names-relay-and-status", "text": "The spawn prompt names the Relay framework and points the next runner at charter.md and status.md, not the full log, so the skill loads downstream with bounded context.", "type": "judgment" }
]
},
{
"id": 2,
"name": "stop-on-blocker-do-not-spawn",
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/log.md and continue. (The next leg hits a condition the charter flagged as requiring the human.)",
"expected_output": "Orients, begins the leg, recognizes the charter's intervention condition has fired. Stops cleanly: logs the blocker clearly, raises the charter's defined intervention signal so the watching human sees it, and does NOT call spawn_session. A clean stop with a clear blocker is the success condition here.",
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/status.md and continue. (The next leg hits a condition the charter flagged as requiring the human.)",
"expected_output": "Orients from charter.md and status.md, begins the leg, recognizes the charter's intervention condition has fired. Stops cleanly: updates status.md with the blocker/intervention state, appends a clear log.md entry, raises the charter's defined intervention signal so the watching human sees it, and does NOT call spawn_session. A clean stop with a clear blocker is the success condition here.",
"files": [],
"assertions": [
{ "name": "blocker-logged", "text": "The agent logs the blocker clearly in the log.", "type": "script" },
{ "name": "status-records-blocker", "text": "status.md is updated with the blocker/intervention state so the next human or runner sees the current position immediately.", "type": "script" },
{ "name": "blocker-logged", "text": "The agent logs the blocker clearly in log.md.", "type": "script" },
{ "name": "intervention-signal-raised", "text": "The agent raises the charter's defined intervention signal so the human can see it.", "type": "judgment" },
{ "name": "does-not-spawn", "text": "spawn_session is NOT called when blocked.", "type": "script" },
{ "name": "no-silent-stall", "text": "The agent does not stop silently; the stop is explained and visible.", "type": "judgment" }
{ "name": "no-silent-stall", "text": "The agent does not stop silently; the stop is explained and visible in status.md/log.md.", "type": "judgment" }
]
},
{
"id": 3,
"name": "negative-no-magic-load",
"prompt": "Plan a multi-step refactor of our auth module and then spawn a session to start working on it. Break it into stages.",
"expected_output": "This prompt mentions a multi-step plan AND spawning a session, but never names the Relay framework, points at a charter/log, or invokes the skill. The relay skill should NOT load. The agent should plan and (optionally) use spawn_session as ordinary tools, without adopting relay ceremony (charter/log/legs/intervention signal).",
"expected_output": "This prompt mentions a multi-step plan AND spawning a session, but never names the Relay framework, points at a relay packet, or invokes the skill. The relay skill should NOT load. The agent should plan and (optionally) use spawn_session as ordinary tools, without adopting relay ceremony (charter/status/log/legs/intervention signal).",
"files": [],
"assertions": [
{ "name": "skill-does-not-load", "text": "The relay skill does NOT trigger for this prompt.", "type": "judgment" },
{ "name": "no-relay-ceremony", "text": "The agent does not create a charter/log or impose relay leg/handoff ceremony.", "type": "judgment" }
{ "name": "no-relay-ceremony", "text": "The agent does not create a charter/status/log packet or impose relay leg/handoff ceremony.", "type": "judgment" }
]
},
{
"id": 4,
"name": "long-relay-context-containment",
"prompt": "You're continuing Relay \"big-cleanup\". The relay has a huge log.md from dozens of prior legs. Read .pi-web/relays/big-cleanup/charter.md and .pi-web/relays/big-cleanup/status.md, then do the next leg without blowing up context.",
"expected_output": "Orients from charter.md and status.md, follows only the relevant context pointers in status.md, and avoids reading the huge log.md end-to-end. If status.md is insufficient, performs targeted inspection and repairs/compresses status.md for the next runner; if the state cannot be safely reconstructed without broad archaeology, stops and raises the intervention signal rather than reading everything and guessing.",
"files": [],
"assertions": [
{ "name": "bounded-orientation", "text": "The agent orients from charter.md and status.md rather than rebuilding full relay history.", "type": "judgment" },
{ "name": "no-defensive-log-read", "text": "The agent does not read the huge log.md end-to-end defensively.", "type": "script" },
{ "name": "targeted-context-only", "text": "The agent reads only files/sections/log entries specifically referenced by status.md or needed for the current leg.", "type": "judgment" },
{ "name": "repairs-status-or-stops", "text": "If status.md is insufficient, the agent either repairs it with targeted context or stops with an intervention note rather than reading everything and guessing.", "type": "judgment" }
]
}
]
skills/relay/evals/live-behavior-testing.md
+134
View File
@@ -0,0 +1,134 @@
# Live behavior testing guide
Use live behavior tests when you want to know how the relay skill behaves **right now** with real agent sessions. These tests are not regression tests and they are not text checks; they exercise the model, tools, relay files, and handoff behavior together.
## Basic idea
Run each eval as a **tracked subsession** so you can inspect what happened afterward. The subsession acts like the agent using the relay skill. The parent session acts as the evaluator.
Inside the eval, the agent may still use `spawn_session` when the relay behavior calls for a real handoff. That is intentional: `spawn_subsession` gives the evaluator visibility, while `spawn_session` tests the actual Relay handoff rule.
## What to test
A useful small live suite covers these behaviors:
- **Planning a relay:** the agent drafts `charter.md`, `status.md`, and `log.md`; asks for missing human choices; does not spawn before approval.
- **Running one leg:** the agent reads `charter.md` and `status.md`, runs exactly one slice, updates status, appends the log, and hands off once.
- **Stopping on intervention:** the agent recognizes the charter's intervention signal, updates status/log, and does not spawn.
- **Long relay containment:** the agent does not read a huge `log.md`; it uses `status.md` plus targeted files only.
- **Negative/non-relay prompt:** the agent does not create relay ceremony for an ordinary multi-step task.
## Sandbox shape
Put throwaway relay files outside the repo or under a clearly temporary path, for example:
```text
/tmp/pi-web-relay-live-evals/iteration-1/<eval-name>/
sandbox/.pi-web/relays/<relay-name>/
charter.md
status.md
log.md
work/...
with_skill/outputs/
```
Keep the sandbox tiny. The point is to test relay behavior, not the complexity of the toy task.
For the handoff eval, make the spawned receiver bounded. The charter can say something like:
```text
If you are the spawned receiver for this eval, do not run another relay leg and do not spawn again. Write spawned-next-runner.txt containing "received", then stop.
```
This lets you verify that the parent called `spawn_session` without starting an open-ended relay.
## Running the evals
For each eval, spawn a tracked subsession with a prompt that says:
- read the skill under test, e.g. `skills/relay/SKILL.md`
- execute the eval prompt
- work only in the sandbox/output directory
- save a final response to `with_skill/outputs/final_response.md`
Example shape:
```text
You are a live behavior eval runner for the relay skill. Act as the target assistant, not as an evaluator.
Use the current skill under test by reading:
/path/to/skills/relay/SKILL.md
Task prompt to execute:
"You're working under the Relay framework. Read /tmp/.../charter.md and /tmp/.../status.md, continue the plan, then dispatch the next agent."
Constraints:
- Work only inside /tmp/.../<eval-name>/ except for reading the skill file.
- Save your final answer to /tmp/.../<eval-name>/with_skill/outputs/final_response.md.
```
Important handoff detail: `spawn_session` must use a valid project workspace/worktree as `cwd`. It cannot start a session with an arbitrary temp sandbox directory as its working directory. During testing, one eval runner tried to hand off with `cwd` set to `/tmp/.../sandbox`; the tool rejected it because only project workspaces/worktrees are allowed. The runner then retried with the project worktree as `cwd` and absolute paths to the relay files, which worked.
So when testing or running a relay whose packet lives outside the repo, keep `cwd` at a valid project workspace/worktree (`<project-root>`) and make the handoff prompt point to the relay files by absolute path:
```text
spawn_session cwd: <project-root>
Prompt:
You are continuing Relay "sandbox".
Read:
- /tmp/pi-web-relay-live-evals/.../sandbox/.pi-web/relays/sandbox/charter.md
- /tmp/pi-web-relay-live-evals/.../sandbox/.pi-web/relays/sandbox/status.md
```
This matters for the "spawn exactly once" assertion: a failed first `spawn_session` call still counts as an attempted handoff. Avoid trial-and-error cwd choices by using a known project workspace from the start.
## Reviewing results
After each subsession finishes, review both transcript and files:
- Did it read `charter.md` and `status.md` before acting?
- Did it avoid reading `log.md` end-to-end unless explicitly targeted?
- Did it do exactly one leg?
- Did it update `status.md` as the next runner's baton?
- Did it append a concise `log.md` entry?
- Did it call `spawn_session` exactly once when handing off?
- Did it avoid spawning when blocked or complete?
- Did any spawned bounded receiver write the expected marker file?
Record a short result summary in the eval workspace, for example:
```text
/tmp/pi-web-relay-live-evals/iteration-1/live-results.md
/tmp/pi-web-relay-live-evals/iteration-1/live-results.json
```
## Interpreting negative tests
If the harness explicitly tells the subsession to read the relay skill, you cannot fairly test whether the skill would have triggered on its own. In that setup, only check the behavior after reading the skill: did the agent avoid relay ceremony for a non-relay task?
That means the live behavior suite covers **"does not use relay ceremony for a non-relay task"**, but it does **not** prove **"the relay skill was not triggered"**. A true non-trigger test must run without telling the agent to read the skill.
## Testing that Relay does not trigger
Use a separate trigger test when you care about whether the skill loads automatically. Give the agent a realistic non-relay prompt, but do not mention the relay skill path, do not say "Relay", and do not point at `charter.md`, `status.md`, or `log.md`.
A good non-trigger prompt is close enough to be tempting:
```text
Plan a multi-step refactor of our auth module and spawn a session to start the first stage. Break it into stages.
```
Review the transcript and outputs for:
- no read of `skills/relay/SKILL.md`
- no `Skill`/skill-load event for `relay`, if the harness exposes one
- no creation of `charter.md`, `status.md`, or `log.md`
- no relay-specific terms such as leg, baton, intervention signal, relay packet, or handoff protocol unless the user used them first
- ordinary `spawn_session` use is allowed if the user asked for it; spawning alone is not Relay
Keep this separate from behavior evals. Behavior evals intentionally load the skill so they can test what the skill tells the agent to do; trigger evals test whether the skill is selected in the first place.
## Why not Docker/static checks?
Static checks can confirm that certain words exist in `SKILL.md`, but they do not show whether an agent follows the skill. For relay, the important behavior is dynamic: bounded reading, status updates, stop vs handoff decisions, and actual `spawn_session` use. Use live subsessions for that.