Archived
Federico Jaramillo Martinez
ab46d7addc
Add the Relay method skill (execute a long plan as a chain of independent sessions handing off via spawn_session) under a top-level skills/ folder that follows the open agent-skills convention, installable with the skills CLI. Also clarify the changeset-changelog skill: no changeset is needed for changes outside the npm release allowlist (package.json files), such as repo-only agent skills under .agents/ and skills/.
63 lines
6.5 KiB
JSON
63 lines
6.5 KiB
JSON
{
|
|
"skill_name": "relay",
|
|
"notes": "Relay is a behavioral framework skill. Test cases are prompts; 'good' is described per case and broken into checkable assertions. Because this project's only spawning primitive is spawn_session (fire-and-forget, real sessions), the standard isolated-subagent benchmark pipeline is not available here. Verify via (a) inline behavioral walkthrough of the skill text and (b) live spawn_session smoke tests against a throwaway sandbox relay, observed by the human in the UI. Assertions tagged \"script\" can be checked by counting spawn_session calls / inspecting files; assertions tagged \"judgment\" need a human or grader read.",
|
|
"evals": [
|
|
{
|
|
"id": 0,
|
|
"name": "plan-a-relay",
|
|
"prompt": "I want to migrate all our REST endpoints to the new validation layer \u2014 there are around 40 of them across src/server/routes. I won't be able to babysit this. Set it up as a relay so an agent can grind through it across sessions and only pull me in when it actually needs me.",
|
|
"expected_output": "Produces a charter (default .pi-web/relays/<name>/charter.md) plus an empty/seeded log. The charter has all four required slots present: goal/finish-line, sizing, handover, intervention signal. The agent ASKS the user to make sizing and the intervention signal concrete rather than inventing strict rules. It does NOT prescribe what a 'good' leg size or cadence is. It may dispatch the first leg only after the charter is agreed.",
|
|
"files": [],
|
|
"assertions": [
|
|
{ "name": "charter-created", "text": "A charter document is created (default under .pi-web/relays/<name>/ unless the user specified a location).", "type": "script" },
|
|
{ "name": "goal-slot-present", "text": "The charter defines a concrete, achievable finish line / goal.", "type": "judgment" },
|
|
{ "name": "sizing-slot-present", "text": "The charter states how much work is one leg (sizing), rather than leaving it undefined.", "type": "judgment" },
|
|
{ "name": "handover-slot-present", "text": "The charter states the handover mechanism (what the spawn prompt says and what the next runner reads).", "type": "judgment" },
|
|
{ "name": "intervention-slot-present", "text": "The charter defines an intervention signal: when/how a runner stops and gets the human.", "type": "judgment" },
|
|
{ "name": "asks-not-prescribes", "text": "For sizing and the intervention signal, the agent asks the user to make them concrete instead of imposing its own strict rules/cadence.", "type": "judgment" },
|
|
{ "name": "no-premature-spawn", "text": "The agent does not spawn the first leg before the charter is agreed with the user.", "type": "script" }
|
|
]
|
|
},
|
|
{
|
|
"id": 1,
|
|
"name": "run-one-leg-and-hand-off",
|
|
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/log.md, continue the plan, then dispatch the next agent.",
|
|
"expected_output": "Loads the relay skill (handoff prompt names the framework). Orients by reading charter+log, re-anchors to the goal, does exactly ONE well-sized leg per the charter's sizing, appends a log entry (what/why/new state/blockers), makes work durable (saves files, commits if the charter calls for it), then calls spawn_session exactly once with a handoff prompt that names Relay and points at the charter+log. Does not do extra legs or spawn more than once.",
|
|
"files": [],
|
|
"assertions": [
|
|
{ "name": "skill-loads-from-handoff", "text": "The agent recognizes it is in a relay and loads/consults the relay skill from the handoff prompt.", "type": "judgment" },
|
|
{ "name": "reads-charter-and-log", "text": "The agent reads both the charter and the log before acting.", "type": "script" },
|
|
{ "name": "exactly-one-leg", "text": "The agent completes exactly one well-sized leg, not several.", "type": "judgment" },
|
|
{ "name": "log-appended", "text": "A new log entry is appended recording what was done, decisions, new state, and any blocker.", "type": "script" },
|
|
{ "name": "work-durable-before-handoff", "text": "Work is saved (and committed if the charter requires it) before spawn_session is called.", "type": "script" },
|
|
{ "name": "spawn-exactly-once", "text": "spawn_session is called exactly once.", "type": "script" },
|
|
{ "name": "handoff-names-relay", "text": "The spawn prompt names the Relay framework and points the next runner at the charter and log so the skill loads downstream.", "type": "judgment" }
|
|
]
|
|
},
|
|
{
|
|
"id": 2,
|
|
"name": "stop-on-blocker-do-not-spawn",
|
|
"prompt": "You're working under the Relay framework. Read .pi-web/relays/<sandbox>/charter.md and .pi-web/relays/<sandbox>/log.md and continue. (The next leg hits a condition the charter flagged as requiring the human.)",
|
|
"expected_output": "Orients, begins the leg, recognizes the charter's intervention condition has fired. Stops cleanly: logs the blocker clearly, raises the charter's defined intervention signal so the watching human sees it, and does NOT call spawn_session. A clean stop with a clear blocker is the success condition here.",
|
|
"files": [],
|
|
"assertions": [
|
|
{ "name": "blocker-logged", "text": "The agent logs the blocker clearly in the log.", "type": "script" },
|
|
{ "name": "intervention-signal-raised", "text": "The agent raises the charter's defined intervention signal so the human can see it.", "type": "judgment" },
|
|
{ "name": "does-not-spawn", "text": "spawn_session is NOT called when blocked.", "type": "script" },
|
|
{ "name": "no-silent-stall", "text": "The agent does not stop silently; the stop is explained and visible.", "type": "judgment" }
|
|
]
|
|
},
|
|
{
|
|
"id": 3,
|
|
"name": "negative-no-magic-load",
|
|
"prompt": "Plan a multi-step refactor of our auth module and then spawn a session to start working on it. Break it into stages.",
|
|
"expected_output": "This prompt mentions a multi-step plan AND spawning a session, but never names the Relay framework, points at a charter/log, or invokes the skill. The relay skill should NOT load. The agent should plan and (optionally) use spawn_session as ordinary tools, without adopting relay ceremony (charter/log/legs/intervention signal).",
|
|
"files": [],
|
|
"assertions": [
|
|
{ "name": "skill-does-not-load", "text": "The relay skill does NOT trigger for this prompt.", "type": "judgment" },
|
|
{ "name": "no-relay-ceremony", "text": "The agent does not create a charter/log or impose relay leg/handoff ceremony.", "type": "judgment" }
|
|
]
|
|
}
|
|
]
|
|
}
|