hope-voice-api/PLAN.md

Plan: Per-user Voice API + Shane Memory System

Goal

Replace the Voice API's single shared Cognee memory connection with a per-user connection to the deployed Shane Memory System. The first Voice user is shane; his server-side profile uses the existing remote memory token pulled from Unraid. Future Voice users can point to separate memory servers and tokens without exposing those credentials to the browser or LLM.

This document is the implementation plan. In this investigation pass, only the bootstrap user registry and requested Azure voice/prompt changes were made; the authentication and per-user runtime described below are not implemented yet.

Investigation summary

Voice API today

• One Docker container runs LiveKit, the Python voice agent, nginx, and web/token_server.py.
• POST /token is unauthenticated. It accepts no real user credential and creates a random browser identity and room.
• The browser has no login/user-selection flow.
• Every agent session calls build_mcp_toolsets() with process-global configuration.
• Memory is hardcoded to the unauthenticated Cognee endpoint at http://192.168.86.2:8003/mcp.
• The prompt uses one global Cognee session ID (hope), so it is not a user isolation boundary.
• Local skills use the shared /skills directory.
• Background tasks use a global /tmp/tasks/events.jsonl and one global tail cursor. With concurrent rooms, one room can consume another room's task event. This must be fixed before calling the product multi-user safe.
• Browser transcript history and voice preference use unscoped localStorage keys, which can expose one user's local history to the next user on a shared browser.
• Rooms are unique, but close_on_disconnect=False leaves their agents alive after the user disconnects. This can accumulate orphan sessions.

Shane Memory System today

Repository: ~/dev/shane-memory-system

Deployed service:

• Host: 192.168.86.2
• Container: shane-memory
• MCP URL: http://192.168.86.2:8790/mcp
• Data: /mnt/user/appdata/shane-memory mounted at /data
• Transport: MCP Streamable HTTP
• Authentication: Authorization: Bearer sm_...
• Isolation: each token resolves to one SQLite user; note files and vector rows are scoped by that user.
• Tools: create_note, read_note, update_note, delete_note, list_notes, search_notes, memory_save, memory_recall, and reindex_all.

The deployed database currently has pi, alice, and alice2. There is no remote account literally named shane. The pi account is Shane's active namespace and has both default and voice-agent tokens. The dedicated voice-agent token was selected for this product.

The selected token was tested against the deployed MCP server: initialization, tool listing, and an authenticated list_notes call all succeeded.

Bootstrap credential stored in this repository

config/users.json now contains the first Voice profile:

• Voice user: shane
• LiveKit identity: shane
• Memory URL: http://192.168.86.2:8790/mcp
• Current remote memory username: pi
• Remote token label: voice-agent
• Token SHA-256 for audit comparison: d6af807287472806769926b1ef5e20fa2afc3e5ae0d6d21b0d5406605309f335

The actual token is deliberately not repeated in this plan. It is stored in plaintext in config/users.json as requested. The file was created with mode 0600, but Git does not preserve read permissions, only the executable bit. Deployment must therefore set ownership/permissions explicitly.

LiveKit MCP compatibility

The locked livekit-agents version is 1.7.0. Its MCPServerHTTP accepts a headers dictionary, so the per-user integration can use:

mcp.MCPServerHTTP(
    url=profile.memory.url,
    transport_type="streamable_http",
    headers={"Authorization": f"Bearer {profile.memory.api_key}"},
)

No custom MCP proxy is required.

Requested voice change completed

The default voice is now:

en-US-Ava:DragonHDOmniLatestNeural

Updated locations:

• agent/agent.py
• docker-compose.yml
• .env.example
• web/app.js
• AGENTS.md

The prompt now teaches Azure HD plain-text delivery markers such as [appreciative], [whispering], [laughter], and [Neutral], with a limit of zero to two markers per response.

This marker syntax is intentional. Azure supports HD styles as plain-text square-bracket markers, and LiveKit's Azure plugin already wraps agent text in SSML. The model should not emit raw XML/SSML. The locked plugin concatenates input text into its SSML wrapper, so a later hardening pass should also XML-escape ordinary model text while preserving bracket markers.

Target architecture

Browser
  |  POST /api/login (username/password)
  |  HttpOnly Secure session cookie
  |  POST /token
  v
Token server -------------------------> config/users.json
  |                                        |
  | signed LiveKit JWT                     | server-only memory URL/token
  | room: voice-shane-<nonce>              |
  v                                        v
LiveKit room -----------------------> Voice agent job
                                         |
                                         | MCP Streamable HTTP
                                         | Authorization: Bearer <Shane token>
                                         v
                              Shane Memory System /mcp
                              token -> isolated remote user

Principles:

1. The browser proves which Voice user it is to the token server.
2. The token server, not the browser, selects the user ID and room name.
3. The memory token remains server-side and is never placed in a LiveKit JWT, room metadata visible to clients, data-channel message, transcript, or log.
4. The agent resolves a trusted Voice user ID and constructs a new memory MCP client for that session.
5. Every mutable local resource is namespaced by user and, where appropriate, room.

User registry design

Evolve config/users.json to this shape:

{
  "version": 1,
  "users": {
    "shane": {
      "enabled": true,
      "display_name": "Shane",
      "livekit_identity": "user:shane",
      "auth": {
        "password_hash": "<scrypt or argon2id hash>"
      },
      "memory": {
        "url": "http://192.168.86.2:8790/mcp",
        "api_key": "<server-only token>",
        "remote_username": "pi",
        "remote_token_label": "voice-agent",
        "allowed_tools": [
          "create_note",
          "read_note",
          "update_note",
          "delete_note",
          "list_notes",
          "search_notes",
          "memory_save",
          "memory_recall"
        ]
      }
    }
  }
}

Validation rules:

• Reject duplicate, malformed, disabled, or unknown user IDs at startup.
• Require HTTPS for non-LAN memory URLs; explicitly allow configured RFC1918 HTTP URLs.
• Require sm_ only as a warning, not as a universal protocol rule, because future users may run a different compatible memory product.
• Never serialize api_key in status responses or exceptions.
• Make startup fail closed if the registry is invalid.

Create shared/user_registry.py so the token server and agent use one parser and validation implementation.

Authentication and identity flow

1. Add Voice login

Add these endpoints to web/token_server.py:

• POST /api/login — verify username/password and issue an opaque session cookie.
• GET /api/session — return only safe profile fields (user_id, display_name).
• POST /api/logout — revoke the current session and clear the cookie.
• POST /token — require a valid session; ignore any client-supplied identity or room.

Cookie requirements:

• HttpOnly
• Secure
• SameSite=Strict
• short idle expiration with a bounded absolute expiration
• random opaque value; only a hash stored server-side

Also enforce same-origin requests and basic login rate limiting. Keep the service LAN-only, but do not treat LAN access as authentication.

2. Bind user to a trusted room

The token server generates a room such as voice-shane-<random nonce> after authenticating Shane. The user slug is not accepted from the browser. The LiveKit participant identity is set from the registry, not request JSON.

The agent parses the trusted room format and looks up the profile. Unknown/malformed rooms fail closed before an LLM or MCP session starts. If supported cleanly by the deployed LiveKit version, signed agent-dispatch metadata may replace room parsing later; the memory token must never be put in that metadata.

3. Update the frontend

• Add a small login state before “Start Conversation.”
• Fetch /api/session on load.
• Fetch /token only after authentication.
• On 401, return to login rather than repeatedly reconnecting.
• Namespace local keys as voice_history:<user_id> and preferred_voice:<user_id>.
• Clear in-memory room, transcript, task, and audio state on logout.
• Never store the memory API key in browser storage.

Per-user agent and memory integration

Refactor construction

Change:

VoiceAssistant()
build_mcp_toolsets()

To:

VoiceAssistant(user_profile, room_context)
build_mcp_toolsets(user_profile, room_context)

For the memory toolset:

• URL comes from user_profile.memory.url.
• Add Authorization: Bearer <api_key> through MCPServerHTTP(headers=...).
• Use Streamable HTTP.
• Apply the profile's allowlist.
• Create one toolset/client per agent session; never mutate shared MCP headers between sessions.
• Redact token-bearing headers from errors and logs.
• Log only user ID, remote hostname, and connection status.

Update memory behavior in the prompt

The new memory server does not expose Cognee's remember, recall, or forget, and does not accept session_id="hope".

Replace those instructions with:

• Personal/history question: call memory_recall before answering.
• Durable fact or preference: call memory_save with a concise title/body and useful tags.
• Precise note lookup: use search_notes, then read_note if full content is needed.
• Correction: find the note and call update_note.
• Forget request: find the exact note, confirm ambiguity when needed, then call delete_note.
• Do not save every utterance; save durable facts, preferences, decisions, and explicit “remember this” requests.

Inject the safe display name into the prompt so the assistant knows it is speaking with Shane. Do not inject the token or raw registry JSON.

Cognee transition

The old Cognee service and the new Shane Memory System are separate stores. Before removing Cognee:

1. Inventory Shane-relevant permanent memories in Cognee.
2. Export and normalize them into markdown notes.
3. Import them through Shane's authenticated memory_save/create_note tools so embeddings are built correctly.
4. Compare representative recall queries against both stores.
5. Keep Cognee as an opt-in read-only fallback for one release.
6. Remove COGNEE_MCP_URL, old tests, and MEMORY_SERVER.md only after parity is accepted.

Do not blindly import Cognee session-cache test probes.

Multi-user isolation work required alongside memory

Skills

Change the skills directory from global /skills to a user namespace, for example /data/users/<user_id>/skills. Pass that path only to that user's stdio MCP process. Decide separately whether any skills are product-global and read-only.

Background tasks

Refactor TaskRegistry so it is instantiated per room with a dedicated path such as /tmp/tasks/<user_id>/<room_nonce>. Pass the same path and user/room IDs into dispatch_mcp.py. Do not use the current process-global registry or cursor.

Every event should include user_id and room_id, and the publisher must reject mismatches before sending data to a room.

Room lifecycle

Because rooms are unique per authenticated conversation, use close_on_disconnect=True or add an explicit idle timeout. Cancel task pollers and close MCP transports when the session ends. This prevents orphan agents and credentials from remaining active indefinitely.

Logs and transcripts

• Never log authorization headers or registry entries.
• Include a safe user_id and room correlation ID in structured logs.
• Namespace browser history by user.
• Do not persist transcripts server-side unless a separate, explicit retention policy is added.

Implementation phases

Phase 1 — Registry and authentication foundation

Files:

• Add shared/user_registry.py.
• Expand config/users.json with enabled/auth fields.
• Add registry unit tests and redaction tests.
• Update Dockerfile to copy the registry with voiceuser ownership and restrictive permissions.
• Update supervisord.conf with VOICE_USERS_FILE=/etc/hope/users.json.

Exit criteria:

• Both Python processes load the same validated profile.
• Invalid config prevents startup.
• No test or log prints the token.

Phase 2 — Login and trusted LiveKit identity

Files:

• Refactor web/token_server.py.
• Add login/session UI to web/index.html, web/app.js, and web/style.css.
• Update token/web tests.

Exit criteria:

• Unauthenticated /token returns 401.
• The browser cannot spoof another identity or room.
• Shane can log in, receive a token, and join only a Shane-bound room.

Phase 3 — Per-user memory MCP

Files:

• Refactor agent/agent.py profile resolution and toolset construction.
• Replace Cognee memory prompt instructions.
• Pass the Bearer header and new tool allowlist.
• Add authenticated MCP fixtures and tests.

Exit criteria:

• Shane can recall and save through the voice-agent token.
• An invalid/revoked token fails only that session and never falls back to another user's memory.
• No request reaches port 8003 during normal operation.

Phase 4 — Isolation hardening

Files:

• Refactor agent/task_registry.py, agent/dispatch_mcp.py, and agent/task_worker.py for per-room storage/routing.
• Namespace SKILLS_DIR.
• Scope frontend local storage.
• Add cleanup/cancellation on disconnect.

Exit criteria:

• Two simultaneous users cannot observe each other's tasks, skills, transcripts, memory results, or voice preferences.

Phase 5 — Migration and deployment

• Back up /mnt/user/appdata/shane-memory and the Cognee store.
• Build and deploy the Voice container.
• Run Shane's end-to-end recall/save checks.
• Import accepted Cognee memories.
• Revoke obsolete test or integration tokens after cutover.

Optional naming cleanup on the memory server:

• Keep the current Voice shane -> remote pi mapping initially; it is functional and preserves the token.
• If consistent naming is desired, stop the memory container, back up SQLite/files, rename /data/pi to /data/shane, update users.username in SQLite, restart, and verify. Existing tokens remain attached to the same numeric user ID, so Shane can retain the same API key. Test startup filesystem reconciliation and semantic search before considering this complete.

Test matrix

Unit

• Registry schema validation and URL rules.
• Password hash verification.
• Token/header redaction.
• Trusted room encode/decode.
• Per-user local storage key generation.
• Prompt contains the new memory tool names and no Cognee session_id instructions.
• Dragon HD Omni is the consistent backend/frontend default.

Integration

• Valid Shane token initializes MCP and lists the expected nine tools.
• Missing, malformed, and revoked memory tokens fail with controlled errors.
• Save -> recall -> update -> delete in Shane's namespace.
• Two remote users create the same note title and cannot read each other's note.
• Two simultaneous LiveKit rooms receive only their own task events.
• Disconnect closes the agent, task poller, and MCP transport.

End to end

1. Log in as Shane.
2. Start a conversation.
3. Ask a known personal question and verify memory_recall runs before the answer.
4. Say “remember…” and verify a note appears under Shane's remote namespace.
5. Restart the Voice container and verify recall persists.
6. Verify [appreciative]/[Neutral] produce expressive speech and are not spoken literally.
7. Log out and confirm local in-memory state is cleared.
8. Log in as a second user and verify no Shane data appears.

Rollback

• Keep the old Cognee URL/config available behind an explicit MEMORY_BACKEND=cognee setting for one release only.
• Roll back the Voice image without changing the remote memory database.
• If remote account renaming is performed, restore both memory.db and the corresponding note directory together.
• Token rotation is independent: issue a new voice-agent token, update config/users.json, rebuild/restart, verify, then revoke the old token.

Acceptance criteria

• Shane authenticates as a stable Voice user rather than a random participant.
• Shane's agent session uses the committed voice-agent API key against :8790/mcp.
• The key never reaches browser-visible data or logs.
• Memory tools use the Shane Memory System API and correct tool schemas.
• Concurrent users have isolated memory, skills, tasks, rooms, transcripts, and browser storage.
• Adding a new user requires one registry entry and no code changes.
• The default TTS voice is Azure Ava Dragon HD Omni and the prompt uses supported expressive delivery markers safely.