feat: provision private Hermes assistants for authenticated Hope accounts

This commit is contained in:
snowspeeder committed 2026-10-01 13:30:18 -04:00
1 parent da65da68b3
commit 2aedea84b5
33 files changed
+3063 -1743

No files matched your search

+17 -32
View File
@@ -1,39 +1,24 @@
# Hope test suite
# Hope multi-user qualification
End-to-end tests that drive the **actual running container** — no mocks.
They run on the host and hit the live web UI, token endpoint, LLM API, MCP
tool servers, and agent process.
## Offline regression suite (no NAS or model calls)
## Prerequisites
- The container is up: `docker compose up -d` (and healthy)
- Host deps installed: `pip install -r tests/requirements.txt`
(or `uv pip install -r tests/requirements.txt`)
- The LLM at `LLM_BASE_URL` (default `http://192.168.86.2:3005/api`, OpenWebUI) is reachable
- Internet access for the weather tool tests (wttr.in)
## Run
```bash
cd ~/dev/hope-voice-api
python -m pytest tests/ -v # everything
python -m pytest tests/ -v -m "not slow" # skip LLM/network-heavy tests
```sh
python -m venv .venv-tests
.venv-tests/bin/python -m pip install pytest fastapi httpx
.venv-tests/bin/python -m pytest --noconftest tests/unit/test_accounts.py tests/unit/test_provisioner.py tests/unit/test_token_bootstrap.py -q
agent/.venv/bin/python -m unittest discover -s tests/unit -p test_hermes_llm.py -v
```
## What each file covers
Covers authorization, revocation, CSRF, lifecycle/restart failures, Docker ownership, authenticated server-derived room routing, SSE framing, content-only output, cancellation, redacted errors, and wait/final-answer callbacks.
| File | Target | Notes |
|------|--------|-------|
| `test_web_ui.py` | nginx HTTPS UI (:9178) + token endpoint (:8091) | static assets, `/token` (direct + via nginx), `/livekit/` proxy |
| `test_llm_api.py` | LLM API directly | completion, tool calling, `enable_thinking=false` latency, streaming |
| `test_mcp_tools.py` | MCP servers via `docker exec` | weather/time, authenticated remote memory save/recall/delete, and local skills — probes are cleaned up |
| `test_media.py` | external Hope Music boundary + media MCP client | LAN service health, stable room scope, split regression guards, local-container MCP probes |
| `test_agent_integration.py` | running container | agent process alive, worker registered with LiveKit, all 4 supervised processes up, container health, LiveKit HTTP API |
| `test_compaction.py` | `HopeLLM._compact_context` in-container | compaction triggers >24 items, system prompt preserved, short context untouched |
## Legacy tests
## Notes
Top-level `test_*.py` and `conftest.py` describe the former OpenWebUI/MCP architecture and default to live Unraid. **Do not run them as the migrated regression suite.** Retained only as historical references: weather/dispatch/local memory and anonymous-token assumptions no longer apply. Hope Music remains an independent deployment.
- LLM and MCP tests are marked `@pytest.mark.slow` (1–30s each).
- The token endpoint is **POST** `/token` → `{"token": "<JWT>"}`.
- Process checks use `/proc/*/cmdline` because the container image has no
`ps`, and the supervisorctl unix socket is not exposed in this build.
## Live qualification
Explicitly opt in with administrator-created test accounts. Never put passwords or bearer tokens in test source/argv. Confirm anonymous denial, browser login/admin creation, ready private containers, direct Hermes content, tool progress and final reply, normal-user administration denial, account deletion/container removal/session revocation, logout room revocation, and account-scoped history isolation.
API/UI/stream evidence is not physical microphone or iPhone audio/interruption qualification. Recorded results and limitations are in `QUALIFICATION.md`.
Keep credentials in protected ignored runtime files; never print response tokens or Docker environment contents.