feat: provision private Hermes assistants for authenticated Hope accounts
This commit is contained in:
1 parent
da65da68b3
commit
2aedea84b5
33 files changed
+3063
-1743
No files matched your search
@@ -0,0 +1,109 @@
|
||||
# Account control plane
|
||||
|
||||
Run `python -m control` with FastAPI and uvicorn installed. Defaults to
|
||||
`127.0.0.1:8092`; configure `CONTROL_HOST` / `CONTROL_PORT` explicitly if needed.
|
||||
Run one control process against a database (lifecycle scheduling is process-local).
|
||||
No Docker logic lives in the account API: production imports
|
||||
`control.provisioner.Provisioner()` (locally aliased HermesProvisioner) lazily
|
||||
on startup.
|
||||
|
||||
## Runtime settings
|
||||
|
||||
- `CONTROL_DB_PATH`: SQLite path, default `/data/control/accounts.sqlite3`.
|
||||
Mount persistent private storage. The DB contains hashes and Hermes keys;
|
||||
protect the directory and its WAL/SHM files, backups, and host access.
|
||||
- `CONTROL_ADMIN_USERNAME`: initial administrator lowercase slug, default `admin`.
|
||||
- `CONTROL_ADMIN_PASSWORD`: **required when DB is empty**, 12–256 characters;
|
||||
no built-in password. Ignored once existing accounts are present. Do not log it.
|
||||
- `CONTROL_SERVICE_SECRET`: >=32-character secret for trusted internal callers.
|
||||
Missing/short secret disables all internal access; send as `X-Control-Secret`.
|
||||
- `CONTROL_ALLOWED_ORIGINS`: exact comma-separated HTTPS browser origins.
|
||||
**Set this behind nginx**; no forwarded host/proto headers are trusted.
|
||||
|
||||
Native login sends `Content-Type: application/json` and no browser Origin.
|
||||
Browser login requires its actual Origin to match. Cookie mutations require a
|
||||
matching Origin; bearer-authenticated requests are exempt. Login is throttled
|
||||
per validated client IP and username (10 failed attempts / 5 minutes; successes
|
||||
release their reservation). Only a direct loopback peer may supply `X-Real-IP`,
|
||||
which nginx must overwrite with `$remote_addr`. Do not expose the control port
|
||||
or preserve untrusted client-supplied identity headers in another proxy.
|
||||
Opaque sessions last 24 hours, are hashed at rest and revoked by logout,
|
||||
password reset or deletion. Cookies are Secure, HttpOnly, SameSite=Strict.
|
||||
|
||||
## Public API
|
||||
|
||||
- `POST /api/auth/login` JSON `{username,password}` →
|
||||
`{token,user,expires_at}` (expiry Unix seconds) plus `hope_session` cookie.
|
||||
- `POST /api/auth/logout` → `{ok:true}`; invalid sessions return 401.
|
||||
- `GET /api/auth/me` → `{user}`.
|
||||
- Administrator-only `GET /api/admin/users` → `{users:[...]}`.
|
||||
- Administrator-only `POST /api/admin/users` JSON
|
||||
`{username,display_name,password,role?:"user"}` → 201 `{user}`.
|
||||
- Administrator-only `DELETE /api/admin/users/{id}` → 202 `{user}` in
|
||||
deleting state. Sessions/rooms revoked synchronously before return; container
|
||||
deletion is background, durable and retriable. Last administrator is protected.
|
||||
- Administrator-only `POST /api/admin/users/{id}/password` JSON `{password}` →
|
||||
`{ok:true}`; revokes all sessions and room grants.
|
||||
- Administrator-only `POST /api/admin/users/{id}/retry` → 202 `{user}`;
|
||||
retries failed provisioning or removal, preserving intended lifecycle.
|
||||
- `GET /health` → `{status:"ok"}` (control HTTP liveness, not Hermes readiness).
|
||||
|
||||
Public user is exactly `{id,username,display_name,role,status,container_name,error}`.
|
||||
IDs are stable UUID hex. Usernames match `[a-z][a-z0-9]*(?:-[a-z0-9]+)*`,
|
||||
1–48 characters. Status is `provisioning`, `ready`, `error` or `deleting`.
|
||||
Deleting failures remain deleting with a sanitized visible error, never become
|
||||
login-eligible. Background reconcile runs on startup and every 30 seconds.
|
||||
|
||||
## Trusted speech boundary
|
||||
|
||||
Every internal route requires `X-Control-Secret`, including room lookups.
|
||||
|
||||
- `POST /internal/resolve-session` JSON `{token}` validates the exact cookie or
|
||||
bearer token; returns `{user,hermes_url,hermes_api_key}` only for ready user.
|
||||
- `GET /internal/users/{id}` returns the same ready-user metadata.
|
||||
- `POST /internal/rooms` JSON `{user_id,room_name,session_token,ttl?:3600}`
|
||||
validates that session belongs to the user and persists a unique server-minted
|
||||
room grant, returning 201 `{room_name,user_id}`. TTL 1–86400 sec, capped at
|
||||
original session expiry.
|
||||
- `GET /internal/rooms/{room_name}` resolves that grant only while unexpired
|
||||
and user still active/ready. Returns the same metadata. Deletion/reset revoke
|
||||
grants immediately; speech agents should poll to stop active conversations.
|
||||
|
||||
401 means absent/invalid auth; 403 forbidden role/Origin; 404 missing/deleted
|
||||
user or invalid room; 503 means not yet ready. Metadata is never in public APIs.
|
||||
Do not proxy internal endpoints publicly. Token service resolves client sessions,
|
||||
mints an opaque room name, persists its grant before issuing LiveKit credentials,
|
||||
and never accepts browser-supplied account IDs, room ownership or Hermes URLs.
|
||||
For colocated FastAPI integration use `Depends(control.current_user)` and
|
||||
`Depends(control.administrator)`; these return public user dicts and require
|
||||
this application's `app.state.accounts`/settings. Separate services use the
|
||||
secret-authenticated internal endpoints instead.
|
||||
|
||||
## Provisioner interface
|
||||
|
||||
`create_app(settings: Settings, provisioner=...)` supports offline injection.
|
||||
The adapter receives a **private trusted dict**, including `id`, `username`,
|
||||
`container_name`, `desired`, and stored `hermes_url` / `hermes_api_key`.
|
||||
|
||||
- `async ensure(user) -> dict`: idempotently create/recover owned container;
|
||||
return `{container_name,hermes_url,hermes_api_key}`. Container name must match
|
||||
`hope-hermes-<username>-<id[:8]>`; metadata is persisted before readiness check.
|
||||
- `async ready(user) -> bool`: owned service health, not existence alone.
|
||||
- `async remove(user) -> None`: verify management/user ownership before any
|
||||
removal; retain private data by default; failure raises and remains retriable.
|
||||
- Optional `async aclose()`: close adapter resources during shutdown.
|
||||
|
||||
Async adapters offload blocking Docker/HTTP calls themselves; synchronous
|
||||
adapters are supported using a worker thread. Account SQLite and scrypt work
|
||||
are offloaded from the ASGI loop. Slow provision/remove run separate tasks;
|
||||
request handlers do not await container startup. Exception text is not logged or
|
||||
returned because provider failures may contain credentials.
|
||||
|
||||
Offline tests:
|
||||
|
||||
```sh
|
||||
python -m pytest tests/unit/test_accounts.py --confcutdir=tests/unit -q
|
||||
```
|
||||
|
||||
`--confcutdir` avoids this repository's deployment-oriented top-level fixtures.
|
||||
Use a temporary DB and fake adapter; no Docker or provider calls are made.
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Trusted account and per-user Hermes control plane."""
|
||||
|
||||
from .app import Settings, create_app, current_user, administrator
|
||||
|
||||
__all__ = ["Settings", "create_app", "current_user", "administrator"]
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Run `python -m control`; never exposes the control plane on LAN by default."""
|
||||
import os
|
||||
import uvicorn
|
||||
|
||||
from .app import create_app
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Access logs include path identifiers; disable to avoid accidental secret
|
||||
# disclosure if a malformed client includes credentials in a URL.
|
||||
uvicorn.run(create_app(), host=os.getenv('CONTROL_HOST', '127.0.0.1'),
|
||||
port=int(os.getenv('CONTROL_PORT', '8092')), access_log=False,
|
||||
proxy_headers=False)
|
||||
@@ -0,0 +1,204 @@
|
||||
"""SQLite persistence. No credential-bearing row is ever a public API schema."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
PUBLIC_FIELDS = ('id', 'username', 'display_name', 'role', 'status', 'container_name', 'error')
|
||||
SLUG = re.compile(r'[a-z][a-z0-9]*(?:-[a-z0-9]+)*\Z')
|
||||
|
||||
|
||||
def public_user(user):
|
||||
return {key: user[key] for key in PUBLIC_FIELDS}
|
||||
|
||||
|
||||
def validate_username(value):
|
||||
if not isinstance(value, str) or not 1 <= len(value) <= 48 or not SLUG.fullmatch(value):
|
||||
raise ValueError('Username must be a lowercase slug (1–48 characters)')
|
||||
return value
|
||||
|
||||
|
||||
def validate_password(value):
|
||||
if not isinstance(value, str) or not 12 <= len(value) <= 256 or len(value.encode('utf-8')) > 1024:
|
||||
raise ValueError('Password must contain 12–256 characters')
|
||||
return value
|
||||
|
||||
|
||||
def password_hash(password):
|
||||
salt = secrets.token_bytes(16)
|
||||
digest = hashlib.scrypt(password.encode(), salt=salt, n=16384, r=8, p=1, dklen=32)
|
||||
return f'scrypt$16384$8$1${salt.hex()}${digest.hex()}'
|
||||
|
||||
|
||||
def password_matches(password, encoded):
|
||||
try:
|
||||
_, n, r, p, salt, digest = encoded.split('$')
|
||||
actual = hashlib.scrypt(password.encode(), salt=bytes.fromhex(salt), n=int(n), r=int(r), p=int(p), dklen=32)
|
||||
return hmac.compare_digest(actual.hex(), digest)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
|
||||
|
||||
def token_hash(token):
|
||||
return hashlib.sha256(token.encode()).hexdigest()
|
||||
|
||||
|
||||
class Accounts:
|
||||
def __init__(self, path):
|
||||
if path != ':memory:':
|
||||
Path(path).parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
self.lock = threading.RLock()
|
||||
if path != ':memory:':
|
||||
# Create privately before SQLite can create journal companions.
|
||||
fd = os.open(path, os.O_CREAT | os.O_RDWR, 0o600)
|
||||
os.close(fd)
|
||||
os.chmod(path, 0o600)
|
||||
self.db = sqlite3.connect(path, check_same_thread=False)
|
||||
self.db.row_factory = sqlite3.Row
|
||||
self.db.execute('PRAGMA foreign_keys=ON')
|
||||
self.db.execute('PRAGMA busy_timeout=5000')
|
||||
self.db.execute('PRAGMA journal_mode=WAL')
|
||||
self.db.executescript('''
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id TEXT PRIMARY KEY, username TEXT NOT NULL UNIQUE,
|
||||
display_name TEXT NOT NULL, role TEXT NOT NULL,
|
||||
status TEXT NOT NULL, container_name TEXT NOT NULL,
|
||||
error TEXT, password_hash TEXT NOT NULL,
|
||||
hermes_url TEXT, hermes_api_key TEXT, desired TEXT NOT NULL DEFAULT 'ready'
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
hash TEXT PRIMARY KEY, user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at REAL NOT NULL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS rooms (
|
||||
room_name TEXT PRIMARY KEY, user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at REAL NOT NULL, session_hash TEXT
|
||||
);
|
||||
''')
|
||||
if 'session_hash' not in {r[1] for r in self.db.execute('PRAGMA table_info(rooms)')}:
|
||||
self.db.execute('ALTER TABLE rooms ADD COLUMN session_hash TEXT')
|
||||
self.db.commit()
|
||||
if path != ':memory:':
|
||||
os.chmod(path, 0o600)
|
||||
for suffix in ('-wal', '-shm'):
|
||||
companion = Path(path + suffix)
|
||||
if companion.exists():
|
||||
os.chmod(companion, 0o600)
|
||||
|
||||
def close(self):
|
||||
self.db.close()
|
||||
|
||||
def users(self):
|
||||
with self.lock:
|
||||
return [dict(r) for r in self.db.execute('SELECT * FROM users ORDER BY username')]
|
||||
|
||||
def get(self, user_id):
|
||||
with self.lock:
|
||||
row = self.db.execute('SELECT * FROM users WHERE id=?', (user_id,)).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
def by_username(self, username):
|
||||
with self.lock:
|
||||
row = self.db.execute('SELECT * FROM users WHERE username=?', (username,)).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
def create(self, username, display_name, role, password):
|
||||
validate_username(username)
|
||||
validate_password(password)
|
||||
if role not in ('admin', 'user'):
|
||||
raise ValueError('Invalid role')
|
||||
if not isinstance(display_name, str) or not 1 <= len(display_name.strip()) <= 100:
|
||||
raise ValueError('Display name must contain 1–100 characters')
|
||||
user_id = uuid.uuid4().hex
|
||||
hashed = password_hash(password)
|
||||
with self.lock, self.db:
|
||||
self.db.execute('INSERT INTO users(id,username,display_name,role,status,container_name,password_hash) VALUES(?,?,?,?,?,?,?)',
|
||||
(user_id, username, display_name.strip(), role, 'provisioning', f'hope-hermes-{username}-{user_id[:8]}', hashed))
|
||||
return self.get(user_id)
|
||||
|
||||
def update(self, user_id, **values):
|
||||
allowed = {'status', 'error', 'hermes_url', 'hermes_api_key', 'container_name'}
|
||||
if not values or not set(values) <= allowed:
|
||||
raise ValueError('Invalid account update')
|
||||
with self.lock, self.db:
|
||||
guard = " AND desired!='deleted'" if values.get('status') in ('ready', 'provisioning', 'error') else ''
|
||||
self.db.execute('UPDATE users SET ' + ','.join(f'{k}=?' for k in values) + ' WHERE id=?' + guard, (*values.values(), user_id))
|
||||
|
||||
def begin_delete(self, user_id):
|
||||
with self.lock, self.db:
|
||||
user = self.get(user_id)
|
||||
if user is None:
|
||||
raise KeyError(user_id)
|
||||
if user['role'] == 'admin' and user['desired'] != 'deleted':
|
||||
count = self.db.execute("SELECT count(*) FROM users WHERE role='admin' AND desired!='deleted'").fetchone()[0]
|
||||
if count <= 1:
|
||||
raise ValueError('Cannot delete the last administrator')
|
||||
self.db.execute("UPDATE users SET desired='deleted',status='deleting',error=NULL WHERE id=?", (user_id,))
|
||||
self.db.execute('DELETE FROM sessions WHERE user_id=?', (user_id,))
|
||||
self.db.execute('DELETE FROM rooms WHERE user_id=?', (user_id,))
|
||||
return self.get(user_id)
|
||||
|
||||
def finish_delete(self, user_id):
|
||||
with self.lock, self.db:
|
||||
self.db.execute("DELETE FROM users WHERE id=? AND desired='deleted'", (user_id,))
|
||||
|
||||
def reset_password(self, user_id, password):
|
||||
hashed = password_hash(validate_password(password))
|
||||
with self.lock, self.db:
|
||||
user = self.get(user_id)
|
||||
if user is None or user['desired'] == 'deleted':
|
||||
raise KeyError(user_id)
|
||||
self.db.execute('UPDATE users SET password_hash=? WHERE id=?', (hashed, user_id))
|
||||
self.db.execute('DELETE FROM sessions WHERE user_id=?', (user_id,))
|
||||
self.db.execute('DELETE FROM rooms WHERE user_id=?', (user_id,))
|
||||
|
||||
def issue_session(self, user_id, ttl, expected_password_hash):
|
||||
token = secrets.token_urlsafe(32)
|
||||
expires = time.time() + ttl
|
||||
with self.lock, self.db:
|
||||
self.db.execute('DELETE FROM sessions WHERE expires_at<=?', (time.time(),))
|
||||
# Single SQL statement atomically compares the verified credential
|
||||
# and inserts; reset/delete cannot interleave with this check.
|
||||
cursor = self.db.execute("INSERT INTO sessions(hash,user_id,expires_at) SELECT ?,id,? FROM users WHERE id=? AND desired!='deleted' AND password_hash=?",
|
||||
(token_hash(token), expires, user_id, expected_password_hash))
|
||||
if cursor.rowcount != 1:
|
||||
raise ValueError('Account credentials changed or unavailable')
|
||||
return token, expires
|
||||
|
||||
def resolve_session(self, token):
|
||||
if not isinstance(token, str) or not 20 <= len(token) <= 256:
|
||||
return None
|
||||
with self.lock:
|
||||
row = self.db.execute("SELECT u.* FROM sessions s JOIN users u ON u.id=s.user_id WHERE s.hash=? AND s.expires_at>? AND u.desired!='deleted'", (token_hash(token), time.time())).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
def revoke(self, token):
|
||||
with self.lock, self.db:
|
||||
self.db.execute('DELETE FROM sessions WHERE hash=?', (token_hash(token),))
|
||||
|
||||
def grant_room(self, user_id, room_name, ttl, session_token):
|
||||
with self.lock, self.db:
|
||||
user = self.get(user_id)
|
||||
if not user or user['desired'] == 'deleted' or user['status'] != 'ready':
|
||||
raise ValueError('Account not ready')
|
||||
if not isinstance(session_token, str) or not 20 <= len(session_token) <= 256:
|
||||
raise ValueError('Invalid session')
|
||||
hashed = token_hash(session_token)
|
||||
session = self.db.execute('SELECT expires_at FROM sessions WHERE hash=? AND user_id=? AND expires_at>?', (hashed, user_id, time.time())).fetchone()
|
||||
if session is None:
|
||||
raise ValueError('Invalid session')
|
||||
self.db.execute('DELETE FROM rooms WHERE expires_at<=?', (time.time(),))
|
||||
self.db.execute('INSERT INTO rooms(room_name,user_id,expires_at,session_hash) VALUES(?,?,?,?)', (room_name, user_id, min(time.time() + ttl, session['expires_at']), hashed))
|
||||
|
||||
def resolve_room(self, room_name):
|
||||
with self.lock:
|
||||
row = self.db.execute("SELECT u.* FROM rooms r JOIN users u ON r.user_id=u.id JOIN sessions s ON s.hash=r.session_hash AND s.user_id=u.id WHERE r.room_name=? AND r.expires_at>? AND s.expires_at>? AND u.desired!='deleted' AND u.status='ready'", (room_name, time.time(), time.time())).fetchone()
|
||||
return dict(row) if row else None
|
||||
+408
@@ -0,0 +1,408 @@
|
||||
"""Account HTTP API; trusted adapters alone hold Hermes connection secrets."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hmac
|
||||
import inspect
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import time
|
||||
from collections import defaultdict, deque
|
||||
from contextlib import asynccontextmanager
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Protocol
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import Depends, FastAPI, HTTPException, Request, Response
|
||||
|
||||
from .accounts import Accounts, password_hash, password_matches, public_user, validate_password, validate_username
|
||||
|
||||
COOKIE = 'hope_session'
|
||||
MAX_BODY = 8192
|
||||
|
||||
|
||||
class Provisioner(Protocol):
|
||||
"""All methods async. remove MUST verify Hope/user ownership before deletion.
|
||||
|
||||
ensure is idempotent and returns {container_name, hermes_url, hermes_api_key}.
|
||||
ready checks the owned Hermes API health, not just container existence.
|
||||
Private home/workspace must be retained by remove unless explicitly erased.
|
||||
"""
|
||||
async def ensure(self, user: dict) -> dict: ...
|
||||
async def remove(self, user: dict) -> None: ...
|
||||
async def ready(self, user: dict) -> bool: ...
|
||||
|
||||
|
||||
@dataclass
|
||||
class Settings:
|
||||
db_path: str = field(default_factory=lambda: os.getenv('CONTROL_DB_PATH', '/data/control/accounts.sqlite3'))
|
||||
admin_username: str = field(default_factory=lambda: os.getenv('CONTROL_ADMIN_USERNAME', 'admin'))
|
||||
admin_password: str | None = field(default_factory=lambda: os.getenv('CONTROL_ADMIN_PASSWORD'))
|
||||
service_secret: str | None = field(default_factory=lambda: os.getenv('CONTROL_SERVICE_SECRET'))
|
||||
allowed_origins: tuple[str, ...] = field(default_factory=lambda: tuple(x.strip().rstrip('/') for x in os.getenv('CONTROL_ALLOWED_ORIGINS', '').split(',') if x.strip()))
|
||||
session_ttl: int = 86400
|
||||
reconcile_interval: float = 30
|
||||
login_limit: int = 10
|
||||
login_window: int = 300
|
||||
|
||||
|
||||
async def bounded_json(request):
|
||||
if request.headers.get('content-type', '').split(';', 1)[0].strip().lower() != 'application/json':
|
||||
raise HTTPException(415, 'Content-Type application/json required')
|
||||
data = bytearray()
|
||||
async for chunk in request.stream():
|
||||
data.extend(chunk)
|
||||
if len(data) > MAX_BODY:
|
||||
raise HTTPException(413, 'Request body too large')
|
||||
try:
|
||||
value = json.loads(data)
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
raise HTTPException(400, 'Invalid JSON') from None
|
||||
if not isinstance(value, dict):
|
||||
raise HTTPException(400, 'JSON object required')
|
||||
return value
|
||||
|
||||
|
||||
def bearer_token(request):
|
||||
authorization = request.headers.get('authorization')
|
||||
if authorization is None:
|
||||
return None
|
||||
parts = authorization.split()
|
||||
if len(parts) != 2 or parts[0].lower() != 'bearer' or not 20 <= len(parts[1]) <= 256:
|
||||
raise HTTPException(401, 'Invalid authorization')
|
||||
return parts[1]
|
||||
|
||||
|
||||
def check_origin(request, *, login=False):
|
||||
"""No forwarded-header trust; explicit origins required behind TLS proxy.
|
||||
|
||||
Native bearer clients are exempt. Login without Origin is allowed only for
|
||||
JSON native clients with no session cookie and no browser Fetch Metadata.
|
||||
"""
|
||||
if bearer_token(request) is not None:
|
||||
return
|
||||
origin = request.headers.get('origin')
|
||||
if not origin:
|
||||
if login and not request.cookies.get(COOKIE) and not request.headers.get('sec-fetch-site'):
|
||||
return
|
||||
raise HTTPException(403, 'Origin required')
|
||||
settings = request.app.state.settings
|
||||
allowed = settings.allowed_origins or (str(request.base_url).rstrip('/'),)
|
||||
if origin not in allowed or origin == 'null':
|
||||
raise HTTPException(403, 'Origin not allowed')
|
||||
|
||||
|
||||
async def current_user(request: Request):
|
||||
token = bearer_token(request) or request.cookies.get(COOKIE)
|
||||
if not token:
|
||||
raise HTTPException(401, 'Authentication required')
|
||||
user = await asyncio.to_thread(request.app.state.accounts.resolve_session, token)
|
||||
if user is None:
|
||||
raise HTTPException(401, 'Session invalid or expired')
|
||||
if request.method not in ('GET', 'HEAD', 'OPTIONS'):
|
||||
check_origin(request)
|
||||
request.state.session_token = token
|
||||
return public_user(user)
|
||||
|
||||
|
||||
async def administrator(user=Depends(current_user)):
|
||||
if user['role'] != 'admin':
|
||||
raise HTTPException(403, 'Administrator required')
|
||||
return user
|
||||
|
||||
|
||||
def internal_authorized(request: Request):
|
||||
secret = request.app.state.settings.service_secret
|
||||
supplied = request.headers.get('x-control-secret', '')
|
||||
if not secret or len(secret) < 32 or not supplied or not hmac.compare_digest(secret.encode('utf-8'), supplied.encode('utf-8')):
|
||||
raise HTTPException(401, 'Internal authorization required')
|
||||
|
||||
|
||||
class Lifecycle:
|
||||
def __init__(self, accounts, provisioner):
|
||||
self.accounts = accounts
|
||||
self.provisioner = provisioner
|
||||
self.tasks = {}
|
||||
self.closed = False
|
||||
|
||||
def schedule(self, user_id):
|
||||
if not self.closed and (user_id not in self.tasks or self.tasks[user_id].done()):
|
||||
self.tasks[user_id] = asyncio.create_task(self.run(user_id))
|
||||
|
||||
async def call(self, method, user):
|
||||
# Native async adapters must offload their own blocking Docker/HTTP calls.
|
||||
# Also tolerate synchronous test/legacy adapters without blocking ASGI.
|
||||
func = getattr(self.provisioner, method)
|
||||
if inspect.iscoroutinefunction(func):
|
||||
return await func(user)
|
||||
result = await asyncio.to_thread(func, user)
|
||||
return await result if inspect.isawaitable(result) else result
|
||||
|
||||
async def run(self, user_id):
|
||||
user = await asyncio.to_thread(self.accounts.get, user_id)
|
||||
if user is None:
|
||||
return
|
||||
try:
|
||||
if user['desired'] == 'deleted':
|
||||
await self.call('remove', user)
|
||||
await asyncio.to_thread(self.accounts.finish_delete, user_id)
|
||||
return
|
||||
await asyncio.to_thread(self.accounts.update, user_id, status='provisioning', error=None)
|
||||
metadata = await self.call('ensure', user)
|
||||
if not isinstance(metadata, dict):
|
||||
raise ValueError('Invalid provisioning metadata')
|
||||
url = metadata.get('hermes_url')
|
||||
key = metadata.get('hermes_api_key')
|
||||
parsed = urlsplit(url or '')
|
||||
if parsed.scheme not in ('http', 'https') or not parsed.hostname or not isinstance(key, str) or not key:
|
||||
raise ValueError('Invalid provisioning metadata')
|
||||
# Deterministic name cannot be overridden by adapter metadata.
|
||||
if metadata.get('container_name', user['container_name']) != user['container_name']:
|
||||
raise ValueError('Unexpected container ownership')
|
||||
await asyncio.to_thread(self.accounts.update, user_id, hermes_url=url, hermes_api_key=key)
|
||||
user.update(hermes_url=url, hermes_api_key=key)
|
||||
if not await self.call('ready', user):
|
||||
raise RuntimeError('Hermes not healthy')
|
||||
current = await asyncio.to_thread(self.accounts.get, user_id)
|
||||
if current and current['desired'] == 'deleted':
|
||||
await self.call('remove', current)
|
||||
await asyncio.to_thread(self.accounts.finish_delete, user_id)
|
||||
else:
|
||||
await asyncio.to_thread(self.accounts.update, user_id, status='ready', error=None)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
# Exception messages can contain upstream URLs, keys or Docker env.
|
||||
current = await asyncio.to_thread(self.accounts.get, user_id)
|
||||
if current:
|
||||
deleting = current['desired'] == 'deleted'
|
||||
await asyncio.to_thread(self.accounts.update, user_id, status='deleting' if deleting else 'error', error='Container removal failed; retry pending' if deleting else 'Hermes provisioning failed; retry pending')
|
||||
|
||||
async def reconcile(self):
|
||||
for user in await asyncio.to_thread(self.accounts.users):
|
||||
if user['desired'] == 'deleted' or user['status'] != 'ready':
|
||||
self.schedule(user['id'])
|
||||
elif user['id'] not in self.tasks or self.tasks[user['id']].done():
|
||||
try:
|
||||
healthy = await self.call('ready', user)
|
||||
except Exception:
|
||||
healthy = False
|
||||
if not healthy:
|
||||
await asyncio.to_thread(self.accounts.update, user['id'], status='error', error='Hermes health check failed; retry pending')
|
||||
self.schedule(user['id'])
|
||||
|
||||
async def loop(self, interval):
|
||||
while True:
|
||||
await self.reconcile()
|
||||
await asyncio.sleep(interval)
|
||||
|
||||
async def stop(self):
|
||||
self.closed = True
|
||||
tasks = list(self.tasks.values())
|
||||
for task in tasks:
|
||||
task.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
|
||||
def create_app(settings=None, provisioner=None):
|
||||
settings = settings or Settings()
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
store = Accounts(settings.db_path)
|
||||
app.state.accounts = store
|
||||
try:
|
||||
if not store.users():
|
||||
if not settings.admin_password:
|
||||
raise RuntimeError('CONTROL_ADMIN_PASSWORD required to bootstrap an empty account database')
|
||||
await asyncio.to_thread(store.create, settings.admin_username, 'Administrator', 'admin', settings.admin_password)
|
||||
adapter = provisioner
|
||||
if adapter is None:
|
||||
from .provisioner import Provisioner as HermesProvisioner
|
||||
adapter = HermesProvisioner()
|
||||
app.state.lifecycle = Lifecycle(store, adapter)
|
||||
# Revalidate even previously-ready containers after control restart.
|
||||
for user in store.users():
|
||||
app.state.lifecycle.schedule(user['id'])
|
||||
app.state.dummy_hash = await asyncio.to_thread(password_hash, 'timing-only-password')
|
||||
loop = asyncio.create_task(app.state.lifecycle.loop(settings.reconcile_interval))
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
loop.cancel()
|
||||
await asyncio.gather(loop, return_exceptions=True)
|
||||
await app.state.lifecycle.stop()
|
||||
if hasattr(adapter, 'aclose'):
|
||||
await adapter.aclose()
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
app = FastAPI(title='Hope account control', lifespan=lifespan, docs_url=None, redoc_url=None, openapi_url=None)
|
||||
app.state.settings = settings
|
||||
attempts = defaultdict(deque)
|
||||
|
||||
@app.middleware('http')
|
||||
async def no_store(request, call_next):
|
||||
response = await call_next(request)
|
||||
response.headers['Cache-Control'] = 'no-store'
|
||||
return response
|
||||
|
||||
@app.post('/api/auth/login')
|
||||
async def login(request: Request, response: Response):
|
||||
check_origin(request, login=True)
|
||||
body = await bounded_json(request)
|
||||
try:
|
||||
username = validate_username(body.get('username'))
|
||||
password = validate_password(body.get('password'))
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc)) from None
|
||||
peer = request.client.host if request.client else 'unknown'
|
||||
# nginx /api/ overwrites X-Real-IP with $remote_addr. Only its direct
|
||||
# loopback connection is trusted; never trust X-Forwarded-For.
|
||||
try:
|
||||
if ipaddress.ip_address(peer).is_loopback:
|
||||
forwarded = request.headers.get('x-real-ip')
|
||||
if forwarded:
|
||||
peer = str(ipaddress.ip_address(forwarded))
|
||||
except ValueError:
|
||||
pass
|
||||
now = time.monotonic()
|
||||
for key in list(attempts):
|
||||
bucket = attempts[key]
|
||||
while bucket and bucket[0] <= now - settings.login_window:
|
||||
bucket.popleft()
|
||||
if not bucket:
|
||||
del attempts[key]
|
||||
scope = (peer, username)
|
||||
bucket = attempts.get(scope)
|
||||
if bucket is not None and len(bucket) >= settings.login_limit:
|
||||
raise HTTPException(429, 'Too many login attempts', headers={'Retry-After': str(settings.login_window)})
|
||||
if bucket is None:
|
||||
if len(attempts) >= 4096:
|
||||
raise HTTPException(429, 'Too many login attempts')
|
||||
bucket = attempts[scope]
|
||||
# Reserve a slot while scrypt is in flight to prevent concurrent bypass.
|
||||
# Successful requests release their own slot; only failures persist.
|
||||
slot = time.monotonic()
|
||||
bucket.append(slot)
|
||||
store = app.state.accounts
|
||||
user = await asyncio.to_thread(store.by_username, username)
|
||||
valid = await asyncio.to_thread(password_matches, password, user['password_hash'] if user else app.state.dummy_hash)
|
||||
if not valid or not user or user['desired'] == 'deleted':
|
||||
raise HTTPException(401, 'Invalid username or password')
|
||||
try:
|
||||
token, expires = await asyncio.to_thread(store.issue_session, user['id'], settings.session_ttl, user['password_hash'])
|
||||
except ValueError:
|
||||
raise HTTPException(401, 'Account unavailable') from None
|
||||
try:
|
||||
bucket.remove(slot)
|
||||
except ValueError:
|
||||
pass # Expired while the request was in flight.
|
||||
if not bucket and attempts.get(scope) is bucket:
|
||||
attempts.pop(scope, None)
|
||||
response.set_cookie(COOKIE, token, max_age=settings.session_ttl, httponly=True, secure=True, samesite='strict', path='/')
|
||||
return {'token': token, 'user': public_user(user), 'expires_at': expires}
|
||||
|
||||
@app.post('/api/auth/logout')
|
||||
async def logout(request: Request, response: Response, user=Depends(current_user)):
|
||||
await asyncio.to_thread(app.state.accounts.revoke, request.state.session_token)
|
||||
response.delete_cookie(COOKIE, path='/', secure=True, httponly=True, samesite='strict')
|
||||
return {'ok': True}
|
||||
|
||||
@app.get('/api/auth/me')
|
||||
async def me(user=Depends(current_user)):
|
||||
return {'user': user}
|
||||
|
||||
@app.get('/api/admin/users')
|
||||
async def list_users(user=Depends(administrator)):
|
||||
return {'users': [public_user(u) for u in await asyncio.to_thread(app.state.accounts.users)]}
|
||||
|
||||
@app.post('/api/admin/users', status_code=201)
|
||||
async def create_user(request: Request, admin=Depends(administrator)):
|
||||
body = await bounded_json(request)
|
||||
try:
|
||||
user = await asyncio.to_thread(app.state.accounts.create, body.get('username'), body.get('display_name'), body.get('role', 'user'), body.get('password'))
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc)) from None
|
||||
except sqlite3.IntegrityError:
|
||||
raise HTTPException(409, 'Username already exists') from None
|
||||
app.state.lifecycle.schedule(user['id'])
|
||||
return {'user': public_user(user)}
|
||||
|
||||
@app.delete('/api/admin/users/{user_id}', status_code=202)
|
||||
async def delete_user(user_id: str, admin=Depends(administrator)):
|
||||
try:
|
||||
user = await asyncio.to_thread(app.state.accounts.begin_delete, user_id)
|
||||
except KeyError:
|
||||
raise HTTPException(404, 'User not found') from None
|
||||
except ValueError as exc:
|
||||
raise HTTPException(409, str(exc)) from None
|
||||
app.state.lifecycle.schedule(user_id)
|
||||
return {'user': public_user(user)}
|
||||
|
||||
@app.post('/api/admin/users/{user_id}/password')
|
||||
async def reset_password(user_id: str, request: Request, admin=Depends(administrator)):
|
||||
body = await bounded_json(request)
|
||||
try:
|
||||
await asyncio.to_thread(app.state.accounts.reset_password, user_id, body.get('password'))
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc)) from None
|
||||
except KeyError:
|
||||
raise HTTPException(404, 'User not found') from None
|
||||
return {'ok': True}
|
||||
|
||||
@app.post('/api/admin/users/{user_id}/retry', status_code=202)
|
||||
async def retry(user_id: str, admin=Depends(administrator)):
|
||||
user = await asyncio.to_thread(app.state.accounts.get, user_id)
|
||||
if not user:
|
||||
raise HTTPException(404, 'User not found')
|
||||
app.state.lifecycle.schedule(user_id)
|
||||
return {'user': public_user(user)}
|
||||
|
||||
def connection(user):
|
||||
if not user or user['desired'] == 'deleted':
|
||||
raise HTTPException(404, 'User not found')
|
||||
if user['status'] != 'ready' or not user['hermes_url'] or not user['hermes_api_key']:
|
||||
raise HTTPException(503, 'User Hermes is not ready')
|
||||
return {'user': public_user(user), 'hermes_url': user['hermes_url'], 'hermes_api_key': user['hermes_api_key']}
|
||||
|
||||
@app.get('/internal/users/{user_id}', dependencies=[Depends(internal_authorized)])
|
||||
async def internal_user(user_id: str):
|
||||
return connection(await asyncio.to_thread(app.state.accounts.get, user_id))
|
||||
|
||||
@app.post('/internal/resolve-session', dependencies=[Depends(internal_authorized)])
|
||||
async def resolve_session(request: Request):
|
||||
body = await bounded_json(request)
|
||||
user = await asyncio.to_thread(app.state.accounts.resolve_session, body.get('token'))
|
||||
if user is None:
|
||||
raise HTTPException(401, 'Session invalid or expired')
|
||||
return connection(user)
|
||||
|
||||
@app.post('/internal/rooms', dependencies=[Depends(internal_authorized)], status_code=201)
|
||||
async def create_room(request: Request):
|
||||
body = await bounded_json(request)
|
||||
user_id, room = body.get('user_id'), body.get('room_name')
|
||||
ttl = body.get('ttl', 3600)
|
||||
if not isinstance(user_id, str) or not isinstance(room, str) or not 1 <= len(room) <= 128 or not all(c.isascii() and (c.isalnum() or c in '-_') for c in room):
|
||||
raise HTTPException(400, 'Invalid room grant')
|
||||
if type(ttl) is not int or not 1 <= ttl <= 86400:
|
||||
raise HTTPException(400, 'Invalid room expiry')
|
||||
try:
|
||||
await asyncio.to_thread(app.state.accounts.grant_room, user_id, room, ttl, body.get('session_token'))
|
||||
except ValueError:
|
||||
raise HTTPException(409, 'Account not ready') from None
|
||||
except sqlite3.IntegrityError:
|
||||
raise HTTPException(409, 'Room already granted') from None
|
||||
return {'room_name': room, 'user_id': user_id}
|
||||
|
||||
@app.get('/internal/rooms/{room_name}', dependencies=[Depends(internal_authorized)])
|
||||
async def resolve_room(room_name: str):
|
||||
return connection(await asyncio.to_thread(app.state.accounts.resolve_room, room_name))
|
||||
|
||||
@app.get('/health')
|
||||
async def health():
|
||||
return {'status': 'ok'}
|
||||
|
||||
return app
|
||||
@@ -0,0 +1,313 @@
|
||||
"""Trusted Docker-UDS control plane for independent Hermes accounts.
|
||||
|
||||
Only this service may receive the host Docker socket. Bind-source paths must be
|
||||
identical in the control service and Docker host (mount the runtime root there).
|
||||
The official image's root/s6 boot is deliberately bypassed: a nonroot Python
|
||||
bootstrap seeds bundled skills, then execs the verified .venv Hermes CLI. Image
|
||||
compatibility still needs deployment qualification before an image pin is promoted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import secrets
|
||||
import tempfile
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
class ProvisioningError(RuntimeError):
|
||||
"""Safe-to-display provisioning failure (never includes Docker env/secrets)."""
|
||||
|
||||
|
||||
class OwnershipError(ProvisioningError):
|
||||
"""An existing resource is not owned by this account."""
|
||||
|
||||
|
||||
# Run as the image's hermes UID without executing root-only s6 startup. No source
|
||||
# checkout/profile is copied: only the image's public bundled skills are seeded.
|
||||
_BOOTSTRAP = """
|
||||
import os, pathlib, shutil
|
||||
home = pathlib.Path('/opt/data')
|
||||
for name in ('home', 'sessions', 'memories', 'skills', 'cron', 'hooks', 'logs', 'cache'):
|
||||
(home / name).mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
source = pathlib.Path('/opt/hermes/skills')
|
||||
if source.is_dir() and not (home / '.hope-skills-seeded').exists():
|
||||
shutil.copytree(source, home / 'skills', dirs_exist_ok=True)
|
||||
(home / '.hope-skills-seeded').touch(mode=0o600)
|
||||
os.makedirs('/tmp/hermes-runtime', mode=0o700, exist_ok=True)
|
||||
os.execv('/opt/hermes/.venv/bin/hermes', ['hermes', 'gateway', 'run'])
|
||||
"""
|
||||
|
||||
|
||||
def container_name(user: dict) -> str:
|
||||
uid, username = user.get('id'), user.get('username')
|
||||
if not isinstance(uid, str) or not re.fullmatch(r'[0-9a-f]{32}', uid):
|
||||
raise ValueError('Account id must be a lowercase UUID hex string')
|
||||
if not isinstance(username, str) or not re.fullmatch(r'[a-z][a-z0-9]*(?:-[a-z0-9]+)*', username) or len(username) > 48:
|
||||
raise ValueError('Account username must be a lowercase slug')
|
||||
return f'hope-hermes-{username}-{uid[:8]}'
|
||||
|
||||
|
||||
class Provisioner:
|
||||
def __init__(self, state_dir=None, *, docker_socket=None, image=None,
|
||||
upstream_api_key=None, docker_client=None, health_client=None,
|
||||
ready_timeout=60, poll_interval=.5, runtime_uid=10000,
|
||||
runtime_gid=10000, memory_bytes=2 * 1024 ** 3, nano_cpus=2_000_000_000):
|
||||
self.state_dir = Path(state_dir or os.environ.get('HOPE_HERMES_DATA_DIR', '.runtime/hermes')).absolute()
|
||||
self.image = image or os.environ.get('HERMES_IMAGE', 'nousresearch/hermes-agent@sha256:d4da4a40cd7a28aba983775d9fd31d94cbf153eeb0cb9e844d6d0f612b7c24db')
|
||||
self.upstream_api_key = upstream_api_key if upstream_api_key is not None else os.environ.get('HERMES_UPSTREAM_API_KEY', '')
|
||||
self.ready_timeout, self.poll_interval = ready_timeout, poll_interval
|
||||
self.runtime_uid, self.runtime_gid = runtime_uid, runtime_gid
|
||||
if runtime_uid <= 0 or runtime_gid <= 0:
|
||||
raise ValueError('Hermes must run nonroot')
|
||||
self.memory_bytes, self.nano_cpus = memory_bytes, nano_cpus
|
||||
self._own_docker, self._own_health = docker_client is None, health_client is None
|
||||
# No TCP Docker endpoint, DOCKER_HOST, subprocess, or proxy inheritance.
|
||||
self.docker = docker_client or httpx.AsyncClient(
|
||||
transport=httpx.AsyncHTTPTransport(uds=docker_socket or os.environ.get('DOCKER_SOCKET', '/var/run/docker.sock')),
|
||||
base_url='http://docker', timeout=20, trust_env=False)
|
||||
self.health = health_client or httpx.AsyncClient(timeout=3, trust_env=False, follow_redirects=False)
|
||||
self._locks = {}
|
||||
|
||||
async def aclose(self):
|
||||
if self._own_docker:
|
||||
await self.docker.aclose()
|
||||
if self._own_health:
|
||||
await self.health.aclose()
|
||||
|
||||
def _lock(self, user):
|
||||
container_name(user)
|
||||
return self._locks.setdefault(user['id'], asyncio.Lock())
|
||||
|
||||
async def _request(self, method, path, *, allowed=(200,), **kwargs):
|
||||
try:
|
||||
response = await self.docker.request(method, path, **kwargs)
|
||||
except httpx.HTTPError as exc:
|
||||
raise ProvisioningError('Docker control socket unavailable; retry provisioning') from exc
|
||||
if response.status_code not in allowed:
|
||||
raise ProvisioningError(f'Docker operation failed (HTTP {response.status_code}); retry provisioning')
|
||||
return response
|
||||
|
||||
async def _inspect(self, name):
|
||||
response = await self._request('GET', f'/containers/{name}/json', allowed=(200, 404))
|
||||
return response.json() if response.status_code == 200 else None
|
||||
|
||||
@staticmethod
|
||||
def _owned(resource, user):
|
||||
labels = resource.get('Config', {}).get('Labels') or resource.get('Labels') or {}
|
||||
if labels.get('hope.managed') != 'true' or labels.get('hope.user_id') != user['id']:
|
||||
raise OwnershipError('Refusing resource whose Hope ownership labels do not match')
|
||||
|
||||
@staticmethod
|
||||
def _directory(path):
|
||||
# Do not follow a user-created symlink into another account's state.
|
||||
if path.is_symlink():
|
||||
raise ProvisioningError('Private runtime directory cannot be a symlink')
|
||||
path.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
os.chmod(path, 0o700)
|
||||
|
||||
@staticmethod
|
||||
def _write_private(path, value):
|
||||
if path.is_symlink():
|
||||
raise ProvisioningError('Private runtime file cannot be a symlink')
|
||||
fd, temp = tempfile.mkstemp(prefix='.hope-', dir=path.parent)
|
||||
try:
|
||||
with os.fdopen(fd, 'w') as stream:
|
||||
stream.write(value)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temp, path)
|
||||
finally:
|
||||
if os.path.exists(temp):
|
||||
temporary = Path(temp)
|
||||
if temporary.resolve().parent != path.parent.resolve() or not temporary.name.startswith('.hope-'):
|
||||
raise ProvisioningError('Refusing unexpected temporary-file cleanup path')
|
||||
temporary.unlink()
|
||||
|
||||
def _paths(self, user):
|
||||
container_name(user)
|
||||
# Every ancestor under the configured root is control-owned, not mounted
|
||||
# into Hermes. HOME cannot access the sibling API credential file.
|
||||
return self.state_dir / user['id'], self.state_dir / user['id'] / 'home', self.state_dir / user['id'] / 'workspace'
|
||||
|
||||
def _read_key(self, user):
|
||||
private, _, _ = self._paths(user)
|
||||
path = private / 'api-key'
|
||||
if path.is_symlink():
|
||||
raise ProvisioningError('Private runtime file cannot be a symlink')
|
||||
try:
|
||||
key = path.read_text().strip()
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
if len(key) < 32 or '\n' in key:
|
||||
raise ProvisioningError('Invalid persisted Hermes API credential')
|
||||
return key
|
||||
|
||||
def _prepare(self, user, exists):
|
||||
private, home, workspace = self._paths(user)
|
||||
self._directory(self.state_dir)
|
||||
for path in (private, home, workspace):
|
||||
self._directory(path)
|
||||
key = self._read_key(user)
|
||||
if not key:
|
||||
if exists:
|
||||
raise ProvisioningError('Owned container has no persisted API credential; refusing unsafe adoption')
|
||||
key = secrets.token_urlsafe(48)
|
||||
self._write_private(private / 'api-key', key + '\n')
|
||||
config = home / 'config.yaml'
|
||||
if not config.exists():
|
||||
# JSON is a YAML subset; no credentials are placed in configuration.
|
||||
self._write_private(config, json.dumps(self.config(), indent=2) + '\n')
|
||||
elif config.is_symlink():
|
||||
raise ProvisioningError('Hermes config cannot be a symlink')
|
||||
if os.geteuid() == 0:
|
||||
# Only seeded directory roots/config, never recursive retained data.
|
||||
for path in (home, workspace, config):
|
||||
os.chown(path, self.runtime_uid, self.runtime_gid)
|
||||
elif home.stat().st_uid != self.runtime_uid:
|
||||
raise ProvisioningError('Control service must prepare private directories for the Hermes runtime UID')
|
||||
return key, home, workspace
|
||||
|
||||
@staticmethod
|
||||
def config():
|
||||
model = os.environ.get('HERMES_UPSTREAM_MODEL', 'auto')
|
||||
return {
|
||||
'model': {'default': model, 'provider': 'custom:hope-upstream'},
|
||||
'providers': {'hope-upstream': {'api': os.environ.get('HERMES_UPSTREAM_URL', 'http://192.168.86.5:2099/v1'),
|
||||
'key_env': 'HERMES_UPSTREAM_API_KEY', 'transport': 'codex_responses', 'default_model': model,
|
||||
'models': [model], 'discover_models': False}},
|
||||
'agent': {'reasoning_effort': 'none'},
|
||||
'terminal': {'backend': 'local', 'cwd': '/workspace'},
|
||||
'platform_toolsets': {'api_server': ['terminal', 'file', 'web', 'browser', 'memory', 'skills', 'delegation']},
|
||||
'gateway': {'multiplex_profiles': False,
|
||||
'api_server': {'enabled': True, 'host': '0.0.0.0', 'port': 8642,
|
||||
'max_concurrent_runs': 1, 'tool_progress_events': True}},
|
||||
}
|
||||
|
||||
async def _network(self, user):
|
||||
name = 'hope-hermes-net-' + user['id']
|
||||
response = await self._request('GET', f'/networks/{name}', allowed=(200, 404))
|
||||
if response.status_code == 404:
|
||||
await self._request('POST', '/networks/create', allowed=(201, 409), json={
|
||||
'Name': name, 'Driver': 'bridge', 'CheckDuplicate': True,
|
||||
'Labels': {'hope.managed': 'true', 'hope.user_id': user['id']}})
|
||||
response = await self._request('GET', f'/networks/{name}')
|
||||
network = response.json()
|
||||
self._owned(network, user)
|
||||
if network.get('Driver') != 'bridge' or network.get('Internal'):
|
||||
raise ProvisioningError('Owned Hermes network is not an outbound-capable dedicated bridge')
|
||||
return name
|
||||
|
||||
def _spec(self, user, key, home, workspace, network):
|
||||
return {
|
||||
'Image': self.image, 'User': f'{self.runtime_uid}:{self.runtime_gid}',
|
||||
'Entrypoint': ['/opt/hermes/.venv/bin/python'], 'Cmd': ['-c', _BOOTSTRAP],
|
||||
'WorkingDir': '/workspace',
|
||||
'Labels': {'hope.managed': 'true', 'hope.user_id': user['id']},
|
||||
'Env': ['HOME=/opt/data/home', 'HERMES_HOME=/opt/data',
|
||||
'HERMES_GATEWAY_NO_SUPERVISE=1', 'API_SERVER_ENABLED=true',
|
||||
'API_SERVER_HOST=0.0.0.0', 'API_SERVER_PORT=8642', f'API_SERVER_KEY={key}',
|
||||
f'HERMES_UPSTREAM_API_KEY={self.upstream_api_key}'],
|
||||
'ExposedPorts': {'8642/tcp': {}},
|
||||
'HostConfig': {
|
||||
'Binds': [f'{home}:/opt/data:rw', f'{workspace}:/workspace:rw'],
|
||||
'NetworkMode': network, 'PortBindings': {'8642/tcp': [{'HostIp': '127.0.0.1', 'HostPort': ''}]},
|
||||
'Init': True, 'Privileged': False, 'CapDrop': ['ALL'],
|
||||
'SecurityOpt': ['no-new-privileges:true'], 'PidsLimit': 256,
|
||||
'Memory': self.memory_bytes, 'MemorySwap': self.memory_bytes, 'NanoCpus': self.nano_cpus,
|
||||
'ShmSize': 256 * 1024 ** 2,
|
||||
'RestartPolicy': {'Name': 'unless-stopped'},
|
||||
'LogConfig': {'Type': 'json-file', 'Config': {'max-size': '10m', 'max-file': '3'}},
|
||||
},
|
||||
}
|
||||
|
||||
def _connection(self, user, info, key):
|
||||
self._owned(info, user)
|
||||
env = info.get('Config', {}).get('Env') or []
|
||||
if f'API_SERVER_KEY={key}' not in env:
|
||||
raise ProvisioningError('Owned container API credential does not match private state')
|
||||
bindings = info.get('NetworkSettings', {}).get('Ports', {}).get('8642/tcp') or []
|
||||
if len(bindings) != 1 or bindings[0].get('HostIp') != '127.0.0.1':
|
||||
raise ProvisioningError('Hermes API must have exactly one loopback-only publication')
|
||||
port = str(bindings[0].get('HostPort', ''))
|
||||
if not port.isdigit() or not 1 <= int(port) <= 65535:
|
||||
raise ProvisioningError('Hermes API has no allocated host port')
|
||||
return {'container_name': container_name(user), 'hermes_url': f'http://127.0.0.1:{port}', 'hermes_api_key': key}
|
||||
|
||||
async def _healthy(self, connection):
|
||||
try:
|
||||
response = await self.health.get(connection['hermes_url'] + '/v1/models',
|
||||
headers={'Authorization': 'Bearer ' + connection['hermes_api_key']})
|
||||
return response.status_code == 200
|
||||
except httpx.HTTPError:
|
||||
return False
|
||||
|
||||
async def ensure(self, user: dict) -> dict:
|
||||
async with self._lock(user):
|
||||
if not self.upstream_api_key:
|
||||
raise ProvisioningError('HERMES_UPSTREAM_API_KEY must be provided at runtime')
|
||||
name = container_name(user)
|
||||
info = await self._inspect(name)
|
||||
if info:
|
||||
self._owned(info, user)
|
||||
key, home, workspace = self._prepare(user, info is not None)
|
||||
if info is None:
|
||||
network = await self._network(user)
|
||||
await self._request('POST', '/containers/create', params={'name': name},
|
||||
json=self._spec(user, key, home, workspace, network), allowed=(201, 409))
|
||||
info = await self._inspect(name)
|
||||
if info is None:
|
||||
raise ProvisioningError('Docker container creation did not persist; retry')
|
||||
self._owned(info, user)
|
||||
if not info.get('State', {}).get('Running'):
|
||||
await self._request('POST', f'/containers/{info["Id"]}/start', allowed=(204, 304))
|
||||
# One overall deadline also bounds Docker inspect / health I/O; no
|
||||
# deletion on failed health, so a retry starts/adopts the same state.
|
||||
try:
|
||||
async with asyncio.timeout(self.ready_timeout):
|
||||
while True:
|
||||
info = await self._inspect(name)
|
||||
if info is None:
|
||||
raise ProvisioningError('Hermes container disappeared; retry')
|
||||
self._owned(info, user)
|
||||
if info.get('State', {}).get('Running'):
|
||||
connection = self._connection(user, info, key)
|
||||
if await self._healthy(connection):
|
||||
return connection
|
||||
await asyncio.sleep(self.poll_interval)
|
||||
except TimeoutError as exc:
|
||||
raise ProvisioningError('Hermes API readiness timed out; retained container and data for retry') from exc
|
||||
|
||||
async def ready(self, user: dict) -> bool:
|
||||
try:
|
||||
info = await self._inspect(container_name(user))
|
||||
if not info or not info.get('State', {}).get('Running'):
|
||||
return False
|
||||
key = self._read_key(user)
|
||||
return bool(key) and await self._healthy(self._connection(user, info, key))
|
||||
except (ProvisioningError, ValueError, OSError):
|
||||
return False
|
||||
|
||||
async def remove(self, user: dict):
|
||||
async with self._lock(user):
|
||||
info = await self._inspect(container_name(user))
|
||||
if info:
|
||||
self._owned(info, user)
|
||||
# Operate on inspected immutable ID, never a raced name. Labels
|
||||
# are immutable for the lifetime of that Docker container ID.
|
||||
await self._request('DELETE', f'/containers/{info["Id"]}',
|
||||
params={'force': 'true', 'v': 'false'}, allowed=(204, 404))
|
||||
name = 'hope-hermes-net-' + user['id']
|
||||
response = await self._request('GET', f'/networks/{name}', allowed=(200, 404))
|
||||
if response.status_code == 200:
|
||||
network = response.json()
|
||||
self._owned(network, user)
|
||||
await self._request('DELETE', f'/networks/{network["Id"]}', allowed=(204, 404))
|
||||
# Intentionally retain private credential, home, workspace and memory.
|
||||
|
||||
|
||||
DockerProvisioner = Provisioner
|
||||
@@ -0,0 +1,3 @@
|
||||
fastapi>=0.115,<1
|
||||
uvicorn>=0.30,<1
|
||||
httpx>=0.27,<1
|
||||
Reference in new issue
Block a user