stackpilot/ROADMAP.md

StackPilot Roadmap — Phases 21–23

Historical record. Phases below describe work as it shipped at the time. The multi-host / agent integration they refer to was removed in 0.48.0 — StackPilot manages a single Docker host. Anything here mentioning agent_app.py, AGENT_TOKEN, /api/agents/* or /ws/agent-* no longer exists; the entries are kept because they record what was actually done, not what is currently true.

Planned 2026-06-09. Status keys: ☐ not started · ◐ in progress · ☑ done. Each phase ships independently following the standing release checklist (bump backend/main.py + backend/agent_app.py AGENT_VERSION + frontend/package.json → build backend→agent→frontend :VERSION+:latest → py_compile + tsc -b && vite build → route smoke-test → push all 3 → README → git commit + push → update memory). Current released version: 0.29.0.

Order: 21 → 22 → 23 (Terminal is highest-value and self-contained; Secrets is most design-ambiguous, left last).

---

Phase 21 — Container terminal (web exec) ☑ DONE — shipped 0.27.0

Result: All backend paths live-verified via TestClient against real containers — local exec (echo round-trip, exit code 0, unmanaged-guard, non-admin→4403, no-token→4401), agent exec (echo, bad-token→4401), and the central proxy (bidirectional: keystrokes browser→proxy→agent→container, output back). docker-py 7.1.0 socket is socket.SocketIO with the real fd at ._sock (as predicted). Frontend tsc -b && vite build green with xterm. Not click-tested in a real browser (the WS/exec plumbing is what's verified).

Interactive shell into a running, compose-managed container over WebSocket + xterm.js, like Portainer's Console. Local + agent. Admin-only (exec is root-equivalent).

Backend

• ☐ services/exec_service.py (new). Reuse container_service._get_managed() to refuse non-compose containers. Functions:
  • create_exec(container_id, cmd, tty=True) → low-level client.api.exec_create(container_id, cmd, stdin=True, tty=True, stdout=True, stderr=True) returns exec_id.
  • start_exec(exec_id) → client.api.exec_start(exec_id, socket=True, tty=True, demux=False) returns the raw socket (a SocketIO/socket; on some docker-py versions the real fd is sock._sock — handle both; verify live, this is the main risk).
  • resize_exec(exec_id, h, w) → client.api.exec_resize(exec_id, height=h, width=w).
  • default cmd = ["/bin/sh"] (frontend may request /bin/bash).
• ☐ routers/ws.py: add _authorize_admin(websocket, token) (decode JWT like _authorize but also require role == "admin"; close 4403 if not admin). Then @router.websocket("/ws/exec/{container_id}"):
  • accept, _authorize_admin, read ?cmd= (default /bin/sh), create+start exec.
  • Bidirectional pump with two asyncio tasks:
    • reader: loop.run_in_executor(None, sock.recv, 4096) → send_bytes/ send_text to browser (decode utf-8, errors="replace").
    • writer: await websocket.receive_text(); JSON control msgs {type:"resize",cols,rows} → resize_exec; otherwise raw keystrokes → sock.sendall(data.encode()) (also in executor).
    • on first task to finish, cancel the other; close sock + ws.
  • audit container.exec.
• ☐ agent_app.py: @app.websocket("/agent/ws/exec/{container_id}") — token via ?token= vs AGENT_TOKEN, same pump (import exec_service).
• ☐ routers/ws.py proxy: @router.websocket("/ws/agent-exec/{agent_id}/{container_id}") — mirror ws_agent_deploy BUT forward both directions (agent-logs/deploy only pump upstream→browser; exec also needs browser→upstream). Use urllib.parse.quote(token, safe=''), InvalidStatus 404 → "update agent" hint.

Frontend

• ☐ Add deps to frontend/package.json: xterm (@xterm/xterm) + @xterm/addon-fit. (rebuild installs them).
• ☐ components/stacks/ContainerTerminal.tsx — modal: xterm.js Terminal + FitAddon, connects to /ws/exec/{id} or /ws/agent-exec/{aid}/{id}; shell <select> (/bin/sh | /bin/bash); term.onData → ws.send; ws.onmessage → term.write; ResizeObserver/fit → send {type:"resize",cols,rows}; status line (connected/closed/error 4403 → "admin only").
• ☐ ContainerCard.tsx: add a Terminal button (admin + running only) next to start/stop/restart → opens ContainerTerminal with the same agentId.

Verify

• ☐ Live: exec /bin/sh into a real throwaway compose container, run ls/pwd, type interactively, resize, Ctrl-C, exit. Bad/non-admin token → 4403.
• ☐ Agent path: same against a local agent container w/ docker.sock.
• ☐ py_compile + tsc/vite build + route smoke-test (expect main +2 WS, agent +1).

Open risks

• docker-py exec_start(socket=True) socket object shape varies by version — verify .recv/.sendall vs needing sock._sock. This is THE thing to nail live.
• TTY mode (tty=True) gives a raw single stream (no 8-byte demux header) — right for xterm. If we ever do non-tty, we'd need demux handling.

---

Phase 22 — Auto-Update (Watchtower-style) ☑ DONE — shipped 0.28.0

Result: All four orchestration paths live-verified against real compose in the 0.28.0 backend image (seeding update_service._CACHE): redeploy→updated (pull+up, stack stays running), notify-only→update-available, no-update→up-to-date, stopped-stack→skipped. Routes present (local + agent + /agent/stacks/{id}/updates). Hooked into update_service.background_loop (lazy import, no circular). Remote path reuses the proven agent_service.call; not separately e2e'd against a live agent this round.

Per-stack policy: when an image used by the stack has a newer registry digest, either auto pull+redeploy or just notify. Builds on the existing update_service (digest check) + compose_service.pull/up + notify_service.

Backend

• ☐ models/auto_update.py — AutoUpdate(id, stack_id, agent_id nullable, enabled bool, redeploy bool [True=pull+up / False=notify-only], last_run, last_status, last_result). (agent_id None = local stack.)
• ☐ services/auto_update_service.py:
  • run_due() — for each enabled policy: resolve the stack's images (local: update_service images for that compose project; remote: via agent), check_image each; if any update_available:
    • redeploy=True local → compose_service.pull(stack_id) then compose_service.up(stack_id); remote → agent_service POST /agent/stacks/{id}/update.
    • redeploy=False → notify only.
    • record last_run/last_status (updated/up-to-date/error), notify EVENT_STACK_AUTO_UPDATED (add to models/setting.py ALL_EVENTS + frontend EVENT_LABELS) on redeploy, reuse pull_failed/stack_error on failure.
  • Hook into the existing update_service.background_loop (it already polls on the settings interval) — after check_all(), call auto_update_service.run_due(). (Or the scheduler_loop; pick background_loop since it already has fresh digest data.)
• ☐ routers/stacks.py: GET /api/stacks/{id}/auto-update + PUT /api/stacks/{id}/auto-update (admin). Agent stacks: add to routers/agents.py GET/PUT /api/agents/{id}/stacks/{sid}/auto-update (policy stored centrally keyed by agent_id+stack_id; the agent itself stays stateless — central drives it).

Frontend

• ☐ StackDetail.tsx + RemoteStackDetail.tsx: an Auto-update toggle in the action bar / a small card (enabled + mode redeploy|notify-only). api/stacks.ts
  • api/agents.ts get/set.
• ☐ Optional: Settings → Auto-updates section listing all policies w/ last run + status (like Scheduled backups).

Verify

• ☐ Enable on a throwaway stack pinned to an old tag, point its tag at a newer digest (or use a 2-tag trick), confirm run_due pulls + redeploys + records status + fires the notification. Notify-only mode notifies without redeploy.
• ☐ py_compile + build + route smoke-test.

Open risks

• Mapping a stack → its images reliably (compose image: refs vs built images; built/untagged images can't be digest-checked — skip them).
• Don't redeploy a stack the user has manually stopped (check status first? decide at impl: probably only auto-update stacks currently running).

---

Phase 23 — Docker Secrets & Configs ☑ DONE — shipped 0.29.0

DECISION (user, 2026-06-09): (A) Compose file-based secrets. Swarm path (B) dropped. Sub-decision (Claude's call): per-stack storage with RELATIVE paths — secret/config files live in <stack_dir>/.secrets/<name> and .configs/<name>, referenced as file: ./.secrets/<name>. Compose resolves file: relative to the compose file (which is in the stack dir = a host bind-mount), so the daemon reads it with NO HOST_ROOT_PREFIX dependency. Secrets are therefore per-stack (matches how compose scopes them), managed from a Secrets tab on StackDetail / RemoteStackDetail (not the new-stack editor, which has no dir yet).

As shipped — deviations from the plan (both simplifications):

• No DB model. Both content and metadata live on disk; list derives name/ size/mtime from the filesystem. A models/secret.py would only duplicate that, so it was dropped — there is nothing to keep in sync.
• Files live in the stack dir, not a separate sandbox. <stack_dir>/.secrets/ and .configs/ (per-stack, relative file: refs) — this is exactly what compose expects and removes the HOST_ROOT_PREFIX dependency. Sandboxing comes from strict name validation (single component, no .., no leading dot, no sep).
• Routes are stack-scoped: /api/stacks/{stack_id}/secrets, not /api/secrets.
• UI is a Secrets tab on Stack/RemoteStackDetail, not a Settings page — a stack must exist (have a dir) before it can hold secrets, matching compose's scoping.
• Swarm path not built (decision A dropped B); no swarm_active() guard.

Backend

• ☑ services/secret_service.py — file-based store under <stack_dir>/.secrets & .configs (dir 0700, file 0600); write_secret/delete_secret/list_all (metadata only, never content)/exists/rel_path/attach/detach. 1 MiB cap; name validation rejects traversal/hidden/separators.
• ☑ compose_edit_service: add_secret/remove_secret (top-level secrets: {<name>: {file: <path>}} + per-service list) and add_config/ remove_config (with source/target mount); top-level defs pruned when unused.
• ☑ routers/secrets.py (prefix /api/stacks/{stack_id}/secrets): list/write/ delete/attach/detach, admin-only, audit secret.*, content write-only.
• ☑ Agent /agent/stacks/{stack_id}/secrets/* (agent_app.py) + proxy /api/agents/{id}/stacks/{stack_id}/secrets/* (agents.py), audit agent.secret.*.

Frontend

• ☑ SecretsPanel (api/secrets.ts + components/stacks/SecretsPanel.tsx): create (type/name/content; content cleared after save, never re-shown), list (name, kind, size), delete, and per-row attach/detach to a service (config rows take a mount target). Admin-gated. Wired as a Secrets tab on both StackDetail and RemoteStackDetail (agentId-aware → multi-host).

Verify

• ☑ Sandbox: traversal/hidden/separator names rejected; dir 0700 / file 0600; list returns metadata only, never content (unit-tested in the backend image).
• ☑ compose round-trip: add secret+config → remove both → back to clean YAML (top-level defs pruned). add/remove for secrets and configs unit-tested.
• ☑ py_compile + backend image build + frontend tsc -b && vite build + route smoke-test (local CRUD/attach/detach, agent, proxy all registered).
• ☐ Live hardware-verify debt: create a secret, attach via the panel, deploy, exec in and confirm /run/secrets/<name> holds the content (needs a running stack on real hardware). Swarm path intentionally not built (decision A).

Open risks (carried)

• Secret file ownership/permissions inside the container vs on host (file-based secrets mount the host file → uid/gid must be readable by the service user).
• Multi-host: the agent stores the file on its own host, so the secret lives where that host's Docker daemon can read it — verified by design (relative file:), pending the live exec check above.

---

Phase 25 — Templates as stack folders ☑ DONE — shipped 0.31.0

Templates reworked from DB rows + manifest.json + {{VAR}} mustache rendering to stack-shaped folders: backend/templates/<slug>/ with compose.yaml, optional .env.example and a template.json (name/description/tags/gpu). "Pull" copies the whole folder into a new stack (.env.example → .env); custom templates live under ${DATA_DIR}/templates/ and are written by "Save as template" (stack detail) or the manual save endpoint.

• Backend: template_service rewritten (folder scan, traversal-guarded resolve, copy_into_stack, save_from_stack); Template DB table dropped with a one-time startup migration ({{VAR}} → ${VAR}, vars → .env.example).
• API: POST /api/templates/from-stack new; instantiate no longer takes values.
• Frontend: Templates page shows compose/env preview + file list, delete for custom templates; StackDetail gains "Save as template".

---

After 23

Remaining un-built ideas from the gap analysis (not chosen this round): Health-monitoring & alerting (Docker-events → notify; note /ws/events already exists), per-container live-stats + historical graphs, GitOps/deploy-from-Git, image-management expansion (pull/remove/build arbitrary images + registry creds UI), 2FA/TOTP + OIDC SSO + API tokens.