stackpilot/ROADMAP.md

# StackPilot Roadmap — Phases 21–23

Planned 2026-06-09. Status keys: ☐ not started · ◐ in progress · ☑ done.
Each phase ships independently following the standing release checklist
(bump `backend/main.py` + `backend/agent_app.py` AGENT_VERSION +
`frontend/package.json` → build backend→agent→frontend `:VERSION`+`:latest` →
py_compile + `tsc -b && vite build` → route smoke-test → push all 3 → README →
git commit + push → update memory). Current released version: **0.29.0**.

Order: **21 → 22 → 23** (Terminal is highest-value and self-contained; Secrets
is most design-ambiguous, left last).

---

## Phase 21 — Container terminal (web exec)   ☑ DONE — shipped 0.27.0

**Result:** All backend paths live-verified via TestClient against real
containers — local exec (echo round-trip, exit code 0, unmanaged-guard,
non-admin→4403, no-token→4401), agent exec (echo, bad-token→4401), and the
central proxy (bidirectional: keystrokes browser→proxy→agent→container, output
back). docker-py 7.1.0 socket is `socket.SocketIO` with the real fd at
`._sock` (as predicted). Frontend `tsc -b && vite build` green with xterm.
Not click-tested in a real browser (the WS/exec plumbing is what's verified).

Interactive shell into a running, compose-managed container over WebSocket +
xterm.js, like Portainer's Console. Local + agent. **Admin-only** (exec is
root-equivalent).

### Backend
- ☐ `services/exec_service.py` (new). Reuse `container_service._get_managed()`
  to refuse non-compose containers. Functions:
  - `create_exec(container_id, cmd, tty=True)` → low-level
    `client.api.exec_create(container_id, cmd, stdin=True, tty=True,
    stdout=True, stderr=True)` returns `exec_id`.
  - `start_exec(exec_id)` → `client.api.exec_start(exec_id, socket=True,
    tty=True, demux=False)` returns the raw socket (a `SocketIO`/`socket`;
    on some docker-py versions the real fd is `sock._sock` — handle both;
    **verify live**, this is the main risk).
  - `resize_exec(exec_id, h, w)` → `client.api.exec_resize(exec_id, height=h,
    width=w)`.
  - default cmd = `["/bin/sh"]` (frontend may request `/bin/bash`).
- ☐ `routers/ws.py`: add `_authorize_admin(websocket, token)` (decode JWT like
  `_authorize` but also require `role == "admin"`; close 4403 if not admin).
  Then `@router.websocket("/ws/exec/{container_id}")`:
  - accept, `_authorize_admin`, read `?cmd=` (default /bin/sh), create+start exec.
  - **Bidirectional pump** with two asyncio tasks:
    - reader: `loop.run_in_executor(None, sock.recv, 4096)` → `send_bytes`/
      `send_text` to browser (decode utf-8, errors="replace").
    - writer: `await websocket.receive_text()`; JSON control msgs
      `{type:"resize",cols,rows}` → `resize_exec`; otherwise raw keystrokes →
      `sock.sendall(data.encode())` (also in executor).
    - on first task to finish, cancel the other; close sock + ws.
  - audit `container.exec`.
- ☐ `agent_app.py`: `@app.websocket("/agent/ws/exec/{container_id}")` — token
  via `?token=` vs AGENT_TOKEN, same pump (import exec_service).
- ☐ `routers/ws.py` proxy: `@router.websocket("/ws/agent-exec/{agent_id}/{container_id}")`
  — mirror `ws_agent_deploy` BUT forward **both directions** (agent-logs/deploy
  only pump upstream→browser; exec also needs browser→upstream). Use
  `urllib.parse.quote(token, safe='')`, InvalidStatus 404 → "update agent" hint.

### Frontend
- ☐ Add deps to `frontend/package.json`: `xterm` (`@xterm/xterm`) +
  `@xterm/addon-fit`. (rebuild installs them).
- ☐ `components/stacks/ContainerTerminal.tsx` — modal: xterm.js Terminal +
  FitAddon, connects to `/ws/exec/{id}` or `/ws/agent-exec/{aid}/{id}`; shell
  `<select>` (/bin/sh | /bin/bash); `term.onData` → ws.send; ws.onmessage →
  `term.write`; ResizeObserver/fit → send `{type:"resize",cols,rows}`; status
  line (connected/closed/error 4403 → "admin only").
- ☐ `ContainerCard.tsx`: add a **Terminal** button (admin + running only) next
  to start/stop/restart → opens `ContainerTerminal` with the same `agentId`.

### Verify
- ☐ Live: exec `/bin/sh` into a real throwaway compose container, run `ls`/`pwd`,
  type interactively, resize, Ctrl-C, exit. Bad/non-admin token → 4403.
- ☐ Agent path: same against a local agent container w/ docker.sock.
- ☐ py_compile + tsc/vite build + route smoke-test (expect main +2 WS, agent +1).

### Open risks
- docker-py `exec_start(socket=True)` socket object shape varies by version —
  verify `.recv`/`.sendall` vs needing `sock._sock`. This is THE thing to nail live.
- TTY mode (tty=True) gives a raw single stream (no 8-byte demux header) — right
  for xterm. If we ever do non-tty, we'd need demux handling.

---

## Phase 22 — Auto-Update (Watchtower-style)   ☑ DONE — shipped 0.28.0

**Result:** All four orchestration paths live-verified against real compose in
the 0.28.0 backend image (seeding `update_service._CACHE`): redeploy→`updated`
(pull+up, stack stays running), notify-only→`update-available`,
no-update→`up-to-date`, stopped-stack→`skipped`. Routes present (local +
agent + `/agent/stacks/{id}/updates`). Hooked into `update_service.background_loop`
(lazy import, no circular). Remote path reuses the proven `agent_service.call`;
not separately e2e'd against a live agent this round.

Per-stack policy: when an image used by the stack has a newer registry digest,
either auto pull+redeploy or just notify. Builds on the existing
`update_service` (digest check) + `compose_service.pull/up` + `notify_service`.

### Backend
- ☐ `models/auto_update.py` — `AutoUpdate(id, stack_id, agent_id nullable,
  enabled bool, redeploy bool [True=pull+up / False=notify-only], last_run,
  last_status, last_result)`. (agent_id None = local stack.)
- ☐ `services/auto_update_service.py`:
  - `run_due()` — for each enabled policy: resolve the stack's images
    (local: `update_service` images for that compose project; remote: via agent),
    `check_image` each; if any `update_available`:
    - redeploy=True local → `compose_service.pull(stack_id)` then
      `compose_service.up(stack_id)`; remote → `agent_service` POST
      `/agent/stacks/{id}/update`.
    - redeploy=False → notify only.
    - record last_run/last_status (`updated`/`up-to-date`/`error`), notify
      `EVENT_STACK_AUTO_UPDATED` (add to `models/setting.py` ALL_EVENTS +
      frontend EVENT_LABELS) on redeploy, reuse `pull_failed`/`stack_error` on
      failure.
  - Hook into the existing **`update_service.background_loop`** (it already polls
    on the settings interval) — after `check_all()`, call
    `auto_update_service.run_due()`. (Or the scheduler_loop; pick background_loop
    since it already has fresh digest data.)
- ☐ `routers/stacks.py`: `GET /api/stacks/{id}/auto-update` +
  `PUT /api/stacks/{id}/auto-update` (admin). Agent stacks: add to
  `routers/agents.py` `GET/PUT /api/agents/{id}/stacks/{sid}/auto-update`
  (policy stored centrally keyed by agent_id+stack_id; the agent itself stays
  stateless — central drives it).

### Frontend
- ☐ `StackDetail.tsx` + `RemoteStackDetail.tsx`: an **Auto-update** toggle in the
  action bar / a small card (enabled + mode redeploy|notify-only). `api/stacks.ts`
  + `api/agents.ts` get/set.
- ☐ Optional: Settings → **Auto-updates** section listing all policies w/ last
  run + status (like Scheduled backups).

### Verify
- ☐ Enable on a throwaway stack pinned to an old tag, point its tag at a newer
  digest (or use a 2-tag trick), confirm `run_due` pulls + redeploys + records
  status + fires the notification. Notify-only mode notifies without redeploy.
- ☐ py_compile + build + route smoke-test.

### Open risks
- Mapping a stack → its images reliably (compose `image:` refs vs built images;
  built/untagged images can't be digest-checked — skip them).
- Don't redeploy a stack the user has manually stopped (check status first?
  decide at impl: probably only auto-update stacks currently `running`).

---

## Phase 23 — Docker Secrets & Configs   ☑ DONE — shipped 0.29.0

**DECISION (user, 2026-06-09): (A) Compose file-based secrets.** Swarm path (B)
dropped. **Sub-decision (Claude's call): per-stack storage with RELATIVE paths**
— secret/config files live in `<stack_dir>/.secrets/<name>` and `.configs/<name>`,
referenced as `file: ./.secrets/<name>`. Compose resolves `file:` relative to the
compose file (which is in the stack dir = a host bind-mount), so the daemon reads
it with NO `HOST_ROOT_PREFIX` dependency. Secrets are therefore per-stack (matches
how compose scopes them), managed from a **Secrets tab on StackDetail /
RemoteStackDetail** (not the new-stack editor, which has no dir yet).

**As shipped — deviations from the plan (both simplifications):**
- **No DB model.** Both content *and* metadata live on disk; `list` derives name/
  size/mtime from the filesystem. A `models/secret.py` would only duplicate that,
  so it was dropped — there is nothing to keep in sync.
- **Files live in the stack dir, not a separate sandbox.** `<stack_dir>/.secrets/`
  and `.configs/` (per-stack, relative `file:` refs) — this is exactly what
  compose expects and removes the `HOST_ROOT_PREFIX` dependency. Sandboxing comes
  from strict name validation (single component, no `..`, no leading dot, no sep).
- **Routes are stack-scoped:** `/api/stacks/{stack_id}/secrets`, not `/api/secrets`.
- **UI is a Secrets tab on Stack/RemoteStackDetail**, not a Settings page — a stack
  must exist (have a dir) before it can hold secrets, matching compose's scoping.
- **Swarm path not built** (decision A dropped B); no `swarm_active()` guard.

### Backend
- ☑ `services/secret_service.py` — file-based store under `<stack_dir>/.secrets`
  & `.configs` (dir 0700, file 0600); `write_secret`/`delete_secret`/`list_all`
  (metadata only, never content)/`exists`/`rel_path`/`attach`/`detach`. 1 MiB cap;
  name validation rejects traversal/hidden/separators.
- ☑ `compose_edit_service`: `add_secret`/`remove_secret` (top-level
  `secrets: {<name>: {file: <path>}}` + per-service list) and `add_config`/
  `remove_config` (with `source`/`target` mount); top-level defs pruned when unused.
- ☑ `routers/secrets.py` (prefix `/api/stacks/{stack_id}/secrets`): list/write/
  delete/attach/detach, admin-only, audit `secret.*`, content write-only.
- ☑ Agent `/agent/stacks/{stack_id}/secrets/*` (agent_app.py) + proxy
  `/api/agents/{id}/stacks/{stack_id}/secrets/*` (agents.py), audit `agent.secret.*`.

### Frontend
- ☑ `SecretsPanel` (api/secrets.ts + components/stacks/SecretsPanel.tsx): create
  (type/name/content; content cleared after save, never re-shown), list (name,
  kind, size), delete, and per-row attach/detach to a service (config rows take a
  mount target). Admin-gated. Wired as a **Secrets** tab on both StackDetail and
  RemoteStackDetail (agentId-aware → multi-host).

### Verify
- ☑ Sandbox: traversal/hidden/separator names rejected; dir 0700 / file 0600;
  `list` returns metadata only, never content (unit-tested in the backend image).
- ☑ compose round-trip: add secret+config → remove both → back to clean YAML
  (top-level defs pruned). add/remove for secrets and configs unit-tested.
- ☑ py_compile + backend image build + frontend `tsc -b && vite build` + route
  smoke-test (local CRUD/attach/detach, agent, proxy all registered).
- ☐ **Live hardware-verify debt:** create a secret, attach via the panel, deploy,
  exec in and confirm `/run/secrets/<name>` holds the content (needs a running
  stack on real hardware). Swarm path intentionally not built (decision A).

### Open risks (carried)
- Secret file ownership/permissions inside the container vs on host (file-based
  secrets mount the host file → uid/gid must be readable by the service user).
- Multi-host: the agent stores the file on its own host, so the secret lives where
  that host's Docker daemon can read it — verified by design (relative `file:`),
  pending the live exec check above.

---

## Phase 25 — Templates as stack folders   ☑ DONE — shipped 0.31.0

Templates reworked from DB rows + `manifest.json` + `{{VAR}}` mustache rendering
to **stack-shaped folders**: `backend/templates/<slug>/` with `compose.yaml`,
optional `.env.example` and a `template.json` (name/description/tags/gpu).
"Pull" copies the whole folder into a new stack (`.env.example` → `.env`);
custom templates live under `${DATA_DIR}/templates/` and are written by
"Save as template" (stack detail) or the manual save endpoint.

- Backend: `template_service` rewritten (folder scan, traversal-guarded resolve,
  `copy_into_stack`, `save_from_stack`); `Template` DB table dropped with a
  one-time startup migration (`{{VAR}}` → `${VAR}`, vars → `.env.example`).
- API: `POST /api/templates/from-stack` new; instantiate no longer takes `values`.
- Frontend: Templates page shows compose/env preview + file list, delete for
  custom templates; StackDetail gains "Save as template".

---

## After 23
Remaining un-built ideas from the gap analysis (not chosen this round):
Health-monitoring & alerting (Docker-events → notify; note `/ws/events` already
exists), per-container live-stats + historical graphs, GitOps/deploy-from-Git,
image-management expansion (pull/remove/build arbitrary images + registry creds
UI), 2FA/TOTP + OIDC SSO + API tokens.