menzeljandClaude Opus 5
51d1998307
StackPilot now manages exactly one Docker host: the one it runs on. The
stackpilot-agent sidecar and everything that proxied to it are gone — 4721
lines deleted against 657 added.
Deleted outright: agent/ (image, compose, env), agent_app.py, models/agent.py,
routers/agents.py (1200 lines), services/agent_service.py, the agent API client,
RemoteStackDetail, the host components and AgentStacksSection. That removes 57
API routes and the three /ws/agent-* proxies.
Threaded out everywhere else, which was the bulk of the work. Every API module
carried an optional agentId that switched the base path; every page that listed
Docker objects rendered one section per host behind a HostHeader; Files had a
host switcher; the New Stack editor and the template dialog had host selectors;
schedules, auto-update policies and stack summaries carried agent_id. All of it
is gone, and the typechecker drove the sweep — 85 files touched, tsc and the
build clean.
Two things the removal exposed as dead weight rather than merely unused:
compose_service kept an in-process busy set purely because the agent needed a
lock and has no database. With the agent gone that was a second source of truth
next to the real DB lock, so it is deleted; compute_status now reports only what
the containers say and the two callers that want "updating" overlay the lock.
StacksTable's linkBase prop only ever existed to point at /hosts/{id}/stacks.
The dashboard's "Hosts 1/1 online" KPI can no longer say anything else, so the
tile and the KPIs behind it are gone and the row is five wide.
Upgrading matters here. An existing install still has an agent table holding
each remote host's URL and bearer token — full Docker control of that host,
sitting in the database with nothing left to use it. _drop_removed_schema drops
it on first start, and drops the agent_id columns where the SQLite build
supports DROP COLUMN. Each statement runs in its own transaction on purpose: a
failed DDL poisons the transaction it is in, so sharing one would let an
unsupported column drop take the table drop down with it. test_agent_removal
covers both branches plus the fresh-install and idempotent cases, and an
end-to-end run against a seeded pre-0.48 database confirms the table is gone and
every /api/agents route answers 404.
Docstrings that justified a design by "shared with the agent, which has no
database" were rewritten rather than left lying: update_service's persistence
callback and image_status_store are still the right split (registry logic stays
testable without a database), but for that reason now, not the old one. The
README's multi-host sections are removed and an upgrade note explains what to do
with running agent containers; ROADMAP keeps its history behind a note saying
the feature it describes no longer exists.
CI no longer builds or pushes stackpilot-agent.
735 tests pass, ruff and tsc clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
242 lines
14 KiB
Markdown
242 lines
14 KiB
Markdown
# StackPilot Roadmap — Phases 21–23
|
||
|
||
> **Historical record.** Phases below describe work as it shipped at the time.
|
||
> The multi-host / agent integration they refer to was removed in 0.48.0 —
|
||
> StackPilot manages a single Docker host. Anything here mentioning
|
||
> `agent_app.py`, `AGENT_TOKEN`, `/api/agents/*` or `/ws/agent-*` no longer
|
||
> exists; the entries are kept because they record what was actually done, not
|
||
> what is currently true.
|
||
|
||
Planned 2026-06-09. Status keys: ☐ not started · ◐ in progress · ☑ done.
|
||
Each phase ships independently following the standing release checklist
|
||
(bump `backend/main.py` + `backend/agent_app.py` AGENT_VERSION +
|
||
`frontend/package.json` → build backend→agent→frontend `:VERSION`+`:latest` →
|
||
py_compile + `tsc -b && vite build` → route smoke-test → push all 3 → README →
|
||
git commit + push → update memory). Current released version: **0.29.0**.
|
||
|
||
Order: **21 → 22 → 23** (Terminal is highest-value and self-contained; Secrets
|
||
is most design-ambiguous, left last).
|
||
|
||
---
|
||
|
||
## Phase 21 — Container terminal (web exec) ☑ DONE — shipped 0.27.0
|
||
|
||
**Result:** All backend paths live-verified via TestClient against real
|
||
containers — local exec (echo round-trip, exit code 0, unmanaged-guard,
|
||
non-admin→4403, no-token→4401), agent exec (echo, bad-token→4401), and the
|
||
central proxy (bidirectional: keystrokes browser→proxy→agent→container, output
|
||
back). docker-py 7.1.0 socket is `socket.SocketIO` with the real fd at
|
||
`._sock` (as predicted). Frontend `tsc -b && vite build` green with xterm.
|
||
Not click-tested in a real browser (the WS/exec plumbing is what's verified).
|
||
|
||
Interactive shell into a running, compose-managed container over WebSocket +
|
||
xterm.js, like Portainer's Console. Local + agent. **Admin-only** (exec is
|
||
root-equivalent).
|
||
|
||
### Backend
|
||
- ☐ `services/exec_service.py` (new). Reuse `container_service._get_managed()`
|
||
to refuse non-compose containers. Functions:
|
||
- `create_exec(container_id, cmd, tty=True)` → low-level
|
||
`client.api.exec_create(container_id, cmd, stdin=True, tty=True,
|
||
stdout=True, stderr=True)` returns `exec_id`.
|
||
- `start_exec(exec_id)` → `client.api.exec_start(exec_id, socket=True,
|
||
tty=True, demux=False)` returns the raw socket (a `SocketIO`/`socket`;
|
||
on some docker-py versions the real fd is `sock._sock` — handle both;
|
||
**verify live**, this is the main risk).
|
||
- `resize_exec(exec_id, h, w)` → `client.api.exec_resize(exec_id, height=h,
|
||
width=w)`.
|
||
- default cmd = `["/bin/sh"]` (frontend may request `/bin/bash`).
|
||
- ☐ `routers/ws.py`: add `_authorize_admin(websocket, token)` (decode JWT like
|
||
`_authorize` but also require `role == "admin"`; close 4403 if not admin).
|
||
Then `@router.websocket("/ws/exec/{container_id}")`:
|
||
- accept, `_authorize_admin`, read `?cmd=` (default /bin/sh), create+start exec.
|
||
- **Bidirectional pump** with two asyncio tasks:
|
||
- reader: `loop.run_in_executor(None, sock.recv, 4096)` → `send_bytes`/
|
||
`send_text` to browser (decode utf-8, errors="replace").
|
||
- writer: `await websocket.receive_text()`; JSON control msgs
|
||
`{type:"resize",cols,rows}` → `resize_exec`; otherwise raw keystrokes →
|
||
`sock.sendall(data.encode())` (also in executor).
|
||
- on first task to finish, cancel the other; close sock + ws.
|
||
- audit `container.exec`.
|
||
- ☐ `agent_app.py`: `@app.websocket("/agent/ws/exec/{container_id}")` — token
|
||
via `?token=` vs AGENT_TOKEN, same pump (import exec_service).
|
||
- ☐ `routers/ws.py` proxy: `@router.websocket("/ws/agent-exec/{agent_id}/{container_id}")`
|
||
— mirror `ws_agent_deploy` BUT forward **both directions** (agent-logs/deploy
|
||
only pump upstream→browser; exec also needs browser→upstream). Use
|
||
`urllib.parse.quote(token, safe='')`, InvalidStatus 404 → "update agent" hint.
|
||
|
||
### Frontend
|
||
- ☐ Add deps to `frontend/package.json`: `xterm` (`@xterm/xterm`) +
|
||
`@xterm/addon-fit`. (rebuild installs them).
|
||
- ☐ `components/stacks/ContainerTerminal.tsx` — modal: xterm.js Terminal +
|
||
FitAddon, connects to `/ws/exec/{id}` or `/ws/agent-exec/{aid}/{id}`; shell
|
||
`<select>` (/bin/sh | /bin/bash); `term.onData` → ws.send; ws.onmessage →
|
||
`term.write`; ResizeObserver/fit → send `{type:"resize",cols,rows}`; status
|
||
line (connected/closed/error 4403 → "admin only").
|
||
- ☐ `ContainerCard.tsx`: add a **Terminal** button (admin + running only) next
|
||
to start/stop/restart → opens `ContainerTerminal` with the same `agentId`.
|
||
|
||
### Verify
|
||
- ☐ Live: exec `/bin/sh` into a real throwaway compose container, run `ls`/`pwd`,
|
||
type interactively, resize, Ctrl-C, exit. Bad/non-admin token → 4403.
|
||
- ☐ Agent path: same against a local agent container w/ docker.sock.
|
||
- ☐ py_compile + tsc/vite build + route smoke-test (expect main +2 WS, agent +1).
|
||
|
||
### Open risks
|
||
- docker-py `exec_start(socket=True)` socket object shape varies by version —
|
||
verify `.recv`/`.sendall` vs needing `sock._sock`. This is THE thing to nail live.
|
||
- TTY mode (tty=True) gives a raw single stream (no 8-byte demux header) — right
|
||
for xterm. If we ever do non-tty, we'd need demux handling.
|
||
|
||
---
|
||
|
||
## Phase 22 — Auto-Update (Watchtower-style) ☑ DONE — shipped 0.28.0
|
||
|
||
**Result:** All four orchestration paths live-verified against real compose in
|
||
the 0.28.0 backend image (seeding `update_service._CACHE`): redeploy→`updated`
|
||
(pull+up, stack stays running), notify-only→`update-available`,
|
||
no-update→`up-to-date`, stopped-stack→`skipped`. Routes present (local +
|
||
agent + `/agent/stacks/{id}/updates`). Hooked into `update_service.background_loop`
|
||
(lazy import, no circular). Remote path reuses the proven `agent_service.call`;
|
||
not separately e2e'd against a live agent this round.
|
||
|
||
Per-stack policy: when an image used by the stack has a newer registry digest,
|
||
either auto pull+redeploy or just notify. Builds on the existing
|
||
`update_service` (digest check) + `compose_service.pull/up` + `notify_service`.
|
||
|
||
### Backend
|
||
- ☐ `models/auto_update.py` — `AutoUpdate(id, stack_id, agent_id nullable,
|
||
enabled bool, redeploy bool [True=pull+up / False=notify-only], last_run,
|
||
last_status, last_result)`. (agent_id None = local stack.)
|
||
- ☐ `services/auto_update_service.py`:
|
||
- `run_due()` — for each enabled policy: resolve the stack's images
|
||
(local: `update_service` images for that compose project; remote: via agent),
|
||
`check_image` each; if any `update_available`:
|
||
- redeploy=True local → `compose_service.pull(stack_id)` then
|
||
`compose_service.up(stack_id)`; remote → `agent_service` POST
|
||
`/agent/stacks/{id}/update`.
|
||
- redeploy=False → notify only.
|
||
- record last_run/last_status (`updated`/`up-to-date`/`error`), notify
|
||
`EVENT_STACK_AUTO_UPDATED` (add to `models/setting.py` ALL_EVENTS +
|
||
frontend EVENT_LABELS) on redeploy, reuse `pull_failed`/`stack_error` on
|
||
failure.
|
||
- Hook into the existing **`update_service.background_loop`** (it already polls
|
||
on the settings interval) — after `check_all()`, call
|
||
`auto_update_service.run_due()`. (Or the scheduler_loop; pick background_loop
|
||
since it already has fresh digest data.)
|
||
- ☐ `routers/stacks.py`: `GET /api/stacks/{id}/auto-update` +
|
||
`PUT /api/stacks/{id}/auto-update` (admin). Agent stacks: add to
|
||
`routers/agents.py` `GET/PUT /api/agents/{id}/stacks/{sid}/auto-update`
|
||
(policy stored centrally keyed by agent_id+stack_id; the agent itself stays
|
||
stateless — central drives it).
|
||
|
||
### Frontend
|
||
- ☐ `StackDetail.tsx` + `RemoteStackDetail.tsx`: an **Auto-update** toggle in the
|
||
action bar / a small card (enabled + mode redeploy|notify-only). `api/stacks.ts`
|
||
+ `api/agents.ts` get/set.
|
||
- ☐ Optional: Settings → **Auto-updates** section listing all policies w/ last
|
||
run + status (like Scheduled backups).
|
||
|
||
### Verify
|
||
- ☐ Enable on a throwaway stack pinned to an old tag, point its tag at a newer
|
||
digest (or use a 2-tag trick), confirm `run_due` pulls + redeploys + records
|
||
status + fires the notification. Notify-only mode notifies without redeploy.
|
||
- ☐ py_compile + build + route smoke-test.
|
||
|
||
### Open risks
|
||
- Mapping a stack → its images reliably (compose `image:` refs vs built images;
|
||
built/untagged images can't be digest-checked — skip them).
|
||
- Don't redeploy a stack the user has manually stopped (check status first?
|
||
decide at impl: probably only auto-update stacks currently `running`).
|
||
|
||
---
|
||
|
||
## Phase 23 — Docker Secrets & Configs ☑ DONE — shipped 0.29.0
|
||
|
||
**DECISION (user, 2026-06-09): (A) Compose file-based secrets.** Swarm path (B)
|
||
dropped. **Sub-decision (Claude's call): per-stack storage with RELATIVE paths**
|
||
— secret/config files live in `<stack_dir>/.secrets/<name>` and `.configs/<name>`,
|
||
referenced as `file: ./.secrets/<name>`. Compose resolves `file:` relative to the
|
||
compose file (which is in the stack dir = a host bind-mount), so the daemon reads
|
||
it with NO `HOST_ROOT_PREFIX` dependency. Secrets are therefore per-stack (matches
|
||
how compose scopes them), managed from a **Secrets tab on StackDetail /
|
||
RemoteStackDetail** (not the new-stack editor, which has no dir yet).
|
||
|
||
**As shipped — deviations from the plan (both simplifications):**
|
||
- **No DB model.** Both content *and* metadata live on disk; `list` derives name/
|
||
size/mtime from the filesystem. A `models/secret.py` would only duplicate that,
|
||
so it was dropped — there is nothing to keep in sync.
|
||
- **Files live in the stack dir, not a separate sandbox.** `<stack_dir>/.secrets/`
|
||
and `.configs/` (per-stack, relative `file:` refs) — this is exactly what
|
||
compose expects and removes the `HOST_ROOT_PREFIX` dependency. Sandboxing comes
|
||
from strict name validation (single component, no `..`, no leading dot, no sep).
|
||
- **Routes are stack-scoped:** `/api/stacks/{stack_id}/secrets`, not `/api/secrets`.
|
||
- **UI is a Secrets tab on Stack/RemoteStackDetail**, not a Settings page — a stack
|
||
must exist (have a dir) before it can hold secrets, matching compose's scoping.
|
||
- **Swarm path not built** (decision A dropped B); no `swarm_active()` guard.
|
||
|
||
### Backend
|
||
- ☑ `services/secret_service.py` — file-based store under `<stack_dir>/.secrets`
|
||
& `.configs` (dir 0700, file 0600); `write_secret`/`delete_secret`/`list_all`
|
||
(metadata only, never content)/`exists`/`rel_path`/`attach`/`detach`. 1 MiB cap;
|
||
name validation rejects traversal/hidden/separators.
|
||
- ☑ `compose_edit_service`: `add_secret`/`remove_secret` (top-level
|
||
`secrets: {<name>: {file: <path>}}` + per-service list) and `add_config`/
|
||
`remove_config` (with `source`/`target` mount); top-level defs pruned when unused.
|
||
- ☑ `routers/secrets.py` (prefix `/api/stacks/{stack_id}/secrets`): list/write/
|
||
delete/attach/detach, admin-only, audit `secret.*`, content write-only.
|
||
- ☑ Agent `/agent/stacks/{stack_id}/secrets/*` (agent_app.py) + proxy
|
||
`/api/agents/{id}/stacks/{stack_id}/secrets/*` (agents.py), audit `agent.secret.*`.
|
||
|
||
### Frontend
|
||
- ☑ `SecretsPanel` (api/secrets.ts + components/stacks/SecretsPanel.tsx): create
|
||
(type/name/content; content cleared after save, never re-shown), list (name,
|
||
kind, size), delete, and per-row attach/detach to a service (config rows take a
|
||
mount target). Admin-gated. Wired as a **Secrets** tab on both StackDetail and
|
||
RemoteStackDetail (agentId-aware → multi-host).
|
||
|
||
### Verify
|
||
- ☑ Sandbox: traversal/hidden/separator names rejected; dir 0700 / file 0600;
|
||
`list` returns metadata only, never content (unit-tested in the backend image).
|
||
- ☑ compose round-trip: add secret+config → remove both → back to clean YAML
|
||
(top-level defs pruned). add/remove for secrets and configs unit-tested.
|
||
- ☑ py_compile + backend image build + frontend `tsc -b && vite build` + route
|
||
smoke-test (local CRUD/attach/detach, agent, proxy all registered).
|
||
- ☐ **Live hardware-verify debt:** create a secret, attach via the panel, deploy,
|
||
exec in and confirm `/run/secrets/<name>` holds the content (needs a running
|
||
stack on real hardware). Swarm path intentionally not built (decision A).
|
||
|
||
### Open risks (carried)
|
||
- Secret file ownership/permissions inside the container vs on host (file-based
|
||
secrets mount the host file → uid/gid must be readable by the service user).
|
||
- Multi-host: the agent stores the file on its own host, so the secret lives where
|
||
that host's Docker daemon can read it — verified by design (relative `file:`),
|
||
pending the live exec check above.
|
||
|
||
---
|
||
|
||
## Phase 25 — Templates as stack folders ☑ DONE — shipped 0.31.0
|
||
|
||
Templates reworked from DB rows + `manifest.json` + `{{VAR}}` mustache rendering
|
||
to **stack-shaped folders**: `backend/templates/<slug>/` with `compose.yaml`,
|
||
optional `.env.example` and a `template.json` (name/description/tags/gpu).
|
||
"Pull" copies the whole folder into a new stack (`.env.example` → `.env`);
|
||
custom templates live under `${DATA_DIR}/templates/` and are written by
|
||
"Save as template" (stack detail) or the manual save endpoint.
|
||
|
||
- Backend: `template_service` rewritten (folder scan, traversal-guarded resolve,
|
||
`copy_into_stack`, `save_from_stack`); `Template` DB table dropped with a
|
||
one-time startup migration (`{{VAR}}` → `${VAR}`, vars → `.env.example`).
|
||
- API: `POST /api/templates/from-stack` new; instantiate no longer takes `values`.
|
||
- Frontend: Templates page shows compose/env preview + file list, delete for
|
||
custom templates; StackDetail gains "Save as template".
|
||
|
||
---
|
||
|
||
## After 23
|
||
Remaining un-built ideas from the gap analysis (not chosen this round):
|
||
Health-monitoring & alerting (Docker-events → notify; note `/ws/events` already
|
||
exists), per-container live-stats + historical graphs, GitOps/deploy-from-Git,
|
||
image-management expansion (pull/remove/build arbitrary images + registry creds
|
||
UI), 2FA/TOTP + OIDC SSO + API tokens.
|