# StackPilot A self-hosted Docker Compose manager for power users and homelab enthusiasts — as intuitive as Dockge, as capable as Portainer for Compose workflows. > **Status:** Phase 1 (Core) + Phase 2 (Volumes & GPU) + Phase 3 (Quality of > Life) + Phase 4 (Operations) + Phase 5 (Multi-host) + Phase 6 (Backup > destinations) + Phase 7 (Scheduled backups) + Phase 8 (Remote-stack backups) > + Phase 9 (Networks) + Phase 10 (iGPU passthrough) + Phase 11 (Remote UX & > network attach) + Phase 12 (File browser) + Phase 13 (Multi-host networks & > images) + Phase 14 (Multi-host file browser) + Phase 15 (Dashboard stack > resource usage) + Phase 16 (Volumes page, multi-host) complete. ## What works today (Phase 1) - **File-first stacks** — every stack is a plain `compose.yaml` (+ optional `.env`) on disk. The DB only stores metadata; nothing is locked in. - **Auth** — JWT access/refresh tokens, bcrypt hashing, admin/user roles, and a first-launch setup wizard that creates the initial admin account. - **Stack lifecycle** — create, edit, clone, delete, and `up / down / start / stop / restart / pull / update` via `docker compose`. - **Live status** — running / partial / stopped / error / updating, computed from Docker container labels. - **Real-time logs** — streamed over WebSocket, color-coded per service. - **Monaco editor** — YAML editing with an `.env` tab and a **`docker run` → compose** converter. - **Dashboard** — system resource bar, stack grid with quick actions, and a recent-activity audit feed. - **Auto-discovery** — stacks created outside the UI (any folder under the stacks dir containing a compose file) are picked up automatically. - **Dark / light theme.** ### Phase 2 — Volumes & GPU - **Volume Wizard** in the editor (right-hand helper panel): Bind / Named / NFS / SMB-CIFS / tmpfs, with a sandboxed **host path browser** for bind mounts and a live YAML preview. NFS/SMB `driver_opts` are generated for you. - **GPU assignment** per service: auto-detects NVIDIA (`nvidia-smi`) and AMD/Intel (`/dev/dri` + sysfs), injects the right YAML (NVIDIA `deploy.reservations`, or `/dev/dri` passthrough + render/video groups + `LIBVA_DRIVER_NAME=iHD` for Intel). - **Device passthrough**: lists host USB / serial-TTY / DRI nodes, add per service, plus a guarded `privileged` toggle. - All wizard edits are merged into the compose YAML **server-side** (robust, validated) and returned to the editor for review before saving. > GPU/device detection needs host visibility. The bundled compose bind-mounts > `/dev:/dev:ro`; NVIDIA additionally requires the NVIDIA container runtime on the host. ### Phase 3 — Quality of Life - **Env editor**: table mode with sensitive-value masking (`PASS`/`SECRET`/`KEY`/… auto-detected) + raw mode, plus PUID/PGID/TZ quick-insert. - **Image update checker**: background task compares the local manifest digest with the registry (Docker Hub / ghcr / lscr / private v2 with token auth); update badges on the Images page + an "updates available" banner on the dashboard. - **Port conflict detector**: pre-deploy check against host-bound ports (`/proc/net/tcp[6]`) and running container bindings, with a confirm dialog. - **Resource limits**: CPU/memory sliders in the editor → `deploy.resources.limits`. - **Template library**: bundled templates (Jellyfin, Vaultwarden, Uptime-Kuma, Paperless-NGX, Gitea) with `{{VARIABLE}}` forms; save any stack as a custom template. - **Healthcheck status** surfaced per container in the stack overview. ### Phase 4 — Operations - **Backup & restore**: per-stack `.tar.gz` backups including named-volume contents (snapshotted via a throwaway helper container); restore via upload with optional rename, volume restore, and overwrite/conflict detection. - **Notification webhooks**: ntfy, Discord, Slack, Gotify, or generic JSON, each subscribed to chosen events (image update available, stack start/stop/error, pull failed). Managed in **Settings → Notifications**; env `NOTIFY_WEBHOOKS` still supported for generic endpoints. - **Settings page**: tune the update-check interval, manage webhooks, and manage users (create/disable/delete, promote/demote, with last-admin safeguards). - **Audit log page**: searchable, paginated view of all recorded actions. - **Mobile-responsive layout**: off-canvas sidebar + adaptive spacing. ### Phase 5 — Multi-host - **Remote agents**: deploy `stackpilot-agent` (same image, different CMD) on any host — it needs only the Docker socket and a shared `AGENT_TOKEN`, and exposes a slim, token-guarded stack/system API (no UI, no DB). - **Central management**: add hosts under **Settings → Remote hosts** (name, agent URL, token) with a live connectivity dot. The Stacks page groups stacks by host ("This host" + one section per agent); remote stacks have their own detail view with full lifecycle (start/stop/restart/pull/update/down), live logs, and compose/.env editing — all proxied to the agent. ### Phase 6 — Backup destinations - **Off-box backups**: define **SFTP** or **S3-compatible** (MinIO, Backblaze B2, AWS S3, …) destinations under **Settings → Backup destinations** (with a Test button; secrets are masked in API responses). - **Push & restore**: the stack Backup dialog can push straight to a destination instead of downloading; the Restore dialog can browse a destination's backups and restore (volumes included) directly from it. Remote backups can also be deleted from the UI. ### Phase 7 — Scheduled backups - **Recurring backups**: schedule a stack to back up to a destination **hourly, daily, or weekly** (UTC) under **Settings → Scheduled backups**. A background scheduler runs due jobs every minute and records last/next run + status. - **Retention**: keep the newest *N* backups per stack on the destination; older ones are pruned automatically. - **Run now** for an on-demand run, plus a `backup_failed` notification event wired into the webhook system. ### Phase 8 — Remote-stack backups - **Back up agent stacks**: the agent exposes its own backup/restore endpoints, and the main app streams a remote stack's backup through to a destination (credentials stay central — agents never see them). Remote stack detail has a **Backup** button; each host section on the Stacks page has a **Restore** button. - **Schedule remote stacks**: a backup schedule can target a remote host; backups are namespaced per host (`backup---…`) so retention never prunes across hosts sharing a destination. ### Phase 9 — Networks - **Network management**: the Networks page lists Docker networks (driver, scope, subnet, attached containers / in-use, owning stack), with **create** (bridge / macvlan / ipvlan / overlay, optional subnet+gateway, internal/attachable), **delete** (default networks protected; in-use guarded by Docker), and **prune unused**. - **Stack delete**: local stacks can now be deleted from the UI (stack detail and the stack card), with a confirm dialog and an optional "keep files on disk". ### Phase 10 — iGPU passthrough - **Render/video group detection**: for a passed-through Intel/AMD iGPU, StackPilot reads the host group ownership of the `/dev/dri` nodes (render node → `render` GID, paired `card` node → `video` GID) and injects them as **numeric** `group_add` entries (e.g. `group_add: ["991", "44"]`). Group names rarely resolve inside images, so the numeric GID is what actually grants access. The GPU selector shows the detected GIDs; removal cleans them (and `LIBVA_DRIVER_NAME`). ### Phase 11 — Remote UX & network attach - **Live remote logs**: remote-stack logs now stream over a WebSocket proxied through the central app to the agent (`/ws/agent-logs/{agent}/{stack}`), instead of polling — same live viewer as local stacks. - **Deploy to a remote host from the UI**: the New Stack editor and the template dialog gained a *host* selector. Pick an online agent and the stack is created (and optionally started) on that host; you land on its remote detail page. - **Network attach/detach**: each network row on the Networks page expands to an inspect view listing connected containers, with admin controls to disconnect a container or connect any container on the host (`POST /api/networks/{id}/connect` / `/disconnect`). ### Phase 16 — Volumes page (multi-host) - **New Volumes page** (sidebar) with per-host sections (local + each online agent, like Networks/Images). Lists Docker volumes with driver, owning stack, in-use containers and mountpoint. - Admin actions: delete a volume (with an in-use warning + force option) and **Prune unused**; an *Only unused* filter. New agent endpoints `/agent/volumes` (list/delete/prune), proxied at `/api/agents/{id}/volumes/*`. - The Volume **Wizard** in the stack editor (bind/named/NFS/SMB/tmpfs YAML generation) is unchanged — the new page is for managing/cleaning up volumes. ### Phase 15 — Dashboard stack resource usage - **The dashboard now lists stacks in a table** (status, services) with live **CPU** and **memory** usage per stack, sampled from `docker stats` and aggregated by compose project. - When a stack has `deploy.resources.limits` assigned, the bar fills toward that limit and shows usage vs the limit (e.g. `0.42 / 1 cores`, `310 MB / 512 MB`); otherwise it shows absolute usage against the host total. Inline start/stop/ restart actions per row for admins. New endpoint `GET /api/stacks/stats`. ### Phase 14 — Multi-host file browser - **The Files page now has a host switcher.** When agents are registered, a *Host* dropdown at the top switches the whole browser between the local host and any online agent; switching resets the path and clipboard. - All file operations (browse, view/edit, create, rename, copy/move, delete, upload files & folders, download) work against the selected agent, sandboxed by *that agent's* `ALLOWED_BROWSE_ROOTS`/`HOST_ROOT_PREFIX`. - New agent endpoints `/agent/files/*`, proxied at `/api/agents/{id}/files/*`. ### Phase 13 — Multi-host networks & images - **Networks and Images are now per-host.** Both pages render a section for the local host plus one for every registered agent (online dot included), exactly like the Stacks page. Each agent section talks to that host's Docker daemon. - **Remote network management**: list, inspect, create, delete, prune, and connect/disconnect containers on an agent host — including a *Prune unused* button, which resolves the common "all predefined address pools have been fully subnetted" deploy error without SSH. - **Remote images**: list image tags (with using-stacks) and run on-demand update checks per host. - New agent endpoints `/agent/networks/*` and `/agent/images/*`, proxied through the central app at `/api/agents/{id}/networks/*` and `/api/agents/{id}/images/*`. ### Phase 12 — File browser - **Files page (sidebar)**: a full host filesystem browser with breadcrumb navigation, clickable browse-root chips, an *Up* control, and a show/hide hidden-files toggle. Listings show size, permissions and modified time. - **View & edit**: clicking a text file opens it in a Monaco editor (with syntax highlighting picked from the extension). Binary and oversized files are detected and offered as a download instead. Admins can edit and **Save**. - **Manage** (admin): create folders/files, rename, delete (recursive for folders), upload files **or whole folders** (the directory tree is recreated server-side), and download any file. **Copy/cut & paste** moves files and folders between directories (clipboard bar + per-row copy/cut, with an overwrite prompt on conflict). Every mutation is audit-logged. - **Sandboxed**: all access is confined to `ALLOWED_BROWSE_ROOTS`; path traversal and deleting a browse root are refused. To reach the real host filesystem, mount it into the backend and set `HOST_ROOT_PREFIX` (see the commented `/:/host_root` volume in `docker-compose.yml`). Endpoints live under `/api/files/*` (`list`, `read`, `write`, `mkdir`, `touch`, `rename`, `copy`, `move`, `upload` — with optional `rel_path` for folder uploads —, `download`, `DELETE`). ## Deploying an agent on another host ```bash cd agent cp .env.example .env # set a strong AGENT_TOKEN docker compose up -d # exposes the agent on :5010 ``` Then in the central UI: **Settings → Remote hosts → Add host** with `http://:5010` and the same `AGENT_TOKEN`. ## Architecture ``` frontend (React + Vite + Tailwind, served by nginx) │ proxies /api and /ws ▼ backend (FastAPI + docker-py + SQLite) │ docker-py + `docker compose` CLI ▼ Docker Engine (via /var/run/docker.sock — never exposed to the browser) ``` ## Quick start ```bash cd stackpilot cp .env.example .env # edit .env and set a strong SECRET_KEY: openssl rand -base64 48 docker compose up -d --build ``` Open and complete the first-launch setup wizard to create your admin account. ### Configuration All backend settings are environment variables (see `backend/config.py`). The most important ones: | Variable | Default | Purpose | |-----------------|--------------------|-------------------------------------------| | `SECRET_KEY` | _(auto, dev only)_ | JWT signing key — **set this in prod** | | `STACKS_DIR` | `/opt/stacks` | Where stack folders live (in-container) | | `DATA_DIR` | `/data` | SQLite DB + app data | | `CORS_ORIGINS` | localhost | Allowed API origins (comma separated) | The host path for stacks is set via `STACKS_HOST_DIR` in `.env`. ## Local development Backend: ```bash cd backend python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt SECRET_KEY=dev STACKS_DIR=../data/stacks DATA_DIR=../data uvicorn main:app --reload --port 5008 ``` Frontend (proxies to the backend on :5008): ```bash cd frontend npm install npm run dev # http://localhost:5173 ``` ## API surface (Phase 1) ``` POST /api/auth/setup | login | refresh GET /api/auth/me | needs-setup GET /api/stacks POST /api/stacks GET /api/stacks/{id} PUT /api/stacks/{id} DELETE /api/stacks/{id} POST /api/stacks/{id}/{start|stop|restart|pull|update|down|clone} GET /api/stacks/{id}/logs GET /api/stacks/{id}/export POST /api/stacks/convert (docker run → compose) GET /api/system/info | gpus | devices GET /api/audit WS /ws/logs/{stack_id}[/{service}] WS /ws/events ``` ### Phase 2 endpoints ``` GET /api/volumes | /orphaned DELETE /api/volumes/{name} POST /api/volumes/prune POST /api/volumes/generate-yaml GET /api/host/paths?path=&show_hidden= (sandboxed browser) POST /api/editor/services | add-volume | set-gpu | add-device | remove-device | set-privileged ``` ### Phase 3 endpoints ``` GET /api/images | /updates POST /api/images/check POST /api/ports/conflicts POST /api/editor/set-resources GET /api/templates | /{id} POST /api/templates/{id}/instantiate POST /api/templates DELETE /api/templates/custom/{slug} ``` ### Phase 4 endpoints ``` GET /api/stacks/{id}/backup?include_volumes=&stop_first= POST /api/stacks/restore GET /api/settings PUT /api/settings GET /api/settings/webhooks POST /api/settings/webhooks PUT /api/settings/webhooks/{id} DELETE /api/settings/webhooks/{id} POST /api/settings/webhooks/{id}/test GET /api/auth/users POST /api/auth/users PATCH /api/auth/users/{id} DELETE /api/auth/users/{id} ``` ### Phase 5 endpoints ``` GET /api/agents POST /api/agents PUT /api/agents/{id} DELETE /api/agents/{id} POST /api/agents/{id}/ping GET /api/agents/{id}/system GET /api/agents/{id}/stacks | /{sid} GET /api/agents/{id}/stacks/{sid}/logs POST /api/agents/{id}/stacks PUT /api/agents/{id}/stacks/{sid} DELETE /api/agents/{id}/stacks/{sid} POST /api/agents/{id}/stacks/{sid}/{action} agent (on the remote host, Bearer AGENT_TOKEN): GET /agent/ping | /system | /stacks | /stacks/{id} | /stacks/{id}/logs GET /agent/stacks/{id}/backup POST /agent/stacks/restore POST /agent/stacks | /stacks/{id}/{action} PUT/DELETE /agent/stacks/{id} ``` ### Phase 6 endpoints ``` GET /api/backups/destinations POST /api/backups/destinations PUT /api/backups/destinations/{id} DELETE /api/backups/destinations/{id} POST /api/backups/destinations/{id}/test GET /api/backups/destinations/{id}/backups DELETE /api/backups/destinations/{id}/backups/{name} POST /api/stacks/{id}/backup/push POST /api/stacks/restore-from ``` ### Phase 7 endpoints ``` GET /api/backups/schedules POST /api/backups/schedules PUT /api/backups/schedules/{id} DELETE /api/backups/schedules/{id} POST /api/backups/schedules/{id}/run ``` ### Phase 8 endpoints (remote-stack backups) ``` GET /api/agents/{id}/stacks/{sid}/backup POST /api/agents/{id}/stacks/{sid}/backup/push POST /api/agents/{id}/stacks/restore POST /api/agents/{id}/stacks/restore-from backup schedules accept an optional agent_id to target a remote host. ``` ### Phase 9 endpoints ``` GET /api/networks | /{id} POST /api/networks DELETE /api/networks/{id} POST /api/networks/prune DELETE /api/stacks/{id}?delete_files= (stack delete, now surfaced in the UI) ``` ### Phase 11 endpoints ``` WS /ws/agent-logs/{agent_id}/{stack_id} (live remote logs, proxied to the agent) GET /api/networks/{id}/containers POST /api/networks/{id}/connect | /disconnect POST /api/agents/{id}/stacks (create a stack on a remote host — now in the UI) POST /api/templates/{id}/instantiate {agent_id} (instantiate a template onto a remote host) ``` ## Security notes - The Docker socket is only ever touched by the backend process; it is never proxied to the browser. - Login is rate-limited (10/min/IP). - Compose files are backed up to `*.bak` before every overwrite. - Generated YAML never includes the obsolete `version:` field and uses Compose v2 (`docker compose`) syntax.