stackpilot/README.md

# StackPilot

A self-hosted Docker Compose manager for power users and homelab enthusiasts —
as intuitive as Dockge, as capable as Portainer for Compose workflows.

> **Status:** Phase 1 (Core) + Phase 2 (Volumes & GPU) + Phase 3 (Quality of
> Life) + Phase 4 (Operations) + Phase 5 (Multi-host) + Phase 6 (Backup
> destinations) + Phase 7 (Scheduled backups) + Phase 8 (Remote-stack backups)
> + Phase 9 (Networks) complete.

## What works today (Phase 1)

- **File-first stacks** — every stack is a plain `compose.yaml` (+ optional `.env`)
  on disk. The DB only stores metadata; nothing is locked in.
- **Auth** — JWT access/refresh tokens, bcrypt hashing, admin/user roles, and a
  first-launch setup wizard that creates the initial admin account.
- **Stack lifecycle** — create, edit, clone, delete, and `up / down / start /
  stop / restart / pull / update` via `docker compose`.
- **Live status** — running / partial / stopped / error / updating, computed from
  Docker container labels.
- **Real-time logs** — streamed over WebSocket, color-coded per service.
- **Monaco editor** — YAML editing with an `.env` tab and a **`docker run` →
  compose** converter.
- **Dashboard** — system resource bar, stack grid with quick actions, and a
  recent-activity audit feed.
- **Auto-discovery** — stacks created outside the UI (any folder under the stacks
  dir containing a compose file) are picked up automatically.
- **Dark / light theme.**

### Phase 2 — Volumes & GPU

- **Volume Wizard** in the editor (right-hand helper panel): Bind / Named / NFS /
  SMB-CIFS / tmpfs, with a sandboxed **host path browser** for bind mounts and a
  live YAML preview. NFS/SMB `driver_opts` are generated for you.
- **GPU assignment** per service: auto-detects NVIDIA (`nvidia-smi`) and AMD/Intel
  (`/dev/dri` + sysfs), injects the right YAML (NVIDIA `deploy.reservations`, or
  `/dev/dri` passthrough + render/video groups + `LIBVA_DRIVER_NAME=iHD` for Intel).
- **Device passthrough**: lists host USB / serial-TTY / DRI nodes, add per service,
  plus a guarded `privileged` toggle.
- All wizard edits are merged into the compose YAML **server-side** (robust,
  validated) and returned to the editor for review before saving.

> GPU/device detection needs host visibility. The bundled compose bind-mounts
> `/dev:/dev:ro`; NVIDIA additionally requires the NVIDIA container runtime on the host.

### Phase 3 — Quality of Life

- **Env editor**: table mode with sensitive-value masking (`PASS`/`SECRET`/`KEY`/…
  auto-detected) + raw mode, plus PUID/PGID/TZ quick-insert.
- **Image update checker**: background task compares the local manifest digest with
  the registry (Docker Hub / ghcr / lscr / private v2 with token auth); update
  badges on the Images page + an "updates available" banner on the dashboard.
- **Port conflict detector**: pre-deploy check against host-bound ports
  (`/proc/net/tcp[6]`) and running container bindings, with a confirm dialog.
- **Resource limits**: CPU/memory sliders in the editor → `deploy.resources.limits`.
- **Template library**: bundled templates (Jellyfin, Vaultwarden, Uptime-Kuma,
  Paperless-NGX, Gitea) with `{{VARIABLE}}` forms; save any stack as a custom template.
- **Healthcheck status** surfaced per container in the stack overview.

### Phase 4 — Operations

- **Backup & restore**: per-stack `.tar.gz` backups including named-volume contents
  (snapshotted via a throwaway helper container); restore via upload with optional
  rename, volume restore, and overwrite/conflict detection.
- **Notification webhooks**: ntfy, Discord, Slack, Gotify, or generic JSON, each
  subscribed to chosen events (image update available, stack start/stop/error,
  pull failed). Managed in **Settings → Notifications**; env `NOTIFY_WEBHOOKS`
  still supported for generic endpoints.
- **Settings page**: tune the update-check interval, manage webhooks, and manage
  users (create/disable/delete, promote/demote, with last-admin safeguards).
- **Audit log page**: searchable, paginated view of all recorded actions.
- **Mobile-responsive layout**: off-canvas sidebar + adaptive spacing.

### Phase 5 — Multi-host

- **Remote agents**: deploy `stackpilot-agent` (same image, different CMD) on any
  host — it needs only the Docker socket and a shared `AGENT_TOKEN`, and exposes a
  slim, token-guarded stack/system API (no UI, no DB).
- **Central management**: add hosts under **Settings → Remote hosts** (name, agent
  URL, token) with a live connectivity dot. The Stacks page groups stacks by host
  ("This host" + one section per agent); remote stacks have their own detail view
  with full lifecycle (start/stop/restart/pull/update/down), live logs, and
  compose/.env editing — all proxied to the agent.

### Phase 6 — Backup destinations

- **Off-box backups**: define **SFTP** or **S3-compatible** (MinIO, Backblaze B2,
  AWS S3, …) destinations under **Settings → Backup destinations** (with a Test
  button; secrets are masked in API responses).
- **Push & restore**: the stack Backup dialog can push straight to a destination
  instead of downloading; the Restore dialog can browse a destination's backups
  and restore (volumes included) directly from it. Remote backups can also be
  deleted from the UI.

### Phase 7 — Scheduled backups

- **Recurring backups**: schedule a stack to back up to a destination **hourly,
  daily, or weekly** (UTC) under **Settings → Scheduled backups**. A background
  scheduler runs due jobs every minute and records last/next run + status.
- **Retention**: keep the newest *N* backups per stack on the destination; older
  ones are pruned automatically.
- **Run now** for an on-demand run, plus a `backup_failed` notification event
  wired into the webhook system.

### Phase 8 — Remote-stack backups

- **Back up agent stacks**: the agent exposes its own backup/restore endpoints,
  and the main app streams a remote stack's backup through to a destination
  (credentials stay central — agents never see them). Remote stack detail has a
  **Backup** button; each host section on the Stacks page has a **Restore** button.
- **Schedule remote stacks**: a backup schedule can target a remote host; backups
  are namespaced per host (`backup-<host>-<stack>-…`) so retention never prunes
  across hosts sharing a destination.

### Phase 9 — Networks

- **Network management**: the Networks page lists Docker networks (driver, scope,
  subnet, attached containers / in-use, owning stack), with **create** (bridge /
  macvlan / ipvlan / overlay, optional subnet+gateway, internal/attachable),
  **delete** (default networks protected; in-use guarded by Docker), and **prune
  unused**.
- **Stack delete**: local stacks can now be deleted from the UI (stack detail and
  the stack card), with a confirm dialog and an optional "keep files on disk".

## Deploying an agent on another host

```bash
cd agent
cp .env.example .env          # set a strong AGENT_TOKEN
docker compose up -d          # exposes the agent on :5010
```

Then in the central UI: **Settings → Remote hosts → Add host** with
`http://<that-host>:5010` and the same `AGENT_TOKEN`.

## Architecture

```
frontend (React + Vite + Tailwind, served by nginx)
    │  proxies /api and /ws
    ▼
backend (FastAPI + docker-py + SQLite)
    │  docker-py + `docker compose` CLI
    ▼
Docker Engine (via /var/run/docker.sock — never exposed to the browser)
```

## Quick start

```bash
cd stackpilot
cp .env.example .env
# edit .env and set a strong SECRET_KEY:  openssl rand -base64 48
docker compose up -d --build
```

Open <http://localhost:5009> and complete the first-launch setup wizard to create
your admin account.

### Configuration

All backend settings are environment variables (see `backend/config.py`). The
most important ones:

| Variable        | Default            | Purpose                                   |
|-----------------|--------------------|-------------------------------------------|
| `SECRET_KEY`    | _(auto, dev only)_ | JWT signing key — **set this in prod**    |
| `STACKS_DIR`    | `/opt/stacks`      | Where stack folders live (in-container)   |
| `DATA_DIR`      | `/data`            | SQLite DB + app data                      |
| `CORS_ORIGINS`  | localhost          | Allowed API origins (comma separated)     |

The host path for stacks is set via `STACKS_HOST_DIR` in `.env`.

## Local development

Backend:

```bash
cd backend
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
SECRET_KEY=dev STACKS_DIR=../data/stacks DATA_DIR=../data uvicorn main:app --reload --port 5008
```

Frontend (proxies to the backend on :5008):

```bash
cd frontend
npm install
npm run dev   # http://localhost:5173
```

## API surface (Phase 1)

```
POST   /api/auth/setup | login | refresh        GET /api/auth/me | needs-setup
GET    /api/stacks                              POST /api/stacks
GET    /api/stacks/{id}                          PUT /api/stacks/{id}   DELETE /api/stacks/{id}
POST   /api/stacks/{id}/{start|stop|restart|pull|update|down|clone}
GET    /api/stacks/{id}/logs                     GET /api/stacks/{id}/export
POST   /api/stacks/convert                       (docker run → compose)
GET    /api/system/info | gpus | devices         GET /api/audit
WS     /ws/logs/{stack_id}[/{service}]           WS  /ws/events
```

### Phase 2 endpoints

```
GET    /api/volumes | /orphaned                  DELETE /api/volumes/{name}
POST   /api/volumes/prune                        POST /api/volumes/generate-yaml
GET    /api/host/paths?path=&show_hidden=        (sandboxed browser)
POST   /api/editor/services | add-volume | set-gpu | add-device | remove-device | set-privileged
```

### Phase 3 endpoints

```
GET    /api/images | /updates                    POST /api/images/check
POST   /api/ports/conflicts                       POST /api/editor/set-resources
GET    /api/templates | /{id}                     POST /api/templates/{id}/instantiate
POST   /api/templates                             DELETE /api/templates/custom/{slug}
```

### Phase 4 endpoints

```
GET    /api/stacks/{id}/backup?include_volumes=&stop_first=   POST /api/stacks/restore
GET    /api/settings                              PUT  /api/settings
GET    /api/settings/webhooks                     POST /api/settings/webhooks
PUT    /api/settings/webhooks/{id}                DELETE /api/settings/webhooks/{id}
POST   /api/settings/webhooks/{id}/test
GET    /api/auth/users                            POST /api/auth/users
PATCH  /api/auth/users/{id}                        DELETE /api/auth/users/{id}
```

### Phase 5 endpoints

```
GET    /api/agents                                POST /api/agents
PUT    /api/agents/{id}                            DELETE /api/agents/{id}
POST   /api/agents/{id}/ping                       GET  /api/agents/{id}/system
GET    /api/agents/{id}/stacks | /{sid}            GET  /api/agents/{id}/stacks/{sid}/logs
POST   /api/agents/{id}/stacks                     PUT  /api/agents/{id}/stacks/{sid}
DELETE /api/agents/{id}/stacks/{sid}               POST /api/agents/{id}/stacks/{sid}/{action}

agent (on the remote host, Bearer AGENT_TOKEN):
GET    /agent/ping | /system | /stacks | /stacks/{id} | /stacks/{id}/logs
GET    /agent/stacks/{id}/backup                   POST /agent/stacks/restore
POST   /agent/stacks | /stacks/{id}/{action}       PUT/DELETE /agent/stacks/{id}
```

### Phase 6 endpoints

```
GET    /api/backups/destinations                  POST /api/backups/destinations
PUT    /api/backups/destinations/{id}              DELETE /api/backups/destinations/{id}
POST   /api/backups/destinations/{id}/test         GET  /api/backups/destinations/{id}/backups
DELETE /api/backups/destinations/{id}/backups/{name}
POST   /api/stacks/{id}/backup/push                POST /api/stacks/restore-from
```

### Phase 7 endpoints

```
GET    /api/backups/schedules                     POST /api/backups/schedules
PUT    /api/backups/schedules/{id}                 DELETE /api/backups/schedules/{id}
POST   /api/backups/schedules/{id}/run
```

### Phase 8 endpoints (remote-stack backups)

```
GET    /api/agents/{id}/stacks/{sid}/backup        POST /api/agents/{id}/stacks/{sid}/backup/push
POST   /api/agents/{id}/stacks/restore             POST /api/agents/{id}/stacks/restore-from
backup schedules accept an optional agent_id to target a remote host.
```

### Phase 9 endpoints

```
GET    /api/networks | /{id}                       POST /api/networks
DELETE /api/networks/{id}                           POST /api/networks/prune
DELETE /api/stacks/{id}?delete_files=             (stack delete, now surfaced in the UI)
```

## Security notes

- The Docker socket is only ever touched by the backend process; it is never
  proxied to the browser.
- Login is rate-limited (10/min/IP).
- Compose files are backed up to `*.bak` before every overwrite.
- Generated YAML never includes the obsolete `version:` field and uses Compose v2
  (`docker compose`) syntax.