An NFSv3 mount with locking makes the kernel register its lock manager
with rpcbind on the Docker host itself. On a host without nfs-common the
mount fails with a bare "connection refused" that reads like the NAS is
unreachable, even though the NAS and its ports are fine.
Only StackPilot ever writes the backup files on an NFS destination, one
writer per file, so nothing needs locking. mount_options() now adds
nolock unless the options already choose (lock, nolock, local_lock=) or
ask for NFSv4, which locks in-protocol and needs no rpcbind. Existing
destinations get their volume recreated once with the new options.
The refused-mount hint now points at rpcbind/nfs-common on the Docker
host first, and "protocol not supported" explains the missing kernel
module (modprobe nfsv4). The form labels the option.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docker-py's containers.run(remove=True) only removes a container after it
ran. A helper whose start failed - an NFS destination refusing the mount -
stayed behind in "Created", holding the destination's volume. Every Test
click and scheduled backup added one (a user found 18), the volume could
not be deleted even with everything stopped, and changed mount options
never took effect because the volume could not be recreated with them.
- docker_client.run_container: create, start, wait, collect output and
always remove, also when the start fails; pulls a missing image. Used
for NFS destination I/O, bind-path inspection, the volume wipe before a
restore and the Trivy scanner.
- Every helper container is labelled stackpilot.helper; on startup the
stopped ones are removed, together with anything left on an NFS
destination volume by older versions.
- When an NFS destination's options change, stopped containers holding
its volume are removed before it is recreated; a running one is now a
clear error instead of silently keeping the old options.
- A refused mount explains itself: NFS enabled and reachable from the
Docker host, NFSv3 needing rpcbind and mountd, and nfsvers=4 needing
only port 2049.
- The scanner waits for Trivy's own 10-minute timeout instead of giving
up after the default client's 60 seconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A review of the backup path found several ways a backup was not one, and a
restore could destroy data it did not contain.
Data loss:
- Restore with overwrite rmtree'd the whole stack directory, including
folders the backup had left out (over 2 GiB, deselected, an NFS share
mounted inside). It now clears only what the backup holds, keeps the
excluded paths (recorded in the manifest) and never crosses into another
filesystem.
- The stack was only stopped when named volumes were selected, so data
folders inside the stack directory were archived while in use. Any
selected data now stops it.
- Retention matched by prefix: pruning "app" deleted "app-db"'s backups.
It now matches backup-<id>-(full|config)-<date>.tar.gz exactly.
- A restore under a new name wrote into the original's volumes when they
had an explicit name, and into its folders outside the stack directory.
Both are skipped now; a volume owned by another stack or mounted by a
running container is never overwritten.
- A truncated archive was only noticed after the old stack was gone. The
whole archive is verified first, and the stack directory is unpacked
next to the old one and swapped in.
Remote storage:
- Bind folders on a mounted share (fstab/autofs), bind-type volumes
pointing at one and shares mounted inside the stack folder are detected
by filesystem type and not selected; a folder du cannot measure in 20s
is no longer selected by default either.
Reliability:
- After a backup exactly the containers that ran are started again with
compose start (falling back to the Docker API), instead of up -d, which
started services stopped on purpose and applied undeployed edits. A
stack that cannot be stopped is not backed up.
- Backup and restore hold the stack lock; a scheduled backup that finds
the stack busy retries after 10 minutes. Restore stops the stack while
replacing its data and brings it back up.
- external: true volumes are backed up; anonymous ones are listed, not
selected. Restored volumes get their original driver and options.
- A backup with uncaptured data or a stack that did not restart is
reported as incomplete: in the dialog, and for schedules as a warning
with a notification and without rotating older backups out.
- Archive writing and restore run off the event loop; downloads delete
their temp file; helper containers use the long-timeout client; S3
listing is paginated; a file changing mid-read no longer corrupts the
archive.
Manifest format 3 records compose_binds/excluded and volume
driver/options/external; format 1/2 backups still restore.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repo had no tests, no lint config, and a CI that went straight from push
to docker push. That is the reason F1 could ship: authorization lives in the
routers, each of 171 routes independently picks require_admin or
get_current_user, and nothing checked the choice was right.
670 tests, no Docker daemon needed. The app is driven through TestClient
without entering it as a context manager, which skips the lifespan — no
background loops, no socket — and conftest points DATA_DIR/STACKS_DIR at a
temp directory before anything is imported.
test_route_authorization.py is the load-bearing one. Rather than 171 implied
decisions it states the policy once — every route requires admin unless it is
listed in USER_READABLE or PUBLIC — and fails on any route that disagrees. A
new route defaults to admin, which is the safe direction; what it catches is a
route written with get_current_user that nobody weighed against "can this
return a credential". Writing the allowlist meant auditing all 53 user-readable
routes, which turned up one more leak: GET /api/templates/{id} returns a
template's env, and "save stack as template" snapshots the stack's real .env
into it. Now admin-only; the listing stays open.
test_agent_authorization.py pins the same invariant on the agent, where the
whole access model is one shared token declared per route and a single
forgotten Depends(verify_token) would hand over the host.
Both were checked by reintroducing the bug: re-opening /api/files/read fails
three tests with actionable messages, dropping a token guard fails two.
test_bundled_templates.py covers the 83 templates — parse, image per service,
.env.example in sync with what compose reads, every bind-mounted file actually
shipped, and no working default password. It found one on its first run:
authentik shipped PG_PASS=change-me and AUTHENTIK_SECRET_KEY=change-me against
a compose that marks both required, so the stack would have come up with a
known password instead of refusing to start. Fixed.
The rest ports the ad-hoc harnesses from 0.44.0 into permanent tests (crypto
round-trip incl. plaintext passthrough and key-loss handling, the browse
sandbox) and covers compose_service's slug/status/file handling and
secret_service's name validation.
ruff is configured as a floor, not a style bar: F, E9 and B only. Import
sorting is deliberately out — it is style, and enabling it would rewrite the
imports of nine files that have nothing else wrong. The 12 findings it did have
are fixed here (unused imports, an unused local, four raise-without-from that
were swallowing exception context).
CI now runs check (ruff, pytest, tsc) and only builds if it passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
F1 — Any authenticated user could read any file the backend could see.
/api/files/read and /download hung on get_current_user, and the sandbox that
should have caught that was open by default: ALLOWED_BROWSE_ROOTS contained
"/", for which _is_allowed() waves through every path. So the `user` role could
download stackpilot.db (password hashes, agent tokens, backup credentials),
every stack's .env and every .secrets/* file — with no audit trail, because
only mutations were logged.
Implementing that turned up three more doors into the same room, all fixed
here since closing only the first would have made the fix cosmetic:
GET /api/stacks/{id} handed the .env to any user, /export tarred the whole
stack dir including .secrets/*, and both the agent file proxies and
/api/agents/{id}/stacks/{id} repeated the leak for every remote host. All 24
filesystem-touching routes are now admin-only; reads and downloads are audited
(listing is not — the Files page polls it). DATA_DIR is refused outright, since
the API deliberately masks agent tokens and destination secrets and the browser
would otherwise be the way around that. "/" is out of the default browse roots.
F2 — Backup destination credentials were plaintext JSON in the DB, which is
what made F1 worth exploiting. They are now Fernet-encrypted at rest behind
parse_config/dump_config, with existing rows migrated at startup.
This needed a prerequisite from F6: the key is derived from SECRET_KEY, which
was regenerated on every boot when unset. Encrypting against a key that changes
per restart would be worse than plaintext, so an auto-generated SECRET_KEY is
now persisted to ${DATA_DIR}/secret_key at mode 0600. Sessions surviving a
restart is a welcome side effect.
F3 — /api/audit is admin-only. Also hidden from the dashboard and the nav for
non-admins, so nobody polls into a 403.
F4 — uvicorn now runs with --proxy-headers, so nginx's X-Forwarded-For is
honoured. Without it request.client.host was the frontend container's IP for
every request, which made the login rate limit global instead of per-IP (10
failures locked out everyone) and filled the audit log's IP column with one
useless value.
Verified: encrypt/decrypt round-trip incl. plaintext passthrough, idempotent
re-encryption and wrong-key handling; sandbox denial for DATA_DIR, traversal
into it, and paths outside the roots, with the allowed roots still reachable.
Both against stubbed settings — there is no Docker here, so nothing was run
end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
Uploading a backup extracted the tar straight into the NFS mount via
put_archive, and the daemon chowns every entry while extracting — an export
with root_squash refuses that ("failed to Lchown ... for UID 0, GID 0:
operation not permitted"), so the upload died with a docker 500 even though
plain writes to the share work (which is why the destination test passed).
The helper container now unpacks into its own filesystem and copies the file
into the mount with cat, which never chowns. Restores hit the same wall when a
volume or bind folder lives on a squashed mount, so import_path/import_volume
fall back to a copy-through-staging when (and only when) the failure is a chown
denial — local restores keep preserving ownership. NFS file names are validated
against the same safe charset as the subdir parts, since both are interpolated
into the helper's shell commands.
Verified against a real root_squash NFS export: test/upload/list/download/delete
round trip, byte-identical download, restore into an NFS-backed volume via the
fallback, and ownership still preserved (1000:1000, 0600) on a local volume.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_ensure_helper_image moved to stack_assets_service, but backup_destination_service
imports it lazily inside _nfs_run/_nfs_helper, so nothing failed at import time —
every NFS destination operation raised ImportError at runtime instead. The helper
is now a public ensure_helper_image() and the NFS helpers import it from its new
home. Verified: every services/ and routers/ module imports, and both NFS helper
paths run through to a Docker call instead of ImportError.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
New destination type 'nfs' alongside SFTP and S3. The StackPilot container
needs no mount privileges: the Docker daemon mounts the export as a named
volume (stackpilot-nfs-dest-<id>, driver local/type nfs, recreated whenever
server/path/options change) and all file I/O runs through throwaway helper
containers (BACKUP_HELPER_IMAGE) — upload via put_archive, list via stat,
download via get_archive, delete/test via short-lived runs. Config: server,
export path, mount options (default rw), optional subdirectory (sanitized;
shell-safe charset). Mount failures surface as clean destination errors.
Settings UI gains the NFS form + summary; works everywhere destinations are
used (push, restore-from, scheduled backups incl. retention).
Verified live against a real kernel NFS server: test, push (file on the
export), list, restore-from incl. volume data, remote delete, config change
recreates the mount volume, unreachable server fails cleanly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- BackupDestination model + backup_destination_service (SFTP via paramiko,
S3-compatible via boto3): upload/list/download/delete/test.
- routers/destinations.py: destinations CRUD (secrets masked, merge-on-update),
test, list/delete remote backups. backups.py: POST /{id}/backup/push and
POST /restore-from (download from a destination + restore, volumes included).
- Frontend: Settings → Backup destinations (SFTP/S3 forms + test); Backup dialog
can push to a destination; Restore dialog can pick a destination + backup.
- deps: paramiko 3.5.0, boto3 1.35.99.
Verified end-to-end against live MinIO + atmoz/sftp: create/test destinations,
push (incl. volumes), list, restore-from to a fresh stack (volume data intact),
delete remote backup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
