A backup holds the stack lock for up to twelve hours — a large volume
legitimately takes that long — and stops the stack's containers for a
consistent snapshot. One that died without unwinding therefore left the
stack stopped and refusing every start, stop, restart and deploy with
"Stack 'npmplus' is busy: backup in progress", with nothing to click
until the TTL ran out.
Backups and restores now register themselves in stack_job_service while
they run, with their phase and the helper containers they stream
through. Cancelling sets a flag and removes those containers, which is
what actually ends the long read: the backup then unwinds through its
own finally — the half-written archive goes, the containers it stopped
are started again, the lock is released. Checks sit between two files
and two volumes, never inside one. A restore is not cancellable at all;
it is halfway through writing a stack's data back.
POST /api/stacks/{id}/unlock does that and waits for it. A backup lock
with no job behind it is a leftover, so it is released on one click; a
deploy lock takes an explicit force, because compose keeps running past
its request. GET /api/stacks/{id}/lock and the stack detail response say
what holds the stack and since when, and the stack page turns that into
"Stop backup" / "Release lock".
Startup now clears every lock rather than only expired ones: after a
restart nothing of ours is in flight, the same reasoning the leftover
helper containers already follow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A review of the backup path found several ways a backup was not one, and a
restore could destroy data it did not contain.
Data loss:
- Restore with overwrite rmtree'd the whole stack directory, including
folders the backup had left out (over 2 GiB, deselected, an NFS share
mounted inside). It now clears only what the backup holds, keeps the
excluded paths (recorded in the manifest) and never crosses into another
filesystem.
- The stack was only stopped when named volumes were selected, so data
folders inside the stack directory were archived while in use. Any
selected data now stops it.
- Retention matched by prefix: pruning "app" deleted "app-db"'s backups.
It now matches backup-<id>-(full|config)-<date>.tar.gz exactly.
- A restore under a new name wrote into the original's volumes when they
had an explicit name, and into its folders outside the stack directory.
Both are skipped now; a volume owned by another stack or mounted by a
running container is never overwritten.
- A truncated archive was only noticed after the old stack was gone. The
whole archive is verified first, and the stack directory is unpacked
next to the old one and swapped in.
Remote storage:
- Bind folders on a mounted share (fstab/autofs), bind-type volumes
pointing at one and shares mounted inside the stack folder are detected
by filesystem type and not selected; a folder du cannot measure in 20s
is no longer selected by default either.
Reliability:
- After a backup exactly the containers that ran are started again with
compose start (falling back to the Docker API), instead of up -d, which
started services stopped on purpose and applied undeployed edits. A
stack that cannot be stopped is not backed up.
- Backup and restore hold the stack lock; a scheduled backup that finds
the stack busy retries after 10 minutes. Restore stops the stack while
replacing its data and brings it back up.
- external: true volumes are backed up; anonymous ones are listed, not
selected. Restored volumes get their original driver and options.
- A backup with uncaptured data or a stack that did not restart is
reported as incomplete: in the dialog, and for schedules as a warning
with a notification and without rotating older backups out.
- Archive writing and restore run off the event loop; downloads delete
their temp file; helper containers use the long-timeout client; S3
listing is paginated; a file changing mid-read no longer corrupts the
archive.
Manifest format 3 records compose_binds/excluded and volume
driver/options/external; format 1/2 backups still restore.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A stack's real state lives in its bind-mounted config directories, and those
were never captured: the backup only tarred the stack folder as this container
sees it. When STACKS_HOST_DIR differs from the container's STACKS_DIR, compose
resolves ./config against the container path and the daemon creates it at that
path on the *host* — invisible here, so the archive held little more than
compose.yaml and .env.
New services/stack_assets_service.py inventories a stack's data (bind sources
merged from container mounts + the compose file, named volumes) and does all
data I/O through a throwaway helper container, i.e. by host path, so unseen
directories are captured anyway. It also detects the host/container stacks-path
mismatch and reports it.
- manifest v2: full inventory, per-asset capture result, skip reasons (v1 still
restores)
- NFS/CIFS-backed volumes are skipped by default and never wiped on restore
- deselected data inside the stack folder no longer sneaks in via compose/
- volume/bind archives stream through temp files instead of RAM
- restore preserves mode, ownership, mtime and symlinks, and writes bind folders
back to their host paths (rewritten when the stack is renamed)
- backup dialog shows the inventory with sizes and per-item checkboxes; restore
gained a "restore bind folders" toggle
- new GET /api/stacks/{id}/backup/inventory (+ agent + proxy), backup endpoints
take include_binds/binds/volumes, restore takes restore_binds
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- backup_filename() crashed with NameError (bare now()) since 0.8.0 —
broke every scheduled backup at the upload step, agent backup download
and the central remote-backup/push endpoints. The local manual path
worked only because the router had its own copy (now an alias).
- restore: the manifest stack_id from an uploaded backup is now slugified
too — a crafted '../../...' id could previously escape STACKS_DIR.
- create_backup no longer starts a previously-stopped stack (stop/restart
only when the stack was actually running).
- overwrite-restore wipes the existing volume contents before extracting,
so files created since the backup no longer survive underneath it.
Verified end-to-end: full/config backup contents (compose, .env, .secrets,
bind dirs, extras, volume tars), delete→restore round-trip incl. volume
data, rename restore with volume re-prefixing, 409 conflict + overwrite,
traversal guard, scheduled run + retention prune + restore-from against
real MinIO, and the complete remote-agent cycle (download/push/restore).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
