docker-py's containers.run(remove=True) only removes a container after it
ran. A helper whose start failed - an NFS destination refusing the mount -
stayed behind in "Created", holding the destination's volume. Every Test
click and scheduled backup added one (a user found 18), the volume could
not be deleted even with everything stopped, and changed mount options
never took effect because the volume could not be recreated with them.
- docker_client.run_container: create, start, wait, collect output and
always remove, also when the start fails; pulls a missing image. Used
for NFS destination I/O, bind-path inspection, the volume wipe before a
restore and the Trivy scanner.
- Every helper container is labelled stackpilot.helper; on startup the
stopped ones are removed, together with anything left on an NFS
destination volume by older versions.
- When an NFS destination's options change, stopped containers holding
its volume are removed before it is recreated; a running one is now a
clear error instead of silently keeping the old options.
- A refused mount explains itself: NFS enabled and reachable from the
Docker host, NFSv3 needing rpcbind and mountd, and nfsvers=4 needing
only port 2049.
- The scanner waits for Trivy's own 10-minute timeout instead of giving
up after the default client's 60 seconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A schedule covered exactly one stack, so backing up a host meant one
schedule per stack, each edited separately, and new stacks were only
backed up once someone remembered to add a schedule.
A schedule now takes a list of stacks, or all_stacks, which includes
stacks added later. They are backed up one after another, each with its
own file, upload, retention and status, so one failing stack does not
cost the others their backup:
- stack_status records each stack's result; last_status summarises
("error: 2 of 5 stacks - a: ...; c: ...") and one notification per run
names the stacks with a problem. The dashboard names them too and shows
an incomplete backup as a warning rather than an error.
- A busy stack (deploy, update) is retried after the others, up to three
times two minutes apart, replacing the 10-minute postponement of the
whole schedule.
- Run now starts the run in the background (202) instead of holding the
request, which a multi-stack run would outlast; the list reports
running and polls, and the row toasts the outcome. A schedule never
runs twice at once.
- Settings: "All stacks" or a checklist with select all/none; stacks can
be changed when editing; deleted stacks stay listed to be unticked.
Schedules from before keep working without a data migration: stack_ids,
all_stacks and stack_status are added with defaults on startup and a row
with only stack_id reads as that one stack. The API still accepts
stack_id and reports it for single-stack schedules.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A review of the backup path found several ways a backup was not one, and a
restore could destroy data it did not contain.
Data loss:
- Restore with overwrite rmtree'd the whole stack directory, including
folders the backup had left out (over 2 GiB, deselected, an NFS share
mounted inside). It now clears only what the backup holds, keeps the
excluded paths (recorded in the manifest) and never crosses into another
filesystem.
- The stack was only stopped when named volumes were selected, so data
folders inside the stack directory were archived while in use. Any
selected data now stops it.
- Retention matched by prefix: pruning "app" deleted "app-db"'s backups.
It now matches backup-<id>-(full|config)-<date>.tar.gz exactly.
- A restore under a new name wrote into the original's volumes when they
had an explicit name, and into its folders outside the stack directory.
Both are skipped now; a volume owned by another stack or mounted by a
running container is never overwritten.
- A truncated archive was only noticed after the old stack was gone. The
whole archive is verified first, and the stack directory is unpacked
next to the old one and swapped in.
Remote storage:
- Bind folders on a mounted share (fstab/autofs), bind-type volumes
pointing at one and shares mounted inside the stack folder are detected
by filesystem type and not selected; a folder du cannot measure in 20s
is no longer selected by default either.
Reliability:
- After a backup exactly the containers that ran are started again with
compose start (falling back to the Docker API), instead of up -d, which
started services stopped on purpose and applied undeployed edits. A
stack that cannot be stopped is not backed up.
- Backup and restore hold the stack lock; a scheduled backup that finds
the stack busy retries after 10 minutes. Restore stops the stack while
replacing its data and brings it back up.
- external: true volumes are backed up; anonymous ones are listed, not
selected. Restored volumes get their original driver and options.
- A backup with uncaptured data or a stack that did not restart is
reported as incomplete: in the dialog, and for schedules as a warning
with a notification and without rotating older backups out.
- Archive writing and restore run off the event loop; downloads delete
their temp file; helper containers use the long-timeout client; S3
listing is paginated; a file changing mid-read no longer corrupts the
archive.
Manifest format 3 records compose_binds/excluded and volume
driver/options/external; format 1/2 backups still restore.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>