Let a stuck backup be stopped instead of stranding the stack (0.67.0)
CI / check (push) Successful in 13m33s
CI / build-and-push (push) Successful in 1m51s

A backup holds the stack lock for up to twelve hours — a large volume
legitimately takes that long — and stops the stack's containers for a
consistent snapshot. One that died without unwinding therefore left the
stack stopped and refusing every start, stop, restart and deploy with
"Stack 'npmplus' is busy: backup in progress", with nothing to click
until the TTL ran out.

Backups and restores now register themselves in stack_job_service while
they run, with their phase and the helper containers they stream
through. Cancelling sets a flag and removes those containers, which is
what actually ends the long read: the backup then unwinds through its
own finally — the half-written archive goes, the containers it stopped
are started again, the lock is released. Checks sit between two files
and two volumes, never inside one. A restore is not cancellable at all;
it is halfway through writing a stack's data back.

POST /api/stacks/{id}/unlock does that and waits for it. A backup lock
with no job behind it is a leftover, so it is released on one click; a
deploy lock takes an explicit force, because compose keeps running past
its request. GET /api/stacks/{id}/lock and the stack detail response say
what holds the stack and since when, and the stack page turns that into
"Stop backup" / "Release lock".

Startup now clears every lock rather than only expired ones: after a
restart nothing of ours is in flight, the same reasoning the leftover
helper containers already follow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
menzeljandClaude Opus 5 committed 2026-09-21 13:39:17 +02:00
1 parent 052a85067b
commit 270ddda51d
19 files changed
+1018 -22

No files matched your search

+18 -1
View File
@@ -469,6 +469,12 @@ it is what your saved destination credentials are encrypted with.
holds across workers and across a restart) and a second one gets `409` while
it is held; auto-update skips a stack somebody is already deploying. Locks
carry an expiry, so a worker killed mid-deploy does not strand a stack.
- **A stack is never stuck behind its own lock** (0.67.0) — the stack page says
what holds it and since when, and offers to stop it. A running backup is
asked to stop, throws its half-written archive away and starts the containers
it stopped; a lock left behind by a backup that is no longer running is
released on one click; anything else can be force-released. A restart clears
every lock, because after it nothing of ours is running.
- **Resilient UI** — a render error shows what broke and offers a way out
instead of blanking the window, and clears itself when you navigate away.
Routes are code-split, so the entry bundle is 377 kB rather than 841 kB and
@@ -599,6 +605,12 @@ it is what your saved destination credentials are encrypted with.
cannot be stopped is not backed up. The stack is locked meanwhile, so a
deploy, update or scheduled backup cannot interleave (a scheduled backup
that finds the stack busy retries after 10 minutes).
- A backup can be stopped while it runs (0.67.0). It is a long operation
that holds the stack — up to twelve hours for a large volume — so the stack
page offers to cancel it: the helper container it streams through is
removed, the archive is thrown away, and the containers it stopped are
started again. A restore is not interruptible (it is halfway through
writing a stack's data back); its lock can only be force-released.
- Remote storage is kept out by what it *is*, not only by how it is declared:
NFS/CIFS volumes, bind folders on a mounted share (`/mnt/nas/…` via fstab or
autofs), bind-type volumes pointing at one, and shares mounted inside the
@@ -1040,7 +1052,12 @@ leaked one.
that moved into the database: that a busy stack answers 409 without ever
reaching Docker, that an expired lock is taken over rather than stranding the
stack, that the stats cache serves repeat callers from one sweep, and that the
update cache and rate limiter survive a restart. One of them asserts that
update cache and rate limiter survive a restart. Since 0.67.0 they also cover
the way out of a lock — a backup that is asked to stop, a leftover backup lock
that is released on one click, a deploy lock that takes an explicit force, and
a restore that refuses to be interrupted at all — and
`tests/test_backup_restore.py` that a cancelled backup starts the stack again,
frees the lock and leaves no half-written archive behind. One of them asserts that
`update_service` never imports the database — it is pure registry logic, and
persistence stays opt-in so the module remains testable without one.