Let a stuck backup be stopped instead of stranding the stack (0.67.0)
A backup holds the stack lock for up to twelve hours — a large volume
legitimately takes that long — and stops the stack's containers for a
consistent snapshot. One that died without unwinding therefore left the
stack stopped and refusing every start, stop, restart and deploy with
"Stack 'npmplus' is busy: backup in progress", with nothing to click
until the TTL ran out.
Backups and restores now register themselves in stack_job_service while
they run, with their phase and the helper containers they stream
through. Cancelling sets a flag and removes those containers, which is
what actually ends the long read: the backup then unwinds through its
own finally — the half-written archive goes, the containers it stopped
are started again, the lock is released. Checks sit between two files
and two volumes, never inside one. A restore is not cancellable at all;
it is halfway through writing a stack's data back.
POST /api/stacks/{id}/unlock does that and waits for it. A backup lock
with no job behind it is a leftover, so it is released on one click; a
deploy lock takes an explicit force, because compose keeps running past
its request. GET /api/stacks/{id}/lock and the stack detail response say
what holds the stack and since when, and the stack page turns that into
"Stop backup" / "Release lock".
Startup now clears every lock rather than only expired ones: after a
restart nothing of ours is in flight, the same reasoning the leftover
helper containers already follow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
052a85067b
commit
270ddda51d
19 files changed
+1018
-22
No files matched your search
@@ -469,6 +469,12 @@ it is what your saved destination credentials are encrypted with.
|
||||
holds across workers and across a restart) and a second one gets `409` while
|
||||
it is held; auto-update skips a stack somebody is already deploying. Locks
|
||||
carry an expiry, so a worker killed mid-deploy does not strand a stack.
|
||||
- **A stack is never stuck behind its own lock** (0.67.0) — the stack page says
|
||||
what holds it and since when, and offers to stop it. A running backup is
|
||||
asked to stop, throws its half-written archive away and starts the containers
|
||||
it stopped; a lock left behind by a backup that is no longer running is
|
||||
released on one click; anything else can be force-released. A restart clears
|
||||
every lock, because after it nothing of ours is running.
|
||||
- **Resilient UI** — a render error shows what broke and offers a way out
|
||||
instead of blanking the window, and clears itself when you navigate away.
|
||||
Routes are code-split, so the entry bundle is 377 kB rather than 841 kB and
|
||||
@@ -599,6 +605,12 @@ it is what your saved destination credentials are encrypted with.
|
||||
cannot be stopped is not backed up. The stack is locked meanwhile, so a
|
||||
deploy, update or scheduled backup cannot interleave (a scheduled backup
|
||||
that finds the stack busy retries after 10 minutes).
|
||||
- A backup can be stopped while it runs (0.67.0). It is a long operation
|
||||
that holds the stack — up to twelve hours for a large volume — so the stack
|
||||
page offers to cancel it: the helper container it streams through is
|
||||
removed, the archive is thrown away, and the containers it stopped are
|
||||
started again. A restore is not interruptible (it is halfway through
|
||||
writing a stack's data back); its lock can only be force-released.
|
||||
- Remote storage is kept out by what it *is*, not only by how it is declared:
|
||||
NFS/CIFS volumes, bind folders on a mounted share (`/mnt/nas/…` via fstab or
|
||||
autofs), bind-type volumes pointing at one, and shares mounted inside the
|
||||
@@ -1040,7 +1052,12 @@ leaked one.
|
||||
that moved into the database: that a busy stack answers 409 without ever
|
||||
reaching Docker, that an expired lock is taken over rather than stranding the
|
||||
stack, that the stats cache serves repeat callers from one sweep, and that the
|
||||
update cache and rate limiter survive a restart. One of them asserts that
|
||||
update cache and rate limiter survive a restart. Since 0.67.0 they also cover
|
||||
the way out of a lock — a backup that is asked to stop, a leftover backup lock
|
||||
that is released on one click, a deploy lock that takes an explicit force, and
|
||||
a restore that refuses to be interrupted at all — and
|
||||
`tests/test_backup_restore.py` that a cancelled backup starts the stack again,
|
||||
frees the lock and leaves no half-written archive behind. One of them asserts that
|
||||
`update_service` never imports the database — it is pure registry logic, and
|
||||
persistence stays opt-in so the module remains testable without one.
|
||||
|
||||
|
||||
Reference in new issue
Block a user