Let a stuck backup be stopped instead of stranding the stack (0.67.0)
CI / check (push) Successful in 13m33s
CI / build-and-push (push) Successful in 1m51s

A backup holds the stack lock for up to twelve hours — a large volume
legitimately takes that long — and stops the stack's containers for a
consistent snapshot. One that died without unwinding therefore left the
stack stopped and refusing every start, stop, restart and deploy with
"Stack 'npmplus' is busy: backup in progress", with nothing to click
until the TTL ran out.

Backups and restores now register themselves in stack_job_service while
they run, with their phase and the helper containers they stream
through. Cancelling sets a flag and removes those containers, which is
what actually ends the long read: the backup then unwinds through its
own finally — the half-written archive goes, the containers it stopped
are started again, the lock is released. Checks sit between two files
and two volumes, never inside one. A restore is not cancellable at all;
it is halfway through writing a stack's data back.

POST /api/stacks/{id}/unlock does that and waits for it. A backup lock
with no job behind it is a leftover, so it is released on one click; a
deploy lock takes an explicit force, because compose keeps running past
its request. GET /api/stacks/{id}/lock and the stack detail response say
what holds the stack and since when, and the stack page turns that into
"Stop backup" / "Release lock".

Startup now clears every lock rather than only expired ones: after a
restart nothing of ours is in flight, the same reasoning the leftover
helper containers already follow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
menzeljandClaude Opus 5 committed 2026-09-21 13:39:17 +02:00
1 parent 052a85067b
commit 270ddda51d
19 files changed
+1018 -22

No files matched your search

+10 -1
View File
@@ -170,6 +170,11 @@ async def _backup_stack(
return "ok"
except backup_service.StackBusyError as exc:
raise _Busy(str(exc)) from exc
except backup_service.BackupCancelled as exc:
# Somebody stopped this backup from the UI. Not a failure of the
# schedule: say so, and don't page anyone with "backup failed".
logger.info("Scheduled backup of %s was cancelled: %s", stack_id, exc)
return f"cancelled: {exc}"
except Exception as exc: # noqa: BLE001
logger.warning("Scheduled backup of %s failed: %s", stack_id, exc)
return f"error: {exc}"
@@ -185,7 +190,11 @@ def _summarise(statuses: dict[str, str]) -> str:
bad = {sid: st for sid, st in statuses.items() if st != "ok"}
if not bad:
return "ok"
worst = "error" if any(not st.startswith("warning") for st in bad.values()) else "warning"
worst = (
"error"
if any(not st.startswith(("warning", "cancelled")) for st in bad.values())
else "warning"
)
if len(statuses) == 1:
return next(iter(statuses.values()))
detail = "; ".join(f"{sid}: {st}" for sid, st in sorted(bad.items()))