Let a stuck backup be stopped instead of stranding the stack (0.67.0)
A backup holds the stack lock for up to twelve hours — a large volume
legitimately takes that long — and stops the stack's containers for a
consistent snapshot. One that died without unwinding therefore left the
stack stopped and refusing every start, stop, restart and deploy with
"Stack 'npmplus' is busy: backup in progress", with nothing to click
until the TTL ran out.
Backups and restores now register themselves in stack_job_service while
they run, with their phase and the helper containers they stream
through. Cancelling sets a flag and removes those containers, which is
what actually ends the long read: the backup then unwinds through its
own finally — the half-written archive goes, the containers it stopped
are started again, the lock is released. Checks sit between two files
and two volumes, never inside one. A restore is not cancellable at all;
it is halfway through writing a stack's data back.
POST /api/stacks/{id}/unlock does that and waits for it. A backup lock
with no job behind it is a leftover, so it is released on one click; a
deploy lock takes an explicit force, because compose keeps running past
its request. GET /api/stacks/{id}/lock and the stack detail response say
what holds the stack and since when, and the stack page turns that into
"Stop backup" / "Release lock".
Startup now clears every lock rather than only expired ones: after a
restart nothing of ours is in flight, the same reasoning the leftover
helper containers already follow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
052a85067b
commit
270ddda51d
19 files changed
+1018
-22
No files matched your search
+10
-3
@@ -92,13 +92,20 @@ async def lifespan(app: FastAPI):
|
||||
except Exception as exc: # noqa: BLE001 - Docker may not be up yet
|
||||
logger.warning("Could not clean up helper containers: %s", exc)
|
||||
# Runtime state that used to live in module dicts and was lost on restart.
|
||||
# Stack locks mean "an operation is running"; after a restart none is, so
|
||||
# all of them go — not just the expired ones. A backup holds its lock for
|
||||
# twelve hours, and one that died with its process used to leave the stack
|
||||
# un-startable for the rest of them.
|
||||
try:
|
||||
with Session(engine) as session:
|
||||
stale = stack_lock_service.prune_expired(session)
|
||||
stale = stack_lock_service.clear_all(session)
|
||||
if stale:
|
||||
logger.info("Cleared %d stale stack lock(s) from a previous run", stale)
|
||||
logger.info(
|
||||
"Cleared %d stack lock(s) left over from a previous run: %s",
|
||||
len(stale), ", ".join(sorted(stale)),
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Could not prune stack locks: %s", exc)
|
||||
logger.warning("Could not clear stack locks: %s", exc)
|
||||
try:
|
||||
restored = image_status_store.install()
|
||||
logger.info("Restored %d cached image update status(es)", restored)
|
||||
|
||||
Reference in new issue
Block a user