Let a stuck backup be stopped instead of stranding the stack (0.67.0)
A backup holds the stack lock for up to twelve hours — a large volume
legitimately takes that long — and stops the stack's containers for a
consistent snapshot. One that died without unwinding therefore left the
stack stopped and refusing every start, stop, restart and deploy with
"Stack 'npmplus' is busy: backup in progress", with nothing to click
until the TTL ran out.
Backups and restores now register themselves in stack_job_service while
they run, with their phase and the helper containers they stream
through. Cancelling sets a flag and removes those containers, which is
what actually ends the long read: the backup then unwinds through its
own finally — the half-written archive goes, the containers it stopped
are started again, the lock is released. Checks sit between two files
and two volumes, never inside one. A restore is not cancellable at all;
it is halfway through writing a stack's data back.
POST /api/stacks/{id}/unlock does that and waits for it. A backup lock
with no job behind it is a leftover, so it is released on one click; a
deploy lock takes an explicit force, because compose keeps running past
its request. GET /api/stacks/{id}/lock and the stack detail response say
what holds the stack and since when, and the stack page turns that into
"Stop backup" / "Release lock".
Startup now clears every lock rather than only expired ones: after a
restart nothing of ours is in flight, the same reasoning the leftover
helper containers already follow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
052a85067b
commit
270ddda51d
19 files changed
+1018
-22
No files matched your search
@@ -132,6 +132,45 @@ def active(session: Session) -> dict[str, str]:
|
||||
}
|
||||
|
||||
|
||||
def info(session: Session, stack_id: str) -> Optional[dict]:
|
||||
"""The lock in force on ``stack_id``, as plain data — or None.
|
||||
|
||||
What the UI needs to explain a stack that will not budge: which operation
|
||||
holds it, since when, and when it would give up on its own.
|
||||
"""
|
||||
lock = session.get(StackLock, stack_id)
|
||||
if lock is None:
|
||||
return None
|
||||
now = _now()
|
||||
expires = _aware(lock.expires_at) or now
|
||||
if expires <= now:
|
||||
return None
|
||||
acquired = _aware(lock.acquired_at) or now
|
||||
return {
|
||||
"action": lock.action,
|
||||
"owner": lock.owner,
|
||||
"acquired_at": acquired.isoformat(),
|
||||
"expires_at": expires.isoformat(),
|
||||
"held_for": int((now - acquired).total_seconds()),
|
||||
}
|
||||
|
||||
|
||||
def clear_all(session: Session) -> list[str]:
|
||||
"""Drop every lock. Returns the stack ids that were locked.
|
||||
|
||||
Called at startup: a lock only means something while the operation holding
|
||||
it runs, and nothing of ours is in flight yet (StackPilot runs a single
|
||||
uvicorn worker — see the Dockerfile's CMD). Before this, a backup whose
|
||||
process died left its stack unusable for the twelve hours of its TTL.
|
||||
"""
|
||||
locks = session.exec(select(StackLock)).all()
|
||||
ids = [lock.stack_id for lock in locks]
|
||||
for lock in locks:
|
||||
session.delete(lock)
|
||||
session.commit()
|
||||
return ids
|
||||
|
||||
|
||||
def is_busy(session: Session, stack_id: str) -> bool:
|
||||
lock = session.get(StackLock, stack_id)
|
||||
return lock is not None and (_aware(lock.expires_at) or _now()) > _now()
|
||||
|
||||
Reference in new issue
Block a user