Let a stuck backup be stopped instead of stranding the stack (0.67.0)
A backup holds the stack lock for up to twelve hours — a large volume
legitimately takes that long — and stops the stack's containers for a
consistent snapshot. One that died without unwinding therefore left the
stack stopped and refusing every start, stop, restart and deploy with
"Stack 'npmplus' is busy: backup in progress", with nothing to click
until the TTL ran out.
Backups and restores now register themselves in stack_job_service while
they run, with their phase and the helper containers they stream
through. Cancelling sets a flag and removes those containers, which is
what actually ends the long read: the backup then unwinds through its
own finally — the half-written archive goes, the containers it stopped
are started again, the lock is released. Checks sit between two files
and two volumes, never inside one. A restore is not cancellable at all;
it is halfway through writing a stack's data back.
POST /api/stacks/{id}/unlock does that and waits for it. A backup lock
with no job behind it is a leftover, so it is released on one click; a
deploy lock takes an explicit force, because compose keeps running past
its request. GET /api/stacks/{id}/lock and the stack detail response say
what holds the stack and since when, and the stack page turns that into
"Stop backup" / "Release lock".
Startup now clears every lock rather than only expired ones: after a
restart nothing of ours is in flight, the same reasoning the leftover
helper containers already follow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
052a85067b
commit
270ddda51d
19 files changed
+1018
-22
No files matched your search
@@ -32,7 +32,7 @@ from docker_client import (
|
||||
run_container,
|
||||
safe_call,
|
||||
)
|
||||
from services import compose_service
|
||||
from services import compose_service, stack_job_service
|
||||
|
||||
logger = logging.getLogger("stackpilot.assets")
|
||||
|
||||
@@ -99,16 +99,22 @@ def ensure_helper_image(client) -> None:
|
||||
|
||||
|
||||
def _create_helper(client, volumes: dict):
|
||||
return safe_call(
|
||||
container = safe_call(
|
||||
client.containers.create,
|
||||
settings.BACKUP_HELPER_IMAGE,
|
||||
command="true",
|
||||
volumes=volumes,
|
||||
labels={HELPER_LABEL: "1"},
|
||||
)
|
||||
# Streaming a large volume through this container is the part of a backup
|
||||
# that runs for hours; cancelling one works by removing it (see
|
||||
# :mod:`services.stack_job_service`), so the job has to know it exists.
|
||||
stack_job_service.register_helper(container.id)
|
||||
return container
|
||||
|
||||
|
||||
def _remove(container) -> None:
|
||||
stack_job_service.drop_helper(getattr(container, "id", ""))
|
||||
try:
|
||||
container.remove(force=True)
|
||||
except Exception: # noqa: BLE001 - cleanup is best effort
|
||||
@@ -160,13 +166,15 @@ def _put_archive(container, dest: str, src_file: str, volumes: dict) -> None:
|
||||
|
||||
def _create_helper_with(client, volumes: dict, command: list[str]):
|
||||
ensure_helper_image(client)
|
||||
return safe_call(
|
||||
container = safe_call(
|
||||
client.containers.create,
|
||||
settings.BACKUP_HELPER_IMAGE,
|
||||
command=command,
|
||||
volumes=volumes,
|
||||
labels={HELPER_LABEL: "1"},
|
||||
)
|
||||
stack_job_service.register_helper(container.id)
|
||||
return container
|
||||
|
||||
|
||||
def _unknown() -> dict:
|
||||
|
||||
Reference in new issue
Block a user