Stop leaking helper containers that fail to start (0.66.1)
docker-py's containers.run(remove=True) only removes a container after it ran. A helper whose start failed - an NFS destination refusing the mount - stayed behind in "Created", holding the destination's volume. Every Test click and scheduled backup added one (a user found 18), the volume could not be deleted even with everything stopped, and changed mount options never took effect because the volume could not be recreated with them. - docker_client.run_container: create, start, wait, collect output and always remove, also when the start fails; pulls a missing image. Used for NFS destination I/O, bind-path inspection, the volume wipe before a restore and the Trivy scanner. - Every helper container is labelled stackpilot.helper; on startup the stopped ones are removed, together with anything left on an NFS destination volume by older versions. - When an NFS destination's options change, stopped containers holding its volume are removed before it is recreated; a running one is now a clear error instead of silently keeping the old options. - A refused mount explains itself: NFS enabled and reachable from the Docker host, NFSv3 needing rpcbind and mountd, and nfsvers=4 needing only port 2049. - The scanner waits for Trivy's own 10-minute timeout instead of giving up after the default client's 60 seconds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9a357d2b92
commit
6c62803dc7
9 files changed
+407
-45
No files matched your search
@@ -13,6 +13,7 @@ from sqlmodel import Session
|
||||
from config import settings
|
||||
from version import APP_VERSION
|
||||
from database import engine, init_db
|
||||
import docker_client
|
||||
from docker_client import DockerError
|
||||
from routers import (
|
||||
audit,
|
||||
@@ -79,6 +80,17 @@ async def lifespan(app: FastAPI):
|
||||
logger.info("Migrated %d custom template(s) from the database to folders", moved)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Legacy template migration failed: %s", exc)
|
||||
# Throwaway containers a previous run left behind (a helper whose start
|
||||
# failed, e.g. on an NFS mount, used to stay in "Created" and keep its
|
||||
# volume in use). Nothing of ours is in flight yet, so all can go.
|
||||
try:
|
||||
client = docker_client.get_client()
|
||||
cleared = docker_client.remove_stale_helpers(client)
|
||||
cleared += backup_destination_service.clear_nfs_leftovers(client)
|
||||
if cleared:
|
||||
logger.info("Removed %d leftover helper container(s)", cleared)
|
||||
except Exception as exc: # noqa: BLE001 - Docker may not be up yet
|
||||
logger.warning("Could not clean up helper containers: %s", exc)
|
||||
# Runtime state that used to live in module dicts and was lost on restart.
|
||||
try:
|
||||
with Session(engine) as session:
|
||||
|
||||
Reference in new issue
Block a user