Stop leaking helper containers that fail to start (0.66.1)

docker-py's containers.run(remove=True) only removes a container after it
ran. A helper whose start failed - an NFS destination refusing the mount -
stayed behind in "Created", holding the destination's volume. Every Test
click and scheduled backup added one (a user found 18), the volume could
not be deleted even with everything stopped, and changed mount options
never took effect because the volume could not be recreated with them.

- docker_client.run_container: create, start, wait, collect output and
  always remove, also when the start fails; pulls a missing image. Used
  for NFS destination I/O, bind-path inspection, the volume wipe before a
  restore and the Trivy scanner.
- Every helper container is labelled stackpilot.helper; on startup the
  stopped ones are removed, together with anything left on an NFS
  destination volume by older versions.
- When an NFS destination's options change, stopped containers holding
  its volume are removed before it is recreated; a running one is now a
  clear error instead of silently keeping the old options.
- A refused mount explains itself: NFS enabled and reachable from the
  Docker host, NFSv3 needing rpcbind and mountd, and nfsvers=4 needing
  only port 2049.
- The scanner waits for Trivy's own 10-minute timeout instead of giving
  up after the default client's 60 seconds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/services/scan_service.py

@@ -26,7 +26,7 @@ from typing import Optional
 from sqlmodel import Session, select
 
 from config import settings
-from docker_client import DockerError, get_client, safe_call
+from docker_client import DockerError, get_client, get_long_client, run_container, safe_call
 from models.image_scan import SEVERITIES, ImageScan
 
 logger = logging.getLogger("stackpilot.scan")
@@ -55,9 +55,15 @@ class ScanError(Exception):
 # --------------------------------------------------------------------------- #
 
 
+#: Wait for the scanner a little longer than its own --timeout of 10m.
+SCAN_WAIT_SECONDS = 720
+
+
 def _run_trivy(image: str) -> str:
     """Run Trivy against a local image and return its JSON report."""
-    client = get_client()
+    # Trivy may take its full --timeout (the first run also downloads its
+    # vulnerability database); the 60s default client gave up long before.
+    client = get_long_client()
     socket_path = settings.DOCKER_SOCKET
     command = [
         "image",
@@ -70,21 +76,19 @@ def _run_trivy(image: str) -> str:
         image,
     ]
     try:
-        output = safe_call(
-            client.containers.run,
+        output = run_container(
+            client,
             settings.SCANNER_IMAGE,
             command,
             volumes={
                 socket_path: {"bind": "/var/run/docker.sock", "mode": "ro"},
                 CACHE_VOLUME: {"bind": "/root/.cache/trivy", "mode": "rw"},
             },
-            remove=True,
-            stdout=True,
-            stderr=False,
+            timeout=SCAN_WAIT_SECONDS,
         )
     except DockerError as exc:
-        raise ScanError(str(exc)) from exc
-    except Exception as exc:  # noqa: BLE001 - docker-py raises ContainerError etc.
+        raise ScanError(_readable(Exception(exc.detail or exc.error))) from exc
+    except Exception as exc:  # noqa: BLE001
         raise ScanError(_readable(exc)) from exc
     return (output or b"").decode("utf-8", "replace")