Run compose like the terminal does, and follow changes made there (0.67.2)

StackPilot called `docker compose -p <folder> -f compose.yaml`, which is
not what `docker compose` typed in the stack's folder does. A
compose.override.yaml or a COMPOSE_FILE in .env applied in a terminal
and was dropped by the next Start in StackPilot, which recreated the
containers without it. A compose file's `name:` or a COMPOSE_PROJECT_NAME
in .env made the terminal's containers invisible: the stack showed as
stopped while running, and Start set up a second copy next to it.

Compose now runs in the stack's folder without -f, so it loads the same
files, and -p is the name compose itself picks there: COMPOSE_PROJECT_NAME
from .env, else the last `name:` among the compose files (${VAR} from
.env), else the folder name. A stack StackPilot already runs under its
folder name keeps that name while it has containers or volumes under it
and none under the declared one: switching would have brought it up
again on empty volumes. Backups record the project, and a restore takes
it from the files it has just put back.

Changes made outside StackPilot now reach every view:

- docker_watch_service reads Docker's events once for the backend. The
  dashboard rollup (cached 25s) and the stats cache are stale as soon as
  the event generation moves; the tiles followed a terminal `compose
  stop` after 15-30s, now within a second.
- Image events re-read the local digests, so a `docker compose pull` in
  a terminal clears the update badge instead of leaving it until the
  next registry check (an hour by default).
- The browser catches up when its event socket reconnects; what changed
  while it was down waited for the 60s poll.
- The stack detail carries a revision of the compose file and .env;
  saving after they changed on disk is refused with 409 and the editor
  offers to overwrite or to load the version on disk. A save used to
  replace a terminal edit without a word.
- A stack whose folder is gone is flagged "folder missing" instead of
  passing for an ordinary stopped stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fx7gcuy8d6nPd95g43rGHu

README.md

@@ -13,6 +13,35 @@ as intuitive as Dockge, as capable as Portainer for Compose workflows.
 > (Auto-update) + Phase 23 (Secrets & configs) + Phase 24 (Design System v2)
 > complete.
 
+## Upgrading to 0.67.2 — StackPilot and the terminal agree
+
+StackPilot now runs `docker compose` the way it runs when typed in the stack's
+folder. Before, it called `docker compose -p <folder> -f compose.yaml`, and the
+two drifted apart:
+
+- **Override files and `COMPOSE_FILE` apply.** A `compose.override.yaml` (or
+  `docker-compose.override.yml`, or `COMPOSE_FILE=` in the stack's `.env`) took
+  effect in a terminal and was dropped again by the next Start in StackPilot,
+  which recreated the containers without it.
+- **The project name is compose's.** `COMPOSE_PROJECT_NAME` in `.env`, else the
+  compose file's `name:`, else the folder name. A stack whose file says
+  `name: immich` in a folder called `photos` showed as stopped while running,
+  and Start set up a second copy. *Nothing to do for existing stacks:* one that
+  StackPilot already runs under its folder name keeps that name for as long as
+  it has containers or volumes under it — switching would have brought it up
+  again next to the old one, on empty volumes.
+- **Changes made elsewhere show up at once.** The dashboard tiles were cached
+  for 25 seconds and the update badge until the next registry check (an hour by
+  default); both now follow Docker's events, so a stack stopped or an image
+  pulled in a terminal is reflected within a second. After the live connection
+  drops (a restart, a laptop asleep, a proxy) the pages catch up on reconnect
+  instead of waiting for their slow poll.
+- **The editor will not overwrite a change made on disk.** Saving after the
+  compose file or `.env` was edited in a terminal (or by a git sync) asks
+  whether to overwrite it or load the version on disk.
+- **A stack whose folder is gone says so** ("folder missing") instead of passing
+  for an ordinary stopped stack.
+
 ## Upgrading to 0.60.0 — single sign-on
 
 **Settings → Single sign-on (OIDC)** configures an identity provider — Authentik,

backend/main.py

@@ -43,6 +43,7 @@ from routers import (
 )
 from services import (
     backup_destination_service,
+    docker_watch_service,
     git_service,
     image_status_store,
     logo_service,
@@ -152,8 +153,11 @@ async def lifespan(app: FastAPI):
     logo_task = asyncio.create_task(logo_service.catalog_loop())
     git_service.ensure_cache_root()
     git_task = asyncio.create_task(git_service.poll_loop())
+    # Keeps the server-side caches in step with changes made outside StackPilot.
+    docker_watch_service.start()
     logger.info("StackPilot backend ready on port %s", settings.PORT)
     yield
+    docker_watch_service.stop()
     update_task.cancel()
     schedule_task.cancel()
     logo_task.cancel()

backend/models/stack.py

@@ -42,6 +42,10 @@ class StackUpdate(SQLModel):
     icon: Optional[str] = None
     yaml: Optional[str] = None
     env: Optional[str] = None
+    #: The ``revision`` the editor loaded. When it no longer matches the files,
+    #: they were changed elsewhere (a terminal, git) and the save is refused
+    #: instead of silently overwriting that change. Omitted: no check.
+    base_revision: Optional[str] = None
 
 
 class StackCloneRequest(SQLModel):

backend/routers/stacks.py

@@ -151,6 +151,10 @@ def _stack_summary(
         "status": status,
         "service_count": total,
         "running_count": running,
+        # The folder is gone (removed in a terminal, a share not mounted): the
+        # row stays — its git source, schedules and settings are still here —
+        # but says so instead of passing for an ordinary stopped stack.
+        "missing": not os.path.isdir(compose_service.stack_dir(stack.id)),
         "created_at": stack.created_at,
         "updated_at": stack.updated_at,
     }
@@ -290,6 +294,7 @@ def get_stack(
         # What is holding the stack, so the page can offer to stop it instead
         # of only showing a spinner that never ends.
         "lock": _lock_view(lock, stack_id),
+        "missing": not os.path.isdir(compose_service.stack_dir(stack_id)),
         # Inline credentials masked for the read-only role, like the .env.
         "yaml": (
             compose_service.read_compose(stack_id)
@@ -300,6 +305,9 @@ def get_stack(
         # from the read-only role — same reasoning as the admin-only file
         # browser. Non-admins still get status, services and the compose file.
         "env": compose_service.read_env(stack_id) if user.role == "admin" else "",
+        # For the editor's save: see StackUpdate.base_revision. Admins only —
+        # it is a hash over the .env, which this role is not shown.
+        "revision": compose_service.stack_revision(stack_id) if user.role == "admin" else None,
         "containers": containers,
         "created_at": stack.created_at,
         "updated_at": stack.updated_at,
@@ -315,6 +323,18 @@ def update_stack(
     user: User = Depends(require_admin),
 ) -> dict:
     stack = _get_stack_or_404(session, stack_id)
+    if body.base_revision and (body.yaml is not None or body.env is not None):
+        current = compose_service.stack_revision(stack_id)
+        if current != body.base_revision:
+            raise HTTPException(
+                status_code=409,
+                detail={
+                    "error": "changed_on_disk",
+                    "detail": f"The compose file or .env of '{stack_id}' was changed outside "
+                    "StackPilot after it was opened here",
+                    "revision": current,
+                },
+            )
     if body.yaml is not None:
         compose_service.write_compose(stack_id, body.yaml)
     if body.env is not None:
@@ -341,7 +361,7 @@ def update_stack(
         session, user=user.username, action="stack.update", target=stack_id,
         ip=_client_ip(request),
     )
-    return _stack_summary(stack)
+    return {**_stack_summary(stack), "revision": compose_service.stack_revision(stack_id)}
 
 
 @router.delete("/{stack_id}")

backend/services/backup_service.py

@@ -507,6 +507,9 @@ async def _create(
         manifest = {
             "format_version": BACKUP_FORMAT_VERSION,
             "stack_id": stack_id,
+            # Volumes are prefixed with the project, which a compose file's
+            # name: or COMPOSE_PROJECT_NAME can make anything at all.
+            "project": compose_service.project_name(stack_id),
             "name": name,
             "created_at": compose_service.now().isoformat(),
             "include_volumes": include_volumes,
@@ -920,9 +923,11 @@ def restore_backup(
     old_id = compose_service.slugify(manifest.get("stack_id") or stack_id)
     renamed = stack_id != old_id
     # Volumes are named after the compose *project*, which differs from the id
-    # for a folder compose had to normalise ("my.app" -> "myapp_data").
-    old_project = compose_service.project_name(manifest.get("stack_id") or stack_id)
-    new_project = compose_service.project_name(stack_id)
+    # for a folder compose had to normalise ("my.app" -> "myapp_data"), and
+    # for one whose files name the project. Backups before 0.67.2 do not say.
+    old_project = manifest.get("project") or compose_service.project_name(
+        manifest.get("stack_id") or stack_id
+    )
     old_dir = manifest.get("stack_dir") or ""
 
     directory = compose_service.stack_dir(stack_id)
@@ -942,6 +947,8 @@ def restore_backup(
 
     with tarfile.open(tar_path, "r:gz") as tar:
         _restore_tree(tar, directory, excluded | held_back, held_back, warnings)
+        # Read from the files just put back: their name: / .env decide it.
+        new_project = compose_service.project_name(stack_id)
         skipped += [{"kind": "bind", "source": rel, "reason": "not requested"} for rel in held_back]
 
         for bind in manifest.get("binds", []):

backend/services/compose_service.py

@@ -37,17 +37,167 @@ class StackFileError(Exception):
 _PROJECT_INVALID_RE = re.compile(r"[^a-z0-9_-]+")
 
 
-def project_name(stack_id: str) -> str:
-    """The compose project name for a stack.
+def _normalize_project(name: str) -> str:
+    return _PROJECT_INVALID_RE.sub("", name.lower()).lstrip("_-")
 
-    The same normalisation compose applies to a directory name when it picks
-    the project name itself (lowercase, only ``a-z0-9_-``, no leading ``-``/``_``).
-    For every stack StackPilot creates that is the id unchanged. A folder found
-    on disk as ``Nextcloud`` or ``my.app`` used to be passed as ``-p Nextcloud``,
-    which compose refuses — so every action on such a stack failed, while
-    ``docker compose up`` in that folder worked.
+
+#: The override files compose loads next to the main file, in its order.
+COMPOSE_OVERRIDE_FILENAMES = (
+    "compose.override.yml",
+    "compose.override.yaml",
+    "docker-compose.override.yml",
+    "docker-compose.override.yaml",
+)
+_ENV_LINE = re.compile(r"^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*?)\s*$")
+_VAR_REF = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::?-([^}]*))?\}|\$([A-Za-z_][A-Za-z0-9_]*)")
+
+
+def _dotenv(directory: str) -> dict[str, str]:
+    """The folder's ``.env`` as compose reads it for its own settings."""
+    values: dict[str, str] = {}
+    try:
+        with open(os.path.join(directory, ".env"), encoding="utf-8", errors="replace") as fh:
+            for raw in fh:
+                if raw.lstrip().startswith("#"):
+                    continue
+                m = _ENV_LINE.match(raw)
+                if not m:
+                    continue
+                value = m.group(2)
+                if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
+                    value = value[1:-1]
+                values[m.group(1)] = value
+    except OSError:
+        pass
+    return values
+
+
+def compose_files(directory: str, env: Optional[dict] = None) -> list[str]:
+    """The files ``docker compose`` loads in this folder, in order.
+
+    ``COMPOSE_FILE`` from the folder's ``.env`` if set, otherwise the first of
+    the default names plus an override file next to it — what running
+    ``docker compose`` in the folder picks.
     """
-    return _PROJECT_INVALID_RE.sub("", stack_id.lower()).lstrip("_-") or "stack"
+    env = _dotenv(directory) if env is None else env
+    if env.get("COMPOSE_FILE"):
+        sep = env.get("COMPOSE_PATH_SEPARATOR") or os.pathsep
+        return [
+            os.path.normpath(os.path.join(directory, part))
+            for part in env["COMPOSE_FILE"].split(sep)
+            if part.strip()
+        ]
+    files = []
+    main = find_compose_file(directory)
+    if main:
+        files.append(main)
+        for name in COMPOSE_OVERRIDE_FILENAMES:
+            candidate = os.path.join(directory, name)
+            if os.path.isfile(candidate):
+                files.append(candidate)
+                break
+    return files
+
+
+#: directory -> (fingerprint of the files that decide it, declared name)
+_DECLARED: dict[str, tuple[tuple, Optional[str]]] = {}
+
+
+def _fingerprint(paths) -> tuple:
+    out = []
+    for path in paths:
+        try:
+            st = os.stat(path)
+            out.append((path, st.st_mtime_ns, st.st_size))
+        except OSError:
+            out.append((path, None, None))
+    return tuple(out)
+
+
+def declared_project(directory: str) -> Optional[str]:
+    """The project name compose takes from the folder itself, if it sets one:
+    ``COMPOSE_PROJECT_NAME`` in ``.env``, else the last ``name:`` among its
+    compose files (``${VAR}`` resolved from ``.env``). Cached until one of
+    those files changes."""
+    candidates = [os.path.join(directory, ".env")]
+    candidates += [os.path.join(directory, n) for n in COMPOSE_FILENAMES + COMPOSE_OVERRIDE_FILENAMES]
+    env = _dotenv(directory)
+    files = compose_files(directory, env)
+    key = _fingerprint(candidates + [f for f in files if f not in candidates])
+    cached = _DECLARED.get(directory)
+    if cached and cached[0] == key:
+        return cached[1]
+    name: Optional[str] = None
+    if env.get("COMPOSE_PROJECT_NAME"):
+        name = env["COMPOSE_PROJECT_NAME"]
+    else:
+        import yaml
+
+        for path in files:
+            try:
+                with open(path, encoding="utf-8") as fh:
+                    doc = yaml.safe_load(fh)
+            except (OSError, yaml.YAMLError):
+                continue
+            if isinstance(doc, dict) and isinstance(doc.get("name"), str) and doc["name"].strip():
+                name = _VAR_REF.sub(
+                    lambda m: env.get(m.group(1) or m.group(3)) or (m.group(2) or ""),
+                    doc["name"].strip(),
+                )
+    declared = _normalize_project(name) if name else None
+    _DECLARED[directory] = (key, declared or None)
+    return declared or None
+
+
+#: project -> (monotonic time, Docker event generation, has containers or volumes)
+_RESOURCES: dict[str, tuple[float, int, bool]] = {}
+_RESOURCES_TTL = 15.0
+
+
+def _has_resources(project: str) -> bool:
+    import time
+
+    from services import docker_watch_service
+
+    gen = docker_watch_service.generation()
+    hit = _RESOURCES.get(project)
+    if hit and hit[1] == gen and time.monotonic() - hit[0] < _RESOURCES_TTL:
+        return hit[2]
+    label = {"label": f"{COMPOSE_LABEL}={project}"}
+    try:
+        client = get_client()
+        found = bool(safe_call(client.containers.list, all=True, filters=label)) or bool(
+            safe_call(client.volumes.list, filters=label)
+        )
+    except DockerError:
+        found = False
+    _RESOURCES[project] = (time.monotonic(), gen, found)
+    return found
+
+
+def project_name(stack_id: str) -> str:
+    """The compose project name for a stack — the one ``docker compose`` uses
+    when it is run in the stack's folder.
+
+    That is ``COMPOSE_PROJECT_NAME`` from the folder's ``.env``, else the
+    compose file's ``name:``, else the folder name, normalised the way compose
+    does it (lowercase, only ``a-z0-9_-``, no leading ``-``/``_``).
+
+    Before 0.67.2 StackPilot always used the folder name. A stack whose file
+    says ``name: immich`` in a folder called ``photos`` therefore ran as two
+    different projects: started in a terminal StackPilot did not see it, and
+    its Start button set up a second copy next to it. A stack StackPilot
+    started that old way keeps its folder-derived name for as long as it has
+    containers or volumes under it and none under the declared one — switching
+    would leave them running and bring up the stack again with empty volumes.
+    """
+    legacy = _normalize_project(stack_id) or "stack"
+    declared = declared_project(stack_dir(stack_id))
+    if not declared or declared == legacy:
+        return legacy
+    if _has_resources(legacy) and not _has_resources(declared):
+        return legacy
+    return declared
 
 
 def stack_ids_by_project(stack_ids) -> dict[str, str]:
@@ -114,6 +264,18 @@ def read_compose(stack_id: str, override: Optional[str] = None) -> str:
         return fh.read()
 
 
+def stack_revision(stack_id: str, override: Optional[str] = None) -> str:
+    """A short fingerprint of the compose file and ``.env`` as they are on
+    disk: what the editor loaded, compared again when it saves."""
+    import hashlib
+
+    digest = hashlib.sha256()
+    digest.update(read_compose(stack_id, override).encode())
+    digest.update(b"\0")
+    digest.update(read_env(stack_id, override).encode())
+    return digest.hexdigest()[:16]
+
+
 def read_env(stack_id: str, override: Optional[str] = None) -> str:
     path = env_path(stack_id, override)
     if not os.path.isfile(path):
@@ -450,8 +612,15 @@ def stack_status_summaries() -> dict[str, dict]:
 
 
 def _compose_base_cmd(stack_id: str, override: Optional[str] = None) -> list[str]:
+    """``docker compose`` as it runs when typed in the stack's folder.
+
+    No ``-f``: with it, compose loaded that one file only — a
+    ``compose.override.yaml`` or a ``COMPOSE_FILE`` in ``.env`` applied in a
+    terminal and was dropped again by the next Start in StackPilot, which
+    recreated the containers without it. The subprocess runs in the folder
+    (:func:`_compose_cwd`), which is where ``COMPOSE_FILE`` paths are read from.
+    """
     directory = stack_dir(stack_id, override)
-    compose_file = find_compose_file(directory) or os.path.join(directory, DEFAULT_COMPOSE_NAME)
     return [
         "docker",
         "compose",
@@ -459,11 +628,14 @@ def _compose_base_cmd(stack_id: str, override: Optional[str] = None) -> list[str
         project_name(stack_id),
         "--project-directory",
         directory,
-        "-f",
-        compose_file,
     ]
 
 
+def _compose_cwd(stack_id: str, override: Optional[str] = None) -> Optional[str]:
+    directory = stack_dir(stack_id, override)
+    return directory if os.path.isdir(directory) else None
+
+
 async def run_compose(
     stack_id: str,
     args: list[str],
@@ -475,6 +647,7 @@ async def run_compose(
     proc = await asyncio.create_subprocess_exec(
         *cmd,
         env=registry_service.cli_env(),
+        cwd=_compose_cwd(stack_id, override),
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.PIPE,
     )
@@ -559,6 +732,7 @@ async def stream_compose(
     proc = await asyncio.create_subprocess_exec(
         *cmd,
         env=registry_service.cli_env(),
+        cwd=_compose_cwd(stack_id, override),
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
     )
@@ -621,6 +795,7 @@ async def _stream_phase(
     proc = await asyncio.create_subprocess_exec(
         *cmd,
         env=registry_service.cli_env(),
+        cwd=_compose_cwd(stack_id, override),
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
     )

backend/services/dashboard_service.py

@@ -16,7 +16,7 @@ from sqlmodel import Session, select
 
 from docker_client import DockerError, get_client, safe_call
 from models.backup_schedule import BackupSchedule, describe_stacks, status_of
-from services import compose_service, update_service
+from services import compose_service, docker_watch_service, update_service
 
 COMPOSE_LABEL = compose_service.COMPOSE_LABEL
 
@@ -25,7 +25,7 @@ FLEET_TTL = 25.0
 DISK_PRESSURE = 0.85  # disk used fraction above which a host needs attention
 MEM_PRESSURE = 0.90   # memory used fraction above which a host needs attention
 
-_fleet_cache: dict = {"data": None, "ts": 0.0}
+_fleet_cache: dict = {"data": None, "ts": 0.0, "gen": None}
 
 
 # --------------------------------------------------------------------------- #
@@ -219,7 +219,17 @@ _SEVERITY_ORDER = {"error": 0, "warn": 1}
 
 async def compute_fleet(session: Session, refresh: bool = False) -> dict:
     now = time.time()
-    if not refresh and _fleet_cache["data"] and now - _fleet_cache["ts"] < FLEET_TTL:
+    # Stale as soon as Docker reported a change: the UI refetches on that very
+    # event, and got the rollup from before it — a stack stopped in a terminal
+    # stayed "running" in the tiles for up to FLEET_TTL. Read before the work,
+    # so a change during it is not mistaken for one already counted.
+    gen = docker_watch_service.generation()
+    if (
+        not refresh
+        and _fleet_cache["data"]
+        and now - _fleet_cache["ts"] < FLEET_TTL
+        and _fleet_cache["gen"] == gen
+    ):
         return _fleet_cache["data"]
 
     local_host, attention = await asyncio.to_thread(_local_host)
@@ -254,4 +264,5 @@ async def compute_fleet(session: Session, refresh: bool = False) -> dict:
     }
     _fleet_cache["data"] = data
     _fleet_cache["ts"] = now
+    _fleet_cache["gen"] = gen
     return data

backend/services/docker_watch_service.py

@@ -0,0 +1,119 @@
+"""One backend-wide reader of Docker's event stream, for server-side caches.
+
+The browser follows Docker on its own (``/ws/events``) and refetches whatever
+an event touched. Two of the answers it refetched were cached here, though,
+and did not know anything had happened:
+
+* the dashboard's fleet rollup, held for 25 seconds — a stack stopped in a
+  terminal showed as running in the dashboard tiles for up to half a minute,
+  and the event that should have refreshed them had come and gone;
+* the image update status, whose local digests were only re-read after a
+  pull *through StackPilot* — after ``docker compose pull && up -d`` in a
+  terminal the "update available" badge stayed until the next registry check,
+  an hour by default.
+
+So: :func:`generation` changes with every container or image event, and a
+cache that remembers the generation it was computed at knows when it is stale.
+Image events also re-read the local digests (no registry calls), debounced so
+a ``pull`` of ten layers is one pass.
+"""
+from __future__ import annotations
+
+import logging
+import threading
+
+logger = logging.getLogger("stackpilot.docker_watch")
+
+_EVENT_TYPES = ["container", "image"]
+#: Container actions that change nothing a cache holds (see routers/ws.py).
+_IGNORED = {
+    "exec_create", "exec_start", "exec_die", "exec_detach",
+    "attach", "top", "resize", "archive-path", "extract-to-dir",
+}
+#: Wait this long after the last image event before re-reading digests.
+DIGEST_DEBOUNCE_S = 2.0
+RECONNECT_MAX_S = 30.0
+
+_generation = 0
+_lock = threading.Lock()
+_stop = threading.Event()
+_thread: threading.Thread | None = None
+_digest_timer: threading.Timer | None = None
+
+
+def generation() -> int:
+    """Changes whenever Docker reports a container or image change."""
+    return _generation
+
+
+def bump() -> None:
+    global _generation
+    with _lock:
+        _generation += 1
+
+
+def _refresh_digests() -> None:
+    from services import update_service
+
+    try:
+        changed = update_service.refresh_local_digests()
+        if changed:
+            logger.info("Re-read local digests after image events: %d changed", changed)
+    except Exception as exc:  # noqa: BLE001 - a cache refresh must never kill the watcher
+        logger.debug("Local digest refresh failed: %s", exc)
+
+
+def _schedule_digest_refresh() -> None:
+    global _digest_timer
+    with _lock:
+        if _digest_timer is not None:
+            _digest_timer.cancel()
+        _digest_timer = threading.Timer(DIGEST_DEBOUNCE_S, _refresh_digests)
+        _digest_timer.daemon = True
+        _digest_timer.start()
+
+
+def handle(event: dict) -> None:
+    """Apply one Docker event to the caches."""
+    kind = event.get("Type")
+    action = (event.get("Action") or "").split(":")[0]
+    if kind == "container" and action in _IGNORED:
+        return
+    bump()
+    if kind == "image":
+        _schedule_digest_refresh()
+
+
+def _run() -> None:
+    from docker_client import get_client
+
+    delay = 1.0
+    while not _stop.is_set():
+        try:
+            stream = get_client().events(decode=True, filters={"type": _EVENT_TYPES})
+            delay = 1.0
+            # Anything may have happened while we were not listening.
+            bump()
+            for event in stream:
+                if _stop.is_set():
+                    break
+                handle(event)
+        except Exception as exc:  # noqa: BLE001 - daemon restarts, socket hiccups
+            logger.debug("Docker event stream ended: %s", exc)
+        if _stop.wait(delay):
+            break
+        delay = min(delay * 2, RECONNECT_MAX_S)
+
+
+def start() -> None:
+    global _thread
+    if _thread is not None and _thread.is_alive():
+        return
+    _stop.clear()
+    _thread = threading.Thread(target=_run, name="docker-watch", daemon=True)
+    _thread.start()
+
+
+def stop() -> None:
+    _stop.set()
+

backend/services/stats_service.py

@@ -19,6 +19,7 @@ import time
 from concurrent.futures import ThreadPoolExecutor
 
 from docker_client import DockerError, get_client, safe_call
+from services import docker_watch_service
 
 COMPOSE_PROJECT_LABEL = "com.docker.compose.project"
 
@@ -85,18 +86,27 @@ def stack_stats(refresh: bool = False) -> dict:
     when none of them have that limit set. Served from a short-lived cache
     unless ``refresh`` is set.
     """
-    if not refresh and _cache["data"] is not None:
-        if time.monotonic() - _cache["ts"] < CACHE_TTL:
+    def fresh() -> bool:
+        # Also stale once Docker reported a change (a container stopped in a
+        # terminal), see services/docker_watch_service.py.
+        return (
+            _cache["data"] is not None
+            and time.monotonic() - _cache["ts"] < CACHE_TTL
+            and _cache.get("gen") == docker_watch_service.generation()
+        )
+
+    if not refresh and fresh():
         return _cache["data"]
 
     with _lock:
         # Somebody may have refreshed it while we waited for the lock.
-        if not refresh and _cache["data"] is not None:
-            if time.monotonic() - _cache["ts"] < CACHE_TTL:
+        if not refresh and fresh():
             return _cache["data"]
+        gen = docker_watch_service.generation()
         data = _sample()
         _cache["data"] = data
         _cache["ts"] = time.monotonic()
+        _cache["gen"] = gen
         return data
 
 

backend/services/update_service.py

@@ -423,6 +423,24 @@ async def stack_updates(stack_id: str, refresh: bool = True) -> dict:
     }
 
 
+def _reconcile_local(image: str, status: UpdateStatus) -> bool:
+    """Re-read one image's local digest against the cached remote one (no
+    registry call). True when that changed what is stored."""
+    local = _local_digest(image)
+    available = bool(local and status.remote_digest and local != status.remote_digest)
+    if local == status.current_digest and available == status.update_available:
+        return False
+    status.current_digest = local
+    status.update_available = available
+    status.checked_at = time.time()
+    if not available:
+        _NOTIFIED.discard(image)
+    # Persisted too, or a restart brought the cleared badge back until the
+    # next background check.
+    _store(status, image in _NOTIFIED)
+    return True
+
+
 def refresh_stack_local(stack_id: str) -> None:
     """Re-read the local digests of one stack's images and reconcile them with
     the cached remote digests (no registry calls). Called right after a manual
@@ -430,19 +448,19 @@ def refresh_stack_local(stack_id: str) -> None:
     until the next background pass."""
     for image in stack_images(stack_id):
         status = _CACHE.get(image)
-        if status is None:
-            continue
-        local = _local_digest(image)
-        status.current_digest = local
-        status.update_available = bool(
-            local and status.remote_digest and local != status.remote_digest
+        if status is not None and not status.local:
+            _reconcile_local(image, status)
+
+
+def refresh_local_digests() -> int:
+    """:func:`refresh_stack_local` for every cached image — what an image
+    event triggers (services/docker_watch_service.py), so a pull made in a
+    terminal clears the badge too. Returns how many changed."""
+    return sum(
+        _reconcile_local(image, status)
+        for image, status in list(_CACHE.items())
+        if not status.local
     )
-        status.checked_at = time.time()
-        if not status.update_available:
-            _NOTIFIED.discard(image)
-        # Persisted too, or a restart brought the cleared badge back until the
-        # next background check.
-        _store(status, image in _NOTIFIED)
 
 
 async def check_all() -> dict[str, dict]:

backend/tests/test_terminal_sync.py

@@ -0,0 +1,263 @@
+"""StackPilot and a terminal looking at the same stack (0.67.2).
+
+Before, StackPilot ran ``docker compose -p <folder> -f compose.yaml``, which is
+not what ``docker compose`` typed in the folder does: an override file or a
+``COMPOSE_FILE`` in ``.env`` was dropped by every Start, and a ``name:`` or
+``COMPOSE_PROJECT_NAME`` made the terminal's containers invisible to it. Its
+caches did not hear about changes made outside it either, and the editor
+overwrote a file changed on disk without a word.
+"""
+from __future__ import annotations
+
+import os
+
+import pytest
+
+
+@pytest.fixture
+def folder(tmp_path):
+    from services import compose_service
+
+    compose_service._DECLARED.clear()
+    d = tmp_path / "photos"
+    d.mkdir()
+    (d / "compose.yaml").write_text("services:\n  app:\n    image: busybox\n")
+    return d
+
+
+def test_the_folder_names_the_project_when_nothing_else_does(folder):
+    from services import compose_service
+
+    assert compose_service.declared_project(str(folder)) is None
+
+
+def test_name_in_the_compose_file(folder):
+    from services import compose_service
+
+    (folder / "compose.yaml").write_text("name: Immich\nservices: {}\n")
+    assert compose_service.declared_project(str(folder)) == "immich"
+
+
+def test_an_override_files_name_wins_and_variables_come_from_dotenv(folder):
+    from services import compose_service
+
+    (folder / "compose.yaml").write_text("name: base\nservices: {}\n")
+    (folder / "docker-compose.override.yml").write_text("name: ${PROJECT:-fallback}\n")
+    (folder / ".env").write_text('PROJECT="from-env"\n')
+    assert compose_service.declared_project(str(folder)) == "from-env"
+
+
+def test_compose_project_name_in_dotenv_wins_over_name(folder):
+    from services import compose_service
+
+    (folder / "compose.yaml").write_text("name: from-file\nservices: {}\n")
+    (folder / ".env").write_text("COMPOSE_PROJECT_NAME=from-dotenv\n")
+    assert compose_service.declared_project(str(folder)) == "from-dotenv"
+
+
+def test_a_changed_file_is_read_again(folder):
+    from services import compose_service
+
+    assert compose_service.declared_project(str(folder)) is None
+    (folder / "compose.yaml").write_text("name: renamed\nservices: {}\n")
+    os.utime(folder / "compose.yaml", ns=(1, 2_000_000_000_000_000_000))
+    assert compose_service.declared_project(str(folder)) == "renamed"
+
+
+def test_the_files_compose_loads(folder):
+    from services import compose_service
+
+    (folder / "compose.override.yaml").write_text("services: {}\n")
+    assert [os.path.basename(f) for f in compose_service.compose_files(str(folder))] == [
+        "compose.yaml",
+        "compose.override.yaml",
+    ]
+    (folder / "gpu.yaml").write_text("services: {}\n")
+    (folder / ".env").write_text("COMPOSE_FILE=compose.yaml:gpu.yaml\n")
+    assert [os.path.basename(f) for f in compose_service.compose_files(str(folder))] == [
+        "compose.yaml",
+        "gpu.yaml",
+    ]
+
+
+@pytest.fixture
+def named_stack(tmp_path, monkeypatch):
+    from config import settings
+    from services import compose_service
+
+    monkeypatch.setattr(settings, "STACKS_DIR", str(tmp_path))
+    compose_service._DECLARED.clear()
+    d = tmp_path / "photos"
+    d.mkdir()
+    (d / "compose.yaml").write_text("name: immich\nservices: {}\n")
+    return d
+
+
+def test_a_declared_name_is_the_project(named_stack, monkeypatch):
+    from services import compose_service
+
+    monkeypatch.setattr(compose_service, "_has_resources", lambda project: False)
+    assert compose_service.project_name("photos") == "immich"
+
+
+def test_a_stack_started_under_its_folder_name_keeps_it(named_stack, monkeypatch):
+    """Switching would orphan its containers and start it again on empty volumes."""
+    from services import compose_service
+
+    monkeypatch.setattr(compose_service, "_has_resources", lambda project: project == "photos")
+    assert compose_service.project_name("photos") == "photos"
+
+
+def test_compose_runs_like_it_does_in_the_folder(named_stack, monkeypatch):
+    from services import compose_service
+
+    monkeypatch.setattr(compose_service, "_has_resources", lambda project: False)
+    cmd = compose_service._compose_base_cmd("photos")
+    assert "-f" not in cmd
+    assert cmd[cmd.index("-p") + 1] == "immich"
+    assert compose_service._compose_cwd("photos") == str(named_stack)
+    assert compose_service._compose_cwd("gone") is None
+
+
+# --------------------------------------------------------------------------- #
+# Saving over a file changed on disk.
+# --------------------------------------------------------------------------- #
+
+
+def _forget(stack_id: str) -> None:
+    from sqlmodel import Session
+
+    from database import engine
+    from models.stack import Stack
+    from services import compose_service
+
+    compose_service.delete_stack_files(stack_id)
+    with Session(engine) as session:
+        row = session.get(Stack, stack_id)
+        if row:
+            session.delete(row)
+            session.commit()
+
+
+def test_a_save_over_a_change_made_on_disk_is_refused(as_admin, db):
+    from services import compose_service
+
+    assert as_admin.post("/api/stacks", json={"name": "raced", "yaml": "services: {}\n"}).status_code == 201
+    try:
+        loaded = as_admin.get("/api/stacks/raced").json()["revision"]
+        with open(compose_service.compose_path("raced"), "a", encoding="utf-8") as fh:
+            fh.write("# edited in a terminal\n")
+        refused = as_admin.put(
+            "/api/stacks/raced", json={"yaml": "services: {}\n# mine\n", "base_revision": loaded}
+        )
+        assert refused.status_code == 409
+        assert refused.json()["detail"]["error"] == "changed_on_disk"
+        assert "edited in a terminal" in compose_service.read_compose("raced")
+
+        # Taking the current revision — or none, to overwrite — goes through.
+        current = refused.json()["detail"]["revision"]
+        saved = as_admin.put("/api/stacks/raced", json={"yaml": "services: {}\n", "base_revision": current})
+        assert saved.status_code == 200
+        assert saved.json()["revision"] == compose_service.stack_revision("raced")
+    finally:
+        _forget("raced")
+
+
+def test_the_revision_is_withheld_from_the_read_only_role(as_admin, as_user, db):
+    assert as_admin.post("/api/stacks", json={"name": "rev-viewer"}).status_code == 201
+    try:
+        assert as_user.get("/api/stacks/rev-viewer").json()["revision"] is None
+    finally:
+        _forget("rev-viewer")
+
+
+def test_a_stack_whose_folder_is_gone_says_so(as_admin, db):
+    import shutil
+
+    from services import compose_service
+
+    assert as_admin.post("/api/stacks", json={"name": "vanished"}).status_code == 201
+    try:
+        shutil.rmtree(compose_service.stack_dir("vanished"))
+        rows = {s["id"]: s for s in as_admin.get("/api/stacks").json()}
+        assert rows["vanished"]["missing"] is True
+        assert as_admin.get("/api/stacks/vanished").json()["missing"] is True
+    finally:
+        _forget("vanished")
+
+
+# --------------------------------------------------------------------------- #
+# Server-side caches following Docker.
+# --------------------------------------------------------------------------- #
+
+
+def test_docker_events_move_the_generation_but_execs_do_not(monkeypatch):
+    from services import docker_watch_service as watch
+
+    monkeypatch.setattr(watch, "_schedule_digest_refresh", lambda: None)
+    before = watch.generation()
+    watch.handle({"Type": "container", "Action": "exec_start: sh"})
+    assert watch.generation() == before
+    watch.handle({"Type": "container", "Action": "die"})
+    assert watch.generation() == before + 1
+
+
+def test_an_image_event_rereads_the_local_digests(monkeypatch):
+    from services import docker_watch_service as watch
+
+    calls = []
+    monkeypatch.setattr(watch, "_schedule_digest_refresh", lambda: calls.append(1))
+    watch.handle({"Type": "image", "Action": "pull"})
+    assert calls == [1]
+
+
+def test_a_pull_made_elsewhere_clears_the_update_badge(monkeypatch):
+    """After `docker compose pull` in a terminal the local digest is the
+    registry's — the badge used to stay until the next registry check."""
+    import time
+
+    from services import update_service
+
+    status = update_service.UpdateStatus(
+        image="nginx:alpine",
+        update_available=True,
+        current_digest="sha256:old",
+        remote_digest="sha256:new",
+        checked_at=time.time(),
+    )
+    monkeypatch.setattr(update_service, "_CACHE", {"nginx:alpine": status})
+    monkeypatch.setattr(update_service, "_store", lambda status, notified: None)
+    monkeypatch.setattr(update_service, "_local_digest", lambda image, attrs=None: "sha256:new")
+    assert update_service.refresh_local_digests() == 1
+    assert status.update_available is False and status.current_digest == "sha256:new"
+    assert update_service.refresh_local_digests() == 0
+
+
+def test_the_fleet_rollup_is_recomputed_after_a_docker_change(monkeypatch, db):
+    import asyncio
+
+    from sqlmodel import Session
+
+    from database import engine
+    from services import dashboard_service, docker_watch_service
+
+    computed = []
+
+    def local_host():
+        computed.append(1)
+        host = {
+            "stacks": {"running": 0, "partial": 0, "total": 0, "stopped": 0, "error": 0},
+            "containers_running": 0, "containers_total": 0, "unhealthy": 0, "updates_available": 0,
+        }
+        return host, []
+
+    monkeypatch.setattr(dashboard_service, "_local_host", local_host)
+    monkeypatch.setattr(dashboard_service, "_backup_attention", lambda session: [])
+    monkeypatch.setattr(dashboard_service, "_fleet_cache", {"data": None, "ts": 0.0, "gen": None})
+    with Session(engine) as session:
+        asyncio.run(dashboard_service.compute_fleet(session))
+        asyncio.run(dashboard_service.compute_fleet(session))
+        assert len(computed) == 1  # cached
+        docker_watch_service.bump()
+        asyncio.run(dashboard_service.compute_fleet(session))
+        assert len(computed) == 2

backend/tests/test_walkthrough_fixes.py

@@ -211,7 +211,15 @@ def test_the_read_only_role_gets_the_masked_compose_file(as_admin, as_user, db):
         assert "hunter2" in as_admin.get("/api/stacks/masked").json()["yaml"]
         assert "hunter2" not in as_user.get("/api/stacks/masked").json()["yaml"]
     finally:
+        from sqlmodel import Session
+
+        from database import engine
+        from models.stack import Stack
+
         compose_service.delete_stack_files("masked")
+        with Session(engine) as session:
+            session.delete(session.get(Stack, "masked"))
+            session.commit()
 
 
 def test_scan_all_covers_only_stackpilots_own_stacks(as_admin, db, monkeypatch):

backend/version.py

@@ -1,3 +1,3 @@
 """Single source of truth for the StackPilot release version."""
 
-APP_VERSION = "0.67.1"
+APP_VERSION = "0.67.2"

frontend/package.json

@@ -1,7 +1,7 @@
 {
   "name": "stackpilot-frontend",
   "private": true,
-  "version": "0.67.1",
+  "version": "0.67.2",
   "type": "module",
   "scripts": {
     "dev": "vite",

frontend/src/api/stacks.ts

@@ -31,8 +31,11 @@ export const stacksApi = {
       icon?: string;
       yaml?: string;
       env?: string;
+      /** The revision the editor loaded; a 409 "changed_on_disk" if the files
+       *  changed since. Left out to overwrite regardless. */
+      base_revision?: string;
     }
-  ) => api.put<StackSummary>(`/api/stacks/${id}`, body).then((r) => r.data),
+  ) => api.put<StackSummary & { revision?: string }>(`/api/stacks/${id}`, body).then((r) => r.data),
   /** `kept` lists folders left in place because another filesystem (a share,
    *  a second disk) is mounted there. */
   remove: (id: string, deleteFiles = true) =>

frontend/src/components/stacks/StacksTable.tsx

@@ -3,7 +3,7 @@ import { Link } from "react-router-dom";
 import { useQueryClient } from "@tanstack/react-query";
 import { Play, Square, RotateCw, ArrowUpCircle, Pencil, Trash2 } from "lucide-react";
 import { toast } from "sonner";
-import { Card, Spinner, Badge } from "@/components/ui";
+import { Card, Spinner, Badge, MissingBadge } from "@/components/ui";
 import { ConfirmDialog } from "@/components/ui/ConfirmDialog";
 import { StackIcon } from "@/components/ui/StackIcon";
 import { formatBytes } from "@/lib/utils";
@@ -171,6 +171,7 @@ function StackRow({
             <StackIcon stack={stack} status={stack.status} />
             <span className="font-medium">{stack.name}</span>
             <Badge status={stack.status}>{stack.status}</Badge>
+            {stack.missing && <MissingBadge />}
             <span className="text-xs text-slate-400">
               {stack.running_count}/{stack.service_count} svc
             </span>

frontend/src/components/ui/index.tsx

@@ -125,6 +125,19 @@ export function Badge({
   );
 }
 
+/** For a stack whose folder no longer exists — deleted outside StackPilot,
+ *  or a share that is not mounted. */
+export function MissingBadge() {
+  return (
+    <span
+      title="The stack's folder no longer exists — deleted outside StackPilot, or its share is not mounted. Delete the stack here to drop it from the list."
+      className="rounded-pill border border-red-500/40 bg-red-500/10 px-2 py-0.5 text-[11px] font-semibold text-red-600 dark:text-red-400"
+    >
+      folder missing
+    </span>
+  );
+}
+
 export function Spinner() {
   return (
     <div className="flex h-full w-full items-center justify-center p-8">

frontend/src/hooks/useDockerEvents.ts

@@ -52,6 +52,8 @@ export function useDockerEvents() {
     let retry: number | null = null;
     let delay = RECONNECT_MIN_MS;
     let closed = false;
+    // Set once a socket has been up: a later "ready" is a reconnect.
+    let connectedBefore = false;
 
     const flush = () => {
       timer.current = null;
@@ -84,6 +86,18 @@ export function useDockerEvents() {
       socket.onmessage = (ev) => {
         try {
           const msg = JSON.parse(ev.data);
+          if (msg.type === "ready") {
+            // Whatever happened while the socket was down (a backend restart,
+            // a laptop asleep, a proxy dropping it) sent no event to us, and
+            // the pages kept showing the state from before until their slow
+            // poll. Catch up once, as if everything had changed.
+            if (connectedBefore) {
+              for (const resource of Object.keys(INVALIDATES)) pending.current.add(resource);
+              if (timer.current === null) timer.current = window.setTimeout(flush, COALESCE_MS);
+            }
+            connectedBefore = true;
+            return;
+          }
           if (msg.type !== "event" || !msg.resource) return;
           pending.current.add(msg.resource);
           if (timer.current === null) {

frontend/src/pages/StackDetail.tsx

@@ -13,7 +13,7 @@ import {
   LayoutTemplate,
 } from "lucide-react";
 import { toast } from "sonner";
-import { Badge, Button, Card, Input, Spinner } from "@/components/ui";
+import { Badge, Button, Card, Input, MissingBadge, Spinner } from "@/components/ui";
 import { ConfirmDialog } from "@/components/ui/ConfirmDialog";
 import { LogViewer } from "@/components/stacks/LogViewer";
 import { ContainerCard } from "@/components/stacks/ContainerCard";
@@ -64,6 +64,7 @@ export function StackDetail() {
             <div className="flex items-center gap-2">
               <h1 className="sp-heading text-xl">{data.name}</h1>
               <Badge status={data.status}>{data.status}</Badge>
+              {data.missing && <MissingBadge />}
             </div>
             {data.description && (
               <p className="mt-1 text-sm text-slate-500">{data.description}</p>

frontend/src/pages/StackEditor.tsx

@@ -78,6 +78,10 @@ function StackEditorForm() {
   // current values below, because the navigation right after a save is checked
   // against a render from before it.
   const saved = useRef({ name: "", description: "", yaml: STARTER, env: "" });
+  // The files' revision when they were loaded (or last saved) here.
+  const baseRevision = useRef<string | null>(null);
+  // A save refused because the files changed on disk meanwhile.
+  const [conflict, setConflict] = useState<{ deploy: boolean } | null>(null);
   const loadedFor = useRef<string | null>(null);
   useEffect(() => {
     if (existing.data && id && loadedFor.current !== id) {
@@ -89,6 +93,7 @@ function StackEditorForm() {
         env: existing.data.env || "",
       };
       saved.current = loaded;
+      baseRevision.current = existing.data.revision ?? null;
       setName(loaded.name);
       setDescription(loaded.description);
       setIcon(existing.data.icon ?? null);
@@ -122,7 +127,7 @@ function StackEditorForm() {
     return () => window.removeEventListener("beforeunload", onBeforeUnload);
   }, [unsaved]);
 
-  const save = async (deploy: boolean) => {
+  const save = async (deploy: boolean, overwrite = false) => {
     if (isNew && !name.trim()) {
       toast.error("Stack name is required");
       return;
@@ -146,7 +151,15 @@ function StackEditorForm() {
         });
         stackId = created.id;
       } else {
-        await stacksApi.update(id!, { name, description, icon: iconField, yaml, env });
+        const updated = await stacksApi.update(id!, {
+          name,
+          description,
+          icon: iconField,
+          yaml,
+          env,
+          base_revision: overwrite ? undefined : baseRevision.current ?? undefined,
+        });
+        baseRevision.current = updated.revision ?? null;
       }
       saved.current = { name, description, yaml, env };
       if (iconFile && stackId) {
@@ -164,12 +177,38 @@ function StackEditorForm() {
       }
       navigate(`/stacks/${stackId}`);
     } catch (err) {
+      const detail = (err as { response?: { status?: number; data?: { detail?: { error?: string } } } })
+        ?.response;
+      if (detail?.status === 409 && detail.data?.detail?.error === "changed_on_disk") {
+        setConflict({ deploy });
+        return;
+      }
       toast.error(apiErrorMessage(err));
     } finally {
       setSaving(false);
     }
   };
 
+  // Take the files as they are on disk now, dropping what was typed here.
+  const loadFromDisk = async () => {
+    const { data } = await existing.refetch();
+    setConflict(null);
+    if (!data) return;
+    const loaded = {
+      name: data.name,
+      description: data.description ?? "",
+      yaml: data.yaml || STARTER,
+      env: data.env || "",
+    };
+    saved.current = loaded;
+    baseRevision.current = data.revision ?? null;
+    setName(loaded.name);
+    setDescription(loaded.description);
+    setYaml(loaded.yaml);
+    setEnv(loaded.env);
+    toast.success("Loaded the version on disk");
+  };
+
   const onDeploy = async () => {
     setChecking(true);
     try {
@@ -406,6 +445,26 @@ function StackEditorForm() {
         />
       )}
 
+      {conflict && (
+        <ConfirmDialog
+          title="Changed outside StackPilot"
+          message="The compose file or .env was changed on disk — in a terminal, by git or another tab — after it was opened here. Saving now would replace that change with this version."
+          confirmLabel={conflict.deploy ? "Overwrite and deploy" : "Overwrite"}
+          danger
+          busy={saving}
+          onConfirm={() => {
+            const { deploy } = conflict;
+            setConflict(null);
+            save(deploy, true);
+          }}
+          onCancel={() => setConflict(null)}
+        >
+          <Button variant="outline" onClick={loadFromDisk}>
+            Load the version on disk (discard the edits here)
+          </Button>
+        </ConfirmDialog>
+      )}
+
       {blocker.state === "blocked" && (
         <ConfirmDialog
           title="Discard unsaved changes?"

frontend/src/types/index.ts

@@ -19,6 +19,8 @@ export interface StackSummary {
   status: StackStatus;
   service_count: number;
   running_count: number;
+  /** Its folder no longer exists (deleted outside StackPilot, share unmounted). */
+  missing?: boolean;
   created_at: string;
   updated_at: string;
 }
@@ -86,8 +88,13 @@ export interface StackDetail {
   status: StackStatus;
   /** Set while an operation holds the stack; null otherwise. */
   lock?: StackLock | null;
+  /** Its folder no longer exists. */
+  missing?: boolean;
   yaml: string;
   env: string;
+  /** Fingerprint of the compose file and .env on disk (admins only); sent
+   *  back on save so a change made elsewhere is not overwritten. */
+  revision?: string | null;
   containers: ContainerInfo[];
   created_at: string;
   updated_at: string;