Add private registry credentials, and stop update checks lying (0.56.0)

This was on the gap list as a missing feature, but it was a bug first. The
update checker asks the registry for a tag's digest over HTTP itself, and could
only do it anonymously. A private repository answers 401, remote_digest returned
None, and None already meant "could not reach registry" — so a private image was
indistinguishable from a network blip. The Images page showed nothing and a
stack pinned to a six-month-old image looked up to date indefinitely.

So AuthRequired is now its own exception, separate from unreachable, and the
error names the registry and which of the two problems it is: "ghcr.io needs
credentials" when there are none, "ghcr.io rejected the stored credentials" when
there are and they are wrong. Those are different fixes, and the message should
say which one you need. The plain unreachable message survives unchanged, with a
test pinning it, because not every failure is an auth failure.

Two consumers need the credentials and they need them in completely different
shapes, which is why this is its own service rather than a field on something
else. StackPilot's own checker wants (user, password) inside async code that has
no database session, so the rows are mirrored into an in-memory cache that
reload() refills on startup and after every write. The Docker CLI wants a
config.json, so reload() writes one into ${DATA_DIR}/docker and compose runs with
DOCKER_CONFIG pointed at it. Generating it from the database every time is what
makes deletion real: removing a registry in the UI revokes the CLI's login
instead of leaving a stale one in ~/.docker.

Host normalization is the join that makes any of it work, and it is easy to
underestimate. parse_ref only ever produces registry-1.docker.io, nobody types
that, and the CLI wants the whole thing under https://index.docker.io/v1/ — three
spellings of one registry across three layers. canonical_host settles on what
parse_ref produces, the config writer translates on the way out, and a bare
nginx:alpine finds credentials entered as "docker.io". Verified end to end:
typed as the v1 URL, stored as registry-1.docker.io, written as the v1 URL.

The password is encrypted at rest with the same key as backup destinations and
never leaves the server, not even masked — the API returns has_password, which
is all the form needs to offer "leave blank to keep". A row that cannot be
decrypted after a SECRET_KEY change is skipped with a warning rather than taking
every other registry down with it. Everything here is admin-only including the
reads, because even masked the rows say which registries this install talks to
and under what account.

The Test button asks the registry rather than validating a string, following the
Bearer challenge with credentials attached the way a real client does. Only an
outright 401 counts as wrong credentials; anything else means reachable and
talking, which is as much as a credentials check can honestly claim. Checked
against the live Docker Hub token endpoint with deliberately wrong credentials.

33 tests: the normalization table, the cache, the generated config.json down to
its 0600 mode and the Docker Hub key, encryption at rest, that no password field
appears in any response, and the 401-is-reported behaviour that started this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/services/compose_service.py

@@ -16,6 +16,7 @@ from typing import Optional
 
 from config import settings
 from docker_client import DockerError, get_client, safe_call
+from services import registry_service
 
 COMPOSE_FILENAMES = ("compose.yaml", "compose.yml", "docker-compose.yml", "docker-compose.yaml")
 DEFAULT_COMPOSE_NAME = "compose.yaml"
@@ -285,6 +286,7 @@ async def run_compose(
     cmd = _compose_base_cmd(stack_id, override) + args
     proc = await asyncio.create_subprocess_exec(
         *cmd,
+        env=registry_service.cli_env(),
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.PIPE,
     )
@@ -322,6 +324,7 @@ async def validate_yaml(content: str, env_content: str = "") -> dict:
         try:
             proc = await asyncio.create_subprocess_exec(
                 *cmd,
+                env=registry_service.cli_env(),
                 stdout=asyncio.subprocess.PIPE,
                 stderr=asyncio.subprocess.PIPE,
             )
@@ -339,6 +342,7 @@ async def stream_compose(
     cmd = _compose_base_cmd(stack_id, override) + args
     proc = await asyncio.create_subprocess_exec(
         *cmd,
+        env=registry_service.cli_env(),
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
     )
@@ -364,6 +368,7 @@ async def supports_json_progress() -> bool:
         try:
             proc = await asyncio.create_subprocess_exec(
                 "docker", "compose", "--progress", "json", "version",
+                env=registry_service.cli_env(),
                 stdout=asyncio.subprocess.DEVNULL,
                 stderr=asyncio.subprocess.DEVNULL,
             )
@@ -386,6 +391,7 @@ async def _stream_phase(
     cmd += args
     proc = await asyncio.create_subprocess_exec(
         *cmd,
+        env=registry_service.cli_env(),
         stdout=asyncio.subprocess.PIPE,
         stderr=asyncio.subprocess.STDOUT,
     )

backend/services/registry_service.py

@@ -0,0 +1,249 @@
+"""Credentials for private container registries.
+
+Two very different consumers need these, which is why this module exists rather
+than the credentials living next to either of them:
+
+* **StackPilot's own update checker** (``services/update_service.py``) talks to
+  the registry v2 API over HTTP itself. Without credentials a private repository
+  answers 401, the check gave up, and the UI said nothing at all — a stack could
+  sit on a stale image for months and look up to date. That was the actual bug
+  here, not a missing feature.
+* **The Docker CLI**, which runs ``docker compose pull``. It reads its own
+  ``config.json``, so this module writes one into ``${DATA_DIR}/docker`` and
+  compose runs with ``DOCKER_CONFIG`` pointed at it.
+
+Lookups happen inside async registry calls that have no database session, so
+the rows are mirrored into a small in-memory cache. :func:`reload` refills it and
+rewrites the CLI config; every write path calls it, and so does startup.
+
+Passwords are encrypted at rest and only ever decrypted into this cache and the
+CLI config file (0600, in the data volume). They are never returned by the API.
+"""
+from __future__ import annotations
+
+import base64
+import json
+import logging
+import os
+import threading
+from typing import Optional
+
+import httpx
+from sqlmodel import Session, select
+
+from config import settings
+from models.registry import Registry
+from services import crypto_service
+
+logger = logging.getLogger("stackpilot.registries")
+
+#: What `parse_ref` calls Docker Hub, and what the CLI calls it. Every spelling
+#: users type — docker.io, index.docker.io, the v1 URL — normalizes to the
+#: first; the second is what has to appear in config.json for `docker pull`.
+DOCKER_HUB = "registry-1.docker.io"
+DOCKER_HUB_CONFIG_KEY = "https://index.docker.io/v1/"
+_DOCKER_HUB_ALIASES = {
+    "docker.io",
+    "index.docker.io",
+    "registry.docker.io",
+    "registry-1.docker.io",
+    "https://index.docker.io/v1/",
+    "index.docker.io/v1/",
+}
+
+TIMEOUT = httpx.Timeout(15.0)
+
+_lock = threading.Lock()
+_credentials: dict[str, tuple[str, str]] = {}
+
+
+class RegistryError(Exception):
+    """A registry that cannot be reached, or credentials it rejects."""
+
+
+# --------------------------------------------------------------------------- #
+# Host normalization
+# --------------------------------------------------------------------------- #
+
+
+def canonical_host(value: str) -> str:
+    """The registry host as an image reference would name it.
+
+    Accepts what people actually paste: a bare host, a URL with a scheme, a
+    trailing slash, Docker Hub under any of its names. Without this, credentials
+    entered as ``docker.io`` would never be found for an image that parses as
+    ``registry-1.docker.io``.
+    """
+    host = (value or "").strip().lower()
+    if not host:
+        raise RegistryError("Registry host is required")
+    if host in _DOCKER_HUB_ALIASES:
+        return DOCKER_HUB
+    for scheme in ("https://", "http://"):
+        if host.startswith(scheme):
+            host = host[len(scheme) :]
+            break
+    host = host.split("/", 1)[0].rstrip("/")
+    if host in _DOCKER_HUB_ALIASES:
+        return DOCKER_HUB
+    if not host:
+        raise RegistryError("Registry host is required")
+    return host
+
+
+def is_docker_hub(host: str) -> bool:
+    return canonical_host(host) == DOCKER_HUB
+
+
+# --------------------------------------------------------------------------- #
+# The cache the async paths read
+# --------------------------------------------------------------------------- #
+
+
+def reload(session: Session) -> int:
+    """Refill the credential cache from the database and rewrite config.json.
+
+    Called at startup and after every write. Returns how many registries are
+    configured.
+    """
+    fresh: dict[str, tuple[str, str]] = {}
+    for row in session.exec(select(Registry)).all():
+        try:
+            password = crypto_service.decrypt(row.password)
+        except crypto_service.DecryptError as exc:
+            # One unreadable row must not take the others down with it.
+            logger.warning("Ignoring credentials for %s: %s", row.host, exc)
+            continue
+        fresh[row.host] = (row.username, password)
+    with _lock:
+        _credentials.clear()
+        _credentials.update(fresh)
+    _write_docker_config(fresh)
+    return len(fresh)
+
+
+def credentials_for(host: str) -> Optional[tuple[str, str]]:
+    """(username, password) for a registry host, or None."""
+    try:
+        key = canonical_host(host)
+    except RegistryError:
+        return None
+    with _lock:
+        return _credentials.get(key)
+
+
+def credentials_for_image(image: str) -> Optional[tuple[str, str]]:
+    """Credentials for whichever registry an image reference points at."""
+    from services import update_service
+
+    registry, _repo, _tag = update_service.parse_ref(image)
+    return credentials_for(registry)
+
+
+def configured_hosts() -> list[str]:
+    with _lock:
+        return sorted(_credentials)
+
+
+# --------------------------------------------------------------------------- #
+# The Docker CLI's config.json
+# --------------------------------------------------------------------------- #
+
+
+def docker_config_dir() -> str:
+    return os.path.join(settings.DATA_DIR, "docker")
+
+
+def _write_docker_config(creds: dict[str, tuple[str, str]]) -> None:
+    """Write the auths file ``docker compose pull`` reads.
+
+    The file is rewritten from the database every time, so removing a registry
+    in the UI actually revokes the CLI's access rather than leaving a stale
+    login behind.
+    """
+    directory = docker_config_dir()
+    path = os.path.join(directory, "config.json")
+    auths = {}
+    for host, (username, password) in creds.items():
+        key = DOCKER_HUB_CONFIG_KEY if host == DOCKER_HUB else host
+        token = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode("ascii")
+        auths[key] = {"auth": token}
+    try:
+        os.makedirs(directory, mode=0o700, exist_ok=True)
+        tmp = f"{path}.tmp"
+        # Written 0600 before it is put in place, so the credentials are never
+        # briefly world-readable.
+        fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+        with os.fdopen(fd, "w", encoding="utf-8") as fh:
+            json.dump({"auths": auths}, fh)
+        os.replace(tmp, path)
+    except OSError as exc:
+        logger.warning("Could not write the Docker CLI credentials file: %s", exc)
+
+
+def cli_env() -> dict:
+    """Environment for a ``docker``/``docker compose`` subprocess.
+
+    Points DOCKER_CONFIG at our generated file rather than writing into
+    ``~/.docker``, so what StackPilot manages stays separate from anything the
+    image ships or an operator put there by hand.
+    """
+    return {**os.environ, "DOCKER_CONFIG": docker_config_dir()}
+
+
+# --------------------------------------------------------------------------- #
+# Verifying credentials
+# --------------------------------------------------------------------------- #
+
+
+async def verify(host: str, username: str, password: str) -> None:
+    """Check that a registry accepts these credentials. Raises RegistryError.
+
+    Asks for a pull-scoped token the way a client would, and treats only an
+    outright 401 as "wrong credentials" — a registry that answers anything else
+    is reachable and talking, which is as much as a credentials check can
+    honestly claim.
+    """
+    registry = canonical_host(host)
+    url = f"https://{registry}/v2/"
+    auth = (username, password)
+    try:
+        async with httpx.AsyncClient(follow_redirects=True, timeout=TIMEOUT) as client:
+            response = await client.get(url, auth=auth)
+            if response.status_code == 401:
+                challenge = response.headers.get("WWW-Authenticate", "")
+                if challenge.lower().startswith("bearer"):
+                    token = await _token(client, challenge, auth)
+                    if not token:
+                        raise RegistryError("The registry rejected these credentials")
+                    return
+                raise RegistryError("The registry rejected these credentials")
+            if response.status_code >= 500:
+                raise RegistryError(
+                    f"The registry answered {response.status_code}; try again later"
+                )
+    except httpx.HTTPError as exc:
+        raise RegistryError(f"Could not reach {registry}: {exc}") from exc
+
+
+async def _token(
+    client: httpx.AsyncClient, challenge: str, auth: tuple[str, str]
+) -> Optional[str]:
+    """Follow a Bearer challenge with credentials attached."""
+    params = {}
+    for part in challenge[len("Bearer ") :].split(","):
+        if "=" in part:
+            key, value = part.split("=", 1)
+            params[key.strip()] = value.strip().strip('"')
+    realm = params.pop("realm", None)
+    if not realm:
+        return None
+    try:
+        response = await client.get(realm, params=params, auth=auth, timeout=TIMEOUT)
+        if response.status_code == 401:
+            return None
+        response.raise_for_status()
+        data = response.json()
+        return data.get("token") or data.get("access_token")
+    except (httpx.HTTPError, ValueError):
+        return None

backend/services/update_service.py

@@ -17,7 +17,7 @@ import httpx
 
 from docker_client import DockerError, get_client, safe_call
 from models.setting import EVENT_UPDATE_AVAILABLE
-from services import notify_service, settings_service
+from services import notify_service, registry_service, settings_service
 
 logger = logging.getLogger("stackpilot.update")
 
@@ -145,7 +145,16 @@ def _local_digest(image: str) -> Optional[str]:
 # --------------------------------------------------------------------------- #
 
 
-async def _get_token(client: httpx.AsyncClient, www_auth: str) -> Optional[str]:
+async def _get_token(
+    client: httpx.AsyncClient,
+    www_auth: str,
+    auth: Optional[tuple[str, str]] = None,
+) -> Optional[str]:
+    """Follow a Bearer challenge, with credentials when we have them.
+
+    A public image gets an anonymous token; a private one only gets a token at
+    all if the request to the token realm is authenticated.
+    """
     # Parse: Bearer realm="...",service="...",scope="..."
     params = {}
     if not www_auth.lower().startswith("bearer"):
@@ -158,7 +167,7 @@ async def _get_token(client: httpx.AsyncClient, www_auth: str) -> Optional[str]:
     if not realm:
         return None
     try:
-        resp = await client.get(realm, params=params, timeout=10)
+        resp = await client.get(realm, params=params, auth=auth, timeout=10)
         resp.raise_for_status()
         data = resp.json()
         return data.get("token") or data.get("access_token")
@@ -166,25 +175,54 @@ async def _get_token(client: httpx.AsyncClient, www_auth: str) -> Optional[str]:
         return None
 
 
+class AuthRequired(Exception):
+    """The registry wants credentials we do not have (or rejected ours).
+
+    Distinct from "could not reach the registry" on purpose: a private image
+    with no configured credentials used to be indistinguishable from a network
+    blip, so the UI said nothing and the stack looked up to date forever.
+    """
+
+
 async def remote_digest(image: str) -> Optional[str]:
+    """The digest the registry currently serves for this tag.
+
+    Raises :class:`AuthRequired` when the registry refuses us; returns None when
+    it could not be reached or answered without a digest.
+    """
     registry, repo, tag = parse_ref(image)
     if tag.startswith("sha256:"):
         return tag
     scheme = "https"
     url = f"{scheme}://{registry}/v2/{repo}/manifests/{tag}"
     headers = {"Accept": _MANIFEST_ACCEPT}
+    auth = registry_service.credentials_for(registry)
     async with httpx.AsyncClient(follow_redirects=True) as client:
         try:
             resp = await client.head(url, headers=headers, timeout=10)
             if resp.status_code == 401:
-                token = await _get_token(client, resp.headers.get("WWW-Authenticate", ""))
-                if not token:
-                    return None
-                headers["Authorization"] = f"Bearer {token}"
-                resp = await client.head(url, headers=headers, timeout=10)
+                challenge = resp.headers.get("WWW-Authenticate", "")
+                if challenge.lower().startswith("basic"):
+                    # A plain htpasswd-protected registry: no token dance.
+                    if not auth:
+                        raise AuthRequired(registry)
+                    resp = await client.head(url, headers=headers, auth=auth, timeout=10)
+                else:
+                    token = await _get_token(client, challenge, auth)
+                    if not token:
+                        raise AuthRequired(registry)
+                    headers["Authorization"] = f"Bearer {token}"
+                    resp = await client.head(url, headers=headers, timeout=10)
+            if resp.status_code in (401, 403):
+                raise AuthRequired(registry)
             if resp.status_code == 405 or "Docker-Content-Digest" not in resp.headers:
                 # Some registries don't support HEAD; fall back to GET.
-                resp = await client.get(url, headers=headers, timeout=10)
+                resp = await client.get(
+                    url, headers=headers, auth=auth if "Authorization" not in headers else None,
+                    timeout=10,
+                )
+                if resp.status_code in (401, 403):
+                    raise AuthRequired(registry)
             digest = resp.headers.get("Docker-Content-Digest")
             return digest
         except httpx.HTTPError as exc:
@@ -199,9 +237,18 @@ async def remote_digest(image: str) -> Optional[str]:
 
 async def check_image(image: str) -> UpdateStatus:
     local = _local_digest(image)
-    remote = await remote_digest(image)
     error = None
-    if remote is None:
+    try:
+        remote = await remote_digest(image)
+    except AuthRequired as exc:
+        # Say which registry, because the fix is to add credentials for it.
+        remote = None
+        error = (
+            f"{exc} needs credentials"
+            if not registry_service.credentials_for(str(exc))
+            else f"{exc} rejected the stored credentials"
+        )
+    if remote is None and error is None:
         error = "could not reach registry"
     update_available = bool(local and remote and local != remote)
     status = UpdateStatus(