Deploy stacks from a Git repository (0.58.0)

StackPilot's stacks were already plain folders on disk, which makes GitOps less
of an architectural change than it would be elsewhere: a sync is "make these
files match that repo, then compose up". Almost all of the design effort went
into the word "these", because getting it wrong destroys data.

A stack folder is not just the compose file. Compose creates bind-mount
directories in it — ./config, ./data — and those hold the live state of whatever
is running. So the obvious implementation, clone into the stack folder and
git reset --hard, is a data-loss bug waiting for its first `git clean`. Instead
the clone lives in a cache under ${DATA_DIR}/git/<stack> where reset and clean
are safe, and the configured subtree is copied across. No .git ends up in the
stack folder, so backups and the file browser are unaffected too.

Deletion is the other half. Making a folder "match" a repo naively means
removing what the repo does not have, which is exactly the application data
above. So each sync records the paths it wrote, and the next sync may delete
only those — a file the repository never provided cannot be touched by any code
path here. Tested directly: a database file and a hand-written .env survive a
sync that replaces the compose file and removes a file the repo dropped.

What the repo does provide is overwritten, hand edits included. That is the
point of GitOps rather than a wart, but it is a surprise if you attach a repo to
a stack you have been editing, so the connect form says it before the first sync
and the first sync is never automatic.

The webhook is the only route in StackPilot with no bearer token, because a Git
forge has none to present. It authenticates with an HMAC over the body —
X-Hub-Signature-256 for GitHub/Gitea/Forgejo, X-Gitlab-Token for GitLab, both
compared in constant time — and answers 404, not 403, to anything unsigned. A
403 would confirm that a given stack exists and is connected to a repository,
which an unauthenticated caller has not earned. The authorization matrix test
caught this route being public and made me write that reasoning down in it,
which is exactly what that test is for.

Credentials never reach a command line: ps is readable by every process on the
host, and this runs in a container next to everything else. The HTTPS token goes
to git through GIT_ASKPASS and the environment, the SSH key through a 0600 file
kept outside the working tree, and everything git prints is scrubbed of both —
plus any credential-carrying URL — before it is stored in last_error or shown.

Auto-deploy takes the same per-stack lock as every other lifecycle action, so a
webhook firing mid-deploy reports "files synced, stack busy" instead of racing a
second compose run at the same project.

The image needed git and openssh-client, which is the only reason this release
touches the Dockerfile.

26 tests against real repositories created with the real git binary, none of
them touching the network — mocking git would mostly test the mock. Verified end
to end as well: connect, sync, a push that changes one file and deletes another,
a wrongly signed webhook, a correctly signed one, and the live data still there
afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/tests/test_git_source.py

@@ -0,0 +1,413 @@
+"""Deploying a stack from Git.
+
+Real repositories, created locally with the real git binary — mocking git would
+mostly test the mock. Nothing here reaches the network.
+
+The test that matters most is the one about deletion. A stack folder holds live
+application data next to the compose file: compose creates bind-mount
+directories like ``./config`` right there. A sync that "makes the folder match
+the repo" by clearing what the repo does not have would destroy exactly that, so
+the rule is that only files the repository has itself provided may ever be
+removed.
+"""
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import subprocess
+
+import pytest
+from sqlmodel import Session, delete
+
+
+@pytest.fixture
+def svc(db):
+    from services import git_service
+
+    return git_service
+
+
+@pytest.fixture(autouse=True)
+def clean_sources(db):
+    from database import engine
+    from models.git_source import GitSource
+
+    def wipe():
+        with Session(engine) as session:
+            session.exec(delete(GitSource))
+            session.commit()
+
+    wipe()
+    yield
+    wipe()
+
+
+def _run(*args, cwd):
+    subprocess.run(
+        args,
+        cwd=cwd,
+        check=True,
+        capture_output=True,
+        env={
+            **os.environ,
+            "GIT_AUTHOR_NAME": "t",
+            "GIT_AUTHOR_EMAIL": "t@example.com",
+            "GIT_COMMITTER_NAME": "t",
+            "GIT_COMMITTER_EMAIL": "t@example.com",
+        },
+    )
+
+
+class _Origin:
+    """A real repository on disk. ``str()`` is its path, so it can be a clone URL."""
+
+    def __init__(self, path):
+        self.path = path
+
+    def __str__(self) -> str:
+        return str(self.path)
+
+    def commit(self, files: dict, message: str = "change") -> None:
+        for name, content in files.items():
+            path = self.path / name
+            if content is None:
+                path.unlink(missing_ok=True)
+                continue
+            path.parent.mkdir(parents=True, exist_ok=True)
+            path.write_text(content)
+        _run("git", "add", "-A", cwd=self.path)
+        _run("git", "commit", "-q", "-m", message, cwd=self.path)
+
+
+@pytest.fixture
+def origin(tmp_path):
+    repo = tmp_path / "origin"
+    repo.mkdir()
+    _run("git", "init", "-q", "-b", "main", cwd=repo)
+    made = _Origin(repo)
+    made.commit({"compose.yaml": "services:\n  app:\n    image: nginx:alpine\n"}, "init")
+    return made
+
+
+@pytest.fixture
+def source(db, origin, tmp_path, monkeypatch):
+    """A stack wired to that repository, with both roots inside tmp_path."""
+    from config import settings
+    from database import engine
+    from models.git_source import GitSource
+    from models.stack import Stack
+    from services import git_service
+
+    monkeypatch.setattr(settings, "DATA_DIR", str(tmp_path / "data"))
+    monkeypatch.setattr(settings, "STACKS_DIR", str(tmp_path / "stacks"))
+    os.makedirs(settings.STACKS_DIR, exist_ok=True)
+    git_service.ensure_cache_root()
+
+    with Session(engine) as session:
+        if (old := session.get(Stack, "gitstack")) is not None:
+            session.delete(old)
+            session.commit()
+        session.add(Stack(id="gitstack", name="Git Stack"))
+        row = GitSource(
+            stack_id="gitstack",
+            url=str(origin),
+            branch="main",
+            webhook_secret="s3cret",
+        )
+        session.add(row)
+        session.commit()
+        session.refresh(row)
+        row_id = row.id
+
+    with Session(engine) as session:
+        yield session, session.get(GitSource, row_id)
+
+
+def _sync(session, row, svc):
+    return asyncio.run(svc.sync(session, row, actor="test"))
+
+
+def _stack_dir() -> str:
+    from services import compose_service
+
+    return compose_service.stack_dir("gitstack")
+
+
+# --------------------------------------------------------------------------- #
+# Syncing
+# --------------------------------------------------------------------------- #
+
+
+def test_the_first_sync_brings_the_repo_into_the_stack(svc, source):
+    session, row = source
+    result = _sync(session, row, svc)
+
+    assert result.changed is True
+    assert result.written == ["compose.yaml"]
+    assert "nginx:alpine" in open(os.path.join(_stack_dir(), "compose.yaml")).read()
+    assert row.last_commit and row.last_error is None
+
+
+def test_syncing_again_with_no_new_commit_changes_nothing(svc, source):
+    session, row = source
+    _sync(session, row, svc)
+    again = _sync(session, row, svc)
+    assert again.changed is False
+    assert again.written == [] and again.removed == []
+
+
+def test_a_new_commit_is_picked_up(svc, source, origin):
+    session, row = source
+    _sync(session, row, svc)
+    origin.commit({"compose.yaml": "services:\n  app:\n    image: nginx:1.27\n"})
+
+    result = _sync(session, row, svc)
+    assert result.changed is True
+    assert "nginx:1.27" in open(os.path.join(_stack_dir(), "compose.yaml")).read()
+
+
+def test_a_hand_edited_file_is_put_back(svc, source):
+    """The repository is the source of truth; that is the whole point."""
+    session, row = source
+    _sync(session, row, svc)
+    path = os.path.join(_stack_dir(), "compose.yaml")
+    with open(path, "w") as fh:
+        fh.write("services: {}\n")
+
+    result = _sync(session, row, svc)
+    assert result.changed is True
+    assert "nginx:alpine" in open(path).read()
+
+
+def test_only_the_configured_subdirectory_is_deployed(svc, source, origin):
+    session, row = source
+    origin.commit({"stacks/web/compose.yaml": "services:\n  web:\n    image: caddy\n"})
+    row.subdir = "stacks/web"
+
+    result = _sync(session, row, svc)
+    assert result.written == ["compose.yaml"]
+    assert "caddy" in open(os.path.join(_stack_dir(), "compose.yaml")).read()
+
+
+def test_a_subdirectory_that_escapes_the_repo_is_refused(svc, source):
+    session, row = source
+    row.subdir = "../../../etc"
+    with pytest.raises(svc.GitError):
+        _sync(session, row, svc)
+
+
+def test_a_missing_subdirectory_is_reported(svc, source):
+    session, row = source
+    row.subdir = "nope"
+    with pytest.raises(svc.GitError):
+        _sync(session, row, svc)
+    assert row.last_error and "nope" in row.last_error
+
+
+def test_an_unreachable_repository_is_recorded_not_raised_away(svc, source, tmp_path):
+    session, row = source
+    row.url = str(tmp_path / "does-not-exist")
+    with pytest.raises(svc.GitError):
+        _sync(session, row, svc)
+    # Stored, so the UI can show why the last attempt failed.
+    assert row.last_error
+
+
+# --------------------------------------------------------------------------- #
+# What may be deleted — the dangerous part
+# --------------------------------------------------------------------------- #
+
+
+def test_a_file_dropped_from_the_repo_is_removed_from_the_stack(svc, source, origin):
+    session, row = source
+    origin.commit({"extra.env": "A=1\n"})
+    _sync(session, row, svc)
+    assert os.path.isfile(os.path.join(_stack_dir(), "extra.env"))
+
+    origin.commit({"extra.env": None}, "drop it")
+    result = _sync(session, row, svc)
+    assert result.removed == ["extra.env"]
+    assert not os.path.exists(os.path.join(_stack_dir(), "extra.env"))
+
+
+def test_files_the_repo_never_had_are_never_touched(svc, source, origin):
+    """Compose puts live application data in the stack folder. It must survive."""
+    session, row = source
+    _sync(session, row, svc)
+
+    data_dir = os.path.join(_stack_dir(), "config")
+    os.makedirs(data_dir, exist_ok=True)
+    with open(os.path.join(data_dir, "app.db"), "w") as fh:
+        fh.write("precious")
+    with open(os.path.join(_stack_dir(), ".env"), "w") as fh:
+        fh.write("SECRET=hunter2\n")
+
+    origin.commit({"compose.yaml": "services:\n  app:\n    image: nginx:1.27\n"})
+    _sync(session, row, svc)
+
+    assert open(os.path.join(data_dir, "app.db")).read() == "precious"
+    assert open(os.path.join(_stack_dir(), ".env")).read() == "SECRET=hunter2\n"
+
+
+def test_a_file_the_repo_stops_providing_is_only_removed_if_it_provided_it(svc, source, origin):
+    """A file the stack folder already had before Git ever touched it."""
+    session, row = source
+    os.makedirs(_stack_dir(), exist_ok=True)
+    with open(os.path.join(_stack_dir(), "notes.txt"), "w") as fh:
+        fh.write("mine")
+    _sync(session, row, svc)
+    # The repo never provided notes.txt, so the first sync left it alone.
+    assert os.path.isfile(os.path.join(_stack_dir(), "notes.txt"))
+    assert "notes.txt" not in json.loads(row.managed_files)
+
+
+def test_managed_files_are_recorded_for_the_next_sync(svc, source, origin):
+    session, row = source
+    origin.commit({"a.yaml": "a", "sub/b.yaml": "b"})
+    _sync(session, row, svc)
+    assert sorted(json.loads(row.managed_files)) == [
+        "a.yaml",
+        "compose.yaml",
+        os.path.join("sub", "b.yaml"),
+    ]
+
+
+def test_a_directory_left_empty_by_a_removal_is_pruned(svc, source, origin):
+    session, row = source
+    origin.commit({"sub/b.yaml": "b"})
+    _sync(session, row, svc)
+    assert os.path.isdir(os.path.join(_stack_dir(), "sub"))
+
+    origin.commit({"sub/b.yaml": None}, "drop")
+    _sync(session, row, svc)
+    assert not os.path.exists(os.path.join(_stack_dir(), "sub"))
+    # But never the stack folder itself.
+    assert os.path.isdir(_stack_dir())
+
+
+# --------------------------------------------------------------------------- #
+# Webhook authorization
+# --------------------------------------------------------------------------- #
+
+
+def _sig(secret: str, body: bytes) -> str:
+    import hmac
+
+    return "sha256=" + hmac.new(secret.encode(), body, "sha256").hexdigest()
+
+
+def test_a_correctly_signed_webhook_is_accepted(svc, source):
+    _session, row = source
+    body = b'{"ref":"refs/heads/main"}'
+    assert svc.verify_webhook(row, body, {"X-Hub-Signature-256": _sig("s3cret", body)})
+
+
+def test_a_wrong_signature_is_rejected(svc, source):
+    _session, row = source
+    body = b'{"ref":"refs/heads/main"}'
+    assert not svc.verify_webhook(row, body, {"X-Hub-Signature-256": _sig("wrong", body)})
+    # A signature over different content must not carry over.
+    assert not svc.verify_webhook(row, b"tampered", {"X-Hub-Signature-256": _sig("s3cret", body)})
+
+
+def test_an_unsigned_webhook_is_rejected(svc, source):
+    _session, row = source
+    assert not svc.verify_webhook(row, b"{}", {})
+
+
+def test_the_gitlab_token_header_works_too(svc, source):
+    _session, row = source
+    assert svc.verify_webhook(row, b"{}", {"X-Gitlab-Token": "s3cret"})
+    assert not svc.verify_webhook(row, b"{}", {"X-Gitlab-Token": "nope"})
+
+
+def test_a_source_with_no_secret_accepts_nothing(svc, source):
+    _session, row = source
+    row.webhook_secret = ""
+    assert not svc.verify_webhook(row, b"{}", {"X-Gitlab-Token": ""})
+
+
+def test_the_webhook_endpoint_hides_whether_a_stack_is_connected(client, source):
+    """Unsigned calls get 404, so the endpoint cannot be used to enumerate."""
+    connected = client.post("/api/git/webhook/gitstack", content=b"{}")
+    unknown = client.post("/api/git/webhook/no-such-stack", content=b"{}")
+    assert connected.status_code == 404
+    assert unknown.status_code == 404
+    assert connected.json() == unknown.json()
+
+
+# --------------------------------------------------------------------------- #
+# Polling
+# --------------------------------------------------------------------------- #
+
+
+def test_polling_is_off_unless_an_interval_is_set(svc, source):
+    _session, row = source
+    row.poll_interval_minutes = None
+    assert svc.due(row) is False
+    row.poll_interval_minutes = 0
+    assert svc.due(row) is False
+
+
+def test_a_source_that_has_never_synced_is_due(svc, source):
+    _session, row = source
+    row.poll_interval_minutes = 15
+    row.last_synced_at = None
+    assert svc.due(row) is True
+
+
+def test_due_respects_the_interval(svc, source):
+    from datetime import datetime, timedelta, timezone
+
+    _session, row = source
+    row.poll_interval_minutes = 15
+    now = datetime.now(timezone.utc)
+    row.last_synced_at = now - timedelta(minutes=5)
+    assert svc.due(row, now) is False
+    row.last_synced_at = now - timedelta(minutes=16)
+    assert svc.due(row, now) is True
+
+
+# --------------------------------------------------------------------------- #
+# Secrets never leak
+# --------------------------------------------------------------------------- #
+
+
+def test_the_api_never_returns_the_stored_secret(as_admin, source):
+    from services import crypto_service
+
+    session, row = source
+    row.auth_type = "token"
+    row.username = "bob"
+    row.secret = crypto_service.encrypt("ghp_supersecret")
+    session.add(row)
+    session.commit()
+
+    body = as_admin.get("/api/stacks/gitstack/git").text
+    assert "ghp_supersecret" not in body
+    assert '"has_secret":true' in body.replace(" ", "")
+
+
+def test_a_token_is_scrubbed_from_error_output(svc):
+    message = "fatal: could not read from https://bob:ghp_supersecret@example.com/x.git"
+    assert "ghp_supersecret" not in svc._redact(message, "ghp_supersecret")
+    # Even without being told the value, a credential-carrying URL is masked.
+    assert "ghp_supersecret" not in svc._redact(message)
+
+
+def test_connecting_and_disconnecting_through_the_api(as_admin, source, origin):
+    connected = as_admin.put(
+        "/api/stacks/gitstack/git",
+        json={"url": str(origin), "branch": "main", "auto_deploy": False},
+    )
+    assert connected.status_code == 200, connected.text
+    assert connected.json()["webhook_url"] == "/api/git/webhook/gitstack"
+
+    assert as_admin.delete("/api/stacks/gitstack/git").status_code == 200
+    assert as_admin.get("/api/stacks/gitstack/git").status_code == 404
+
+
+def test_the_read_only_role_cannot_touch_git_settings(as_user, source):
+    assert as_user.get("/api/stacks/gitstack/git").status_code == 403
+    assert as_user.post("/api/stacks/gitstack/git/sync").status_code == 403

backend/tests/test_route_authorization.py

@@ -39,6 +39,12 @@ PUBLIC = {
     # Only drops the refresh cookie. Requiring a valid token would mean you
     # cannot sign out once the session has already gone stale.
     "POST /api/auth/logout",
+    # A Git forge has no StackPilot credentials to present, so this one cannot
+    # be behind a bearer token. It is authorized instead by an HMAC over the
+    # request body against a per-stack secret, and answers 404 — not 403 — to
+    # anything unsigned, so it cannot be used to discover which stacks exist or
+    # which are connected to a repository. See routers/git.py.
+    "POST /api/git/webhook/{stack_id}",
 }
 
 #: Reachable by the read-only ``user`` role. Everything here has been checked