menzeljonas/stackpilot
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Compare commits

merge into: menzeljonas/stackpilot:feature/phase18-22
Branches Tags
menzeljonas/stackpilot:main menzeljonas/stackpilot:feature/phase18-22
...
pull from: menzeljonas/stackpilot:main
Branches Tags
menzeljonas/stackpilot:main menzeljonas/stackpilot:feature/phase18-22
51 Commits
feature/phase18-22 ... main
Author SHA1 Message Date
menzeljandClaude Opus 5 9247ff9621 Deploy stacks from a Git repository (0.58.0)
CI / build-and-push (push) Blocked by required conditions
Details
CI / check (push) In progress
Details 
StackPilot's stacks were already plain folders on disk, which makes GitOps less
of an architectural change than it would be elsewhere: a sync is "make these
files match that repo, then compose up". Almost all of the design effort went
into the word "these", because getting it wrong destroys data.

A stack folder is not just the compose file. Compose creates bind-mount
directories in it — ./config, ./data — and those hold the live state of whatever
is running. So the obvious implementation, clone into the stack folder and
git reset --hard, is a data-loss bug waiting for its first `git clean`. Instead
the clone lives in a cache under ${DATA_DIR}/git/<stack> where reset and clean
are safe, and the configured subtree is copied across. No .git ends up in the
stack folder, so backups and the file browser are unaffected too.

Deletion is the other half. Making a folder "match" a repo naively means
removing what the repo does not have, which is exactly the application data
above. So each sync records the paths it wrote, and the next sync may delete
only those — a file the repository never provided cannot be touched by any code
path here. Tested directly: a database file and a hand-written .env survive a
sync that replaces the compose file and removes a file the repo dropped.

What the repo does provide is overwritten, hand edits included. That is the
point of GitOps rather than a wart, but it is a surprise if you attach a repo to
a stack you have been editing, so the connect form says it before the first sync
and the first sync is never automatic.

The webhook is the only route in StackPilot with no bearer token, because a Git
forge has none to present. It authenticates with an HMAC over the body —
X-Hub-Signature-256 for GitHub/Gitea/Forgejo, X-Gitlab-Token for GitLab, both
compared in constant time — and answers 404, not 403, to anything unsigned. A
403 would confirm that a given stack exists and is connected to a repository,
which an unauthenticated caller has not earned. The authorization matrix test
caught this route being public and made me write that reasoning down in it,
which is exactly what that test is for.

Credentials never reach a command line: ps is readable by every process on the
host, and this runs in a container next to everything else. The HTTPS token goes
to git through GIT_ASKPASS and the environment, the SSH key through a 0600 file
kept outside the working tree, and everything git prints is scrubbed of both —
plus any credential-carrying URL — before it is stored in last_error or shown.

Auto-deploy takes the same per-stack lock as every other lifecycle action, so a
webhook firing mid-deploy reports "files synced, stack busy" instead of racing a
second compose run at the same project.

The image needed git and openssh-client, which is the only reason this release
touches the Dockerfile.

26 tests against real repositories created with the real git binary, none of
them touching the network — mocking git would mostly test the mock. Verified end
to end as well: connect, sync, a push that changes one file and deletes another,
a wrongly signed webhook, a correctly signed one, and the live data still there
afterwards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-18 01:06:29 +02:00
menzeljandClaude Opus 5 e650aa6833 Add API tokens for scripts and CI (0.57.0)
CI / check (push) Successful in 12m31s
Details
CI / build-and-push (push) Successful in 1m56s
Details 
A session token is the wrong credential for automation. It expires in an hour,
it is minted by typing a password, and revoking it signs every one of that
person's devices out. So automation gets its own credential, revocable on its
own, and showing up in the audit log as itself.

Three decisions worth recording, because each one is a place this could have
been built wrong.

**Only a hash is stored.** This is the opposite call from registry passwords one
release ago, and for a concrete reason: a registry password has to be handed
back to the registry, so it must be recoverable and is encrypted. A token is
only ever compared against, so it does not need to be — and not keeping it is
the difference between leaking the database and leaking everything the database
protects. It is shown once and cannot be recovered; a readable prefix is kept so
rows are still identifiable in the UI and the audit log. The hash is SHA-256,
deliberately not bcrypt: bcrypt is slow to make guessing low-entropy human
passwords expensive, and a token is 256 bits of secrets output, so the cost
would buy nothing and would land on every single API request.

**The scope is not folded into the User object.** get_current_user returns a
session-attached row; downgrading its role in place to represent a read-only
token would be written back to the database the next time anything committed
that user — logout-everywhere does exactly that. So the token row is stashed on
request.state and require_admin consults it, leaving the User untouched. The
same lookup caps a token at its owner's authority rather than trusting the scope
alone, so a demoted admin's token drops to read-only with them and a disabled
account's tokens stop working.

**A token cannot make itself permanent.** Creating tokens and creating users now
require a signed-in session, via a require_session dependency that rejects
token-authenticated requests. Without it, a leaked CI credential could mint a
second one and survive its own revocation — the failure mode where revoking the
leak does nothing. This is the one behaviour change for existing installs:
scripted user creation now needs a login.

The WebSocket routes still take JWTs only. They carry logs, the terminal and the
deploy console, which a CI job has no use for, and leaving them alone keeps the
token surface to the REST API.

19 tests, covering what is stored, that a read token really is read-only while
its owner is an admin, that demoting and disabling the owner both take effect,
expiry, tampering, the throttle on last-used writes, and that a token can
neither mint another nor create a user. Verified end to end against a running
app: two tokens, both scopes, revocation, and no plaintext anywhere in the
database or the list response.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-18 00:53:58 +02:00
menzeljandClaude Opus 5 95e03f031f Add private registry credentials, and stop update checks lying (0.56.0)
CI / check (push) Successful in 12m33s
Details
CI / build-and-push (push) Successful in 2m1s
Details 
This was on the gap list as a missing feature, but it was a bug first. The
update checker asks the registry for a tag's digest over HTTP itself, and could
only do it anonymously. A private repository answers 401, remote_digest returned
None, and None already meant "could not reach registry" — so a private image was
indistinguishable from a network blip. The Images page showed nothing and a
stack pinned to a six-month-old image looked up to date indefinitely.

So AuthRequired is now its own exception, separate from unreachable, and the
error names the registry and which of the two problems it is: "ghcr.io needs
credentials" when there are none, "ghcr.io rejected the stored credentials" when
there are and they are wrong. Those are different fixes, and the message should
say which one you need. The plain unreachable message survives unchanged, with a
test pinning it, because not every failure is an auth failure.

Two consumers need the credentials and they need them in completely different
shapes, which is why this is its own service rather than a field on something
else. StackPilot's own checker wants (user, password) inside async code that has
no database session, so the rows are mirrored into an in-memory cache that
reload() refills on startup and after every write. The Docker CLI wants a
config.json, so reload() writes one into ${DATA_DIR}/docker and compose runs with
DOCKER_CONFIG pointed at it. Generating it from the database every time is what
makes deletion real: removing a registry in the UI revokes the CLI's login
instead of leaving a stale one in ~/.docker.

Host normalization is the join that makes any of it work, and it is easy to
underestimate. parse_ref only ever produces registry-1.docker.io, nobody types
that, and the CLI wants the whole thing under https://index.docker.io/v1/ — three
spellings of one registry across three layers. canonical_host settles on what
parse_ref produces, the config writer translates on the way out, and a bare
nginx:alpine finds credentials entered as "docker.io". Verified end to end:
typed as the v1 URL, stored as registry-1.docker.io, written as the v1 URL.

The password is encrypted at rest with the same key as backup destinations and
never leaves the server, not even masked — the API returns has_password, which
is all the form needs to offer "leave blank to keep". A row that cannot be
decrypted after a SECRET_KEY change is skipped with a warning rather than taking
every other registry down with it. Everything here is admin-only including the
reads, because even masked the rows say which registries this install talks to
and under what account.

The Test button asks the registry rather than validating a string, following the
Bearer challenge with credentials attached the way a real client does. Only an
outright 401 counts as wrong credentials; anything else means reachable and
talking, which is as much as a credentials check can honestly claim. Checked
against the live Docker Hub token endpoint with deliberately wrong credentials.

33 tests: the normalization table, the cache, the generated config.json down to
its 0600 mode and the Docker Hub key, encryption at rest, that no password field
appears in any response, and the 401-is-reported behaviour that started this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-18 00:42:27 +02:00
menzeljandClaude Opus 5 a2adb59526 Group Images and Networks by stack, like Volumes (0.55.0)
CI / check (push) Successful in 12m23s
Details
CI / build-and-push (push) Successful in 2m4s
Details 
The same argument as 0.54.0: these pages already sorted a stack's resources next
to each other, and still made you read prefixes to work out where one stack
ended and the next began. Networks are the clearest win — compose names a
stack's own network <project>_default, so the column was almost entirely prefix.

Doing it a second and third time made the shape obvious, so the grouping is now
one function and one heading component shared by all three pages rather than
three copies drifting apart. A stack looks and sorts the same wherever it turns
up. volumeGroups.ts became stackGroups.ts on the way, and the Volumes page moved
onto it with no behaviour change.

Images forced the model to grow, and this is the part worth reading. An image
carries no compose label — the backend derives its owners from the containers
running it, so ownership is a *list*, and postgres:16 may belong to four stacks
at once. Listing it under each owner would show the same image four times with
four sizes, and a page that adds up to more disk than the host has. So anything
with more than one owner is listed once in a "Shared by several stacks" group,
and the Used by column names them. Each resource appears exactly once, which
keeps the per-group totals honest.

Networks needed a second new kind. bridge, host and none belong to no stack, but
they are not leftovers either, and dropping them into the unassigned group pads
the exact list people scan for junk. They get their own group below it. The
built-in check runs before ownership is even considered, so they can never be
counted as unclaimed.

Ordering is unchanged and now stated once: stacks by display name, then shared,
then unclaimed, then built-ins. The trailing three are appended, never sorted
in, so "unassigned at the bottom" holds whatever anything is called — there is a
test for the case where the only real stack sorts after them.

Group headings gained the counts each page can actually produce: volumes show
unused and total size, images total size and how many have an update waiting,
networks how many are idle.

Both row bodies moved into their own components. Nesting them a level deeper
inside the group left the indentation adrift, and the networks row had a second
<tr> for its expanded detail riding along inside an already doubled map.

Fifteen tests on the shared grouping, including the shared and built-in groups
and a stack that has been deleted since. No server change: every one of these
already knew its stack.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-18 00:11:26 +02:00
menzeljandClaude Opus 5 d7c4f06e67 Group the Volumes page by the stack that owns each volume (0.54.0)
CI / check (push) Successful in 12m18s
Details
CI / build-and-push (push) Successful in 1m55s
Details 
A flat volume list is sorted by name, and because Docker names a compose volume
<project>_<name> that already puts a stack's volumes next to each other. What it
does not do is say so: you read prefixes down the column to work out where one
stack's volumes end and the next begins, and `arr-stack_config`,
`arr-stack_downloads`, `immich_model-cache` is exactly as hard to scan as it
looks.

So the stack becomes a heading instead of a prefix repeated on every row. Each
group carries the stack's icon and name, its volume count, how many are unused,
and — once sizes have been computed — what the stack costs on disk, which is the
number you actually want when you are deciding what to clear out. The rows below
drop the prefix and show the part that differs: `pgdata`, not `immich_pgdata`.
The full name stays in the row's title attribute, since that is what you need
when typing a docker command.

Ordering: stacks by the name the user gave them rather than by the slug (an id
of "zz-project" for a stack called "Alpha" should sort under A), and volumes
belonging to no stack appended last, never sorted in — they are the ones you
scroll past rather than look for.

The case that turned out to be worth building for is the third one. A volume
keeps its compose label after the stack is gone, so it is neither owned nor
loose. Putting it in the unassigned group would hide it among portainer_data and
friends; instead it keeps its own heading, marked "stack removed". A flat list
made that invisible, and it is precisely where forgotten data sits.

No server change: the owning stack has been on every volume all along, from the
com.docker.compose.project label. The grouping is a pure function in lib/ rather
than logic inside the page, so the ordering rules are tested directly — ten
cases, including the deleted-stack group and that the unassigned group stays
last when the only real stack sorts after it.

The row markup moved into its own component on the way past. Nesting it one
level deeper inside the group left its indentation two stops adrift, and a 70-
line <tr> inline in a double map was already the least readable thing in the file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-17 15:02:34 +02:00
menzeljandClaude Opus 5 76a228314a Plate only the logos that need one, instead of all of them (0.53.0)
CI / check (push) Successful in 12m3s
Details
CI / build-and-push (push) Successful in 1m56s
Details 
0.52.0 put every app logo on a white tile. The reason was real — a good number
of logos are solid black line art and vanish into a dark surface — but the cure
was applied to all of them, and in dark mode that makes each row look like it
has a sticker pasted on it.

So the tile goes back to the same neutral surface everything else uses, and the
decision is made per image. The browser already holds the bytes, so it draws
each one into a 24px canvas once and measures it: no new dependency, no second
request, and it covers uploads as well as catalog logos.

Measuring luminance alone was the first attempt and it is wrong. It plates Home
Assistant, whose logo is a mid-blue house that reads on anything, and it plates
Plex, which is dark *orange* — mean luminance cannot see that hue is doing the
work. So chroma is measured too, and a plate requires low contrast **and** art
with essentially no colour of its own. The threshold sits between the logos that
only look monochrome (Sonarr 0.115, Uptime Kuma 0.129) and the ones that are
(MinIO 0.065, Memos 0.037).

Checked against 66 real logos rather than guessed: six get a plate in dark mode
(Vaultwarden, Tailscale, Frigate, Heimdall, Miniflux, MinIO, all solid black),
two in light mode (Ollama, Open-WebUI, solid white). The other ~90% sit bare.
Mid-grey art like Bazarr is deliberately left alone — it already has contrast
against both grounds, and a plate would be noise.

Every failure path returns "no plate": jsdom with no canvas, a blocked canvas, an
image that will not decode. Being wrong in that direction costs contrast on a
handful of icons; being wrong the other way is the sticker problem again. The
unit test is built from the measured luminance/chroma pairs, so it tests the
rule against the art it actually has to handle.

0.53.0 because 0.52.0's images are already in the registry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-17 11:08:08 +02:00
menzeljandClaude Opus 5 b629d1b2c2 Use the apps' real logos as stack icons, fetched server-side (0.52.0)
CI / check (push) Successful in 12m8s
Details
CI / build-and-push (push) Successful in 2m1s
Details 
0.51.0 gave every stack an icon, but a generic one: jellyfin got a clapperboard,
not the Jellyfin logo. Glyphs make a list readable; they do not make a stack
recognisable, which was the point. This resolves stacks against the selfh.st
icon catalog (~2900 self-hosted apps, the set Homarr and Homepage draw on), so
the row shows the thing people already recognise. All 83 bundled templates
resolve to their own logo.

The whole design question was *who* talks to the CDN. If the <img> points at
jsdelivr, then every client needs internet, every page load leaks the names of
somebody's stacks to a third party, and an air-gapped box gets nothing. So the
backend does it: the catalog on startup and weekly after, each logo once on
first use, both into ${DATA_DIR}/stack-icons/. Browsers keep reading icons from
the authenticated endpoint that already existed for uploads, and after the first
fetch the feature is fully offline. Logos are cached per *app*, not per stack —
verified: two stacks resolving to jellyfin produce one download.

Nothing here can fail loudly. Every entry point returns None rather than raising
when the network is absent, the catalog refresh is a task the lifespan does not
await, and an install with no outbound internet simply keeps 0.51.0's glyphs.
That fallback is also what covers a name the catalog does not know
("Mediaserver Wohnzimmer" is still a clapperboard), and the seconds after a
fresh install before the catalog lands. The glyph is derived even for stacks
that *do* have a logo, so an image that cannot be fetched degrades to something
meaningful instead of a box.

Matching gained a second source that turned out to matter more than expected:
the compose images. A stack called "medienserver" says nothing, but it pulls
lscr.io/linuxserver/jellyfin — strip the registry, the vendor and the tag and
the app is right there. Name first, then the longest run of words inside it,
then the images. It is deliberately cautious: a single word shorter than four
characters never claims a logo, because "web", "app" and "db" are all catalog
entries and a *wrong* logo is worse than a neutral glyph. A short alias table
covers what the catalog spells differently from Docker Hub (postgres →
postgresql, pihole → pi-hole, wg-easy → wireguard).

A slug arrives from the database and from query strings and then becomes a
filename, so it is pattern-checked before it is ever joined to a path, catalog
entries that are not slug-shaped are dropped on load, and a downloaded logo is
verified to start with the PNG magic bytes before being cached.

The picker searches the catalog too — pre-seeded with the stack's own name, so
opening it on "jellyfin" offers the Jellyfin logo first — which is how a wrong
match gets corrected, and how a stack can be given any app's logo on purpose.

Verified end to end against the live catalog and real downloads: list rows carry
the resolved logo, the icon endpoint serves real PNG bytes, an unmatched stack
404s (and falls through to its glyph), a hand-picked logo round-trips, reset
clears it, and a traversal slug 404s. 30 new backend tests and 12 new frontend
ones run without any network at all.

0.52.0 rather than amending 0.51.0: those images are already in the registry,
and rebuilding a published version tag with different content is exactly what
breaks the self-update checker.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-17 10:39:14 +02:00
menzeljandClaude Opus 5 7682460b4f Give every stack an icon, and put the status on it (0.51.0)
CI / check (push) Successful in 12m14s
Details
CI / build-and-push (push) Successful in 3m37s
Details 
Stacks were a name and a coloured dot. The dot carried the status but nothing
carried identity, so a list of twenty stacks read as twenty identical rows.
This gives each one an icon in front of its name and moves the status onto that
icon as a halo in the status colour, which is the thing the eye lands on anyway.

The constraint that shaped the design: people already have stacks. Asking them
to pick an icon for each one before the feature does anything would mean it
never gets used, so the icon is *derived* from the stack's name and the column
stays empty until somebody overrides it. ~700 keywords in 79 groups cover the
self-hosted long tail (jellyfin -> clapperboard, vaultwarden -> key,
home-assistant -> house) plus generic English and German terms; the longest
match wins, so photoprism beats a bare photo, and short keywords like "tv" only
match as whole words. No backfill, no migration, and a rename moves the icon
with it.

That is also why the catalog and the matcher live in the frontend. It is the
only place that can render an icon, so a copy in the backend would be a list to
keep in sync and nothing else. The server validates the shape of the stored
value and stores uploads; it never needs to know what "lucide:database" looks
like. An icon name that later leaves the catalog falls back to the derived one
rather than blanking the row.

Overriding happens in two places, because there are two moments: the editor
(holding a chosen file until the stack exists, since uploading needs an id) and
a click on the icon on the detail page, which is how a stack that has existed
for a year gets one without a trip through the editor.

Uploads are classified by their bytes, not by the filename or Content-Type the
browser claims, and land in ${DATA_DIR}/stack-icons/ under the stack id. SVG is
allowed — <img> does not execute it — but the endpoint serves every icon as an
attachment so one can never be opened as a document in the API's own origin. A
client-supplied "custom:" value is refused: the server mints those, so a stack
cannot be pointed at a file it does not own. Files follow the stack: replaced on
re-upload (including across formats, or the old one orphans), copied on clone,
removed on delete.

The one piece of plumbing worth knowing about: the icon endpoint needs the
bearer token like everything else, and an <img src> would not carry it. So
StackIcon fetches the bytes through the API client and renders the blob, keyed
on the stored value — which carries an upload timestamp precisely so a re-upload
changes the key and retires the cached image.

Covered by 22 backend tests (the value rules, byte-sniffing, the file lifecycle,
the API round-trip, and that the read-only role cannot change an icon) and 29
frontend ones for the matcher. The schema change was verified against a
hand-built pre-0.51 database: the column is added on start and existing rows
come back NULL, i.e. automatic. Not click-tested in a browser — no Docker in
this environment — so the row height the taller icon produces is unverified.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-09-17 10:03:10 +02:00
menzeljandClaude Opus 5 a25741f579 Add an error boundary and split the bundle (0.50.0)
CI / check (push) Successful in 7m23s
Details
CI / build-and-push (push) Successful in 1m56s
Details 
F15 — Two unrelated frontend weaknesses.

A render error unmounted the whole React tree: a white window, no navigation,
no indication of what happened, and the only way out was knowing to reload.
ErrorBoundary now shows the message with a retry and a reload, and clears itself
when resetKey (the route) changes, so navigating to a working page just works
instead of staying stuck. There are two: one inside AppShell around the routed
pages, one at the root for the shell itself and the login screen, which sit
outside it.

The bundle was one 841 kB file (234 kB gzipped) every visitor downloaded in
full, with Vite warning about it on every build. Routes are lazy now and the
entry chunk is 377 kB (120 kB gzipped) — a 55% cut, warning gone.

Measuring first changed what to split. Monaco turned out not to be in the bundle
at all: @monaco-editor/react loads it from cdn.jsdelivr.net, so only the small
wrapper ships. xterm.js *is* bundled, all 294 kB of it, and it was reachable
from ContainerCard — which renders on every stack detail page — so every visitor
paid for a terminal most never open. It is lazy now and lands in its own chunk.

(Worth knowing separately: the compose editor therefore needs jsdelivr.net
reachable. For a self-hosted tool on an air-gapped network that is a real
limitation, but vendoring Monaco means +3 MB and is its own change.)

Adding a boundary whose behaviour I could only reason about was not good enough,
and the missing frontend test runner was already flagged as the gap from 0.49.0.
So this also sets up vitest + jsdom + testing-library and covers the boundary:
that it renders the error rather than a blank page, offers a way out, clears on
navigation, and stays put on an unrelated re-render. CI runs `npm test` next to
pytest.

One snag worth recording: installing the dev dependencies triggered npm's
optional-dependency pruning and dropped @rollup/rollup-linux-x64-gnu, which
broke the build. Reinstalling it directly put a linux-x64-glibc binary in
package.json, which would have broken `npm ci` on every other platform — so that
was backed out and the lockfile now carries the bindings as rollup's optional
deps, where they belong. Verified with a clean `npm ci` in a scratch copy:
install, typecheck, build and test all pass from the committed lockfile.

758 backend tests, 7 frontend tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
 2026-08-31 15:38:14 +02:00
menzeljandClaude Opus 5 fb2eefb0e1 Refresh the UI from Docker events instead of polling (0.49.0)
CI / check (push) Successful in 7m7s
Details
CI / build-and-push (push) Successful in 1m39s
Details 
F16 — /ws/events was implemented and nothing consumed it, while thirty polling
intervals across the pages asked for state that only changes when Docker does
something. The endpoint was the answer; it just was not usable as it stood, so
this is three fixes and a client, not a wiring job.

The endpoint forwarded the whole firehose. Three exec_* events fire per web
terminal session and top/attach fire whenever anything inspects a container, so
a client invalidating on each would have been noisier than the polling it
replaces. Now the daemon filters by resource type and the handler drops the
actions that say nothing about rendered state — matching on the verb before the
colon, since Docker reports these as "exec_create: /bin/sh".

It never said *what* changed, so there was nothing to decide which caches to
drop. The payload now carries the resource type.

And it leaked its reader thread. Cancelling the executor future does not
interrupt a thread already inside a blocking read; closing the underlying
CancellableStream is what does. Every page load left one behind holding a socket
open. A test asserts the close, because this is invisible until the process has
been up for a week.

Client side, useDockerEvents holds one connection for the session and maps
resource types to query keys. Bursts are coalesced over 300ms — a ten-service
compose up emits dozens of events in a second, and refetching per event would
reintroduce exactly the load being removed. Reconnects back off to 30s, and any
close reconnects including 4401, since the access token is short-lived and gets
refreshed out from under the socket.

Intervals drop from the mechanism to the safety net: 5s becomes 30-60s. Two
deliberately stay fast. Live CPU/memory drifts continuously with no event to
announce it, and that one is served from the 4s server-side cache added in
0.47.0, so it costs one sample per interval regardless of how many tabs are
open. The audit feed polls because its entries come from people, not Docker.

Net effect is both cheaper and faster: no fixed floor of requests per second
against the daemon, and a stack that finishes starting shows up immediately
rather than up to five seconds later.

23 new tests (758 total), driven against a fake daemon. Both nets were checked
by reverting the fix: dropping the filter fails one, dropping the stream close
fails the leak test.

Not covered: the hook itself has no test — there is no frontend test runner yet.
Its contract with the backend is tested; its own behaviour is only typechecked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
 2026-08-31 15:25:10 +02:00
menzeljandClaude Opus 5 51d1998307 Remove the remote-host (agent) integration (0.48.0)
CI / check (push) Successful in 7m17s
Details
CI / build-and-push (push) Successful in 1m45s
Details 
StackPilot now manages exactly one Docker host: the one it runs on. The
stackpilot-agent sidecar and everything that proxied to it are gone — 4721
lines deleted against 657 added.

Deleted outright: agent/ (image, compose, env), agent_app.py, models/agent.py,
routers/agents.py (1200 lines), services/agent_service.py, the agent API client,
RemoteStackDetail, the host components and AgentStacksSection. That removes 57
API routes and the three /ws/agent-* proxies.

Threaded out everywhere else, which was the bulk of the work. Every API module
carried an optional agentId that switched the base path; every page that listed
Docker objects rendered one section per host behind a HostHeader; Files had a
host switcher; the New Stack editor and the template dialog had host selectors;
schedules, auto-update policies and stack summaries carried agent_id. All of it
is gone, and the typechecker drove the sweep — 85 files touched, tsc and the
build clean.

Two things the removal exposed as dead weight rather than merely unused:

compose_service kept an in-process busy set purely because the agent needed a
lock and has no database. With the agent gone that was a second source of truth
next to the real DB lock, so it is deleted; compute_status now reports only what
the containers say and the two callers that want "updating" overlay the lock.
StacksTable's linkBase prop only ever existed to point at /hosts/{id}/stacks.

The dashboard's "Hosts 1/1 online" KPI can no longer say anything else, so the
tile and the KPIs behind it are gone and the row is five wide.

Upgrading matters here. An existing install still has an agent table holding
each remote host's URL and bearer token — full Docker control of that host,
sitting in the database with nothing left to use it. _drop_removed_schema drops
it on first start, and drops the agent_id columns where the SQLite build
supports DROP COLUMN. Each statement runs in its own transaction on purpose: a
failed DDL poisons the transaction it is in, so sharing one would let an
unsupported column drop take the table drop down with it. test_agent_removal
covers both branches plus the fresh-install and idempotent cases, and an
end-to-end run against a seeded pre-0.48 database confirms the table is gone and
every /api/agents route answers 404.

Docstrings that justified a design by "shared with the agent, which has no
database" were rewritten rather than left lying: update_service's persistence
callback and image_status_store are still the right split (registry logic stays
testable without a database), but for that reason now, not the old one. The
README's multi-host sections are removed and an upgrade note explains what to do
with running agent containers; ROADMAP keeps its history behind a note saying
the feature it describes no longer exists.

CI no longer builds or pushes stackpilot-agent.

735 tests pass, ruff and tsc clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
 2026-08-31 14:11:54 +02:00
menzeljandClaude Opus 5 09bed274eb Lock stacks during compose runs, cache stats, persist runtime state (0.47.0)
CI / check (push) Successful in 7m4s
Details
CI / build-and-push (push) Successful in 1m47s
Details 
F7 — Nothing stopped two compose operations landing on the same stack. There
was a busy flag, but is_busy() was only ever read to colour the status column;
no lifecycle handler consulted it before acting. Two tabs, or auto-update
picking up a stack somebody had just clicked, both ran pull + up -d against the
same project and raced over recreating containers.

Lifecycle calls, the two deploy WebSockets and the auto-update pass now take a
real lock; a second caller gets 409 (or an error frame and close 4409) and
auto-update skips and retries next cycle. The lock is a row rather than a set
in one worker's memory, so it holds across workers and across a restart, and it
carries an expiry — a worker killed mid-deploy would otherwise strand the stack
with no fix short of editing the database.

F10 — /api/stacks/stats sampled every running container on every call, one
blocking daemon request each, and both the dashboard and the stacks list poll
it every five seconds. Two tabs on a 40-container host meant a sustained ~16
samples a second. Cached for 4s behind a lock so concurrent callers share one
sweep, the same shape dashboard_service already used for its fleet aggregate.

F11 — Three module dicts assumed exactly one uvicorn worker without saying so
and were lost on restart. The busy set is the lock above. The image update
cache is now mirrored to SQLite, so a restart shows the badges immediately
instead of blanking them for up to an hour, and the already-notified marks come
back with them rather than re-announcing the same updates. The login rate
limiter is a table, so it cannot be cleared by getting the process to restart
and no longer multiplies by the worker count.

The constraint that shaped this: compose_service and update_service are shared
with the agent, which has no database. Neither may import one. So the lock is a
separate service the central app enforces at its own entry points, and update
persistence is an opt-in callback the central app registers in its lifespan —
the agent registers nothing and behaves exactly as before. A test asserts
update_service never imports the database, since that is the kind of thing a
later change breaks silently.

Both new nets were checked by reverting the fix: dropping the lock from
_lifecycle fails six tests, removing the stats cache fails the one that names
the behaviour.

Also wires up cache pruning in the same sweep — without it both the dict and
the table grew one entry per image tag ever run, for the life of the install.

31 new tests (729 total).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
 2026-08-31 13:43:39 +02:00
menzeljandClaude Opus 5 41a21b5a25 Make tokens revocable and move the refresh token out of localStorage (0.46.0)
CI / check (push) Successful in 7m7s
Details
CI / build-and-push (push) Successful in 1m44s
Details 
F5 — A token was valid until it expired, full stop. Resetting a compromised
account's password changed nothing for whoever held its tokens (up to 30 days
for a refresh token), demoting or disabling an account only took effect once
the same clock ran out, and logout was purely client-side.

Every account now has a token_version, every token is minted carrying it, and
every request compares the two. Bumping it is the revoke switch, pulled on the
three changes that alter what an account may do: password, role, active flag.
"Sign out everywhere" in the user menu bumps your own. Plain "Sign out" only
drops the cookie, because signing out on your phone should not kill your
desktop session.

The refresh token left localStorage for an httpOnly cookie (SameSite=Lax,
scoped to /api/auth), and the access token is now held in memory only. A
successful XSS can still act inside the open page but can no longer walk off
with 30 days of access. The cookie is marked Secure only when the request
arrived over HTTPS — request.url.scheme is trustworthy since the F4 fix — so a
plain-HTTP homelab keeps working. Any refresh token an older build left in
localStorage is deleted on first load. Scripted clients that cannot hold a
cookie can still ask for it in the body with ?in_body=true.

F9 comes with it, as predicted: the WebSocket helpers read the role off the
live user instead of the token's claim. /ws/exec is root-equivalent on the
host, and a token minted while the account was an admin stayed syntactically
valid after a demotion.

The sharp edge was the migration, not the feature. _ensure_model_columns emits
ADD COLUMN without a DEFAULT, so SQLite would have filled token_version with
NULL on every existing install, every version check would have failed against
it, and the upgrade would have locked out every user everywhere. The helper now
renders NOT NULL DEFAULT <literal> for scalar defaults; test_schema_migration
builds a genuinely old-shaped user table and asserts the backfill. The version
comparison also tolerates NULL as 1, so a database migrated by some other route
still works.

Writing that test surfaced an undocumented precondition: _ensure_model_columns
does nothing unless `models` has been imported, since SQLModel.metadata is
empty until then. It holds in production because init_db imports first; now it
says so.

The authorization matrix did its job — adding two auth routes failed the suite
until both were classified, which is exactly the review moment it exists for.

30 new tests (698 total). Upgrading signs everyone out once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
 2026-08-31 13:31:00 +02:00
menzeljandClaude Opus 5 60a7ccff93 Add a test suite, a linter and a CI gate in front of the build (0.45.0)
CI / check (push) Successful in 7m40s
Details
CI / build-and-push (push) Successful in 1m55s
Details 
The repo had no tests, no lint config, and a CI that went straight from push
to docker push. That is the reason F1 could ship: authorization lives in the
routers, each of 171 routes independently picks require_admin or
get_current_user, and nothing checked the choice was right.

670 tests, no Docker daemon needed. The app is driven through TestClient
without entering it as a context manager, which skips the lifespan — no
background loops, no socket — and conftest points DATA_DIR/STACKS_DIR at a
temp directory before anything is imported.

test_route_authorization.py is the load-bearing one. Rather than 171 implied
decisions it states the policy once — every route requires admin unless it is
listed in USER_READABLE or PUBLIC — and fails on any route that disagrees. A
new route defaults to admin, which is the safe direction; what it catches is a
route written with get_current_user that nobody weighed against "can this
return a credential". Writing the allowlist meant auditing all 53 user-readable
routes, which turned up one more leak: GET /api/templates/{id} returns a
template's env, and "save stack as template" snapshots the stack's real .env
into it. Now admin-only; the listing stays open.

test_agent_authorization.py pins the same invariant on the agent, where the
whole access model is one shared token declared per route and a single
forgotten Depends(verify_token) would hand over the host.

Both were checked by reintroducing the bug: re-opening /api/files/read fails
three tests with actionable messages, dropping a token guard fails two.

test_bundled_templates.py covers the 83 templates — parse, image per service,
.env.example in sync with what compose reads, every bind-mounted file actually
shipped, and no working default password. It found one on its first run:
authentik shipped PG_PASS=change-me and AUTHENTIK_SECRET_KEY=change-me against
a compose that marks both required, so the stack would have come up with a
known password instead of refusing to start. Fixed.

The rest ports the ad-hoc harnesses from 0.44.0 into permanent tests (crypto
round-trip incl. plaintext passthrough and key-loss handling, the browse
sandbox) and covers compose_service's slug/status/file handling and
secret_service's name validation.

ruff is configured as a floor, not a style bar: F, E9 and B only. Import
sorting is deliberately out — it is style, and enabling it would rewrite the
imports of nine files that have nothing else wrong. The 12 findings it did have
are fixed here (unused imports, an unused local, four raise-without-from that
were swallowing exception context).

CI now runs check (ruff, pytest, tsc) and only builds if it passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
 2026-08-31 13:16:41 +02:00
menzeljandClaude Opus 5 54c835b032 Close the read-side privilege escalation and fix proxy-aware IPs (0.44.0)
CI / build-and-push (push) Successful in 3m53s
Details 
F1 — Any authenticated user could read any file the backend could see.
/api/files/read and /download hung on get_current_user, and the sandbox that
should have caught that was open by default: ALLOWED_BROWSE_ROOTS contained
"/", for which _is_allowed() waves through every path. So the `user` role could
download stackpilot.db (password hashes, agent tokens, backup credentials),
every stack's .env and every .secrets/* file — with no audit trail, because
only mutations were logged.

Implementing that turned up three more doors into the same room, all fixed
here since closing only the first would have made the fix cosmetic:
GET /api/stacks/{id} handed the .env to any user, /export tarred the whole
stack dir including .secrets/*, and both the agent file proxies and
/api/agents/{id}/stacks/{id} repeated the leak for every remote host. All 24
filesystem-touching routes are now admin-only; reads and downloads are audited
(listing is not — the Files page polls it). DATA_DIR is refused outright, since
the API deliberately masks agent tokens and destination secrets and the browser
would otherwise be the way around that. "/" is out of the default browse roots.

F2 — Backup destination credentials were plaintext JSON in the DB, which is
what made F1 worth exploiting. They are now Fernet-encrypted at rest behind
parse_config/dump_config, with existing rows migrated at startup.

This needed a prerequisite from F6: the key is derived from SECRET_KEY, which
was regenerated on every boot when unset. Encrypting against a key that changes
per restart would be worse than plaintext, so an auto-generated SECRET_KEY is
now persisted to ${DATA_DIR}/secret_key at mode 0600. Sessions surviving a
restart is a welcome side effect.

F3 — /api/audit is admin-only. Also hidden from the dashboard and the nav for
non-admins, so nobody polls into a 403.

F4 — uvicorn now runs with --proxy-headers, so nginx's X-Forwarded-For is
honoured. Without it request.client.host was the frontend container's IP for
every request, which made the login rate limit global instead of per-IP (10
failures locked out everyone) and filled the audit log's IP column with one
useless value.

Verified: encrypt/decrypt round-trip incl. plaintext passthrough, idempotent
re-encryption and wrong-key handling; sandbox denial for DATA_DIR, traversal
into it, and paths outside the roots, with the allowed roots still reachable.
Both against stubbed settings — there is no Docker here, so nothing was run
end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
 2026-08-31 13:01:53 +02:00
menzeljandClaude Opus 5 b3af0c2109 Grow the bundled template library to 83 homelab apps (0.43.0)
CI / build-and-push (push) Successful in 1m46s
Details 
The library shipped five templates. This adds 78 more, covering what
homelab lists and the self-hosted community actually run: media servers
and the *arr automation chain, DNS ad-blocking, reverse proxies, VPN,
SSO, monitoring and dashboards, files/backup, notes and wikis, home
automation, dev tooling, databases, local AI, finance and notifications.

Every template follows the existing shape — compose.yaml, .env.example,
template.json — with PUID/PGID/TZ/DATA_PATH/HTTP_PORT knobs and no
literal secrets: anything that must be set uses ${VAR:?...} so deploy
fails loudly instead of coming up with a default password. Six ship the
extra config file their app needs (prometheus.yml, Caddyfile,
mosquitto.conf, frigate config.yml, Authelia's two files,
zigbee2mqtt configuration.yaml), which the folder-copy pull already
carries into the new stack.

All 95 referenced images were verified pullable against their
registries. Default host ports were deconflicted so several templates
can be pulled side by side; the only remaining overlaps are between
services you would never run together anyway (two DNS blockers on 53,
three reverse proxies on 80/443).

The Templates page would have been an unusable 83-card grid, so it now
has a search box and tag filter chips, with the tags on each card
clickable to filter by.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
 2026-08-31 09:12:34 +02:00
menzeljandClaude Opus 5 2afec08c4f Fix "Add variable" doing nothing, and size the editors to the viewport (0.42.2)
CI / build-and-push (push) Successful in 1m45s
Details 
The env table derived its rows from the serialized text on every render,
and serialize() drops rows with an empty key — so a freshly added blank
row was discarded before it could be typed into. The rows are now owned
by the component and re-parsed only when `value` changes from outside,
with stable per-row ids so deleting a row no longer shifts the reveal
state onto its neighbour.

AppShell's <main> is content-height, so the editor page's `h-full`
collapsed to auto: Monaco and the raw .env textarea fell back to their
intrinsic size, the textarea to a two-row default. The page is now sized
against the viewport minus the top bar and page padding, so both editors
fill the screen, and the textarea gets min-h-0 so flex-1 can grow it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pMmFFkdfxkoYjcEcpZTa5
 2026-08-31 00:49:40 +02:00
menzeljandClaude Opus 5 f6f82245f7 Keep the progress bar on one line, and portal modals out of the top bar (0.42.1)
CI / build-and-push (push) Successful in 1m48s
Details 
The stacks-list bar sat below the name and grew the row when an action
started. It now runs inline to the right of the name and service count,
filling the space before the CPU column, so the row keeps its height.
Label, percentage and byte detail sit on that same line.

Also fixes the self-update prompt being cut off at the top. The top bar
is backdrop-blurred, and a non-none backdrop-filter makes an element the
containing block for `position: fixed` descendants — so the dialog
centred itself in the 60px header instead of the viewport and overflowed
off-screen. ConfirmDialog and the update overlay now render through a
portal on document.body. ConfirmDialog also scrolls itself rather than
its backdrop, which would otherwise strand its top edge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pMmFFkdfxkoYjcEcpZTa5
 2026-08-31 00:43:10 +02:00
menzeljandClaude Opus 5 1e8d4248fd Stream update progress into a bar on the stack's row (0.42.0)
CI / build-and-push (push) Successful in 1m55s
Details 
Update ran as a blocking POST with nothing to show but a spinner, so the
status added in 86c67df could only sit above the table as a banner.

Adds /ws/update/{stack_id}, streaming `compose pull` then `up -d` with
--progress json, and feeds it through the existing DeployTracker — the
same weighting the deploy console uses. The result renders as a progress
bar inside the stack's own row: percentage, phase label, and layer/byte
detail. Non-streaming actions (start/stop/restart/pull/down) reuse the
bar in its indeterminate form, so every row action looks consistent.

compose_service gains _stream_phase, shared by stream_up and the new
stream_update; a failed pull short-circuits before `up`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pMmFFkdfxkoYjcEcpZTa5
 2026-08-31 00:33:34 +02:00
menzeljandClaude Opus 5 86c67dfcea Show action status on the stacks list and dashboard, not just detail (0.41.0)
CI / build-and-push (push) Successful in 1m56s
Details 
The status banner added in 9d28e12 only rendered on the stack detail
page, but Update is most often clicked from the stacks list — so in
practice the status was invisible. Render it on the stacks list and
dashboard too.

Actions on different stacks run concurrently from the list, so busy
state and status are now keyed by stack id instead of a single value:
previously the first action to finish cleared every row's spinner, and
each new action overwrote the previous one's status. StacksTable takes
an isBusy(id) predicate in place of the single busyId prop.

Also bumps the version so the newly version-tagged CI images (f8bfc91)
actually differ from the running release — self-update compares tags
against APP_VERSION, so shipping without a bump shows no update.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pMmFFkdfxkoYjcEcpZTa5
 2026-08-31 00:20:03 +02:00
menzeljandClaude Sonnet 5 f8bfc911f8 ci: also push a version-tagged image alongside :latest
CI / build-and-push (push) Successful in 29s
Details 
self_update_service compares registry version tags against APP_VERSION
to detect a newer release; with only :latest pushed, it always reported
"No version tags found" and the update pill never appeared. Read the
version from backend/version.py and push it as an extra tag for all
three images.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pMmFFkdfxkoYjcEcpZTa5
 2026-08-31 00:12:26 +02:00
menzeljandClaude Sonnet 5 9d28e12cd7 Add persistent action status banner and fix stacked toast overlap
CI / build-and-push (push) Successful in 1m54s
Details 
Stack actions (start/stop/pull/update/…) now surface a dismissible
status banner on the stack detail page instead of relying on the
transient top-right toast alone. Also enable toast expand mode so
multiple notifications no longer collapse behind each other.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pMmFFkdfxkoYjcEcpZTa5
 2026-08-30 20:04:10 +02:00
menzelj d81c48a5c0 ci: point images at git.menzel.center and add build-and-push workflow
CI / build-and-push (push) Successful in 4m5s
Details 
Image references still pointed at the old server (10.10.6.10:3020/menzelj);
menzelj was never a valid namespace there either, the actual account is
menzeljonas. Also adds .gitea/workflows/ci.yml to build and push
backend, frontend and agent on push to main.
 2026-08-25 08:40:18 +00:00
menzeljandClaude Opus 5 adfd77a983 Fix NFS uploads against root_squash exports (0.40.2) 
Uploading a backup extracted the tar straight into the NFS mount via
put_archive, and the daemon chowns every entry while extracting — an export
with root_squash refuses that ("failed to Lchown ... for UID 0, GID 0:
operation not permitted"), so the upload died with a docker 500 even though
plain writes to the share work (which is why the destination test passed).

The helper container now unpacks into its own filesystem and copies the file
into the mount with cat, which never chowns. Restores hit the same wall when a
volume or bind folder lives on a squashed mount, so import_path/import_volume
fall back to a copy-through-staging when (and only when) the failure is a chown
denial — local restores keep preserving ownership. NFS file names are validated
against the same safe charset as the subdir parts, since both are interpolated
into the helper's shell commands.

Verified against a real root_squash NFS export: test/upload/list/download/delete
round trip, byte-identical download, restore into an NFS-backed volume via the
fallback, and ownership still preserved (1000:1000, 0600) on a local volume.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 18:53:25 +00:00
menzeljandClaude Opus 5 6af02a1367 Default STACKS_HOST_DIR to /opt/stacks so host and container paths match 
The shipped default (./data/stacks) guarantees the mismatch that hid stack data
from the file browser, the editor and (before 0.40.0) from backups: compose
resolves ./config against the container path, so the daemon creates the data
directories at /opt/stacks/<stack>/... on the host regardless of where
STACKS_HOST_DIR points. Same change for the agent, plus the reasoning in
.env.example and the README config table. Images unchanged (0.40.1).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 18:35:22 +00:00
menzeljandClaude Opus 5 4c158e9407 Fix NFS backup destinations broken by the 0.40.0 refactor (0.40.1) 
_ensure_helper_image moved to stack_assets_service, but backup_destination_service
imports it lazily inside _nfs_run/_nfs_helper, so nothing failed at import time —
every NFS destination operation raised ImportError at runtime instead. The helper
is now a public ensure_helper_image() and the NFS helpers import it from its new
home. Verified: every services/ and routers/ module imports, and both NFS helper
paths run through to a Docker call instead of ImportError.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 18:29:10 +00:00
menzeljandClaude Opus 5 5347a36eaf Back up bind-mount data, not just the compose file (0.40.0) 
A stack's real state lives in its bind-mounted config directories, and those
were never captured: the backup only tarred the stack folder as this container
sees it. When STACKS_HOST_DIR differs from the container's STACKS_DIR, compose
resolves ./config against the container path and the daemon creates it at that
path on the *host* — invisible here, so the archive held little more than
compose.yaml and .env.

New services/stack_assets_service.py inventories a stack's data (bind sources
merged from container mounts + the compose file, named volumes) and does all
data I/O through a throwaway helper container, i.e. by host path, so unseen
directories are captured anyway. It also detects the host/container stacks-path
mismatch and reports it.

- manifest v2: full inventory, per-asset capture result, skip reasons (v1 still
  restores)
- NFS/CIFS-backed volumes are skipped by default and never wiped on restore
- deselected data inside the stack folder no longer sneaks in via compose/
- volume/bind archives stream through temp files instead of RAM
- restore preserves mode, ownership, mtime and symlinks, and writes bind folders
  back to their host paths (rewritten when the stack is renamed)
- backup dialog shows the inventory with sizes and per-item checkboxes; restore
  gained a "restore bind folders" toggle
- new GET /api/stacks/{id}/backup/inventory (+ agent + proxy), backup endpoints
  take include_binds/binds/volumes, restore takes restore_binds

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 18:20:19 +00:00
menzeljandClaude Opus 5 ecf780c5e6 Deploy console: real progress bar for image pulls (0.39.0) 
Compose is now run with `--progress json` (probed once, falls back to the
plain text stream on older compose/agents). The console folds the event
stream into a weighted progress bar — download bytes per layer, then
container create/start — with a per-image bar and a byte/layer counter,
and renders the raw output one line per layer (updated in place) instead
of a wall of scrolling text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-16 14:19:27 +00:00
menzeljandClaude Fable 5 9119f94536 Clear the stack update pill immediately after a manual/auto update (0.38.4) 
The amber image-update indicator is fed from update_service._CACHE, which
only the background loop refreshed — after a per-stack Update/Pull the
stale digests kept the pill on until the next pass. Now the local digests
are reconciled with the cached remote digests right after a successful
pull/update (local backend, agent lifecycle, auto-update pass), and the
frontend invalidates the stack-updates queries after actions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-04 17:52:31 +00:00
menzeljandClaude Opus 4.8 e651029ab2 db: auto-add missing model columns on startup (fix backupschedule.agent_id) (0.38.3) 
create_all never ALTERs an existing table, so installs predating the
backupschedule.agent_id column kept the old schema and any ORM query
naming it failed with "no such column" — which the new fleet dashboard
(and the schedules list / scheduler loop) hit. init_db now diffs each
mapped table against the live schema and ADD COLUMNs the missing
nullable/defaulted ones. Idempotent and self-healing for similar drift.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-24 11:43:17 +00:00
menzeljandClaude Opus 4.8 d399caadc9 Dashboard: surface the real compute_fleet error in the response (0.38.2) 
The fleet endpoint returned a bare 500, so the error banner only showed
"status code 500" with no cause. Wrap the call to log the full traceback
server-side and return the exception type, message and originating
file:line in the HTTP detail, so the dashboard banner pinpoints the
failure for an authenticated user.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-24 11:31:19 +00:00
menzeljandClaude Opus 4.8 0b95d7d4a2 Dashboard: surface fleet load errors instead of infinite skeletons (0.38.1) 
The cockpit cards gated purely on `fleet.data`, so any failed
/api/dashboard/fleet request (e.g. a stale backend returning 404, or a
500) left the new components stuck on skeletons forever. Render a clear
error banner with the API message and a Retry button when the query
errors with no data, so the actual cause is visible.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-24 11:17:32 +00:00
menzeljandClaude Opus 4.8 c830d28b65 Dashboard: rebuild into an operator cockpit (0.38.0) 
Replace the analytics-style dashboard (stack-health funnel, uptime %,
operations/day grid, AI pill) with an attention-driven fleet cockpit:

- New /api/dashboard/fleet endpoint: server-side fan-out across the local
  host and every agent into one payload — a prioritized "needs attention"
  list, headline KPIs, an honest stack-status breakdown and a per-host
  resource rollup. Each agent uses its own DB session so the fan-out is
  concurrency-safe; failures degrade to "offline" instead of stalling.
- New frontend: AttentionStrip, FleetKpiRow, StackStatusBar and
  HostResourceTable; Dashboard.tsx rewritten around them.
- Remove the funnel/summary endpoints, the uptime sampler loop and the
  ops-activity machinery; delete the now-unused chart components.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-24 11:04:57 +00:00
menzeljandClaude Opus 4.8 5c46e40866 0.37.7: surface folder-upload diagnostics (find why it does nothing) 
Folder upload still reported as doing nothing, and without browser access
the failure point is invisible. Make every outcome visible on-screen:

- onChange: if the folder picker returns 0 files, toast an error; otherwise
  toast "Starting folder upload: N file(s)…" so it's clear the upload fired
  (independent of the progress bar rendering).
- Per-file failures are no longer swallowed: capture the first error and
  show it in the result toast ("Uploaded X, Y failed — <path>: <reason>").

This pinpoints whether the picker returns nothing, the upload never starts,
or the requests fail (and why). Frontend-only; all 3 images pushed 0.37.7.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 21:30:46 +00:00
menzeljandClaude Opus 4.8 415ebb733a 0.37.6: remember the "Show hidden" toggle across reloads 
Uploaded dotfiles (.env) were persisting fine, but "Show hidden" is
component state that reset to off on every reload — so after refreshing,
hidden files disappeared from view and looked lost. Persist the toggle in
localStorage (sp.files.showHidden) so it survives reloads; combined with
0.37.5's auto-reveal, an uploaded .env now stays visible.

Frontend-only; all 3 images rebuilt+pushed 0.37.6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 21:26:03 +00:00
menzeljandClaude Opus 4.8 a40dd0de3e 0.37.5: overwrite prompt for file upload + auto-reveal uploaded hidden files 
The reported "upload doesn't work, file never appears" was two things, both
hit when uploading config files like .env:

1. Single-file upload used overwrite=false and dead-ended on "Already
   exists: .env — rename or remove the existing file first." with no way to
   replace the file. Now a conflict opens an Overwrite confirmation dialog
   (mirroring the copy/paste conflict flow) that retries with overwrite=true.
2. .env (and any dotfile) is hidden, so even a successful upload stayed
   invisible unless "Show hidden" was on. After an upload whose name/path
   has a dot-segment, "Show hidden" is now auto-enabled so the file shows.

The single-file upload mutation now takes {file, overwrite}; folder upload
(already overwrite=true) also auto-reveals hidden results.

Frontend-only; all 3 images rebuilt+pushed 0.37.5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 21:20:15 +00:00
menzeljandClaude Opus 4.8 98d756faf6 0.37.4: fix folder upload doing nothing (set webkitdirectory reliably) 
"Upload folder" silently did nothing: the directory-selection attribute
was set on the hidden <input> via a JSX spread
({...{webkitdirectory:"", directory:""}}), which React doesn't reliably
apply to the DOM — and if isAdmin resolves after first render, a one-shot
effect would miss the input mounting entirely. Without the attribute the
picker is a plain file picker where no folder can be selected, so the user
picks nothing and nothing happens.

- Set webkitdirectory/directory/mozdirectory imperatively through a
  callback ref, which runs whenever the input mounts. folderInput is now a
  MutableRefObject so the callback can populate it.
- Folder upload now shows the progress bar immediately on start (small
  files can finish before the browser emits any upload-progress event, so
  don't wait for the first one to render feedback).

Frontend-only; all 3 images rebuilt+pushed 0.37.4.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 20:52:50 +00:00
menzeljandClaude Opus 4.8 a43e6b48f0 0.37.3: byte-accurate upload progress + file counter for folder uploads 
Reviewed the upload path. Single-file and folder uploads already drove the
progress bar, but folder progress was file-COUNT based ((i + filePct)/total),
which jumps around when a folder mixes tiny files with large ones and gives
no sense of total size.

- Folder upload: progress is now byte-weighted (sum of all file sizes), so
  the bar tracks real transfer. Added a detail line "<i> / <n> files ·
  <sent> / <total>" and the bar shows the current file name.
- Single file: added the same byte detail ("<sent> / <size>").
- Progress component gained an optional detail sub-line (shared by the
  download bar too).

Frontend-only; all 3 images rebuilt+pushed 0.37.3 for tag consistency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 20:41:42 +00:00
menzeljandClaude Opus 4.8 5ac9f15de4 0.37.2: stream folder zip-downloads to fix 504 on large folders 
A big folder hit a 504 Gateway Timeout: the zip was built into a temp
file *before* any response was sent, so for large folders the backend
stayed silent past nginx's proxy_read_timeout.

Now the zip is streamed as it's built, end to end:
- file_service.open_archive() returns (filename, byte iterator); _iter_zip
  walks the dir and yields zip bytes incrementally via a small drain
  buffer, writing each file in 1 MiB chunks (bounded memory, valid CRCs).
  Same hardening as before — only real regular files; FIFOs/sockets/
  devices/symlinks skipped without open(); per-file read errors skipped.
- /api/files/download and /agent/files/download return a StreamingResponse
  (no temp file). The agent proxy streams the agent response straight
  through (agent_service.stream_download), pulling the first chunk eagerly
  so an offline/bad-token agent still yields a clean status before 200.
- Files page: streamed downloads have no Content-Length, so the progress
  bar shows the running downloaded byte count ("Downloading … 12.3 MB")
  instead of a percentage, after the initial "Preparing …".

Verified end to end via TestClient (200, application/zip, valid zip,
2 MiB file intact, FIFO skipped, no hang).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 20:28:49 +00:00
menzeljandClaude Opus 4.8 0dc430bb2a 0.37.1: fix folder-download hang/501 on special files + add download progress 
Two issues with the 0.37.0 folder zip-download:

1. Hang / server error (reported as 501) on "some folders". archive_dir
   tried to zip every entry, including non-regular files. Opening a FIFO
   blocks forever (no writer); a unix socket / unreadable file raised an
   OSError that aborted the whole archive. Now only real regular files are
   zipped — FIFOs, sockets, devices and symlinks are skipped without ever
   open()-ing them, and a per-file read error skips just that file instead
   of failing the download.

2. No feedback while a large folder is being prepared. The zip is built
   server-side before any bytes flow, so the click felt dead. The Files
   page now shows an indeterminate "Preparing <name>…" bar from click,
   switching to a real percentage during the transfer (Content-Length is
   known for the finished zip). filesApi.download forwards onDownloadProgress.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 20:11:50 +00:00
menzeljandClaude Opus 4.8 f1782eca0e 0.37.0: download whole folders (recursive) as a .zip from the file browser 
The file browser/editor could only download individual files. Add a
recursive directory download that streams the folder as a zip archive,
on the local host and on every remote agent.

- file_service.archive_dir(): zip a directory recursively into a temp
  file, preserving the folder name as the archive root and empty
  subdirectories; symlinks are skipped (no sandbox escape / loops).
- /api/files/download and /agent/files/download branch on directories
  and return application/zip, cleaning up the temp file afterwards.
- Files page: show the download button for folders too (as <name>.zip).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 19:54:50 +00:00
menzeljandClaude Opus 4.8 5bcec06bbd 0.36.1: app logo as browser-tab favicon 
Add frontend/public/favicon.svg (the TopNav LogoMark glyph as a
standalone SVG) and link it from index.html so the StackPilot logo
shows in browser tabs. Vite copies public/ into dist on build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-16 21:24:07 +00:00
menzeljandClaude Opus 4.8 cd15cdc75e 0.36.0: per-stack image-update indicator on the Stacks overview 
Shows an amber "Update" pill next to a stack's status (and highlights the
inline Update button) when any of the stack's images has a newer digest in
the registry. Reuses the existing background image-update check — a new
update_service.stacks_update_summary() reads the cached digests in a single
container sweep (no extra registry calls), exposed as GET /api/stacks/updates
and proxied per agent at GET /api/agents/{id}/stacks/updates. The Stacks page
and each remote-host section poll it every 60s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-16 18:52:51 +00:00
menzeljandClaude Opus 4.8 a4b1bbcdd1 0.35.0: per-stack Update button on the Stacks page 
Adds an inline "Update (pull latest images & recreate)" action to each
row of the stacks table, next to start/stop/restart/edit — for both the
local host and remote agents. Wires the existing updateImages action and
the agent "update" lifecycle action through StacksTable's new onUpdate prop.

Also bumps backend/version.py to 0.35.0 so it tracks the frontend version
again (it had drifted to 0.33.0 while package.json moved to 0.34.x).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-16 18:44:21 +00:00
menzeljandClaude Opus 4.8 844655d1c8 0.34.1: populate the log container filter (parse compose prefix) 
The whole-stack log stream sends service:null on every line, so the
container filter dropdown only ever showed "All containers". docker compose
logs already prefixes each line with the container name (and an RFC3339
timestamp via --timestamps); parse that prefix client-side to recover the
container, populate the filter, and render time + container + message
separately (cleaner than the raw prefixed line).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-16 07:58:26 +00:00
menzeljandClaude Opus 4.8 efb468560e 0.34.0: stack log viewer — fixed height, container filter, severity coloring 
Frontend-only release. Overhauls the stack Logs tab:
- Fix the log panel growing down the page (AppShell <main> has no definite
  height, so the page h-full/flex-1 chain collapsed to auto): the scroll
  area now uses a fixed h-[65vh] instead of flex-1.
- Filter by container (service <select>) plus a free-text search; the line
  count shows filtered / total.
- Dozzle-style per-line severity coloring (error/warn/debug via regex).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-16 07:49:57 +00:00
menzeljandClaude Fable 5 786c346c40 0.33.0: NFS share as backup destination 
New destination type 'nfs' alongside SFTP and S3. The StackPilot container
needs no mount privileges: the Docker daemon mounts the export as a named
volume (stackpilot-nfs-dest-<id>, driver local/type nfs, recreated whenever
server/path/options change) and all file I/O runs through throwaway helper
containers (BACKUP_HELPER_IMAGE) — upload via put_archive, list via stat,
download via get_archive, delete/test via short-lived runs. Config: server,
export path, mount options (default rw), optional subdirectory (sanitized;
shell-safe charset). Mount failures surface as clean destination errors.

Settings UI gains the NFS form + summary; works everywhere destinations are
used (push, restore-from, scheduled backups incl. retention).

Verified live against a real kernel NFS server: test, push (file on the
export), list, restore-from incl. volume data, remote delete, config change
recreates the mount volume, unreachable server fails cleanly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-12 12:39:35 +00:00
menzeljandClaude Fable 5 79d82361d8 0.32.1: backup/restore fixes (audit findings, all paths live-verified) 
- backup_filename() crashed with NameError (bare now()) since 0.8.0 —
  broke every scheduled backup at the upload step, agent backup download
  and the central remote-backup/push endpoints. The local manual path
  worked only because the router had its own copy (now an alias).
- restore: the manifest stack_id from an uploaded backup is now slugified
  too — a crafted '../../...' id could previously escape STACKS_DIR.
- create_backup no longer starts a previously-stopped stack (stop/restart
  only when the stack was actually running).
- overwrite-restore wipes the existing volume contents before extracting,
  so files created since the backup no longer survive underneath it.

Verified end-to-end: full/config backup contents (compose, .env, .secrets,
bind dirs, extras, volume tars), delete→restore round-trip incl. volume
data, rename restore with volume re-prefixing, 409 conflict + overwrite,
traversal guard, scheduled run + retention prune + restore-from against
real MinIO, and the complete remote-agent cycle (download/push/restore).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-12 12:10:45 +00:00
menzeljandClaude Fable 5 a0dda120f5 0.32.0: StackPilot self-update (check on page load + one-click update) 
- backend/version.py is now the single version source (main.py, agent).
- GET /api/system/update: reads the version tags of the backend's own image
  repo (anonymous v2 token flow, https→http fallback for insecure
  registries), compares the highest semver tag against APP_VERSION; reports
  update_supported from the container's compose labels. 10 min cache.
- POST /api/system/update (admin, audited): spawns a detached helper
  container from the current backend image that runs docker compose pull &&
  up -d on StackPilot's own compose project (project name, working dir and
  config files resolved from its own container labels) — the helper
  outlives the backend being recreated. Non-compose installs get a 400.
- /api/health now returns the version so the UI can detect the switchover.
- TopNav version badge: queries the update status on page load; when a
  newer release exists an amber pill shows the version — one click (admin)
  confirms, triggers the update and overlays a wait screen that polls
  /api/health and reloads once the new version answers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-12 09:01:08 +00:00
menzeljandClaude Fable 5 11effdc2ca 0.31.1: make dashboard metrics honest 
- Container card now compares compose-only counts across hosts: agents
  report compose_running in /agent/system (pre-0.31.1 agents fall back to
  the all-containers number); card retitled, ResourceBar stat labelled
  'Containers (all)'.
- Uptime is sampled every 5 min (background loop + opportunistic on read)
  and charted as daily averages instead of a once-a-day snapshot; no
  sample is written when no compose containers exist (was: fake 100%).
  Legacy daily entries in uptime.jsonl still count; file pruned at startup.
- Funnel stage 'monitored' is now per-stack and real: stacks with an
  enabled local auto-update policy (was: global webhook-exists toggle).
  Frontend label renamed to 'Auto-managed'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-12 08:13:16 +00:00
menzeljandClaude Fable 5 1609b8bcc3 Phase 25: templates as stack folders (0.31.0) 
Templates are now stack-shaped folders (compose.yaml + .env.example +
template.json) instead of DB rows + manifest.json + {{VAR}} rendering.
Pull copies the folder into a new stack; custom templates persist under
DATA_DIR/templates. Adds POST /api/templates/from-stack and a one-time
startup migration for pre-0.31 DB templates (drops the template table).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-12 07:29:54 +00:00
431 changed files with 22183 additions and 6353 deletions
Expand all files Collapse all files
.env.example
+11 -4
View File
@@ -3,8 +3,13 @@
SECRET_KEY=change-me-to-a-long-random-string
# Host directory where stack folders (compose.yaml + .env) are stored.
# This MUST be the same path on the host and is bind-mounted into the backend.
STACKS_HOST_DIR=./data/stacks
# It MUST be the same path as STACKS_DIR inside the container (/opt/stacks):
# compose runs in the backend container and resolves a stack's relative bind
# mounts (./config) against the *container* path, so the daemon creates those
# data directories at that path on the host. With a different host path here,
# every stack's data lands outside StackPilot's view — the file browser and the
# editor won't see it (backups capture it either way, via a helper container).
STACKS_HOST_DIR=/opt/stacks
# Allowed CORS origin(s) for the API (comma separated). The bundled frontend
# proxies /api, so this only matters if you call the API from another origin.
@@ -18,11 +23,13 @@ NOTIFY_WEBHOOKS=
# Throwaway image used to read/write named-volume contents during backups.
BACKUP_HELPER_IMAGE=alpine:latest
# File browser (sidebar) + volume host-path picker.
# File browser (sidebar) + volume host-path picker. Admin-only.
# ALLOWED_BROWSE_ROOTS: comma-separated paths the browser may reach (sandbox).
# A single "/" in this list disables the sandbox -- it makes every path
# allowed. StackPilot's own DATA_DIR is refused regardless of this setting.
# HOST_ROOT_PREFIX: where the host filesystem is mounted inside the backend
# container. Leave empty to browse the container's own filesystem. To browse
# the real host, uncomment the "/:/host_root" volume in docker-compose.yml and
# set HOST_ROOT_PREFIX=/host_root here (mount without :ro to allow edits).
ALLOWED_BROWSE_ROOTS=/,/mnt,/media,/srv,/opt
ALLOWED_BROWSE_ROOTS=/mnt,/media,/srv,/opt,/home
HOST_ROOT_PREFIX=
.gitea/workflows/ci.yml
+110
View File
@@ -0,0 +1,110 @@
# Continuous integration on git.menzel.center (Gitea Actions).
#
# Two jobs: `check` runs both test suites (pytest, vitest), the linter and the
# frontend typecheck; `build-and-push` only starts once `check` is green, so a
# red suite never reaches the registry (and never reaches the self-update
# checker, which would happily offer a broken release).
#
# Builds and pushes both images to this instance's container registry on every
# push to main: backend and frontend. Each image gets both a ":latest" tag and
# a ":{APP_VERSION}" tag, the latter read
# from backend/version.py (the single source of truth for the release
# version) - self_update_service compares registry version *tags* against the
# running APP_VERSION to decide whether an update is available, so without a
# version tag it would never see one, no matter how far behind :latest is.
#
# Runs on ubuntu-latest, not the docker label: that label's image is a bare
# docker:24-dind with no Node/bash, which breaks actions/checkout (a JS
# action). ubuntu-latest has both a shell and the Docker CLI, talking to the
# host daemon through the socket the runner passes in.
name: CI
on:
push:
branches: [main]
env:
REGISTRY: git.menzel.center/menzeljonas
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.CI_TOKEN }}
- uses: actions/setup-python@v5
with:
python-version: "3.12" # matches backend/Dockerfile
- name: Install backend + test dependencies
working-directory: backend
run: pip install -r requirements-dev.txt
- name: Lint (ruff)
working-directory: backend
run: ruff check .
# The suite runs without a Docker daemon on purpose: it drives the app
# through TestClient without the lifespan, so no background loops and no
# socket. See backend/tests/conftest.py.
- name: Test (pytest)
working-directory: backend
run: pytest
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: npm
cache-dependency-path: frontend/package-lock.json
- name: Install frontend dependencies
working-directory: frontend
run: npm ci
- name: Typecheck (tsc)
working-directory: frontend
run: npx tsc --noEmit -p tsconfig.json
- name: Test (vitest)
working-directory: frontend
run: npm test
build-and-push:
needs: check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.CI_TOKEN }}
- name: Read app version
id: version
run: |
VERSION=$(grep -oP '(?<=APP_VERSION = ")[^"]+' backend/version.py)
echo "Building version $VERSION"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
- name: Log in to the registry
run: |
echo "${{ secrets.CI_TOKEN }}" | docker login git.menzel.center -u menzeljonas --password-stdin
- name: Build and push backend
run: |
docker build \
-t "$REGISTRY/stackpilot-backend:latest" \
-t "$REGISTRY/stackpilot-backend:${{ steps.version.outputs.version }}" \
./backend
docker push "$REGISTRY/stackpilot-backend:latest"
docker push "$REGISTRY/stackpilot-backend:${{ steps.version.outputs.version }}"
- name: Build and push frontend
run: |
docker build \
-t "$REGISTRY/stackpilot-frontend:latest" \
-t "$REGISTRY/stackpilot-frontend:${{ steps.version.outputs.version }}" \
./frontend
docker push "$REGISTRY/stackpilot-frontend:latest"
docker push "$REGISTRY/stackpilot-frontend:${{ steps.version.outputs.version }}"
README.md
+683 -162
View File
File diff suppressed because it is too large Load Diff
ROADMAP.md
+25
View File
@@ -1,5 +1,12 @@
# StackPilot Roadmap — Phases 21–23
> **Historical record.** Phases below describe work as it shipped at the time.
> The multi-host / agent integration they refer to was removed in 0.48.0 —
> StackPilot manages a single Docker host. Anything here mentioning
> `agent_app.py`, `AGENT_TOKEN`, `/api/agents/*` or `/ws/agent-*` no longer
> exists; the entries are kept because they record what was actually done, not
> what is currently true.
Planned 2026-06-09. Status keys: ☐ not started · ◐ in progress · ☑ done.
Each phase ships independently following the standing release checklist
(bump `backend/main.py` + `backend/agent_app.py` AGENT_VERSION +
@@ -208,6 +215,24 @@ RemoteStackDetail** (not the new-stack editor, which has no dir yet).
---
## Phase 25 — Templates as stack folders ☑ DONE — shipped 0.31.0
Templates reworked from DB rows + `manifest.json` + `{{VAR}}` mustache rendering
to **stack-shaped folders**: `backend/templates/<slug>/` with `compose.yaml`,
optional `.env.example` and a `template.json` (name/description/tags/gpu).
"Pull" copies the whole folder into a new stack (`.env.example` → `.env`);
custom templates live under `${DATA_DIR}/templates/` and are written by
"Save as template" (stack detail) or the manual save endpoint.
- Backend: `template_service` rewritten (folder scan, traversal-guarded resolve,
`copy_into_stack`, `save_from_stack`); `Template` DB table dropped with a
one-time startup migration (`{{VAR}}` → `${VAR}`, vars → `.env.example`).
- API: `POST /api/templates/from-stack` new; instantiate no longer takes `values`.
- Frontend: Templates page shows compose/env preview + file list, delete for
custom templates; StackDetail gains "Save as template".
---
## After 23
Remaining un-built ideas from the gap analysis (not chosen this round):
Health-monitoring & alerting (Docker-events → notify; note `/ws/events` already
agent/.env.example
-7
View File
@@ -1,7 +0,0 @@
# Shared secret the central StackPilot must present to manage this host.
# Generate with: openssl rand -base64 32
# Enter the SAME value when adding this host under Settings → Remote hosts.
AGENT_TOKEN=change-me-to-a-long-random-shared-secret
# Host directory where this host's stack folders live.
STACKS_HOST_DIR=./data/stacks
agent/Dockerfile
-14
View File
@@ -1,14 +0,0 @@
# The agent reuses the backend image (same compose/Docker code + deps) and
# just runs a different ASGI app. Build the backend image first.
ARG BACKEND_IMAGE=10.10.6.10:3020/menzelj/stackpilot-backend:latest
FROM ${BACKEND_IMAGE}
ENV STACKS_DIR=/opt/stacks \
PORT=5010
EXPOSE 5010
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s \
CMD curl -fsS http://localhost:5010/agent/health || exit 1
CMD ["uvicorn", "agent_app:app", "--host", "0.0.0.0", "--port", "5010"]
agent/docker-compose.yml
-23
View File
@@ -1,23 +0,0 @@
# StackPilot agent — deploy this on each remote host you want to manage.
# It needs only the Docker socket and a shared AGENT_TOKEN (must match the
# token you enter when adding this host in the central StackPilot UI).
services:
agent:
image: 10.10.6.10:3020/menzelj/stackpilot-agent:latest
build:
context: .
args:
BACKEND_IMAGE: 10.10.6.10:3020/menzelj/stackpilot-backend:latest
restart: unless-stopped
environment:
- AGENT_TOKEN=${AGENT_TOKEN:?set AGENT_TOKEN in .env}
- STACKS_DIR=/opt/stacks
- HOST_PROC_PATH=/host_proc
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ${STACKS_HOST_DIR:-./data/stacks}:/opt/stacks
- /proc:/host_proc:ro
# Read-only host devices for status/detection parity with the main host.
- /dev:/dev:ro
ports:
- "5010:5010"
backend/.dockerignore
+7
View File
@@ -4,3 +4,10 @@ __pycache__
data
*.db
.env
# Test + lint tooling: run in CI, never needed in the runtime image.
tests
pyproject.toml
requirements-dev.txt
.pytest_cache
.ruff_cache
backend/Dockerfile
+15 -2
View File
@@ -1,8 +1,10 @@
FROM python:3.12-slim
# Docker CLI + compose plugin are required for lifecycle commands.
# git + openssh-client are required for deploying stacks from a Git repository
# (services/git_service.py); ssh only for repositories reached over SSH.
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl gnupg \
&& apt-get install -y --no-install-recommends ca-certificates curl gnupg git openssh-client \
&& install -m 0755 -d /etc/apt/keyrings \
&& curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc \
&& chmod a+r /etc/apt/keyrings/docker.asc \
@@ -27,4 +29,15 @@ EXPOSE 5008
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s \
CMD curl -fsS http://localhost:5008/api/health || exit 1
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "5008"]
# --proxy-headers: nginx setzt X-Forwarded-For, ohne dieses Flag ignoriert
# uvicorn den Header und request.client.host ist fuer JEDE Anfrage die IP des
# Frontend-Containers -- was das Login-Rate-Limit global statt pro IP wirken
# laesst und die IP-Spalte im Audit-Log wertlos macht.
#
# forwarded-allow-ips=* vertraut dem Header von jedem Absender. Das ist hier
# richtig, weil der Backend-Port nur im Docker-Netz erreichbar ist (siehe
# "expose" statt "ports" in docker-compose.yml). Wer 5008 direkt nach aussen
# gibt, muss den Wert auf die IP des eigenen Proxys einschraenken -- sonst
# kann ein Client seine eigene Herkunfts-IP faelschen.
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "5008", \
"--proxy-headers", "--forwarded-allow-ips", "*"]
backend/agent_app.py
-813
View File
@@ -1,813 +0,0 @@
"""StackPilot agent — a slim, token-guarded Docker Compose API for one host.
The agent runs on each remote host (same image as the backend, different CMD).
It has no users, no database and no UI: it exposes just enough of the stack /
system surface for a central StackPilot to manage this host's compose stacks,
authenticated by a single shared bearer token (``AGENT_TOKEN``).
All compose/Docker logic is reused from the backend's ``compose_service`` and
``docker_client`` so behaviour matches the local host exactly.
"""
from __future__ import annotations
import logging
import os
import shutil
from dataclasses import asdict
import tempfile
import json
from fastapi import (
Depends,
FastAPI,
File,
Form,
Header,
HTTPException,
Query,
Request,
UploadFile,
WebSocket,
WebSocketDisconnect,
)
from fastapi.responses import FileResponse, JSONResponse
from pydantic import BaseModel
from config import settings
from docker_client import DockerError, get_client, safe_call
from services import (
backup_service,
compose_edit_service,
compose_service,
container_service,
device_service,
exec_service,
file_service,
image_service,
network_service,
secret_service,
stats_service,
update_service,
volume_service,
)
logger = logging.getLogger("stackpilot.agent")
# Map network_service's DockerError codes to HTTP status. forbidden is mapped to
# 400 (not 403) so the central proxy doesn't misread it as a token failure.
_DOCKER_STATUS = {"invalid_request": 400, "forbidden": 400, "not_found": 404}
def _map_docker(exc: DockerError):
code = _DOCKER_STATUS.get(exc.error)
if code:
raise HTTPException(status_code=code, detail=exc.detail or exc.error)
raise exc # falls through to the global 502 DockerError handler
AGENT_VERSION = "0.30.0"
# --------------------------------------------------------------------------- #
# Auth
# --------------------------------------------------------------------------- #
def verify_token(authorization: str = Header(default="")) -> None:
expected = settings.AGENT_TOKEN
if not expected:
raise HTTPException(status_code=503, detail="Agent token not configured")
if authorization != f"Bearer {expected}":
raise HTTPException(status_code=401, detail="Invalid agent token")
# --------------------------------------------------------------------------- #
# Schemas
# --------------------------------------------------------------------------- #
class StackBody(BaseModel):
name: str | None = None
yaml: str | None = None
env: str | None = None
class NetworkCreateBody(BaseModel):
name: str
driver: str = "bridge"
subnet: str | None = None
gateway: str | None = None
internal: bool = False
attachable: bool = True
class ContainerRefBody(BaseModel):
container: str
aliases: list[str] | None = None
force: bool = False
class FileWriteBody(BaseModel):
path: str
content: str
class FileNameBody(BaseModel):
path: str
name: str
class FileRenameBody(BaseModel):
path: str
new_name: str
class FileTransferBody(BaseModel):
src: str
dest_dir: str
overwrite: bool = False
def _file_guard(fn, *args, **kwargs):
try:
return fn(*args, **kwargs)
except file_service.BrowseError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
# --------------------------------------------------------------------------- #
# Helpers
# --------------------------------------------------------------------------- #
def _summary(stack_id: str, summaries: dict | None = None) -> dict:
if summaries is None:
try:
containers = compose_service.containers_for_stack(stack_id)
total = len(containers)
running = sum(1 for c in containers if c.state == "running")
status = compose_service.compute_status(stack_id, containers)
except DockerError:
total = running = 0
status = "unknown"
else:
info = summaries.get(stack_id)
total = info["total"] if info else 0
running = info["running"] if info else 0
if compose_service.is_busy(stack_id):
status = "updating"
else:
status = info["status"] if info else "stopped"
return {
"id": stack_id,
"name": stack_id,
"description": None,
"status": status,
"service_count": total,
"running_count": running,
"created_at": None,
"updated_at": None,
}
def _hostname() -> str:
return os.uname().nodename
def _mem_info() -> tuple[int, int]:
"""Return (total_bytes, used_bytes) from meminfo (used = total - available)."""
for base in (settings.HOST_PROC_PATH, "/proc"):
try:
vals: dict[str, int] = {}
with open(os.path.join(base, "meminfo"), "r", encoding="utf-8") as fh:
for line in fh:
parts = line.split(":")
if len(parts) == 2 and parts[0] in ("MemTotal", "MemAvailable", "MemFree"):
try:
vals[parts[0]] = int(parts[1].split()[0]) * 1024 # kB -> bytes
except ValueError:
pass
total = vals.get("MemTotal", 0)
available = vals.get("MemAvailable", vals.get("MemFree", 0))
return total, max(total - available, 0)
except OSError:
continue
return 0, 0
def _disk_info() -> tuple[int, int]:
"""Return (total_bytes, used_bytes) for the host disk backing the stacks dir."""
for path in (settings.STACKS_DIR, "/"):
try:
usage = shutil.disk_usage(path)
return usage.total, usage.used
except OSError:
continue
return 0, 0
def _system_info() -> dict:
docker_version = ""
host_os = ""
running = total = 0
try:
client = get_client()
docker_version = safe_call(client.version).get("Version", "")
info = safe_call(client.info)
host_os = info.get("OperatingSystem", "")
running = info.get("ContainersRunning", 0)
total = info.get("Containers", 0)
except DockerError as exc:
docker_version = f"unavailable ({exc.error})"
mem_total, mem_used = _mem_info()
disk_total, disk_used = _disk_info()
return {
"hostname": _hostname(),
"docker_version": docker_version,
"host_os": host_os,
"cpu_cores": os.cpu_count() or 0,
"mem_total": mem_total,
"mem_used": mem_used,
"disk_total": disk_total,
"disk_used": disk_used,
"containers_running": running,
"containers_total": total,
}
# --------------------------------------------------------------------------- #
# App
# --------------------------------------------------------------------------- #
app = FastAPI(title="StackPilot Agent", version=AGENT_VERSION)
@app.exception_handler(DockerError)
async def _docker_error(_request: Request, exc: DockerError):
return JSONResponse(status_code=502, content={"error": exc.error, "detail": exc.detail})
@app.get("/agent/ping", dependencies=[Depends(verify_token)])
def ping() -> dict:
return {"ok": True, "hostname": _hostname(), "version": AGENT_VERSION}
@app.get("/agent/system", dependencies=[Depends(verify_token)])
def system() -> dict:
return _system_info()
@app.get("/agent/stacks", dependencies=[Depends(verify_token)])
def list_stacks() -> list[dict]:
try:
summaries = compose_service.stack_status_summaries()
except DockerError:
summaries = {}
return [_summary(sid, summaries) for sid in compose_service.discover_stacks()]
@app.get("/agent/stacks/stats", dependencies=[Depends(verify_token)])
def stacks_stats() -> dict:
return stats_service.stack_stats()
@app.get("/agent/stacks/{stack_id}", dependencies=[Depends(verify_token)])
def get_stack(stack_id: str) -> dict:
directory = compose_service.stack_dir(stack_id)
if not os.path.isdir(directory):
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
try:
raw = compose_service.containers_for_stack(stack_id)
containers = [asdict(c) for c in raw]
status = compose_service.compute_status(stack_id, raw)
except DockerError:
containers = []
status = "unknown"
return {
"id": stack_id,
"name": stack_id,
"description": None,
"status": status,
"yaml": compose_service.read_compose(stack_id),
"env": compose_service.read_env(stack_id),
"containers": containers,
"created_at": None,
"updated_at": None,
}
@app.post("/agent/stacks", dependencies=[Depends(verify_token)], status_code=201)
def create_stack(body: StackBody) -> dict:
if not body.name:
raise HTTPException(status_code=400, detail="name is required")
stack_id = compose_service.slugify(body.name)
if os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=409, detail=f"Stack '{stack_id}' already exists")
compose_service.write_compose(stack_id, body.yaml or "services:\n")
if body.env:
compose_service.write_env(stack_id, body.env)
return _summary(stack_id)
@app.put("/agent/stacks/{stack_id}", dependencies=[Depends(verify_token)])
def update_stack(stack_id: str, body: StackBody) -> dict:
if not os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
if body.yaml is not None:
compose_service.write_compose(stack_id, body.yaml)
if body.env is not None:
compose_service.write_env(stack_id, body.env)
return _summary(stack_id)
@app.delete("/agent/stacks/{stack_id}", dependencies=[Depends(verify_token)])
async def delete_stack(stack_id: str, delete_files: bool = Query(True)) -> dict:
if not os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
try:
await compose_service.down(stack_id)
except Exception: # noqa: BLE001 - best-effort teardown
pass
if delete_files:
compose_service.delete_stack_files(stack_id)
return {"ok": True}
_ACTIONS = {
"start": compose_service.up,
"stop": compose_service.stop,
"restart": compose_service.restart,
"pull": compose_service.pull,
"update": compose_service.update,
"down": compose_service.down,
}
@app.post("/agent/stacks/{stack_id}/{action}", dependencies=[Depends(verify_token)])
async def lifecycle(stack_id: str, action: str) -> dict:
fn = _ACTIONS.get(action)
if not fn:
raise HTTPException(status_code=400, detail=f"Unknown action '{action}'")
if not os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
result = await fn(stack_id)
if result.get("returncode") not in (0, None):
raise HTTPException(
status_code=500,
detail={
"error": f"compose {action} failed",
"detail": result.get("stderr", "").strip()[-2000:],
},
)
return result
@app.get("/agent/stacks/{stack_id}/logs", dependencies=[Depends(verify_token)])
async def stack_logs(stack_id: str, tail: int = Query(200, le=2000)) -> dict:
if not os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
result = await compose_service.logs(stack_id, tail=tail)
return {"logs": result.get("stdout", "") + result.get("stderr", "")}
@app.get("/agent/stacks/{stack_id}/updates", dependencies=[Depends(verify_token)])
async def stack_updates(stack_id: str, refresh: bool = Query(True)) -> dict:
"""Update status for this stack's images (used by central auto-update)."""
return await update_service.stack_updates(stack_id, refresh=refresh)
# --------------------------------------------------------------------------- #
# Secrets & configs (per-stack, file-based)
# --------------------------------------------------------------------------- #
class SecretWriteBody(BaseModel):
kind: str = "secret"
name: str
content: str
class SecretAttachBody(BaseModel):
kind: str = "secret"
name: str
service: str
target: str | None = None
class SecretDetachBody(BaseModel):
kind: str = "secret"
name: str
service: str
def _ensure_stack(stack_id: str) -> None:
if not os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
def _secret_guard(fn, *args, **kwargs):
try:
return fn(*args, **kwargs)
except (secret_service.SecretError, compose_edit_service.EditError) as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@app.get("/agent/stacks/{stack_id}/secrets", dependencies=[Depends(verify_token)])
def agent_list_secrets(stack_id: str) -> list:
_ensure_stack(stack_id)
return secret_service.list_all(stack_id)
@app.put("/agent/stacks/{stack_id}/secrets", dependencies=[Depends(verify_token)])
def agent_write_secret(stack_id: str, body: SecretWriteBody) -> dict:
_ensure_stack(stack_id)
return _secret_guard(secret_service.write_secret, stack_id, body.kind, body.name, body.content)
@app.delete("/agent/stacks/{stack_id}/secrets/{kind}/{name}", dependencies=[Depends(verify_token)])
def agent_delete_secret(stack_id: str, kind: str, name: str) -> dict:
_ensure_stack(stack_id)
_secret_guard(secret_service.delete_secret, stack_id, kind, name)
return {"ok": True}
@app.post("/agent/stacks/{stack_id}/secrets/attach", dependencies=[Depends(verify_token)])
def agent_attach_secret(stack_id: str, body: SecretAttachBody) -> dict:
_ensure_stack(stack_id)
if not secret_service.exists(stack_id, body.kind, body.name):
raise HTTPException(status_code=404, detail="secret not found")
new_yaml = _secret_guard(secret_service.attach, stack_id, body.kind, body.name, body.service, body.target)
return {"ok": True, "yaml": new_yaml}
@app.post("/agent/stacks/{stack_id}/secrets/detach", dependencies=[Depends(verify_token)])
def agent_detach_secret(stack_id: str, body: SecretDetachBody) -> dict:
_ensure_stack(stack_id)
new_yaml = _secret_guard(secret_service.detach, stack_id, body.kind, body.name, body.service)
return {"ok": True, "yaml": new_yaml}
@app.get("/agent/stacks/{stack_id}/backup", dependencies=[Depends(verify_token)])
async def backup_stack(
stack_id: str,
include_volumes: bool = Query(True),
stop_first: bool = Query(True),
):
if not os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
try:
path = await backup_service.create_backup(
stack_id, stack_id, include_volumes=include_volumes, stop_first=stop_first,
)
except backup_service.BackupError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return FileResponse(
path,
media_type="application/gzip",
filename=backup_service.backup_filename(stack_id, include_volumes),
)
@app.post("/agent/stacks/restore", dependencies=[Depends(verify_token)])
async def restore_stack(
file: UploadFile = File(...),
target_id: str | None = Form(None),
overwrite: bool = Form(False),
restore_volumes: bool = Form(True),
) -> dict:
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".tar.gz")
try:
while chunk := await file.read(1024 * 1024):
tmp.write(chunk)
tmp.close()
target = compose_service.slugify(target_id) if target_id else None
try:
return backup_service.restore_backup(
tmp.name, target_id=target, overwrite=overwrite, restore_volumes=restore_volumes,
)
except backup_service.BackupError as exc:
code = 409 if "already exists" in str(exc) else 400
raise HTTPException(status_code=code, detail=str(exc)) from exc
finally:
if os.path.exists(tmp.name):
os.unlink(tmp.name)
# --------------------------------------------------------------------------- #
# Networks
# --------------------------------------------------------------------------- #
@app.get("/agent/networks", dependencies=[Depends(verify_token)])
def list_networks() -> list[dict]:
return network_service.list_networks()
@app.get("/agent/networks/{network_id}", dependencies=[Depends(verify_token)])
def inspect_network(network_id: str) -> dict:
try:
return network_service.inspect_network(network_id)
except DockerError as exc:
_map_docker(exc)
@app.get("/agent/networks/{network_id}/containers", dependencies=[Depends(verify_token)])
def network_containers(network_id: str) -> list[dict]:
try:
return network_service.connectable_containers(network_id)
except DockerError as exc:
_map_docker(exc)
@app.post("/agent/networks/{network_id}/connect", dependencies=[Depends(verify_token)])
def connect_container(network_id: str, body: ContainerRefBody) -> dict:
try:
network_service.connect_container(network_id, body.container, body.aliases)
except DockerError as exc:
_map_docker(exc)
return {"ok": True}
@app.post("/agent/networks/{network_id}/disconnect", dependencies=[Depends(verify_token)])
def disconnect_container(network_id: str, body: ContainerRefBody) -> dict:
try:
network_service.disconnect_container(network_id, body.container, body.force)
except DockerError as exc:
_map_docker(exc)
return {"ok": True}
@app.post("/agent/networks", dependencies=[Depends(verify_token)], status_code=201)
def create_network(body: NetworkCreateBody) -> dict:
try:
return network_service.create_network(body.model_dump())
except DockerError as exc:
_map_docker(exc)
@app.delete("/agent/networks/{network_id}", dependencies=[Depends(verify_token)])
def delete_network(network_id: str) -> dict:
try:
network_service.delete_network(network_id)
except DockerError as exc:
_map_docker(exc)
return {"ok": True}
@app.post("/agent/networks/prune", dependencies=[Depends(verify_token)])
def prune_networks() -> dict:
return network_service.prune_networks()
# --------------------------------------------------------------------------- #
# Images
# --------------------------------------------------------------------------- #
@app.get("/agent/images", dependencies=[Depends(verify_token)])
def list_images() -> list[dict]:
return image_service.list_images()
@app.get("/agent/images/updates", dependencies=[Depends(verify_token)])
def image_updates() -> dict:
return update_service.get_cache()
@app.post("/agent/images/check", dependencies=[Depends(verify_token)])
async def image_check() -> dict:
return await update_service.check_all()
@app.post("/agent/images/prune", dependencies=[Depends(verify_token)])
def image_prune(all_unused: bool = Query(False, alias="all")) -> dict:
return image_service.prune_images(all_unused)
# --------------------------------------------------------------------------- #
# Containers (single-container inspect + lifecycle)
# --------------------------------------------------------------------------- #
@app.get("/agent/containers/{container_id}", dependencies=[Depends(verify_token)])
def inspect_container(container_id: str) -> dict:
return container_service.inspect_container(container_id)
@app.post("/agent/containers/{container_id}/{action}", dependencies=[Depends(verify_token)])
def container_action(container_id: str, action: str) -> dict:
return container_service.container_action(container_id, action)
# --------------------------------------------------------------------------- #
# Volumes
# --------------------------------------------------------------------------- #
@app.get("/agent/volumes", dependencies=[Depends(verify_token)])
def list_volumes() -> list[dict]:
return volume_service.list_volumes()
@app.get("/agent/volumes/sizes", dependencies=[Depends(verify_token)])
def volume_sizes(force: bool = Query(False)) -> dict:
return volume_service.volume_sizes(force=force)
@app.delete("/agent/volumes/{name}", dependencies=[Depends(verify_token)])
def delete_volume(name: str, force: bool = Query(False)) -> dict:
vols = {v["name"]: v for v in volume_service.list_volumes()}
if name in vols and vols[name]["in_use"] and not force:
raise HTTPException(
status_code=409,
detail={
"error": "volume_in_use",
"detail": f"Volume '{name}' is used by: {', '.join(vols[name]['used_by'])}",
},
)
volume_service.remove_volume(name, force=force)
return {"ok": True}
@app.post("/agent/volumes/prune", dependencies=[Depends(verify_token)])
def prune_volumes() -> dict:
return volume_service.prune_volumes()
# --------------------------------------------------------------------------- #
# File browser (sandboxed by this agent's ALLOWED_BROWSE_ROOTS/HOST_ROOT_PREFIX)
# --------------------------------------------------------------------------- #
@app.get("/agent/files/list", dependencies=[Depends(verify_token)])
def files_list(path: str = Query("/"), show_hidden: bool = Query(False)) -> dict:
return _file_guard(device_service.browse, path, show_hidden)
@app.get("/agent/files/read", dependencies=[Depends(verify_token)])
def files_read(path: str = Query(...)) -> dict:
return _file_guard(file_service.read_file, path)
@app.get("/agent/files/download", dependencies=[Depends(verify_token)])
def files_download(path: str = Query(...)):
real, filename = _file_guard(file_service.resolve_download, path)
return FileResponse(real, filename=filename, media_type="application/octet-stream")
@app.put("/agent/files/write", dependencies=[Depends(verify_token)])
def files_write(body: FileWriteBody) -> dict:
return _file_guard(file_service.write_file, body.path, body.content)
@app.post("/agent/files/mkdir", dependencies=[Depends(verify_token)])
def files_mkdir(body: FileNameBody) -> dict:
return _file_guard(file_service.create_dir, body.path, body.name)
@app.post("/agent/files/touch", dependencies=[Depends(verify_token)])
def files_touch(body: FileNameBody) -> dict:
return _file_guard(file_service.create_file, body.path, body.name)
@app.post("/agent/files/rename", dependencies=[Depends(verify_token)])
def files_rename(body: FileRenameBody) -> dict:
return _file_guard(file_service.rename, body.path, body.new_name)
@app.post("/agent/files/copy", dependencies=[Depends(verify_token)])
def files_copy(body: FileTransferBody) -> dict:
return _file_guard(file_service.copy, body.src, body.dest_dir, body.overwrite)
@app.post("/agent/files/move", dependencies=[Depends(verify_token)])
def files_move(body: FileTransferBody) -> dict:
return _file_guard(file_service.move, body.src, body.dest_dir, body.overwrite)
@app.delete("/agent/files", dependencies=[Depends(verify_token)])
def files_delete(path: str = Query(...), recursive: bool = Query(False)) -> dict:
return _file_guard(file_service.delete, path, recursive)
@app.post("/agent/files/upload", dependencies=[Depends(verify_token)])
async def files_upload(
path: str = Form(...),
overwrite: bool = Form(False),
rel_path: str = Form(""),
file: UploadFile = File(...),
) -> dict:
real = _file_guard(
file_service.upload_target, path, file.filename or "", overwrite, rel_path or None
)
tmp = tempfile.NamedTemporaryFile(delete=False, dir=os.path.dirname(real))
try:
while chunk := await file.read(1024 * 1024):
tmp.write(chunk)
tmp.close()
os.replace(tmp.name, real)
except OSError as exc:
if os.path.exists(tmp.name):
os.unlink(tmp.name)
raise HTTPException(status_code=400, detail=f"Upload failed: {exc}") from exc
return {"ok": True, "name": rel_path or file.filename}
@app.websocket("/agent/ws/logs/{stack_id}")
async def ws_logs(websocket: WebSocket, stack_id: str, token: str | None = Query(default=None)):
"""Stream `docker compose logs -f` to the central app (token via query param)."""
await websocket.accept()
expected = settings.AGENT_TOKEN
if not expected or token != expected:
await websocket.close(code=4401)
return
if not os.path.isdir(compose_service.stack_dir(stack_id)):
await websocket.send_text(json.dumps({"type": "error", "detail": "stack not found"}))
await websocket.close()
return
args = ["logs", "--no-color", "--tail", "200", "--timestamps", "-f"]
try:
async for line in compose_service.stream_compose(stack_id, args):
await websocket.send_text(
json.dumps({"type": "log", "stack_id": stack_id, "service": None, "line": line})
)
except WebSocketDisconnect:
pass
except Exception as exc: # noqa: BLE001
try:
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
except Exception: # noqa: BLE001
pass
@app.websocket("/agent/ws/deploy/{stack_id}")
async def ws_deploy(websocket: WebSocket, stack_id: str, token: str | None = Query(default=None)):
"""Run `docker compose up -d` and stream its output to the central app so the
browser sees deploy progress live (token via query param)."""
await websocket.accept()
expected = settings.AGENT_TOKEN
if not expected or token != expected:
await websocket.close(code=4401)
return
if not os.path.isdir(compose_service.stack_dir(stack_id)):
await websocket.send_text(json.dumps({"type": "error", "detail": "stack not found"}))
await websocket.close()
return
compose_service.mark_busy(stack_id)
try:
async for kind, payload in compose_service.stream_up(stack_id):
if kind == "log":
await websocket.send_text(json.dumps({"type": "log", "line": payload}))
else:
await websocket.send_text(json.dumps({"type": "done", "returncode": payload}))
except WebSocketDisconnect:
# Browser navigated away; the compose subprocess keeps running.
pass
except Exception as exc: # noqa: BLE001
try:
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
except Exception: # noqa: BLE001
pass
finally:
compose_service.clear_busy(stack_id)
@app.websocket("/agent/ws/exec/{container_id}")
async def ws_exec(
websocket: WebSocket,
container_id: str,
token: str | None = Query(default=None),
cmd: str | None = Query(default=None),
):
"""Interactive shell into a compose-managed container (token via query)."""
await websocket.accept()
expected = settings.AGENT_TOKEN
if not expected or token != expected:
await websocket.close(code=4401)
return
shell = cmd or exec_service.DEFAULT_SHELL
try:
exec_id = exec_service.create_exec(container_id, [shell])
holder, raw = exec_service.start_exec(exec_id)
except Exception as exc: # noqa: BLE001
try:
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
except Exception: # noqa: BLE001
pass
await websocket.close()
return
try:
await exec_service.pump_exec(websocket, exec_id, holder, raw)
except WebSocketDisconnect:
pass
finally:
try:
await websocket.close()
except Exception: # noqa: BLE001
pass
@app.get("/agent/health")
def health() -> dict:
return {"status": "ok"}
backend/auth.py
+107 -16
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from datetime import datetime, timedelta, timezone
from typing import Optional
from fastapi import Depends, HTTPException, status
from fastapi import Depends, HTTPException, Request, status
from fastapi.security import OAuth2PasswordBearer
from jose import JWTError, jwt
from passlib.context import CryptContext
@@ -13,6 +13,7 @@ from sqlmodel import Session, select
from config import settings
from database import get_session
from models.user import User
from services import api_token_service
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="/api/auth/login", auto_error=False)
@@ -32,12 +33,20 @@ def verify_password(plain: str, hashed: str) -> bool:
# --- token helpers ---
def _create_token(sub: str, role: str, token_type: str, expires: timedelta) -> str:
def token_version_of(user: User) -> int:
"""A user's current token version, tolerating a NULL from an older schema."""
return int(user.token_version or 1)
def _create_token(user: User, token_type: str, expires: timedelta) -> str:
now = datetime.now(timezone.utc)
payload = {
"sub": sub,
"role": role,
"sub": user.username,
"role": user.role,
"type": token_type,
# Minted-at authority version. Checked on every request, so bumping it
# revokes every token this user already holds.
"ver": token_version_of(user),
"iat": now,
"exp": now + expires,
}
@@ -46,22 +55,26 @@ def _create_token(sub: str, role: str, token_type: str, expires: timedelta) -> s
def create_access_token(user: User) -> str:
return _create_token(
user.username,
user.role,
"access",
timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES),
user, "access", timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
)
def create_refresh_token(user: User) -> str:
return _create_token(
user.username,
user.role,
"refresh",
timedelta(days=settings.REFRESH_TOKEN_EXPIRE_DAYS),
user, "refresh", timedelta(days=settings.REFRESH_TOKEN_EXPIRE_DAYS)
)
def bump_token_version(user: User) -> None:
"""Invalidate every token this user currently holds.
Called whenever their authority changes — password, role, active flag — so
a compromised account is actually cut off instead of staying usable until
the tokens expire on their own. The caller commits.
"""
user.token_version = token_version_of(user) + 1
def decode_token(token: str, expected_type: str = "access") -> dict:
try:
payload = jwt.decode(
@@ -83,6 +96,21 @@ def decode_token(token: str, expected_type: str = "access") -> dict:
# --- user lookups ---
def resolve_token_user(session: Session, payload: dict) -> Optional[User]:
"""The live user a token payload refers to, or None if it is no longer valid.
Deliberately re-reads the database rather than trusting the token's claims:
the role in a token is a snapshot from when it was minted, and an account
can be disabled or have its password reset at any point afterwards.
"""
user = get_user(session, payload.get("sub", ""))
if not user or not user.is_active:
return None
if int(payload.get("ver", 0)) != token_version_of(user):
return None
return user
def get_user(session: Session, username: str) -> Optional[User]:
return session.exec(select(User).where(User.username == username)).first()
@@ -104,6 +132,7 @@ def users_exist(session: Session) -> bool:
def get_current_user(
request: Request,
token: Optional[str] = Depends(oauth2_scheme),
session: Session = Depends(get_session),
) -> User:
@@ -113,20 +142,82 @@ def get_current_user(
detail="Not authenticated",
headers={"WWW-Authenticate": "Bearer"},
)
# An API token is not a JWT and must not be fed to the decoder — it is
# recognised by its prefix and looked up instead.
if api_token_service.looks_like_token(token):
resolved = api_token_service.resolve(session, token)
if not resolved:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="This API token is not valid (unknown, expired or revoked)",
)
row, user = resolved
api_token_service.touch(session, row)
# Stashed rather than folded into the User: mutating the role on a
# session-attached row would be written back to the database the next
# time anything commits that user.
request.state.api_token = row
return user
request.state.api_token = None
payload = decode_token(token, "access")
user = get_user(session, payload.get("sub", ""))
if not user or not user.is_active:
user = resolve_token_user(session, payload)
if not user:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found or inactive",
detail="Session is no longer valid — sign in again",
)
return user
def require_admin(user: User = Depends(get_current_user)) -> User:
def current_api_token(request: Request):
"""The API token this request was authenticated with, if any."""
return getattr(request.state, "api_token", None)
def require_admin_role(user: User) -> User:
"""Role check split out so the WebSocket routes can reuse it."""
if user.role != "admin":
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Admin privileges required",
)
return user
def require_admin(
request: Request, user: User = Depends(get_current_user)
) -> User:
require_admin_role(user)
row = current_api_token(request)
if row and api_token_service.effective_role(row, user) != "admin":
# The owner is an admin but this token was issued read-only, which is
# the whole point of handing one to a monitoring script.
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="This API token is read-only",
)
return user
def require_session(
request: Request, user: User = Depends(get_current_user)
) -> User:
"""An interactive session, not an API token.
Guards the routes that mint or revoke credentials — API tokens and user
accounts. A leaked CI token should be able to do the job it was issued for,
not quietly grant itself permanent access that outlives its own revocation.
"""
if current_api_token(request) is not None:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="This action requires a signed-in session, not an API token",
)
return user
def require_admin_session(
request: Request, user: User = Depends(require_admin)
) -> User:
return require_session(request, user)
backend/config.py
+40 -10
View File
@@ -1,11 +1,13 @@
"""Application settings, loaded from environment variables."""
from __future__ import annotations
import os
import secrets
import stat
from functools import lru_cache
from typing import Annotated
from pydantic import field_validator
from pydantic import ValidationInfo, field_validator
from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict
@@ -17,7 +19,8 @@ class Settings(BaseSettings):
DATA_DIR: str = "/opt/stackpilot/data"
# Security
SECRET_KEY: str = "" # Auto-generated if empty (dev only); set in prod.
# Auto-generated and persisted to ${DATA_DIR}/secret_key when left empty.
SECRET_KEY: str = ""
ALGORITHM: str = "HS256"
ACCESS_TOKEN_EXPIRE_MINUTES: int = 60
REFRESH_TOKEN_EXPIRE_DAYS: int = 30
@@ -35,13 +38,11 @@ class Settings(BaseSettings):
# Throwaway image used to read/write named-volume contents during backup.
BACKUP_HELPER_IMAGE: str = "alpine:latest"
# Multi-host agent: shared bearer token the agent requires on every request.
# Only used when running the agent app (agent_app:app).
AGENT_TOKEN: str = ""
# Host browser sandbox roots
# Host browser sandbox roots. Deliberately does NOT contain "/": that entry
# makes _is_allowed() wave through every path, i.e. it switches the sandbox
# off. Add it back explicitly if you really want the whole filesystem.
ALLOWED_BROWSE_ROOTS: Annotated[list[str], NoDecode] = [
"/", "/mnt", "/media", "/srv", "/opt",
"/mnt", "/media", "/srv", "/opt", "/home",
]
HOST_ROOT_PREFIX: str = "" # e.g. "/host_root" when host / is bind-mounted
@@ -55,8 +56,37 @@ class Settings(BaseSettings):
@field_validator("SECRET_KEY", mode="after")
@classmethod
def _ensure_secret(cls, v: str) -> str:
return v or secrets.token_urlsafe(48)
def _ensure_secret(cls, v: str, info: ValidationInfo) -> str:
"""Return the configured key, or a persisted auto-generated one.
Generating a fresh key per process (the old behaviour) silently
invalidated every session on each restart, and would now also make the
encrypted backup-destination credentials undecryptable. So the
generated key is written next to the database instead, mode 0600, and
read back on the next start. An explicitly configured SECRET_KEY always
wins and nothing is written.
"""
if v:
return v
data_dir = info.data.get("DATA_DIR") or "/opt/stackpilot/data"
key_file = os.path.join(data_dir, "secret_key")
try:
with open(key_file, "r", encoding="utf-8") as fh:
if existing := fh.read().strip():
return existing
except OSError:
pass
generated = secrets.token_urlsafe(48)
try:
os.makedirs(data_dir, exist_ok=True)
with open(key_file, "w", encoding="utf-8") as fh:
fh.write(generated + "\n")
os.chmod(key_file, stat.S_IRUSR | stat.S_IWUSR)
except OSError:
# Read-only data dir: fall back to the old per-process behaviour
# rather than refusing to boot. Sessions won't survive a restart.
pass
return generated
@field_validator(
"NOTIFY_WEBHOOKS", "ALLOWED_BROWSE_ROOTS", "CORS_ORIGINS", mode="before"
backend/database.py
+124
View File
@@ -1,13 +1,19 @@
"""SQLModel database setup."""
from __future__ import annotations
import logging
import os
from collections.abc import Generator
from typing import Optional
from sqlalchemy import inspect, text
from sqlalchemy.exc import OperationalError
from sqlmodel import Session, SQLModel, create_engine
from config import settings
logger = logging.getLogger("stackpilot.database")
os.makedirs(settings.DATA_DIR, exist_ok=True)
_DB_PATH = os.path.join(settings.DATA_DIR, "stackpilot.db")
_DB_URL = f"sqlite:///{_DB_PATH}"
@@ -19,11 +25,129 @@ engine = create_engine(
)
def _default_literal(col) -> Optional[str]:
"""SQL literal for a column's scalar default, or None if it has none.
Only plain values are rendered — a callable default (``default_factory``,
e.g. a timestamp) has no fixed literal, so those columns are added nullable
as before and filled by the ORM on the next write.
"""
default = col.default
if default is None or not getattr(default, "is_scalar", False):
return None
value = default.arg
if isinstance(value, bool):
return "1" if value else "0"
if isinstance(value, (int, float)):
return str(value)
if isinstance(value, str):
escaped = value.replace("'", "''")
return f"'{escaped}'"
return None
def _ensure_model_columns() -> None:
"""Add columns that models define but a pre-existing table is missing.
``SQLModel.create_all`` creates missing *tables* but never ALTERs an
existing one, so installs that predate a newly-added column keep the old
schema — and every ORM query that names the column fails with
``OperationalError: no such column``. For each mapped table we diff the
model's columns against the live table and ``ADD COLUMN`` the safe
(nullable, or defaulted) ones. Idempotent: on a fresh DB create_all already
made every column, so this is a no-op.
Requires ``models`` to have been imported, or ``SQLModel.metadata`` is empty
and this silently does nothing. :func:`init_db` imports it first.
A column with a scalar default is added ``NOT NULL DEFAULT <value>`` so
existing rows are backfilled in the same statement. Without that clause
SQLite fills them with NULL, which is how a new non-nullable field turns
into a runtime surprise — for ``User.token_version`` it would have meant
every existing session failing its version check after the upgrade.
"""
insp = inspect(engine)
live_tables = set(insp.get_table_names())
with engine.begin() as conn:
for table_name, table in SQLModel.metadata.tables.items():
if table_name not in live_tables:
continue
existing = {c["name"] for c in insp.get_columns(table_name)}
for col in table.columns:
if col.name in existing:
continue
# SQLite can only ADD a NOT NULL column if it has a default to
# backfill existing rows; skip the rest rather than crash.
if not col.nullable and col.default is None and col.server_default is None:
logger.warning(
"Cannot auto-add non-nullable column %s.%s (no default); "
"manual migration needed", table_name, col.name
)
continue
ddl = f'ALTER TABLE "{table_name}" ADD COLUMN "{col.name}" '
ddl += col.type.compile(dialect=engine.dialect)
if (literal := _default_literal(col)) is not None:
# Backfills existing rows and satisfies SQLite's rule that a
# NOT NULL column may only be added together with a default.
ddl += f" NOT NULL DEFAULT {literal}"
conn.execute(text(ddl))
logger.info("Schema migration: added column %s.%s", table_name, col.name)
#: Tables and columns left behind when the remote-host (agent) integration was
#: removed in 0.48.0. SQLite before 3.35 cannot DROP COLUMN, and the rows are
#: harmless dead weight either way — so the table goes and the columns are only
#: dropped where the SQLite build supports it.
_REMOVED_TABLES = ("agent",)
_REMOVED_COLUMNS = (("autoupdate", "agent_id"), ("backupschedule", "agent_id"))
def _drop_removed_schema() -> None:
"""Clean up schema left over from features that no longer exist.
Without this an upgraded install keeps an ``agent`` table full of host URLs
and bearer tokens for a feature that is gone — credentials sitting in the
database with nothing to use them.
Each statement runs in its own transaction on purpose: a failed DDL poisons
the transaction it is in, so sharing one would mean a single unsupported
DROP COLUMN takes the table drop down with it.
"""
insp = inspect(engine)
live = set(insp.get_table_names())
for table in _REMOVED_TABLES:
if table not in live:
continue
with engine.begin() as conn:
conn.execute(text(f'DROP TABLE "{table}"'))
logger.info("Schema migration: dropped obsolete table %s", table)
for table, column in _REMOVED_COLUMNS:
if table not in live:
continue
if column not in {c["name"] for c in insp.get_columns(table)}:
continue
try:
with engine.begin() as conn:
conn.execute(text(f'ALTER TABLE "{table}" DROP COLUMN "{column}"'))
logger.info("Schema migration: dropped obsolete column %s.%s", table, column)
except OperationalError:
# SQLite < 3.35 has no DROP COLUMN. The column is nullable and
# nothing reads it any more, so leaving it is harmless.
logger.info(
"Leaving obsolete column %s.%s in place (this SQLite cannot "
"drop columns); it is unused", table, column
)
def init_db() -> None:
# Import models so they are registered on SQLModel.metadata.
import models # noqa: F401
SQLModel.metadata.create_all(engine)
_ensure_model_columns()
_drop_removed_schema()
def get_session() -> Generator[Session, None, None]:
backend/main.py
+67 -5
View File
@@ -11,10 +11,10 @@ from fastapi.responses import JSONResponse
from sqlmodel import Session
from config import settings
from version import APP_VERSION
from database import engine, init_db
from docker_client import DockerError
from routers import (
agents,
audit,
auth,
backups,
@@ -23,19 +23,32 @@ from routers import (
destinations,
editor,
files,
git,
images,
networks,
ports,
registries,
schedules,
secrets,
settings as settings_router,
stacks,
system,
templates,
tokens,
volumes,
ws,
)
from services import schedule_service, update_service
from services import (
backup_destination_service,
git_service,
image_status_store,
logo_service,
registry_service,
schedule_service,
stack_lock_service,
template_service,
update_service,
)
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("stackpilot")
@@ -50,15 +63,61 @@ async def lifespan(app: FastAPI):
stacks.sync_discovered_stacks(session)
except Exception as exc: # noqa: BLE001
logger.warning("Stack discovery failed: %s", exc)
# One-off: encrypt backup-destination credentials written before they were
# stored encrypted (see services/crypto_service.py).
try:
with Session(engine) as session:
encrypted = backup_destination_service.migrate_plaintext_configs(session)
if encrypted:
logger.info("Encrypted %d backup destination config(s) at rest", encrypted)
except Exception as exc: # noqa: BLE001
logger.warning("Destination config encryption migration failed: %s", exc)
try:
moved = template_service.migrate_legacy_db_templates()
if moved:
logger.info("Migrated %d custom template(s) from the database to folders", moved)
except Exception as exc: # noqa: BLE001
logger.warning("Legacy template migration failed: %s", exc)
# Runtime state that used to live in module dicts and was lost on restart.
try:
with Session(engine) as session:
stale = stack_lock_service.prune_expired(session)
if stale:
logger.info("Cleared %d stale stack lock(s) from a previous run", stale)
except Exception as exc: # noqa: BLE001
logger.warning("Could not prune stack locks: %s", exc)
try:
restored = image_status_store.install()
logger.info("Restored %d cached image update status(es)", restored)
except Exception as exc: # noqa: BLE001
logger.warning("Could not restore the image update cache: %s", exc)
# Private registry credentials: into the in-memory cache the update checker
# reads, and into the config.json the Docker CLI reads.
try:
with Session(engine) as session:
known = registry_service.reload(session)
if known:
logger.info("Loaded credentials for %d registr%s", known, "y" if known == 1 else "ies")
except Exception as exc: # noqa: BLE001
logger.warning("Could not load registry credentials: %s", exc)
update_task = asyncio.create_task(update_service.background_loop())
schedule_task = asyncio.create_task(schedule_service.scheduler_loop())
# App logos. Deliberately a task and not awaited: the catalog is a network
# download, and a box with no outbound internet must still start instantly
# (it just keeps the built-in glyphs).
logo_task = asyncio.create_task(logo_service.catalog_loop())
git_service.ensure_cache_root()
git_task = asyncio.create_task(git_service.poll_loop())
logger.info("StackPilot backend ready on port %s", settings.PORT)
yield
update_task.cancel()
schedule_task.cancel()
logo_task.cancel()
git_task.cancel()
app = FastAPI(title="StackPilot", version="0.30.0", lifespan=lifespan)
app = FastAPI(title="StackPilot", version=APP_VERSION, lifespan=lifespan)
app.add_middleware(
CORSMiddleware,
@@ -79,6 +138,10 @@ async def docker_error_handler(_request: Request, exc: DockerError):
app.include_router(auth.router)
app.include_router(stacks.router)
app.include_router(git.router)
app.include_router(git.hook_router)
app.include_router(tokens.router)
app.include_router(registries.router)
app.include_router(secrets.router)
app.include_router(containers.router)
app.include_router(dashboard.router)
@@ -95,10 +158,9 @@ app.include_router(backups.router)
app.include_router(destinations.router)
app.include_router(schedules.router)
app.include_router(networks.router)
app.include_router(agents.router)
app.include_router(ws.router)
@app.get("/api/health")
def health() -> dict:
return {"status": "ok"}
return {"status": "ok", "version": APP_VERSION}
backend/models/__init__.py
+6 -3
View File
@@ -1,15 +1,18 @@
"""SQLModel table models. Importing this package registers all tables."""
from models.agent import Agent
from models.api_token import ApiToken
from models.audit import AuditLog
from models.auto_update import AutoUpdate
from models.backup_destination import BackupDestination
from models.backup_schedule import BackupSchedule
from models.git_source import GitSource
from models.registry import Registry
from models.runtime_state import ImageStatus, LoginAttempt, StackLock
from models.setting import Setting, Webhook
from models.stack import Stack
from models.template import Template
from models.user import User
__all__ = [
"User", "Stack", "AuditLog", "Template", "Setting", "Webhook", "Agent",
"User", "Stack", "AuditLog", "Setting", "Webhook",
"BackupDestination", "BackupSchedule", "AutoUpdate",
"StackLock", "ImageStatus", "LoginAttempt", "Registry", "ApiToken", "GitSource",
]
backend/models/agent.py
-49
View File
@@ -1,49 +0,0 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Optional
from sqlmodel import Field, SQLModel
def _now() -> datetime:
return datetime.now(timezone.utc)
class Agent(SQLModel, table=True):
"""A remote host running stackpilot-agent."""
id: Optional[int] = Field(default=None, primary_key=True)
name: str
url: str # e.g. http://10.0.0.5:5010
token: str # shared AGENT_TOKEN of that host
status: str = "unknown" # online | offline | unauthorized | unknown
hostname: Optional[str] = None # reported by the agent on ping
last_seen: Optional[datetime] = None
created_at: datetime = Field(default_factory=_now)
# --- API schemas ---
class AgentCreate(SQLModel):
name: str
url: str
token: str
class AgentUpdate(SQLModel):
name: Optional[str] = None
url: Optional[str] = None
token: Optional[str] = None
class AgentRead(SQLModel):
id: int
name: str
url: str
status: str
hostname: Optional[str]
last_seen: Optional[datetime]
created_at: datetime
token_set: bool
backend/models/api_token.py
+66
View File
@@ -0,0 +1,66 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Optional
from sqlmodel import Field, SQLModel
def _now() -> datetime:
return datetime.now(timezone.utc)
#: What a token is allowed to do. "read" matches the read-only user role even
#: when the owner is an admin, so a monitoring script can be handed a token that
#: cannot change anything.
SCOPES = ["read", "admin"]
class ApiToken(SQLModel, table=True):
"""A long-lived bearer token for scripts and CI, owned by a user.
Only a hash is stored — the token itself is shown once, when it is created,
and cannot be recovered afterwards. ``prefix`` is the readable front of the
token (``sp_`` plus eight characters); it identifies the row in the UI and
in the audit log without being enough to authenticate with.
"""
id: Optional[int] = Field(default=None, primary_key=True)
name: str
prefix: str = Field(index=True, unique=True)
token_hash: str
scope: str = Field(default="read")
#: The account the token acts as. Its role caps the token's scope, and a
#: disabled account disables its tokens.
user_id: int = Field(index=True)
expires_at: Optional[datetime] = None
last_used_at: Optional[datetime] = None
created_at: datetime = Field(default_factory=_now)
# --- API schemas ---
class ApiTokenCreate(SQLModel):
name: str
scope: str = "read"
#: Days until it expires. None means it does not.
expires_in_days: Optional[int] = None
class ApiTokenRead(SQLModel):
id: int
name: str
prefix: str
scope: str
username: str
expires_at: Optional[datetime]
last_used_at: Optional[datetime]
created_at: datetime
expired: bool
class ApiTokenCreated(ApiTokenRead):
"""The create response, and the only time the token itself is returned."""
token: str
backend/models/auto_update.py
-4
View File
@@ -16,12 +16,10 @@ class AutoUpdate(SQLModel, table=True):
When the background image-update check finds a newer registry digest for one
of the stack's images, the stack is either pulled + redeployed
(``redeploy=True``) or merely notified about (``redeploy=False``).
``agent_id`` None = local host, otherwise a remote agent's stack.
"""
id: Optional[int] = Field(default=None, primary_key=True)
stack_id: str
agent_id: Optional[int] = None
enabled: bool = True
redeploy: bool = True # True = pull + up -d; False = notify only
last_run: Optional[datetime] = None
@@ -41,8 +39,6 @@ class AutoUpdateWrite(SQLModel):
class AutoUpdateRead(SQLModel):
id: Optional[int]
stack_id: str
agent_id: Optional[int]
agent_name: Optional[str] = None
enabled: bool
redeploy: bool
last_run: Optional[datetime]
backend/models/backup_destination.py
+2 -2
View File
@@ -10,14 +10,14 @@ def _now() -> datetime:
return datetime.now(timezone.utc)
DESTINATION_TYPES = ["sftp", "s3"]
DESTINATION_TYPES = ["sftp", "s3", "nfs"]
# config keys that hold secrets — masked in API responses.
SECRET_KEYS = {"password", "private_key", "secret_key"}
class BackupDestination(SQLModel, table=True):
"""A remote target for stack backups (SFTP or S3-compatible)."""
"""A remote target for stack backups (SFTP, S3-compatible or NFS)."""
id: Optional[int] = Field(default=None, primary_key=True)
name: str
backend/models/backup_schedule.py
-4
View File
@@ -19,7 +19,6 @@ class BackupSchedule(SQLModel, table=True):
id: Optional[int] = Field(default=None, primary_key=True)
stack_id: str
destination_id: int
agent_id: Optional[int] = None # None = local host; otherwise a remote agent
frequency: str = "daily" # one of FREQUENCIES
hour: int = 3 # UTC, used for daily/weekly
minute: int = 0
@@ -40,7 +39,6 @@ class BackupSchedule(SQLModel, table=True):
class ScheduleCreate(SQLModel):
stack_id: str
destination_id: int
agent_id: Optional[int] = None
frequency: str = "daily"
hour: int = 3
minute: int = 0
@@ -68,8 +66,6 @@ class ScheduleRead(SQLModel):
stack_id: str
destination_id: int
destination_name: Optional[str]
agent_id: Optional[int]
agent_name: Optional[str]
frequency: str
hour: int
minute: int
backend/models/git_source.py
+93
View File
@@ -0,0 +1,93 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Optional
from sqlmodel import Field, SQLModel
def _now() -> datetime:
return datetime.now(timezone.utc)
#: How to reach a private repository. "token" is an HTTPS username + personal
#: access token; "ssh" is a private key.
AUTH_TYPES = ["none", "token", "ssh"]
class GitSource(SQLModel, table=True):
"""A Git repository that a stack's files are deployed from.
The repository is the source of truth: a sync overwrites the stack's files
with what the repo says, which is the whole point of GitOps and also the
thing to be careful about. Only files the repo has ever provided are touched
— see ``services/git_service.py`` — so the data directories compose creates
inside a stack folder are never at risk.
"""
id: Optional[int] = Field(default=None, primary_key=True)
stack_id: str = Field(index=True, unique=True)
url: str
branch: str = "main"
#: Subdirectory inside the repository holding the compose file. Empty means
#: the repository root, which is the common case for one-stack repos.
subdir: str = ""
auth_type: str = "none"
username: Optional[str] = None
#: Encrypted: the access token, or the SSH private key.
secret: Optional[str] = None
#: Run `compose up -d` after a sync that actually changed something.
auto_deploy: bool = True
#: Poll the repository this often. None means only manual syncs and webhooks.
poll_interval_minutes: Optional[int] = None
#: Shared secret for the webhook endpoint (HMAC, or GitLab's token header).
webhook_secret: str = ""
#: JSON list of the paths the last sync wrote, relative to the stack folder.
#: The only files a later sync is allowed to delete.
managed_files: str = "[]"
last_commit: Optional[str] = None
last_synced_at: Optional[datetime] = None
last_error: Optional[str] = None
created_at: datetime = Field(default_factory=_now)
updated_at: datetime = Field(default_factory=_now)
# --- API schemas ---
class GitSourceWrite(SQLModel):
url: str
branch: str = "main"
subdir: str = ""
auth_type: str = "none"
username: Optional[str] = None
#: Omitted on update keeps the stored one.
secret: Optional[str] = None
auto_deploy: bool = True
poll_interval_minutes: Optional[int] = None
class GitSourceRead(SQLModel):
stack_id: str
url: str
branch: str
subdir: str
auth_type: str
username: Optional[str]
has_secret: bool
auto_deploy: bool
poll_interval_minutes: Optional[int]
webhook_url: str
last_commit: Optional[str]
last_synced_at: Optional[datetime]
last_error: Optional[str]
managed_file_count: int
class SyncResult(SQLModel):
changed: bool
commit: Optional[str] = None
written: list[str] = []
removed: list[str] = []
deployed: bool = False
detail: Optional[str] = None
backend/models/registry.py
+68
View File
@@ -0,0 +1,68 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Optional
from sqlmodel import Field, SQLModel
def _now() -> datetime:
return datetime.now(timezone.utc)
class Registry(SQLModel, table=True):
"""Credentials for one container registry.
``host`` is the canonical registry hostname as
:func:`services.registry_service.canonical_host` produces it, so the lookup
from an image reference is a dict hit and Docker Hub's several spellings all
land on one row.
The password is encrypted at rest (see ``services/crypto_service.py``) and
never leaves the API — reads return it masked.
"""
id: Optional[int] = Field(default=None, primary_key=True)
name: str
host: str = Field(index=True)
username: str
password: str # encrypted
created_at: datetime = Field(default_factory=_now)
updated_at: datetime = Field(default_factory=_now)
# --- API schemas ---
class RegistryCreate(SQLModel):
name: Optional[str] = None
host: str
username: str
password: str
class RegistryUpdate(SQLModel):
name: Optional[str] = None
host: Optional[str] = None
username: Optional[str] = None
# Omitted leaves the stored password alone, so the UI can save a row it only
# ever received masked.
password: Optional[str] = None
class RegistryRead(SQLModel):
id: int
name: str
host: str
username: str
has_password: bool
created_at: datetime
updated_at: datetime
class RegistryTestRequest(SQLModel):
"""An unsaved set of credentials to try, for the "Test" button."""
host: str
username: str
password: Optional[str] = None
backend/models/runtime_state.py
+73
View File
@@ -0,0 +1,73 @@
"""Runtime state that used to live in module-level dicts.
Three things were kept in process memory: which stacks are mid-deploy, the
registry digests behind the "update available" badges, and the login rate
limiter's counters. All three assumed exactly one uvicorn worker — nothing said
so, and ``--workers 2`` would have silently given each worker its own copy —
and all three were lost on restart.
They are tables now. SQLite is already here; this needs no new dependency.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Optional
from sqlmodel import Field, SQLModel
def _now() -> datetime:
return datetime.now(timezone.utc)
class StackLock(SQLModel, table=True):
"""A stack is mid-operation and must not be touched concurrently.
``docker compose`` has no locking of its own, so two simultaneous ``update``
calls — two browser tabs, or auto-update racing a manual click — would both
run ``pull`` and ``up`` against the same project and fight over recreating
containers.
``expires_at`` is what keeps a crashed worker from locking a stack forever:
an expired row is simply taken over by the next caller.
"""
stack_id: str = Field(primary_key=True)
action: str # "update", "start", "backup", …
#: Free-form owner, for the log when a lock is stolen. Not a security control.
owner: str = ""
acquired_at: datetime = Field(default_factory=_now)
expires_at: datetime
class ImageStatus(SQLModel, table=True):
"""Cached result of one image's registry digest check.
Persisted so a restart does not blank every update badge until the next
background sweep (up to an hour), and so ``notified`` survives with it —
otherwise every restart re-announced the same pending updates.
"""
image: str = Field(primary_key=True)
update_available: bool = False
current_digest: Optional[str] = None
remote_digest: Optional[str] = None
checked_at: float = 0.0
error: Optional[str] = None
#: Whether an "update available" notification already went out for this
#: image at its current state.
notified: bool = False
class LoginAttempt(SQLModel, table=True):
"""One login attempt, for the rate limiter.
In memory this reset on every restart, so an attacker could clear their own
budget by getting the process to restart — and with more than one worker the
limit multiplied by the worker count. Rows are pruned as they age out.
"""
id: Optional[int] = Field(default=None, primary_key=True)
ip: str = Field(index=True)
at: datetime = Field(default_factory=_now, index=True)
backend/models/stack.py
+7
View File
@@ -15,6 +15,10 @@ class Stack(SQLModel, table=True):
id: str = Field(primary_key=True)
name: str
description: Optional[str] = None
# None = automatic (the UI derives one from the name), "lucide:<name>" for a
# built-in icon, "custom:<ext>:<version>" for an uploaded image.
# See services/icon_service.py.
icon: Optional[str] = None
stacks_dir_override: Optional[str] = None
created_at: datetime = Field(default_factory=_now)
updated_at: datetime = Field(default_factory=_now)
@@ -26,6 +30,7 @@ class Stack(SQLModel, table=True):
class StackCreate(SQLModel):
name: str
description: Optional[str] = None
icon: Optional[str] = None # "lucide:<name>", or None/"" for automatic
yaml: Optional[str] = None # initial compose content
env: Optional[str] = None
@@ -33,6 +38,8 @@ class StackCreate(SQLModel):
class StackUpdate(SQLModel):
name: Optional[str] = None
description: Optional[str] = None
# Omitted leaves the icon alone; "" resets it to automatic.
icon: Optional[str] = None
yaml: Optional[str] = None
env: Optional[str] = None
backend/models/template.py
+15 -31
View File
@@ -1,34 +1,11 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Optional
from sqlmodel import Field, SQLModel
from sqlmodel import SQLModel
def _now() -> datetime:
return datetime.now(timezone.utc)
class Template(SQLModel, table=True):
"""User-saved custom template (bundled ones live on disk)."""
id: Optional[int] = Field(default=None, primary_key=True)
slug: str = Field(index=True, unique=True)
name: str
description: Optional[str] = None
tags: str = "" # comma separated
yaml: str = ""
created_at: datetime = Field(default_factory=_now)
# --- API schemas ---
class TemplateVariable(SQLModel):
name: str
description: str = ""
default: str = ""
# Templates are stored as stack-shaped folders on disk (see
# services/template_service.py), not in the database. These are API schemas only.
class TemplateSummary(SQLModel):
@@ -41,18 +18,25 @@ class TemplateSummary(SQLModel):
class TemplateDetail(TemplateSummary):
yaml: str
variables: list[TemplateVariable] = []
compose: str = ""
env: str = ""
files: list[str] = [] # relative paths the template ships
class TemplateSaveRequest(SQLModel):
name: str
description: Optional[str] = None
tags: list[str] = []
yaml: str
gpu: Optional[str] = None
compose: str
env: str = ""
class TemplateFromStackRequest(SQLModel):
stack_id: str
name: str
description: Optional[str] = None
class TemplateInstantiateRequest(SQLModel):
name: str # new stack name
values: dict[str, str] = {}
agent_id: int | None = None # None = local host; otherwise deploy to a remote agent
backend/models/user.py
+18 -2
View File
@@ -17,6 +17,12 @@ class User(SQLModel, table=True):
role: str = Field(default="user") # "admin" | "user"
is_active: bool = Field(default=True)
created_at: datetime = Field(default_factory=_now)
#: Bumped whenever this account's authority changes — password, role or
#: active flag. Every token carries the value it was minted with, so a
#: bump makes all outstanding tokens for this user fail their next check.
#: Without it a password reset left the old tokens usable for their full
#: lifetime (up to 30 days for a refresh token).
token_version: int = Field(default=1)
# --- API schemas ---
@@ -47,10 +53,20 @@ class LoginRequest(SQLModel):
class TokenPair(SQLModel):
"""Login/refresh response.
``refresh_token`` is optional in the body: the API sets it as an httpOnly
cookie, and browsers never need (or should) see it. It is still returned
when the caller opts in with ``?in_body=true`` so scripted clients that
cannot hold a cookie jar keep working.
"""
access_token: str
refresh_token: str
token_type: str = "bearer"
refresh_token: Optional[str] = None
class RefreshRequest(SQLModel):
refresh_token: str
"""Body for ``/api/auth/refresh``. Optional — the cookie is preferred."""
refresh_token: Optional[str] = None
backend/pyproject.toml
+38
View File
@@ -0,0 +1,38 @@
# Tooling config only — StackPilot's backend is not packaged, it runs from
# source in the image (see backend/Dockerfile). Runtime deps stay in
# requirements.txt; test/lint deps in requirements-dev.txt.
[tool.pytest.ini_options]
testpaths = ["tests"]
# Import test modules without needing tests/ to be a package, and make the
# backend root importable so `from services import ...` works the same way it
# does at runtime.
pythonpath = ["."]
addopts = "-q --strict-markers"
filterwarnings = [
# passlib 1.7.4 reads bcrypt.__about__, which bcrypt 4.x removed. Cosmetic,
# and tracked as F8 (migrate off passlib).
"ignore:.*error reading bcrypt version.*:UserWarning",
]
[tool.ruff]
target-version = "py312"
line-length = 100
exclude = ["templates"]
[tool.ruff.lint]
# Deliberately a floor, not a style bar: these are the rules that catch real
# defects (undefined names, unused imports, shadowed builtins, mutable
# defaults, swallowed exception context) without demanding a reformat of the
# existing 12k lines. Import sorting ("I") is left out on purpose — it is
# style, and turning it on would rewrite the import block of nine files that
# have nothing else wrong with them. Tighten over time rather than landing a
# big-bang cleanup.
select = ["F", "E9", "B"]
ignore = [
"B008", # Depends()/Query() in defaults is how FastAPI is written.
]
[tool.ruff.lint.per-file-ignores]
# Routers re-export request models for the agent proxy to reuse.
"routers/agents.py" = ["F401"]
backend/requirements-dev.txt
+5
View File
@@ -0,0 +1,5 @@
# Test + lint tooling. Not installed into the runtime image (see Dockerfile);
# used by `pytest` locally and by the CI's test job.
-r requirements.txt
pytest==8.3.4
ruff==0.9.2
backend/requirements.txt
+2
View File
@@ -5,6 +5,8 @@ sqlmodel==0.0.22
pydantic==2.10.4
pydantic-settings==2.7.1
python-jose[cryptography]==3.3.0
# Direct dependency: services/crypto_service encrypts DB-stored secrets.
cryptography==44.0.0
passlib[bcrypt]==1.7.4
bcrypt==4.2.1
python-multipart==0.0.20
backend/routers/agents.py
-1131
View File
File diff suppressed because it is too large Load Diff
backend/routers/audit.py
+8 -3
View File
@@ -1,4 +1,9 @@
"""Audit log query endpoint."""
"""Audit log query endpoint.
Admin-only: the log is security telemetry (who did what, from which IP,
including every administrator's activity) and has no business being readable
by an account with the ``user`` role.
"""
from __future__ import annotations
from typing import Optional
@@ -6,7 +11,7 @@ from typing import Optional
from fastapi import APIRouter, Depends, Query
from sqlmodel import Session, select
from auth import get_current_user
from auth import require_admin
from database import get_session
from models.audit import AuditLog
from models.user import User
@@ -20,7 +25,7 @@ def list_audit(
offset: int = 0,
stack_id: Optional[str] = None,
session: Session = Depends(get_session),
_user: User = Depends(get_current_user),
_admin: User = Depends(require_admin),
) -> list[AuditLog]:
stmt = select(AuditLog).order_by(AuditLog.timestamp.desc())
if stack_id:
backend/routers/auth.py
+148 -30
View File
@@ -1,14 +1,14 @@
"""Authentication routes + first-launch setup wizard."""
from __future__ import annotations
import time
from collections import defaultdict, deque
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends, HTTPException, Request, status
from sqlmodel import Session, select
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
from sqlmodel import Session, delete, select
import auth as auth_mod
from database import get_session
from models.runtime_state import LoginAttempt
from models.user import (
LoginRequest,
RefreshRequest,
@@ -18,33 +18,83 @@ from models.user import (
UserRead,
UserUpdate,
)
from config import settings
from services import audit_service
router = APIRouter(prefix="/api/auth", tags=["auth"])
# Simple in-memory rate limiter for login (max 10 / minute / IP).
_LOGIN_HITS: dict[str, deque] = defaultdict(deque)
# Login rate limit: max 10 attempts per minute per client IP.
#
# Kept in the database rather than a module dict. In memory it reset on every
# restart — so an attacker could clear their own budget by getting the process
# to restart — and with more than one uvicorn worker each worker enforced its
# own limit, multiplying the real allowance by the worker count.
#
# The IP is only meaningful because uvicorn runs with --proxy-headers; without
# that every request looks like it comes from the frontend container and this
# would throttle all users together.
_RATE_LIMIT = 10
_RATE_WINDOW = 60.0
_RATE_WINDOW = timedelta(seconds=60)
#: Attempts older than this are deleted while we are in the table anyway.
_RATE_RETENTION = timedelta(hours=1)
def _check_rate_limit(ip: str) -> None:
now = time.monotonic()
hits = _LOGIN_HITS[ip]
while hits and now - hits[0] > _RATE_WINDOW:
hits.popleft()
if len(hits) >= _RATE_LIMIT:
def _check_rate_limit(session: Session, ip: str) -> None:
now = datetime.now(timezone.utc)
session.exec(delete(LoginAttempt).where(LoginAttempt.at < now - _RATE_RETENTION))
recent = session.exec(
select(LoginAttempt).where(
LoginAttempt.ip == ip, LoginAttempt.at >= now - _RATE_WINDOW
)
).all()
if len(recent) >= _RATE_LIMIT:
session.commit()
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="Too many login attempts, slow down.",
)
hits.append(now)
session.add(LoginAttempt(ip=ip, at=now))
session.commit()
def _tokens_for(user: User) -> TokenPair:
#: The refresh cookie is scoped to the two endpoints that consume it, so it is
#: not attached to every API call the way a "/" cookie would be.
REFRESH_COOKIE = "stackpilot_refresh"
REFRESH_COOKIE_PATH = "/api/auth"
def _issue(
user: User, response: Response, request: Request, in_body: bool = False
) -> TokenPair:
"""Mint a token pair, putting the refresh token in an httpOnly cookie.
Keeping the long-lived token out of JavaScript's reach means a successful
XSS can no longer walk off with 30 days of access — it is limited to
whatever it can do in the live page. The short-lived access token still
goes to the client, which holds it in memory only.
``in_body`` returns it in the response as well, for scripted clients that
have no cookie jar.
"""
refresh = auth_mod.create_refresh_token(user)
response.set_cookie(
REFRESH_COOKIE,
refresh,
max_age=settings.REFRESH_TOKEN_EXPIRE_DAYS * 24 * 3600,
httponly=True,
# Lax rather than Strict so following a link into StackPilot keeps you
# signed in; the cookie is only ever read by same-site POSTs anyway.
samesite="lax",
# Only when the request actually arrived over TLS — marking it Secure on
# a plain-HTTP homelab deployment would make the browser drop it and
# nobody could stay signed in. request.url.scheme is trustworthy here
# because uvicorn runs with --proxy-headers.
secure=request.url.scheme == "https",
path=REFRESH_COOKIE_PATH,
)
return TokenPair(
access_token=auth_mod.create_access_token(user),
refresh_token=auth_mod.create_refresh_token(user),
refresh_token=refresh if in_body else None,
)
@@ -56,7 +106,11 @@ def needs_setup(session: Session = Depends(get_session)) -> dict:
@router.post("/setup", response_model=TokenPair)
def setup(
body: UserCreate, session: Session = Depends(get_session)
body: UserCreate,
request: Request,
response: Response,
in_body: bool = False,
session: Session = Depends(get_session),
) -> TokenPair:
if auth_mod.users_exist(session):
raise HTTPException(status_code=400, detail="Setup already completed")
@@ -71,17 +125,19 @@ def setup(
audit_service.record(
session, user=user.username, action="user.setup", target=user.username
)
return _tokens_for(user)
return _issue(user, response, request, in_body)
@router.post("/login", response_model=TokenPair)
def login(
body: LoginRequest,
request: Request,
response: Response,
in_body: bool = False,
session: Session = Depends(get_session),
) -> TokenPair:
ip = request.client.host if request.client else "unknown"
_check_rate_limit(ip)
_check_rate_limit(session, ip)
user = auth_mod.authenticate(session, body.username, body.password)
if not user:
raise HTTPException(
@@ -91,20 +147,71 @@ def login(
audit_service.record(
session, user=user.username, action="auth.login", target=user.username, ip=ip
)
return _tokens_for(user)
return _issue(user, response, request, in_body)
@router.post("/refresh", response_model=TokenPair)
def refresh(
body: RefreshRequest, session: Session = Depends(get_session)
request: Request,
response: Response,
body: RefreshRequest | None = None,
in_body: bool = False,
session: Session = Depends(get_session),
) -> TokenPair:
payload = auth_mod.decode_token(body.refresh_token, "refresh")
user = auth_mod.get_user(session, payload.get("sub", ""))
if not user or not user.is_active:
"""Exchange a refresh token for a fresh pair.
Reads the httpOnly cookie; a body is accepted as a fallback for clients
that cannot hold one. The token is re-validated against the live user, so a
password reset or a disabled account takes effect here too rather than at
the end of the token's 30-day life.
"""
token = request.cookies.get(REFRESH_COOKIE) or (body.refresh_token if body else None)
if not token:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid refresh token"
status_code=status.HTTP_401_UNAUTHORIZED, detail="No refresh token"
)
return _tokens_for(user)
payload = auth_mod.decode_token(token, "refresh")
user = auth_mod.resolve_token_user(session, payload)
if not user:
response.delete_cookie(REFRESH_COOKIE, path=REFRESH_COOKIE_PATH)
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Session is no longer valid — sign in again",
)
return _issue(user, response, request, in_body)
@router.post("/logout")
def logout(request: Request, response: Response) -> dict:
"""End the session on this device by dropping the refresh cookie.
Deliberately does not bump ``token_version``: signing out on your phone
should not kill the session on your desktop. Use "sign out everywhere"
for that. The access token is held in memory by the client and dies with
the tab; it stays technically valid for the rest of its hour, which is why
it is short-lived.
"""
response.delete_cookie(REFRESH_COOKIE, path=REFRESH_COOKIE_PATH)
return {"ok": True}
@router.post("/logout-everywhere")
def logout_everywhere(
request: Request,
response: Response,
session: Session = Depends(get_session),
user: User = Depends(auth_mod.get_current_user),
) -> dict:
"""Revoke every token this account holds, on every device."""
auth_mod.bump_token_version(user)
session.add(user)
session.commit()
response.delete_cookie(REFRESH_COOKIE, path=REFRESH_COOKIE_PATH)
audit_service.record(
session, user=user.username, action="auth.logout_everywhere",
target=user.username, ip=_ip(request),
)
return {"ok": True}
@router.get("/me", response_model=UserRead)
@@ -124,7 +231,7 @@ def _ip(request: Request) -> str:
@router.get("/users", response_model=list[UserRead])
def list_users(
session: Session = Depends(get_session),
_admin: User = Depends(auth_mod.require_admin),
_admin: User = Depends(auth_mod.require_admin_session),
) -> list[User]:
return session.exec(select(User).order_by(User.id)).all()
@@ -134,7 +241,7 @@ def create_user(
body: UserCreate,
request: Request,
session: Session = Depends(get_session),
admin: User = Depends(auth_mod.require_admin),
admin: User = Depends(auth_mod.require_admin_session),
) -> User:
if not body.username.strip() or not body.password:
raise HTTPException(status_code=400, detail="Username and password required")
@@ -162,7 +269,7 @@ def update_user(
body: UserUpdate,
request: Request,
session: Session = Depends(get_session),
admin: User = Depends(auth_mod.require_admin),
admin: User = Depends(auth_mod.require_admin_session),
) -> User:
user = session.get(User, user_id)
if not user:
@@ -175,6 +282,15 @@ def update_user(
).first()
if not other_admins:
raise HTTPException(status_code=400, detail="Cannot demote or disable the last active admin")
# Any of these three changes what this account is allowed to do, so the
# tokens it already holds must stop working. Without the bump a password
# reset was cosmetic: whoever had the old tokens kept full access for up to
# 30 days, and a demotion or a disable only took effect once they expired.
authority_changed = (
bool(body.password)
or (body.role is not None and body.role != user.role)
or (body.is_active is not None and body.is_active != user.is_active)
)
if body.password:
user.hashed_password = auth_mod.hash_password(body.password)
if body.role is not None:
@@ -183,6 +299,8 @@ def update_user(
user.role = body.role
if body.is_active is not None:
user.is_active = body.is_active
if authority_changed:
auth_mod.bump_token_version(user)
session.add(user)
session.commit()
session.refresh(user)
@@ -198,7 +316,7 @@ def delete_user(
user_id: int,
request: Request,
session: Session = Depends(get_session),
admin: User = Depends(auth_mod.require_admin),
admin: User = Depends(auth_mod.require_admin_session),
) -> dict:
user = session.get(User, user_id)
if not user:
backend/routers/backups.py
+62 -11
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import asyncio
import json
import os
import tempfile
@@ -29,10 +30,40 @@ def _ip(request: Request) -> str:
return request.client.host if request.client else "unknown"
def _backup_filename(stack_id: str, include_volumes: bool) -> str:
date = compose_service.now().strftime("%Y%m%d-%H%M%S")
suffix = "full" if include_volumes else "config"
return f"backup-{stack_id}-{suffix}-{date}.tar.gz"
_backup_filename = backup_service.backup_filename
def _compact(report: dict) -> dict:
"""The parts of a backup report worth showing the user."""
return {
"size": report.get("size"),
"binds": report.get("binds", []),
"volumes": report.get("volumes", []),
"skipped": report.get("skipped", []),
"path_mismatch": report.get("path_mismatch"),
}
def _summary(report: dict) -> str:
return (
f"binds={len(report.get('binds', []))} "
f"volumes={len(report.get('volumes', []))} "
f"skipped={len(report.get('skipped', []))}"
)
@router.get("/{stack_id}/backup/inventory")
async def backup_inventory(
stack_id: str,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
"""What a backup of this stack would capture: bind-mount sources (with size
and whether they are reachable at all), named volumes, and anything that is
skipped by default with the reason why."""
if not session.get(Stack, stack_id):
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
return await asyncio.to_thread(backup_service.plan, stack_id)
@router.get("/{stack_id}/backup")
@@ -40,7 +71,10 @@ async def backup_stack(
stack_id: str,
request: Request,
include_volumes: bool = Query(True),
include_binds: bool = Query(True),
stop_first: bool = Query(True),
binds: list[str] | None = Query(None),
volumes: list[str] | None = Query(None),
session: Session = Depends(get_session),
user: User = Depends(require_admin),
):
@@ -48,19 +82,24 @@ async def backup_stack(
if not stack:
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
try:
path = await backup_service.create_backup(
stack_id, stack.name, include_volumes=include_volumes, stop_first=stop_first,
path, report = await backup_service.create_backup_ex(
stack_id, stack.name, include_volumes=include_volumes,
stop_first=stop_first, include_binds=include_binds,
binds=binds, volumes=volumes,
)
except backup_service.BackupError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
audit_service.record(
session, user=user.username, action="stack.backup", target=stack_id,
detail=f"volumes={include_volumes}", ip=_ip(request),
detail=_summary(report), ip=_ip(request),
)
return FileResponse(
path,
media_type="application/gzip",
filename=_backup_filename(stack_id, include_volumes),
# The browser downloads a blob, so the summary of what actually made it
# into the archive rides along in a header.
headers={"X-Stackpilot-Backup": json.dumps(_compact(report))},
)
@@ -71,6 +110,7 @@ async def restore_stack(
target_id: str | None = Form(None),
overwrite: bool = Form(False),
restore_volumes: bool = Form(True),
restore_binds: bool = Form(True),
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
@@ -87,6 +127,7 @@ async def restore_stack(
target_id=target,
overwrite=overwrite,
restore_volumes=restore_volumes,
restore_binds=restore_binds,
)
except backup_service.BackupError as exc:
# 409 for the "already exists" conflict, 400 for malformed backups.
@@ -100,7 +141,8 @@ async def restore_stack(
session.commit()
audit_service.record(
session, user=user.username, action="stack.restore", target=stack_id,
detail=f"volumes={result['volumes_restored']}", ip=_ip(request),
detail=f"volumes={result['volumes_restored']} binds={result['binds_restored']}",
ip=_ip(request),
)
return result
finally:
@@ -116,7 +158,10 @@ async def restore_stack(
class PushBody(BaseModel):
destination_id: int
include_volumes: bool = True
include_binds: bool = True
stop_first: bool = True
binds: list[str] | None = None
volumes: list[str] | None = None
class RestoreFromBody(BaseModel):
@@ -125,6 +170,7 @@ class RestoreFromBody(BaseModel):
target_id: str | None = None
overwrite: bool = False
restore_volumes: bool = True
restore_binds: bool = True
def _get_dest(session: Session, dest_id: int) -> BackupDestination:
@@ -147,9 +193,10 @@ async def push_backup(
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
dest = _get_dest(session, body.destination_id)
try:
path = await backup_service.create_backup(
path, report = await backup_service.create_backup_ex(
stack_id, stack.name,
include_volumes=body.include_volumes, stop_first=body.stop_first,
include_binds=body.include_binds, binds=body.binds, volumes=body.volumes,
)
except backup_service.BackupError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@@ -165,9 +212,12 @@ async def push_backup(
audit_service.record(
session, user=user.username, action="stack.backup.push",
target=stack_id, detail=f"{dest.name}:{filename}", ip=_ip(request),
target=stack_id, detail=f"{dest.name}:{filename} {_summary(report)}", ip=_ip(request),
)
return {"ok": True, "destination": dest.name, "name": filename, "remote": remote}
return {
"ok": True, "destination": dest.name, "name": filename,
"remote": remote, "report": _compact(report),
}
@router.post("/restore-from")
@@ -191,6 +241,7 @@ async def restore_from_destination(
result = backup_service.restore_backup(
tmp.name, target_id=target,
overwrite=body.overwrite, restore_volumes=body.restore_volumes,
restore_binds=body.restore_binds,
)
except backup_service.BackupError as exc:
code = 409 if "already exists" in str(exc) else 400
backend/routers/dashboard.py
+23 -13
View File
@@ -1,7 +1,10 @@
"""Dashboard aggregation endpoints (funnel + summary widgets)."""
"""Dashboard aggregation endpoint (fleet-wide cockpit data)."""
from __future__ import annotations
from fastapi import APIRouter, Depends
import logging
import traceback
from fastapi import APIRouter, Depends, HTTPException
from sqlmodel import Session
from auth import get_current_user
@@ -9,21 +12,28 @@ from database import get_session
from models.user import User
from services import dashboard_service
logger = logging.getLogger("stackpilot.dashboard")
router = APIRouter(prefix="/api/dashboard", tags=["dashboard"])
@router.get("/funnel")
async def funnel(
@router.get("/fleet")
async def fleet(
refresh: bool = False,
session: Session = Depends(get_session),
_user: User = Depends(get_current_user),
) -> dict:
return await dashboard_service.compute_funnel(session, refresh=refresh)
@router.get("/summary")
async def summary(
session: Session = Depends(get_session),
_user: User = Depends(get_current_user),
) -> dict:
return await dashboard_service.compute_summary(session)
"""Fleet-wide 'needs attention' list, KPIs and per-host rollup across the
the host — the data behind the operator cockpit."""
try:
return await dashboard_service.compute_fleet(session, refresh=refresh)
except Exception as exc: # noqa: BLE001 — surface the real cause for diagnosis
logger.exception("compute_fleet failed")
# Deepest frame pinpoints where it broke; safe to expose to the
# authenticated user and it makes the dashboard error banner actionable.
tb = traceback.extract_tb(exc.__traceback__)
where = f" at {tb[-1].filename.split('/')[-1]}:{tb[-1].lineno}" if tb else ""
raise HTTPException(
status_code=500,
detail=f"{type(exc).__name__}: {exc}{where}",
) from exc
backend/routers/destinations.py
+5 -4
View File
@@ -2,12 +2,11 @@
from __future__ import annotations
import asyncio
import json
from fastapi import APIRouter, Depends, HTTPException, Request
from sqlmodel import Session, select
from auth import get_current_user, require_admin
from auth import require_admin
from database import get_session
from models.backup_destination import (
DESTINATION_TYPES,
@@ -66,7 +65,9 @@ def create_destination(
) -> DestinationRead:
if body.type not in DESTINATION_TYPES:
raise HTTPException(status_code=400, detail=f"Unknown type '{body.type}'")
d = BackupDestination(name=body.name, type=body.type, config=json.dumps(body.config))
d = BackupDestination(
name=body.name, type=body.type, config=dest_service.dump_config(body.config)
)
session.add(d)
session.commit()
session.refresh(d)
@@ -95,7 +96,7 @@ def update_destination(
if k in SECRET_KEYS and (v == "" or v == "••••••"):
continue # keep existing secret
existing[k] = v
d.config = json.dumps(existing)
d.config = dest_service.dump_config(existing)
session.add(d)
session.commit()
session.refresh(d)
backend/routers/files.py
+36 -8
View File
@@ -1,7 +1,10 @@
"""Full host filesystem browser: list, read, edit, manage, up/download.
Listing and reads require an authenticated user; every mutating operation
(write, mkdir, rename, delete, upload) requires admin and is audit-logged.
Every operation requires admin. Reads are not less dangerous than writes here:
the browser reaches whatever the backend container can see, which includes
every stack's ``.env`` and ``.secrets/*``. Reading a file and downloading one
are audit-logged just like the mutating operations; directory listing is not,
because the Files page polls it and would drown the log.
All paths are sandboxed by :mod:`services.file_service`.
"""
from __future__ import annotations
@@ -19,11 +22,11 @@ from fastapi import (
Request,
UploadFile,
)
from fastapi.responses import FileResponse
from fastapi.responses import FileResponse, StreamingResponse
from pydantic import BaseModel
from sqlmodel import Session
from auth import get_current_user, require_admin
from auth import require_admin
from database import get_session
from models.user import User
from services import audit_service, device_service, file_service
@@ -31,6 +34,12 @@ from services import audit_service, device_service, file_service
router = APIRouter(prefix="/api/files", tags=["files"])
def _attachment(filename: str) -> str:
"""A safe ``Content-Disposition`` value for an arbitrary filename."""
safe = filename.replace("\\", "_").replace('"', "_")
return f'attachment; filename="{safe}"'
def _ip(request: Request) -> str:
return request.client.host if request.client else "unknown"
@@ -51,24 +60,43 @@ def _guard(fn, *args, **kwargs):
def list_dir(
path: str = Query("/"),
show_hidden: bool = Query(False),
_user: User = Depends(get_current_user),
_admin: User = Depends(require_admin),
) -> dict:
return _guard(device_service.browse, path, show_hidden)
@router.get("/read")
def read_file(
request: Request,
path: str = Query(...),
_user: User = Depends(get_current_user),
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
return _guard(file_service.read_file, path)
result = _guard(file_service.read_file, path)
audit_service.record(
session, user=user.username, action="file.read", target=path, ip=_ip(request)
)
return result
@router.get("/download")
def download(
request: Request,
path: str = Query(...),
_user: User = Depends(get_current_user),
session: Session = Depends(get_session),
user: User = Depends(require_admin),
):
audit_service.record(
session, user=user.username, action="file.download", target=path, ip=_ip(request)
)
if _guard(file_service.is_dir, path):
filename, chunks = _guard(file_service.open_archive, path)
# Stream the zip as it's built so the response starts immediately
# (large folders no longer hit the proxy's read timeout).
return StreamingResponse(
chunks, media_type="application/zip",
headers={"Content-Disposition": _attachment(filename)},
)
real, filename = _guard(file_service.resolve_download, path)
return FileResponse(real, filename=filename, media_type="application/octet-stream")
backend/routers/git.py
+239
View File
@@ -0,0 +1,239 @@
"""Deploying stacks from Git.
Everything here is admin-only except the webhook, which cannot be: a Git forge
has no StackPilot credentials to present. It authenticates with an HMAC over the
request body instead, against a secret generated per stack — see
``git_service.verify_webhook``.
"""
from __future__ import annotations
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, Request
from sqlmodel import Session, select
from auth import require_admin
from database import get_session
from models.git_source import (
AUTH_TYPES,
GitSource,
GitSourceRead,
GitSourceWrite,
SyncResult,
)
from models.stack import Stack
from models.user import User
from services import audit_service, crypto_service, git_service
router = APIRouter(prefix="/api/stacks/{stack_id}/git", tags=["git"])
def _ip(request: Request) -> str:
return request.client.host if request.client else "unknown"
def _stack_or_404(session: Session, stack_id: str) -> Stack:
stack = session.get(Stack, stack_id)
if not stack:
raise HTTPException(status_code=404, detail=f"Stack '{stack_id}' not found")
return stack
def _source(session: Session, stack_id: str) -> GitSource:
row = session.exec(select(GitSource).where(GitSource.stack_id == stack_id)).first()
if not row:
raise HTTPException(
status_code=404, detail=f"Stack '{stack_id}' is not connected to a repository"
)
return row
def _to_read(row: GitSource) -> GitSourceRead:
return GitSourceRead(
stack_id=row.stack_id,
url=row.url,
branch=row.branch,
subdir=row.subdir,
auth_type=row.auth_type,
username=row.username,
has_secret=bool(row.secret),
auto_deploy=row.auto_deploy,
poll_interval_minutes=row.poll_interval_minutes,
# Relative on purpose: StackPilot does not know its own external URL,
# and guessing one into a forge's webhook settings would be worse than
# letting the UI prefix the address the admin is already looking at.
webhook_url=f"/api/git/webhook/{row.stack_id}",
last_commit=row.last_commit,
last_synced_at=row.last_synced_at,
last_error=row.last_error,
managed_file_count=len(git_service._managed(row)),
)
@router.get("", response_model=GitSourceRead)
def get_source(
stack_id: str,
session: Session = Depends(get_session),
_user: User = Depends(require_admin),
) -> GitSourceRead:
return _to_read(_source(session, stack_id))
@router.put("", response_model=GitSourceRead)
def connect(
stack_id: str,
body: GitSourceWrite,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> GitSourceRead:
"""Connect a stack to a repository, or change how it is connected.
Does not sync — the caller decides when, because the first sync overwrites
the stack's compose file with whatever the repository says.
"""
_stack_or_404(session, stack_id)
if body.auth_type not in AUTH_TYPES:
raise HTTPException(status_code=400, detail=f"Unknown auth type '{body.auth_type}'")
if not (body.url or "").strip():
raise HTTPException(status_code=400, detail="A repository URL is required")
row = session.exec(select(GitSource).where(GitSource.stack_id == stack_id)).first()
if row is None:
row = GitSource(stack_id=stack_id, url="", webhook_secret=git_service.new_webhook_secret())
row.url = body.url.strip()
row.branch = (body.branch or "main").strip() or "main"
row.subdir = (body.subdir or "").strip().strip("/")
row.auth_type = body.auth_type
row.username = body.username
if body.secret:
row.secret = crypto_service.encrypt(body.secret)
elif body.auth_type == "none":
row.secret = None
row.auto_deploy = body.auto_deploy
row.poll_interval_minutes = body.poll_interval_minutes or None
row.updated_at = datetime.now(timezone.utc)
session.add(row)
session.commit()
session.refresh(row)
audit_service.record(
session, user=user.username, action="stack.git-connect", target=stack_id,
detail=f"{row.url}#{row.branch}", ip=_ip(request),
)
return _to_read(row)
@router.delete("")
def disconnect(
stack_id: str,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
"""Stop tracking the repository. The stack's files are left exactly as they are."""
row = _source(session, stack_id)
session.delete(row)
session.commit()
git_service.forget(stack_id)
audit_service.record(
session, user=user.username, action="stack.git-disconnect", target=stack_id,
ip=_ip(request),
)
return {"ok": True}
@router.post("/sync", response_model=SyncResult)
async def sync_now(
stack_id: str,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> SyncResult:
row = _source(session, stack_id)
try:
result = await git_service.sync(session, row, actor=user.username)
except git_service.GitError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
audit_service.record(
session, user=user.username, action="stack.git-sync", target=stack_id,
detail=f"{(result.commit or '')[:8]} {'changed' if result.changed else 'no change'}",
ip=_ip(request),
)
return result
@router.get("/webhook-secret")
def reveal_webhook_secret(
stack_id: str,
session: Session = Depends(get_session),
_user: User = Depends(require_admin),
) -> dict:
"""The secret to paste into the forge's webhook settings.
Readable rather than shown-once: it lives in the forge's configuration too,
so hiding it here would only mean re-pointing the webhook to see it again.
"""
return {"secret": _source(session, stack_id).webhook_secret}
@router.post("/webhook-secret")
def rotate_webhook_secret(
stack_id: str,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
row = _source(session, stack_id)
row.webhook_secret = git_service.new_webhook_secret()
row.updated_at = datetime.now(timezone.utc)
session.add(row)
session.commit()
audit_service.record(
session, user=user.username, action="stack.git-rotate-secret", target=stack_id,
ip=_ip(request),
)
return {"secret": row.webhook_secret}
# --------------------------------------------------------------------------- #
# The webhook
# --------------------------------------------------------------------------- #
hook_router = APIRouter(prefix="/api/git", tags=["git"])
@hook_router.post("/webhook/{stack_id}")
async def webhook(
stack_id: str,
request: Request,
session: Session = Depends(get_session),
) -> dict:
"""Push webhook from a Git forge.
Unauthenticated in the usual sense — a forge holds no StackPilot session —
and authorized by an HMAC over the body instead. An unsigned or wrongly
signed call is a 404, not a 403: without credentials to present, telling a
caller that a given stack *is* connected to a repository is information it
has not earned.
"""
row = session.exec(select(GitSource).where(GitSource.stack_id == stack_id)).first()
body = await request.body()
if not row or not git_service.verify_webhook(row, body, request.headers):
raise HTTPException(status_code=404, detail="Not found")
try:
result = await git_service.sync(session, row, actor="webhook")
except git_service.GitError as exc:
raise HTTPException(status_code=502, detail=str(exc)) from exc
audit_service.record(
session, user="webhook", action="stack.git-sync", target=stack_id,
detail=f"{(result.commit or '')[:8]} {'changed' if result.changed else 'no change'}",
ip=_ip(request),
)
return {
"ok": True,
"changed": result.changed,
"commit": result.commit,
"deployed": result.deployed,
}
backend/routers/registries.py
+186
View File
@@ -0,0 +1,186 @@
"""Private registry credentials.
Admin-only throughout, including the reads: even masked, the rows say which
registries this install talks to and under what account.
"""
from __future__ import annotations
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, Request
from sqlmodel import Session, select
from auth import require_admin
from database import get_session
from models.registry import (
Registry,
RegistryCreate,
RegistryRead,
RegistryTestRequest,
RegistryUpdate,
)
from models.user import User
from services import audit_service, crypto_service, registry_service
router = APIRouter(prefix="/api/registries", tags=["registries"])
def _ip(request: Request) -> str:
return request.client.host if request.client else "unknown"
def _to_read(row: Registry) -> RegistryRead:
# The password never leaves the server, not even masked — the UI only needs
# to know whether one is stored, so it can leave the field blank on edit.
return RegistryRead(
id=row.id,
name=row.name,
host=row.host,
username=row.username,
has_password=bool(row.password),
created_at=row.created_at,
updated_at=row.updated_at,
)
def _get_or_404(session: Session, registry_id: int) -> Registry:
row = session.get(Registry, registry_id)
if not row:
raise HTTPException(status_code=404, detail=f"Registry {registry_id} not found")
return row
def _canonical(host: str) -> str:
try:
return registry_service.canonical_host(host)
except registry_service.RegistryError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@router.get("", response_model=list[RegistryRead])
def list_registries(
session: Session = Depends(get_session),
_user: User = Depends(require_admin),
) -> list[RegistryRead]:
rows = session.exec(select(Registry).order_by(Registry.host)).all()
return [_to_read(r) for r in rows]
@router.post("", response_model=RegistryRead, status_code=201)
def create_registry(
body: RegistryCreate,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> RegistryRead:
host = _canonical(body.host)
if session.exec(select(Registry).where(Registry.host == host)).first():
# One set of credentials per registry: two rows for the same host would
# make "which account are we using" unanswerable.
raise HTTPException(
status_code=409, detail=f"Credentials for '{host}' already exist"
)
if not body.username or not body.password:
raise HTTPException(status_code=400, detail="Username and password are required")
row = Registry(
name=body.name or host,
host=host,
username=body.username,
password=crypto_service.encrypt(body.password),
)
session.add(row)
session.commit()
session.refresh(row)
registry_service.reload(session)
audit_service.record(
session, user=user.username, action="registry.create", target=host,
detail=f"as {body.username}", ip=_ip(request),
)
return _to_read(row)
@router.put("/{registry_id}", response_model=RegistryRead)
def update_registry(
registry_id: int,
body: RegistryUpdate,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> RegistryRead:
row = _get_or_404(session, registry_id)
if body.host is not None:
host = _canonical(body.host)
clash = session.exec(select(Registry).where(Registry.host == host)).first()
if clash and clash.id != row.id:
raise HTTPException(
status_code=409, detail=f"Credentials for '{host}' already exist"
)
row.host = host
if body.name is not None:
row.name = body.name
if body.username is not None:
row.username = body.username
# An omitted password keeps the stored one: the UI never received it, so it
# cannot send it back.
if body.password:
row.password = crypto_service.encrypt(body.password)
row.updated_at = datetime.now(timezone.utc)
session.add(row)
session.commit()
session.refresh(row)
registry_service.reload(session)
audit_service.record(
session, user=user.username, action="registry.update", target=row.host,
ip=_ip(request),
)
return _to_read(row)
@router.delete("/{registry_id}")
def delete_registry(
registry_id: int,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
row = _get_or_404(session, registry_id)
host = row.host
session.delete(row)
session.commit()
# Rewrites config.json without this host, so the CLI loses the login too.
registry_service.reload(session)
audit_service.record(
session, user=user.username, action="registry.delete", target=host,
ip=_ip(request),
)
return {"ok": True}
@router.post("/test")
async def test_credentials(
body: RegistryTestRequest,
session: Session = Depends(get_session),
_user: User = Depends(require_admin),
) -> dict:
"""Try a set of credentials against the registry.
With no password in the body, the stored one for that host is used — that is
how the UI can re-test a saved registry it never received the password for.
"""
host = _canonical(body.host)
password = body.password
username = body.username
if not password:
stored = session.exec(select(Registry).where(Registry.host == host)).first()
if not stored:
raise HTTPException(status_code=400, detail="A password is required")
try:
password = crypto_service.decrypt(stored.password)
except crypto_service.DecryptError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
username = username or stored.username
try:
await registry_service.verify(host, username, password)
except registry_service.RegistryError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return {"ok": True, "host": host}
backend/routers/schedules.py
+1 -9
View File
@@ -6,7 +6,6 @@ from sqlmodel import Session, select
from auth import require_admin
from database import get_session
from models.agent import Agent
from models.backup_destination import BackupDestination
from models.backup_schedule import (
FREQUENCIES,
@@ -28,14 +27,11 @@ def _ip(request: Request) -> str:
def _to_read(session: Session, s: BackupSchedule) -> ScheduleRead:
dest = session.get(BackupDestination, s.destination_id)
agent = session.get(Agent, s.agent_id) if s.agent_id is not None else None
return ScheduleRead(
id=s.id,
stack_id=s.stack_id,
destination_id=s.destination_id,
destination_name=dest.name if dest else None,
agent_id=s.agent_id,
agent_name=agent.name if agent else None,
frequency=s.frequency,
hour=s.hour,
minute=s.minute,
@@ -63,11 +59,7 @@ def _validate(session: Session, schedule: BackupSchedule) -> None:
raise HTTPException(status_code=400, detail=f"Unknown frequency '{schedule.frequency}'")
if not session.get(BackupDestination, schedule.destination_id):
raise HTTPException(status_code=404, detail=f"Destination {schedule.destination_id} not found")
if schedule.agent_id is not None:
# Remote stack: validate the agent exists; the stack is checked at run time.
if not session.get(Agent, schedule.agent_id):
raise HTTPException(status_code=404, detail=f"Agent {schedule.agent_id} not found")
elif not session.get(Stack, schedule.stack_id):
if not session.get(Stack, schedule.stack_id):
raise HTTPException(status_code=404, detail=f"Stack '{schedule.stack_id}' not found")
backend/routers/secrets.py
+1 -2
View File
@@ -7,7 +7,7 @@ to take effect on running containers.
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from sqlmodel import Session
@@ -51,7 +51,6 @@ def _guard(fn, *args, **kwargs):
raise HTTPException(status_code=400, detail=str(exc)) from exc
# --- These functions are shared verbatim by the agent (see agent_app.py). ---
def list_secrets(stack_id: str) -> list[dict]:
backend/routers/stacks.py
+218 -21
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
import os
from dataclasses import asdict
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi import APIRouter, Depends, File, HTTPException, Query, Request, UploadFile
from fastapi.responses import FileResponse
from sqlmodel import Session, select
@@ -27,7 +27,17 @@ from models.setting import (
)
from models.auto_update import AutoUpdateRead, AutoUpdateWrite
from models.user import User
from services import audit_service, auto_update_service, compose_service, notify_service, stats_service
from services import (
audit_service,
auto_update_service,
compose_service,
icon_service,
logo_service,
notify_service,
stack_lock_service,
stats_service,
update_service,
)
from services.convert_service import convert_docker_run
router = APIRouter(prefix="/api/stacks", tags=["stacks"])
@@ -59,13 +69,25 @@ def _get_stack_or_404(session: Session, stack_id: str) -> Stack:
return stack
def _stack_summary(stack: Stack, summaries: dict | None = None) -> dict:
def _auto_icon(stack: Stack) -> str | None:
"""The logo a stack gets when nothing is configured, as an icon value."""
if stack.icon:
return None
slug = logo_service.auto_slug(stack.id, stack.name)
return f"logo:{slug}" if slug else None
def _stack_summary(
stack: Stack, summaries: dict | None = None, busy: dict[str, str] | None = None
) -> dict:
"""Build a list-row summary.
Pass ``summaries`` (from :func:`compose_service.stack_status_summaries`) to
serve the whole stacks list from a single Docker call. Without it (single
Pass ``summaries`` (from :func:`compose_service.stack_status_summaries`) and
``busy`` (from :func:`stack_lock_service.active`) to serve the whole stacks
list from one Docker call and one query. Without them (single
create/update/clone responses), fall back to one direct query for this stack.
"""
busy = busy or {}
if summaries is None:
try:
containers = compose_service.containers_for_stack(stack.id)
@@ -79,7 +101,7 @@ def _stack_summary(stack: Stack, summaries: dict | None = None) -> dict:
info = summaries.get(stack.id)
total = info["total"] if info else 0
running = info["running"] if info else 0
if compose_service.is_busy(stack.id):
if stack.id in busy:
status = "updating"
else:
status = info["status"] if info else "stopped"
@@ -87,6 +109,10 @@ def _stack_summary(stack: Stack, summaries: dict | None = None) -> dict:
"id": stack.id,
"name": stack.name,
"description": stack.description,
"icon": stack.icon,
# With no explicit choice, the app logo the name resolves to (the
# frontend falls back to a name-derived glyph when this is null).
"auto_icon": _auto_icon(stack),
"status": status,
"service_count": total,
"running_count": running,
@@ -111,7 +137,8 @@ def list_stacks(
summaries = compose_service.stack_status_summaries()
except DockerError:
summaries = {}
return [_stack_summary(s, summaries) for s in stacks]
busy = stack_lock_service.active(session)
return [_stack_summary(s, summaries, busy) for s in stacks]
@router.post("", status_code=201)
@@ -124,10 +151,16 @@ def create_stack(
stack_id = compose_service.slugify(body.name)
if session.get(Stack, stack_id) or os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=409, detail=f"Stack '{stack_id}' already exists")
try:
icon = icon_service.normalize_choice(body.icon or "")
except icon_service.IconError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
compose_service.write_compose(stack_id, body.yaml or "services:\n")
if body.env:
compose_service.write_env(stack_id, body.env)
stack = Stack(id=stack_id, name=body.name, description=body.description)
stack = Stack(
id=stack_id, name=body.name, description=body.description, icon=icon
)
session.add(stack)
session.commit()
session.refresh(stack)
@@ -144,11 +177,48 @@ def stacks_stats(_user: User = Depends(get_current_user)) -> dict:
return stats_service.stack_stats()
@router.get("/updates")
def stacks_updates(_user: User = Depends(get_current_user)) -> dict:
"""Per-stack image-update availability, read from the cached registry
digests (no live registry calls — safe for the list to poll)."""
return update_service.stacks_update_summary()
@router.get("/icons/search")
def search_app_logos(
q: str = Query("", max_length=64),
limit: int = Query(60, ge=1, le=200),
_user: User = Depends(get_current_user),
) -> dict:
"""Search the app-logo catalog (Jellyfin, Postgres, Gitea, …).
``ready`` is false when the catalog has not been downloaded yet — a box with
no outbound internet, or the very first minute after a fresh install. The
picker says so instead of looking empty and broken.
"""
return {
"ready": logo_service.load_catalog() is not None,
"icons": logo_service.search(q, limit),
}
@router.get("/icons/logo/{slug}")
async def get_app_logo(
slug: str,
_user: User = Depends(get_current_user),
) -> FileResponse:
"""One catalog logo by slug, for the picker's result grid."""
path = await logo_service.ensure_logo(slug)
if not path:
raise HTTPException(status_code=404, detail=f"No logo for '{slug}'")
return _icon_response(path, "image/png", f"{slug}.png")
@router.get("/{stack_id}")
def get_stack(
stack_id: str,
session: Session = Depends(get_session),
_user: User = Depends(get_current_user),
user: User = Depends(get_current_user),
) -> dict:
stack = _get_stack_or_404(session, stack_id)
try:
@@ -158,13 +228,21 @@ def get_stack(
except DockerError:
containers = []
status = "unknown"
# An operation in flight outranks whatever the containers currently say.
if stack_lock_service.is_busy(session, stack_id):
status = "updating"
return {
"id": stack.id,
"name": stack.name,
"description": stack.description,
"icon": stack.icon,
"auto_icon": _auto_icon(stack),
"status": status,
"yaml": compose_service.read_compose(stack_id),
"env": compose_service.read_env(stack_id),
# The .env is where credentials live by convention, so it is withheld
# from the read-only role — same reasoning as the admin-only file
# browser. Non-admins still get status, services and the compose file.
"env": compose_service.read_env(stack_id) if user.role == "admin" else "",
"containers": containers,
"created_at": stack.created_at,
"updated_at": stack.updated_at,
@@ -188,6 +266,16 @@ def update_stack(
stack.name = body.name
if body.description is not None:
stack.description = body.description
if body.icon is not None:
try:
icon = icon_service.normalize_choice(body.icon)
except icon_service.IconError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
# Switching to a built-in icon (or back to automatic) makes any
# uploaded image dead weight, so it goes with the choice.
if icon_service.custom_ext(stack.icon) and icon != stack.icon:
icon_service.remove(stack_id)
stack.icon = icon
stack.updated_at = compose_service.now()
session.add(stack)
session.commit()
@@ -214,6 +302,8 @@ async def delete_stack(
pass
if delete_files:
compose_service.delete_stack_files(stack_id)
icon_service.remove(stack_id)
logo_service.forget(stack_id)
session.delete(stack)
session.commit()
audit_service.record(
@@ -231,7 +321,7 @@ def clone_stack(
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
_get_stack_or_404(session, stack_id)
source = _get_stack_or_404(session, stack_id)
new_id = compose_service.slugify(body.name)
if session.get(Stack, new_id):
raise HTTPException(status_code=409, detail=f"Stack '{new_id}' already exists")
@@ -239,7 +329,11 @@ def clone_stack(
compose_service.clone_stack_files(stack_id, new_id)
except compose_service.StackFileError as exc:
raise HTTPException(status_code=409, detail=str(exc)) from exc
stack = Stack(id=new_id, name=body.name)
stack = Stack(
id=new_id,
name=body.name,
icon=icon_service.copy(stack_id, new_id, source.icon),
)
session.add(stack)
session.commit()
session.refresh(stack)
@@ -250,6 +344,94 @@ def clone_stack(
return _stack_summary(stack)
# --------------------------------------------------------------------------- #
# icon
# --------------------------------------------------------------------------- #
@router.get("/{stack_id}/icon")
async def get_stack_icon(
stack_id: str,
session: Session = Depends(get_session),
_user: User = Depends(get_current_user),
) -> FileResponse:
"""Serve a stack's image icon — an upload, or the app logo it resolved to.
Authenticated like everything else, which is why the frontend fetches it
through the API client and renders the blob rather than pointing an
``<img src>`` straight at this URL (that request would carry no token).
It is also what keeps the browser off the icon CDN: an app logo is
downloaded once by this process and served from disk from then on.
"""
stack = _get_stack_or_404(session, stack_id)
if (path := icon_service.file_for(stack_id, stack.icon)):
ext = icon_service.custom_ext(stack.icon) or ""
return _icon_response(path, icon_service.content_type(ext), f"{stack_id}.{ext}")
slug = icon_service.logo_slug(stack.icon) or icon_service.logo_slug(_auto_icon(stack))
if slug and (path := await logo_service.ensure_logo(slug)):
return _icon_response(path, "image/png", f"{slug}.png")
raise HTTPException(status_code=404, detail="This stack has no image icon")
def _icon_response(path: str, media_type: str, filename: str) -> FileResponse:
return FileResponse(
path,
media_type=media_type,
# An SVG opened as a top-level document would run its own script in the
# API's origin. Nothing here is ever meant to be a document.
headers={"Content-Disposition": f'attachment; filename="{filename}"'},
)
@router.post("/{stack_id}/icon")
async def upload_stack_icon(
stack_id: str,
request: Request,
file: UploadFile = File(...),
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
"""Replace a stack's icon with an uploaded image."""
stack = _get_stack_or_404(session, stack_id)
data = await file.read(icon_service.MAX_ICON_BYTES + 1)
try:
stack.icon = icon_service.store_upload(stack_id, data)
except icon_service.IconError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
stack.updated_at = compose_service.now()
session.add(stack)
session.commit()
session.refresh(stack)
audit_service.record(
session, user=user.username, action="stack.icon", target=stack_id,
detail=f"uploaded {file.filename or 'image'}", ip=_client_ip(request),
)
return _stack_summary(stack)
@router.delete("/{stack_id}/icon")
def reset_stack_icon(
stack_id: str,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
"""Drop any explicit choice and go back to the name-derived icon."""
stack = _get_stack_or_404(session, stack_id)
icon_service.remove(stack_id)
stack.icon = None
stack.updated_at = compose_service.now()
session.add(stack)
session.commit()
session.refresh(stack)
audit_service.record(
session, user=user.username, action="stack.icon", target=stack_id,
detail="reset to automatic", ip=_client_ip(request),
)
return _stack_summary(stack)
# --------------------------------------------------------------------------- #
# lifecycle
# --------------------------------------------------------------------------- #
@@ -286,7 +468,17 @@ async def _notify_lifecycle(action_name: str, stack_id: str, ok: bool, detail: s
async def _lifecycle(action_fn, action_name, stack_id, request, session, user):
_get_stack_or_404(session, stack_id)
result = await action_fn(stack_id)
# One compose operation per stack. Without this two tabs (or auto-update
# landing on a stack somebody just clicked) both run pull + up -d against
# the same project and race over recreating containers.
try:
with stack_lock_service.hold(session, stack_id, action_name, user.username):
result = await action_fn(stack_id)
except stack_lock_service.StackBusy as exc:
raise HTTPException(
status_code=409,
detail=f"Stack '{stack_id}' is busy: {exc.action} in progress",
) from exc
audit_service.record(
session, user=user.username, action=f"stack.{action_name}", target=stack_id,
detail=f"rc={result.get('returncode')}", ip=_client_ip(request),
@@ -322,12 +514,16 @@ async def restart_stack(stack_id: str, request: Request, session: Session = Depe
@router.post("/{stack_id}/pull")
async def pull_stack(stack_id: str, request: Request, session: Session = Depends(get_session), user: User = Depends(require_admin)):
return await _lifecycle(compose_service.pull, "pull", stack_id, request, session, user)
result = await _lifecycle(compose_service.pull, "pull", stack_id, request, session, user)
update_service.refresh_stack_local(stack_id)
return result
@router.post("/{stack_id}/update")
async def update_stack_images(stack_id: str, request: Request, session: Session = Depends(get_session), user: User = Depends(require_admin)):
return await _lifecycle(compose_service.update, "update", stack_id, request, session, user)
result = await _lifecycle(compose_service.update, "update", stack_id, request, session, user)
update_service.refresh_stack_local(stack_id)
return result
@router.post("/{stack_id}/down")
@@ -369,13 +565,14 @@ async def service_logs(
def export_stack(
stack_id: str,
session: Session = Depends(get_session),
_user: User = Depends(get_current_user),
_admin: User = Depends(require_admin),
):
import io
"""Download the whole stack folder as a tarball. Admin only: the archive
contains the ``.env`` and every ``.secrets/*`` file verbatim."""
import tarfile
import tempfile
stack = _get_stack_or_404(session, stack_id)
_get_stack_or_404(session, stack_id) # 404s if unknown
directory = compose_service.stack_dir(stack_id)
if not os.path.isdir(directory):
raise HTTPException(status_code=404, detail="Stack directory missing")
@@ -413,7 +610,7 @@ def get_auto_update(
_user: User = Depends(get_current_user),
) -> dict:
policy = auto_update_service.get_policy(session, stack_id)
return auto_update_service.to_read(session, policy, stack_id)
return auto_update_service.to_read(policy, stack_id)
@router.put("/{stack_id}/auto-update", response_model=AutoUpdateRead)
@@ -430,7 +627,7 @@ def set_auto_update(
target=stack_id, detail=f"enabled={body.enabled} redeploy={body.redeploy}",
ip=_client_ip(request),
)
return auto_update_service.to_read(session, policy, stack_id)
return auto_update_service.to_read(policy, stack_id)
@router.post("/{stack_id}/auto-update/run", response_model=AutoUpdateRead)
@@ -444,4 +641,4 @@ async def run_auto_update(
raise HTTPException(status_code=404, detail="No auto-update policy for this stack")
await auto_update_service.run_policy(session, policy)
session.refresh(policy)
return auto_update_service.to_read(session, policy, stack_id)
return auto_update_service.to_read(policy, stack_id)
backend/routers/system.py
+33 -3
View File
@@ -4,13 +4,15 @@ from __future__ import annotations
import os
import shutil
from fastapi import APIRouter, Depends
from fastapi import APIRouter, Depends, HTTPException, Request
from sqlmodel import Session
from auth import get_current_user
from auth import get_current_user, require_admin
from config import settings
from database import get_session
from docker_client import DockerError, get_client, safe_call
from models.user import User
from services import device_service, gpu_service
from services import audit_service, device_service, gpu_service, self_update_service
router = APIRouter(prefix="/api/system", tags=["system"])
@@ -103,3 +105,31 @@ def gpus(_user: User = Depends(get_current_user)) -> list[dict]:
def devices(_user: User = Depends(get_current_user)) -> dict:
"""List host USB / serial / DRI devices for passthrough."""
return device_service.detect_devices()
@router.get("/update")
async def self_update_status(
refresh: bool = False,
_user: User = Depends(get_current_user),
) -> dict:
"""Is a newer StackPilot release available? (registry check, cached)"""
return await self_update_service.get_status(refresh=refresh)
@router.post("/update")
def self_update_apply(
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
"""Update this StackPilot in place via a detached compose helper."""
try:
result = self_update_service.apply_update()
except self_update_service.SelfUpdateError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
audit_service.record(
session, user=user.username, action="system.update",
target=result.get("helper", ""), detail=result.get("command"),
ip=request.client.host if request.client else "",
)
return result
backend/routers/templates.py
+55 -41
View File
@@ -1,4 +1,8 @@
"""Template library endpoints."""
"""Template library endpoints.
Templates are stack-shaped folders on disk. Listing reads them; "instantiate"
(pull) copies the whole folder into a new stack, which is then editable.
"""
from __future__ import annotations
import os
@@ -8,12 +12,14 @@ from sqlmodel import Session
from auth import get_current_user, require_admin
from database import get_session
from models.agent import Agent
from models.stack import Stack
from models.template import TemplateInstantiateRequest, TemplateSaveRequest
from models.template import (
TemplateFromStackRequest,
TemplateInstantiateRequest,
TemplateSaveRequest,
)
from models.user import User
from services import agent_service, audit_service, compose_service, template_service
from services.agent_service import AgentError
from services import audit_service, compose_service, template_service
router = APIRouter(prefix="/api/templates", tags=["templates"])
@@ -24,19 +30,21 @@ def _ip(request: Request) -> str:
@router.get("")
def list_templates(
session: Session = Depends(get_session),
_user: User = Depends(get_current_user),
) -> list[dict]:
return template_service.list_templates(session)
return template_service.list_templates()
@router.get("/{template_id}")
def get_template(
template_id: str,
session: Session = Depends(get_session),
_user: User = Depends(get_current_user),
_admin: User = Depends(require_admin),
) -> dict:
tpl = template_service.get_template(session, template_id)
"""Full template incl. compose and env. Admin only: "save stack as
template" snapshots the stack's real ``.env`` into the template, so this
can carry live credentials. Only admins can instantiate a template anyway;
the listing above stays open to everyone."""
tpl = template_service.get_template(template_id)
if not tpl:
raise HTTPException(status_code=404, detail="Template not found")
return tpl
@@ -49,13 +57,32 @@ def save_template(
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
tpl = template_service.save_custom(
session, body.name, body.yaml, body.description or "", body.tags
slug = template_service.save_custom(
body.name, body.compose, body.env, body.description or "", body.tags, body.gpu
)
audit_service.record(
session, user=user.username, action="template.save", target=tpl.slug, ip=_ip(request)
session, user=user.username, action="template.save", target=slug, ip=_ip(request)
)
return {"id": f"custom:{tpl.slug}", "name": tpl.name}
return {"id": f"custom:{slug}", "name": body.name}
@router.post("/from-stack")
def save_from_stack(
body: TemplateFromStackRequest,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
if not session.get(Stack, body.stack_id) and not os.path.isdir(
compose_service.stack_dir(body.stack_id)
):
raise HTTPException(status_code=404, detail=f"Stack '{body.stack_id}' not found")
slug = template_service.save_from_stack(body.stack_id, body.name, body.description or "")
audit_service.record(
session, user=user.username, action="template.save",
target=slug, detail=body.stack_id, ip=_ip(request),
)
return {"id": f"custom:{slug}", "name": body.name}
@router.delete("/custom/{slug}")
@@ -65,7 +92,7 @@ def delete_template(
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
if not template_service.delete_custom(session, slug):
if not template_service.delete_custom(slug):
raise HTTPException(status_code=404, detail="Custom template not found")
audit_service.record(
session, user=user.username, action="template.delete", target=slug, ip=_ip(request)
@@ -74,44 +101,31 @@ def delete_template(
@router.post("/{template_id}/instantiate", status_code=201)
async def instantiate(
def instantiate(
template_id: str,
body: TemplateInstantiateRequest,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin),
) -> dict:
tpl = template_service.get_template(session, template_id)
tpl = template_service.get_template(template_id)
if not tpl:
raise HTTPException(status_code=404, detail="Template not found")
rendered = template_service.render(tpl["yaml"], body.values)
if body.agent_id is not None:
agent = session.get(Agent, body.agent_id)
if not agent:
raise HTTPException(status_code=404, detail=f"Agent {body.agent_id} not found")
try:
result = await agent_service.call(
session, agent, "POST", "/agent/stacks",
json={"name": body.name, "yaml": rendered, "env": None},
)
except AgentError as exc:
raise HTTPException(
status_code=exc.status if exc.status >= 400 else 502,
detail={"error": exc.error, "detail": exc.detail},
)
audit_service.record(
session, user=user.username, action="template.instantiate",
target=f"{agent.name}/{result.get('id')}", detail=template_id, ip=_ip(request),
)
return {"id": result.get("id"), "name": body.name, "agent_id": agent.id}
# Copy the whole template folder into a new stack.
stack_id = compose_service.slugify(body.name)
if session.get(Stack, stack_id) or os.path.isdir(compose_service.stack_dir(stack_id)):
raise HTTPException(status_code=409, detail=f"Stack '{stack_id}' already exists")
compose_service.write_compose(stack_id, rendered)
try:
template_service.copy_into_stack(template_id, stack_id)
except FileExistsError as exc:
raise HTTPException(
status_code=409, detail=f"Stack '{stack_id}' already exists"
) from exc
except FileNotFoundError as exc:
raise HTTPException(status_code=404, detail="Template not found") from exc
stack = Stack(id=stack_id, name=body.name, description=tpl.get("description"))
session.add(stack)
session.commit()
@@ -119,4 +133,4 @@ async def instantiate(
session, user=user.username, action="template.instantiate",
target=stack_id, detail=template_id, ip=_ip(request),
)
return {"id": stack_id, "name": body.name, "agent_id": None}
return {"id": stack_id, "name": body.name}
backend/routers/tokens.py
+104
View File
@@ -0,0 +1,104 @@
"""API tokens for scripts and CI.
Managing tokens needs a signed-in session, never another API token: a leaked CI
credential should be able to do the job it was issued for, not mint itself a
second one that survives the first being revoked.
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException, Request
from sqlmodel import Session, select
from auth import require_admin_session
from database import get_session
from models.api_token import (
SCOPES,
ApiToken,
ApiTokenCreate,
ApiTokenCreated,
ApiTokenRead,
)
from models.user import User
from services import api_token_service, audit_service
router = APIRouter(prefix="/api/auth/tokens", tags=["auth"])
def _ip(request: Request) -> str:
return request.client.host if request.client else "unknown"
def _to_read(row: ApiToken, session: Session) -> ApiTokenRead:
owner = session.get(User, row.user_id)
return ApiTokenRead(
id=row.id,
name=row.name,
prefix=row.prefix,
scope=row.scope,
username=owner.username if owner else "(deleted)",
expires_at=row.expires_at,
last_used_at=row.last_used_at,
created_at=row.created_at,
expired=api_token_service.is_expired(row),
)
@router.get("", response_model=list[ApiTokenRead])
def list_tokens(
session: Session = Depends(get_session),
_user: User = Depends(require_admin_session),
) -> list[ApiTokenRead]:
rows = session.exec(select(ApiToken).order_by(ApiToken.created_at.desc())).all()
return [_to_read(r, session) for r in rows]
@router.post("", response_model=ApiTokenCreated, status_code=201)
def create_token(
body: ApiTokenCreate,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin_session),
) -> ApiTokenCreated:
name = (body.name or "").strip()
if not name:
raise HTTPException(status_code=400, detail="A name is required")
if body.scope not in SCOPES:
raise HTTPException(
status_code=400, detail=f"Scope must be one of {', '.join(SCOPES)}"
)
if body.expires_in_days is not None and body.expires_in_days < 1:
raise HTTPException(status_code=400, detail="Expiry must be at least a day")
row, token = api_token_service.mint(
session,
name=name,
user=user,
scope=body.scope,
expires_in_days=body.expires_in_days,
)
audit_service.record(
session, user=user.username, action="token.create", target=row.prefix,
detail=f"{name} ({row.scope})", ip=_ip(request),
)
# The only time the token itself is ever returned.
return ApiTokenCreated(**_to_read(row, session).model_dump(), token=token)
@router.delete("/{token_id}")
def revoke_token(
token_id: int,
request: Request,
session: Session = Depends(get_session),
user: User = Depends(require_admin_session),
) -> dict:
row = session.get(ApiToken, token_id)
if not row:
raise HTTPException(status_code=404, detail=f"Token {token_id} not found")
prefix = row.prefix
session.delete(row)
session.commit()
audit_service.record(
session, user=user.username, action="token.revoke", target=prefix,
detail=row.name, ip=_ip(request),
)
return {"ok": True}
backend/routers/volumes.py
+4 -1
View File
@@ -98,8 +98,11 @@ def generate_yaml(
def host_paths(
path: str = Query("/"),
show_hidden: bool = Query(False),
_user: User = Depends(get_current_user),
_admin: User = Depends(require_admin),
) -> dict:
"""Directory picker for the volume wizard. Same browse() as the file
browser, so it carries the same admin requirement — and only admins can
create a volume with the result anyway."""
try:
return device_service.browse(path, show_hidden)
except device_service.BrowseError as exc:
backend/routers/ws.py
+157 -249
View File
@@ -4,34 +4,54 @@ from __future__ import annotations
import asyncio
import json
import logging
import urllib.parse
import contextlib
import websockets
from fastapi import APIRouter, Query, WebSocket, WebSocketDisconnect
from jose import JWTError
from sqlmodel import Session
from auth import decode_token
from auth import decode_token, resolve_token_user
from database import engine
from models.agent import Agent
from models.setting import EVENT_STACK_ERROR, EVENT_STACK_START
from services import audit_service, compose_service, exec_service, notify_service
from models.setting import EVENT_PULL_FAILED, EVENT_STACK_ERROR, EVENT_STACK_START
from services import (
audit_service,
compose_service,
exec_service,
notify_service,
stack_lock_service,
update_service,
)
logger = logging.getLogger("stackpilot.ws")
router = APIRouter(tags=["ws"])
def _user_for(token: str | None):
"""The live user behind a socket's token, or None.
Resolves against the database rather than reading the role straight off the
JWT: a socket can outlive a demotion, a disabled account or a password
reset, and the exec endpoint below is root-equivalent on the host. Same
check the HTTP routes make.
"""
if not token:
return None
try:
payload = decode_token(token, "access")
except (JWTError, Exception): # noqa: BLE001
return None
with Session(engine) as session:
user = resolve_token_user(session, payload)
if user:
session.expunge(user)
return user
async def _authorize(websocket: WebSocket, token: str | None) -> bool:
"""Validate the JWT supplied as a query param. Closes socket on failure."""
if not token:
await websocket.close(code=4401)
return False
try:
decode_token(token, "access")
except (JWTError, Exception): # noqa: BLE001
if _user_for(token) is None:
await websocket.close(code=4401)
return False
return True
@@ -41,15 +61,11 @@ async def _authorize_admin(websocket: WebSocket, token: str | None) -> bool:
"""Like _authorize but also requires the admin role (exec is root-equivalent).
Closes 4401 on a missing/invalid token, 4403 on a valid non-admin token."""
if not token:
user = _user_for(token)
if user is None:
await websocket.close(code=4401)
return False
try:
payload = decode_token(token, "access")
except (JWTError, Exception): # noqa: BLE001
await websocket.close(code=4401)
return False
if payload.get("role") != "admin":
if user.role != "admin":
await websocket.close(code=4403)
return False
return True
@@ -127,7 +143,18 @@ async def ws_deploy(
rc: int | None = None
disconnected = False
compose_service.mark_busy(stack_id)
# Same guard the REST lifecycle uses — the deploy console runs the very
# same `compose up`, so it has to queue behind an in-flight operation
# rather than race it.
lock_session = Session(engine)
try:
stack_lock_service.acquire(lock_session, stack_id, "start", username)
except stack_lock_service.StackBusy as exc:
lock_session.close()
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
await websocket.close(code=4409)
return
try:
async for kind, payload in compose_service.stream_up(stack_id):
if kind == "log":
@@ -143,7 +170,9 @@ async def ws_deploy(
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
finally:
compose_service.clear_busy(stack_id)
with contextlib.suppress(Exception):
stack_lock_service.release(lock_session, stack_id)
lock_session.close()
ok = rc in (0, None)
try:
@@ -169,155 +198,88 @@ async def ws_deploy(
await websocket.close()
@router.websocket("/ws/agent-logs/{agent_id}/{stack_id}")
async def ws_agent_logs(
@router.websocket("/ws/update/{stack_id}")
async def ws_update(
websocket: WebSocket,
agent_id: int,
stack_id: str,
token: str | None = Query(default=None),
):
"""Proxy live compose logs from a remote agent through to the browser."""
"""Run `docker compose pull && up -d` and stream its output, so the stacks
list can render real update progress. Same audit/notify contract as the
REST `/update` endpoint, which stays for non-interactive callers."""
await websocket.accept()
if not await _authorize(websocket, token):
return
with Session(engine) as session:
agent = session.get(Agent, agent_id)
if not agent:
await websocket.send_text(json.dumps({"type": "error", "detail": "agent not found"}))
await websocket.close()
return
base = agent.url.rstrip("/")
ws_url = ("wss://" + base[8:] if base.startswith("https://")
else "ws://" + base[7:] if base.startswith("http://")
else "ws://" + base)
# URL-encode the token: agent tokens may contain base64 chars (+ / =) that
# would otherwise be mangled in the query string and rejected as 4401.
ws_url += f"/agent/ws/logs/{stack_id}?token={urllib.parse.quote(agent.token, safe='')}"
async def _err(detail: str) -> None:
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": detail}))
# Connect to the agent. Surface connection problems (agent down, wrong URL,
# an outdated agent that lacks /agent/ws/logs, TLS issues) instead of
# silently dropping the socket.
try:
upstream = await websockets.connect(ws_url, open_timeout=10, ping_interval=20)
except websockets.InvalidStatus as exc:
code = getattr(getattr(exc, "response", None), "status_code", None)
hint = " — the agent may be running an old version without live-log support; update it." if code == 404 else ""
logger.warning("Agent log proxy: handshake to %s failed (%s)", agent.name, code)
await _err(f"Agent '{agent.name}' rejected the log stream (HTTP {code}){hint}")
with contextlib.suppress(Exception):
await websocket.close()
return
except Exception as exc: # noqa: BLE001
logger.warning("Agent log proxy: cannot reach %s at %s: %s", agent.name, agent.url, exc)
await _err(f"Could not connect to agent '{agent.name}' at {agent.url}: {exc}")
with contextlib.suppress(Exception):
await websocket.close()
return
try:
async for message in upstream:
await websocket.send_text(
message if isinstance(message, str) else message.decode("utf-8", "replace")
)
except WebSocketDisconnect:
pass
except websockets.ConnectionClosed as exc:
# Abnormal upstream close (e.g. 4401 bad token, or agent-side error).
if exc.code not in (1000, 1001):
await _err(f"Agent log stream closed unexpectedly (code {exc.code}).")
except Exception as exc: # noqa: BLE001
logger.warning("Agent log proxy: stream error from %s: %s", agent.name, exc)
await _err(str(exc))
finally:
with contextlib.suppress(Exception):
await upstream.close()
with contextlib.suppress(Exception):
await websocket.close()
@router.websocket("/ws/agent-deploy/{agent_id}/{stack_id}")
async def ws_agent_deploy(
websocket: WebSocket,
agent_id: int,
stack_id: str,
token: str | None = Query(default=None),
):
"""Proxy a remote agent's `compose up` deploy stream through to the browser,
then record the same audit entry as the REST agent lifecycle endpoint."""
await websocket.accept()
if not await _authorize(websocket, token):
if not await _authorize_admin(websocket, token):
return
username = decode_token(token, "access").get("sub", "unknown") if token else "unknown"
with Session(engine) as session:
agent = session.get(Agent, agent_id)
if not agent:
await websocket.send_text(json.dumps({"type": "error", "detail": "agent not found"}))
await websocket.close()
return
base = agent.url.rstrip("/")
ws_url = ("wss://" + base[8:] if base.startswith("https://")
else "ws://" + base[7:] if base.startswith("http://")
else "ws://" + base)
ws_url += f"/agent/ws/deploy/{stack_id}?token={urllib.parse.quote(agent.token, safe='')}"
async def _err(detail: str) -> None:
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": detail}))
try:
upstream = await websockets.connect(ws_url, open_timeout=10, ping_interval=20)
except websockets.InvalidStatus as exc:
code = getattr(getattr(exc, "response", None), "status_code", None)
hint = " — the agent may be running an old version without deploy-console support; update it." if code == 404 else ""
logger.warning("Agent deploy proxy: handshake to %s failed (%s)", agent.name, code)
await _err(f"Agent '{agent.name}' rejected the deploy stream (HTTP {code}){hint}")
with contextlib.suppress(Exception):
await websocket.close()
return
except Exception as exc: # noqa: BLE001
logger.warning("Agent deploy proxy: cannot reach %s at %s: %s", agent.name, agent.url, exc)
await _err(f"Could not connect to agent '{agent.name}' at {agent.url}: {exc}")
with contextlib.suppress(Exception):
await websocket.close()
return
rc: int | None = None
disconnected = False
lock_session = Session(engine)
try:
async for message in upstream:
text = message if isinstance(message, str) else message.decode("utf-8", "replace")
with contextlib.suppress(Exception):
msg = json.loads(text)
if msg.get("type") == "done":
rc = msg.get("returncode")
await websocket.send_text(text)
stack_lock_service.acquire(lock_session, stack_id, "update", username)
except stack_lock_service.StackBusy as exc:
lock_session.close()
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
await websocket.close(code=4409)
return
try:
async for kind, payload in compose_service.stream_update(stack_id):
if kind == "log":
await websocket.send_text(json.dumps({"type": "log", "line": payload}))
else:
rc = payload
await websocket.send_text(json.dumps({"type": "done", "returncode": rc}))
except WebSocketDisconnect:
pass
except websockets.ConnectionClosed as exc:
if exc.code not in (1000, 1001):
await _err(f"Agent deploy stream closed unexpectedly (code {exc.code}).")
# Client navigated away; compose keeps running so the update finishes.
disconnected = True
except Exception as exc: # noqa: BLE001
logger.warning("Agent deploy proxy: stream error from %s: %s", agent.name, exc)
await _err(str(exc))
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
finally:
with contextlib.suppress(Exception):
await upstream.close()
stack_lock_service.release(lock_session, stack_id)
lock_session.close()
ok = rc in (0, None)
try:
with Session(engine) as session:
audit_service.record(
session, user=username, action="stack.update", target=stack_id,
detail=f"rc={rc} (update stream)", ip="ws",
)
if ok:
await notify_service.notify(
EVENT_STACK_START, f"Stack '{stack_id}' updated",
"compose pull + up completed successfully.", session,
)
else:
await notify_service.notify(
EVENT_PULL_FAILED, f"Stack '{stack_id}' update failed",
"compose pull/up returned a non-zero exit code.", session,
)
except Exception: # noqa: BLE001 - audit/notify are best-effort
pass
if not disconnected:
with contextlib.suppress(Exception):
await websocket.close()
with contextlib.suppress(Exception):
with Session(engine) as session:
audit_service.record(
session, user=username, action="agent.stack.start",
target=f"{agent.name}/{stack_id}", detail=f"rc={rc} (deploy console)", ip="ws",
)
if ok:
update_service.refresh_stack_local(stack_id)
#: Docker event types worth telling the UI about. Filtered daemon-side, so the
#: bulk of the firehose never crosses the socket.
_EVENT_TYPES = ["container", "image", "network", "volume"]
#: Container actions that say nothing about state a page renders. exec_* alone
#: is three events per web-terminal keystroke session, and `top`/`attach` fire
#: whenever something inspects a container — invalidating queries on those would
#: make the stream noisier than the polling it replaces.
_IGNORED_ACTIONS = {
"exec_create", "exec_start", "exec_die", "exec_detach",
"attach", "top", "resize", "archive-path", "extract-to-dir",
}
@router.websocket("/ws/events")
@@ -325,7 +287,20 @@ async def ws_events(
websocket: WebSocket,
token: str | None = Query(default=None),
):
"""Stream global Docker events (decoded subset)."""
"""Stream Docker events so the UI can refresh on change instead of polling.
Every page used to poll its own endpoint every few seconds. Almost all of
that state only changes when Docker does something, which is exactly what
this reports — so the client refreshes on an event and keeps a slow poll as
a safety net.
Payload per event::
{"type": "event", "resource": "container", "action": "start",
"container": "jellyfin", "stack": "jellyfin"}
``resource`` is what the client needs to decide which queries to drop.
"""
await websocket.accept()
if not await _authorize(websocket, token):
return
@@ -333,28 +308,40 @@ async def ws_events(
loop = asyncio.get_event_loop()
queue: asyncio.Queue = asyncio.Queue()
stop = asyncio.Event()
stream = None
def reader():
"""Blocking read of the event stream, handed to the loop thread-safely."""
nonlocal stream
try:
client = get_client()
for event in client.events(decode=True):
if stop.is_set():
break
stream = get_client().events(decode=True, filters={"type": _EVENT_TYPES})
for event in stream:
loop.call_soon_threadsafe(queue.put_nowait, event)
except Exception: # noqa: BLE001
except Exception: # noqa: BLE001 - a closed stream lands here on teardown
pass
finally:
loop.call_soon_threadsafe(queue.put_nowait, None)
task = loop.run_in_executor(None, reader)
try:
await websocket.send_text(json.dumps({"type": "ready"}))
while True:
event = await queue.get()
if event is None: # reader finished — daemon gone or stream closed
await websocket.send_text(
json.dumps({"type": "error", "detail": "Docker event stream ended"})
)
break
action = (event.get("Action") or "").split(":")[0]
if action in _IGNORED_ACTIONS:
continue
actor = event.get("Actor", {}) or {}
attrs = actor.get("Attributes", {}) or {}
await websocket.send_text(
json.dumps(
{
"type": "event",
"resource": event.get("Type"),
"action": event.get("Action"),
"container": attrs.get("name"),
"stack": attrs.get("com.docker.compose.project"),
@@ -363,8 +350,17 @@ async def ws_events(
)
except WebSocketDisconnect:
pass
except Exception as exc: # noqa: BLE001
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
finally:
stop.set()
# Closing the stream is what actually unblocks the reader thread.
# Cancelling the executor future does not: a thread already inside a
# blocking read keeps that read, and the thread leaks for the life of
# the process — once per page load, with the socket held open.
if stream is not None:
with contextlib.suppress(Exception):
stream.close()
task.cancel()
@@ -408,91 +404,3 @@ async def ws_exec(
await websocket.close()
@router.websocket("/ws/agent-exec/{agent_id}/{container_id}")
async def ws_agent_exec(
websocket: WebSocket,
agent_id: int,
container_id: str,
token: str | None = Query(default=None),
cmd: str | None = Query(default=None),
):
"""Proxy an interactive exec session to a remote agent (admin only).
Unlike the log/deploy proxies this forwards in BOTH directions so keystrokes
reach the container and its output streams back."""
await websocket.accept()
if not await _authorize_admin(websocket, token):
return
with Session(engine) as session:
agent = session.get(Agent, agent_id)
if not agent:
await websocket.send_text(json.dumps({"type": "error", "detail": "agent not found"}))
await websocket.close()
return
base = agent.url.rstrip("/")
ws_url = ("wss://" + base[8:] if base.startswith("https://")
else "ws://" + base[7:] if base.startswith("http://")
else "ws://" + base)
ws_url += f"/agent/ws/exec/{container_id}?token={urllib.parse.quote(agent.token, safe='')}"
if cmd:
ws_url += f"&cmd={urllib.parse.quote(cmd, safe='')}"
async def _err(detail: str) -> None:
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": detail}))
try:
upstream = await websockets.connect(ws_url, open_timeout=10, ping_interval=20)
except websockets.InvalidStatus as exc:
code = getattr(getattr(exc, "response", None), "status_code", None)
hint = " — the agent may be running an old version without terminal support; update it." if code == 404 else ""
logger.warning("Agent exec proxy: handshake to %s failed (%s)", agent.name, code)
await _err(f"Agent '{agent.name}' rejected the terminal (HTTP {code}){hint}")
with contextlib.suppress(Exception):
await websocket.close()
return
except Exception as exc: # noqa: BLE001
logger.warning("Agent exec proxy: cannot reach %s at %s: %s", agent.name, agent.url, exc)
await _err(f"Could not connect to agent '{agent.name}' at {agent.url}: {exc}")
with contextlib.suppress(Exception):
await websocket.close()
return
with contextlib.suppress(Exception):
with Session(engine) as session:
username = decode_token(token, "access").get("sub", "unknown") if token else "unknown"
audit_service.record(
session, user=username, action="agent.container.exec",
target=f"{agent.name}/{container_id[:12]}", ip="ws",
)
async def browser_to_agent() -> None:
try:
while True:
msg = await websocket.receive_text()
await upstream.send(msg)
except (WebSocketDisconnect, websockets.ConnectionClosed):
pass
async def agent_to_browser() -> None:
try:
async for message in upstream:
await websocket.send_text(
message if isinstance(message, str) else message.decode("utf-8", "replace")
)
except (WebSocketDisconnect, websockets.ConnectionClosed):
pass
b2a = asyncio.create_task(browser_to_agent())
a2b = asyncio.create_task(agent_to_browser())
done, pending = await asyncio.wait({b2a, a2b}, return_when=asyncio.FIRST_COMPLETED)
for task in pending:
task.cancel()
with contextlib.suppress(Exception):
await asyncio.gather(*pending, return_exceptions=True)
with contextlib.suppress(Exception):
await upstream.close()
with contextlib.suppress(Exception):
await websocket.close()
backend/services/agent_service.py
-182
View File
@@ -1,182 +0,0 @@
"""Talk to remote stackpilot-agent hosts over HTTP.
The central app stores an ``Agent`` row per remote host and proxies stack /
system calls to it using the agent's shared token. Connectivity state
(``status``, ``hostname``, ``last_seen``) is refreshed on every successful or
failed call so the UI can show a live dot per host.
"""
from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import Any, Optional
import httpx
from sqlmodel import Session
from models.agent import Agent
logger = logging.getLogger("stackpilot.agent_proxy")
_TIMEOUT = 30.0
class AgentError(Exception):
def __init__(self, status: int, error: str, detail: str = ""):
self.status = status
self.error = error
self.detail = detail
super().__init__(f"{error}: {detail}" if detail else error)
def _now() -> datetime:
return datetime.now(timezone.utc)
def _mark(session: Session, agent: Agent, status: str, hostname: Optional[str] = None) -> None:
agent.status = status
if status == "online":
agent.last_seen = _now()
if hostname:
agent.hostname = hostname
session.add(agent)
session.commit()
session.refresh(agent)
async def _request(
agent: Agent,
method: str,
path: str,
*,
params: Optional[dict] = None,
json: Any = None,
) -> httpx.Response:
url = agent.url.rstrip("/") + path
headers = {"Authorization": f"Bearer {agent.token}"}
async with httpx.AsyncClient(follow_redirects=True) as client:
return await client.request(
method, url, headers=headers, params=params, json=json, timeout=_TIMEOUT
)
async def call(
session: Session,
agent: Agent,
method: str,
path: str,
*,
params: Optional[dict] = None,
json: Any = None,
) -> Any:
"""Proxy a request to the agent, updating its status, returning parsed JSON."""
try:
resp = await _request(agent, method, path, params=params, json=json)
except httpx.HTTPError as exc:
_mark(session, agent, "offline")
raise AgentError(502, "agent_unreachable", str(exc)) from exc
if resp.status_code in (401, 403):
_mark(session, agent, "unauthorized")
raise AgentError(resp.status_code, "agent_unauthorized", "Invalid agent token")
_mark(session, agent, "online")
if resp.status_code >= 400:
detail = ""
try:
body = resp.json()
detail = body.get("detail") if isinstance(body, dict) else str(body)
if isinstance(detail, dict):
detail = detail.get("detail") or detail.get("error") or str(detail)
except ValueError:
detail = resp.text[:500]
raise AgentError(resp.status_code, "agent_error", str(detail))
if resp.content:
try:
return resp.json()
except ValueError:
return resp.text
return None
def _handle_status(session: Session, agent: Agent, status_code: int, body_text: str = "") -> None:
"""Update agent status from a response code; raise AgentError on failure."""
if status_code in (401, 403):
_mark(session, agent, "unauthorized")
raise AgentError(status_code, "agent_unauthorized", "Invalid agent token")
_mark(session, agent, "online")
if status_code >= 400:
raise AgentError(status_code, "agent_error", body_text[:500])
async def download_to_file(
session: Session,
agent: Agent,
path: str,
dest_path: str,
*,
params: Optional[dict] = None,
) -> None:
"""Stream a GET from the agent into ``dest_path``."""
url = agent.url.rstrip("/") + path
headers = {"Authorization": f"Bearer {agent.token}"}
try:
async with httpx.AsyncClient(follow_redirects=True) as client:
async with client.stream("GET", url, headers=headers, params=params, timeout=None) as resp:
if resp.status_code >= 400:
text = (await resp.aread()).decode("utf-8", "replace")
_handle_status(session, agent, resp.status_code, text)
_handle_status(session, agent, resp.status_code)
with open(dest_path, "wb") as fh:
async for chunk in resp.aiter_bytes(1024 * 256):
fh.write(chunk)
except httpx.HTTPError as exc:
_mark(session, agent, "offline")
raise AgentError(502, "agent_unreachable", str(exc)) from exc
async def upload_file(
session: Session,
agent: Agent,
path: str,
file_path: str,
filename: str,
data: dict,
) -> Any:
"""Stream a multipart POST (file + form fields) to the agent, returning JSON."""
url = agent.url.rstrip("/") + path
headers = {"Authorization": f"Bearer {agent.token}"}
try:
async with httpx.AsyncClient(follow_redirects=True) as client:
with open(file_path, "rb") as fh:
files = {"file": (filename, fh, "application/gzip")}
resp = await client.post(url, headers=headers, files=files, data=data, timeout=None)
except httpx.HTTPError as exc:
_mark(session, agent, "offline")
raise AgentError(502, "agent_unreachable", str(exc)) from exc
detail = ""
if resp.status_code >= 400:
try:
body = resp.json()
detail = body.get("detail") if isinstance(body, dict) else str(body)
except ValueError:
detail = resp.text[:500]
_handle_status(session, agent, resp.status_code, str(detail))
return resp.json() if resp.content else None
async def ping(session: Session, agent: Agent) -> dict:
"""Health-check an agent and refresh its status + hostname. Never raises."""
try:
data = await call(session, agent, "GET", "/agent/ping")
if isinstance(data, dict) and data.get("hostname"):
agent.hostname = data["hostname"]
session.add(agent)
session.commit()
session.refresh(agent)
return {"status": agent.status, "hostname": agent.hostname, "data": data}
except AgentError:
return {"status": agent.status, "hostname": agent.hostname, "data": None}
backend/services/api_token_service.py
+134
View File
@@ -0,0 +1,134 @@
"""Long-lived API tokens for scripts and CI.
A session token is the wrong credential for automation: it expires in an hour,
it is minted by typing a password, and revoking it means signing every one of
that person's devices out. So a CI job gets its own credential, which can be
revoked on its own, is capped to read-only if that is all it needs, and shows up
in the audit log as itself.
**Only a hash is stored.** Unlike a registry password — which has to be handed
back to the registry, so it is encrypted and recoverable — a token is only ever
compared against. It is shown once at creation and cannot be recovered, which is
the difference between leaking the database and leaking everything it protects.
The hash is a plain SHA-256 and deliberately not bcrypt. Bcrypt is slow on
purpose, to make guessing low-entropy human passwords expensive; a token is 256
bits of ``secrets`` output, where guessing is not the threat and the cost would
instead land on every single API request.
"""
from __future__ import annotations
import hashlib
import secrets
from datetime import datetime, timedelta, timezone
from typing import Optional
from sqlmodel import Session, select
from models.api_token import ApiToken
from models.user import User
#: Marks a StackPilot token at a glance — in a log, in a CI settings page, or to
#: a secret scanner. It is also how the auth dependency tells a token from a JWT
#: without trying to decode it.
PREFIX = "sp_"
#: How stale last_used_at may get before a request writes it again. Without a
#: floor this would be a database write on every single API call.
_TOUCH_INTERVAL = timedelta(minutes=5)
def looks_like_token(value: str) -> bool:
return (value or "").startswith(PREFIX)
def _hash(token: str) -> str:
return hashlib.sha256(token.encode("utf-8")).hexdigest()
def _aware(value: Optional[datetime]) -> Optional[datetime]:
"""SQLite hands back naive datetimes; compare them as UTC."""
if value is None:
return None
return value if value.tzinfo else value.replace(tzinfo=timezone.utc)
def is_expired(row: ApiToken, now: Optional[datetime] = None) -> bool:
expires = _aware(row.expires_at)
if expires is None:
return False
return expires <= (now or datetime.now(timezone.utc))
def mint(
session: Session,
*,
name: str,
user: User,
scope: str = "read",
expires_in_days: Optional[int] = None,
) -> tuple[ApiToken, str]:
"""Create a token. Returns the row and the secret, which is shown once."""
token = PREFIX + secrets.token_urlsafe(32)
expires_at = (
datetime.now(timezone.utc) + timedelta(days=expires_in_days)
if expires_in_days
else None
)
row = ApiToken(
name=name,
prefix=token[: len(PREFIX) + 8],
token_hash=_hash(token),
scope=scope if scope in ("read", "admin") else "read",
user_id=user.id,
expires_at=expires_at,
)
session.add(row)
session.commit()
session.refresh(row)
return row, token
def resolve(session: Session, token: str) -> Optional[tuple[ApiToken, User]]:
"""The token row and its owner, or None if it cannot be used.
None covers every reason equally — unknown, expired, owner disabled — so a
caller cannot learn which by watching the responses.
"""
if not looks_like_token(token):
return None
prefix = token[: len(PREFIX) + 8]
row = session.exec(select(ApiToken).where(ApiToken.prefix == prefix)).first()
if not row:
return None
# Constant-time, so a wrong token cannot be narrowed down by timing.
if not secrets.compare_digest(row.token_hash, _hash(token)):
return None
if is_expired(row):
return None
user = session.get(User, row.user_id)
if not user or not user.is_active:
return None
return row, user
def effective_role(row: ApiToken, user: User) -> str:
"""What this token may do, which is never more than its owner may.
A token keeps working when its owner is demoted, but drops to read-only with
them — the alternative is an admin token outliving the admin.
"""
if row.scope == "admin" and user.role == "admin":
return "admin"
return "user"
def touch(session: Session, row: ApiToken) -> None:
"""Record that the token was used, at most once every few minutes."""
now = datetime.now(timezone.utc)
last = _aware(row.last_used_at)
if last and now - last < _TOUCH_INTERVAL:
return
row.last_used_at = now
session.add(row)
session.commit()
backend/services/auto_update_service.py
+31 -73
View File
@@ -5,9 +5,8 @@ Runs once per image-update-check cycle (called from
cache). For each enabled policy whose stack has a newer image available, either
pulls + redeploys the stack or just notifies, recording the outcome.
Central-only / DB-aware. Image resolution + digest comparison live in the
DB-free ``update_service`` so the agent can answer ``/agent/stacks/{id}/updates``
with the same logic.
Image resolution and digest comparison live in ``update_service``; this module
adds the policy layer on top.
"""
from __future__ import annotations
@@ -17,10 +16,14 @@ from datetime import datetime, timezone
from sqlmodel import Session, select
from database import engine
from models.agent import Agent
from models.auto_update import AutoUpdate
from models.setting import EVENT_PULL_FAILED, EVENT_STACK_AUTO_UPDATED
from services import agent_service, compose_service, notify_service, update_service
from services import (
compose_service,
notify_service,
stack_lock_service,
update_service,
)
logger = logging.getLogger("stackpilot.autoupdate")
@@ -33,23 +36,18 @@ def _now() -> datetime:
# --------------------------------------------------------------------------- #
# Policy CRUD helpers (shared by the stacks + agents routers)
# Policy CRUD helpers
# --------------------------------------------------------------------------- #
def get_policy(session: Session, stack_id: str, agent_id: int | None = None) -> AutoUpdate | None:
stmt = select(AutoUpdate).where(AutoUpdate.stack_id == stack_id)
stmt = stmt.where(AutoUpdate.agent_id == agent_id) if agent_id is not None \
else stmt.where(AutoUpdate.agent_id.is_(None))
return session.exec(stmt).first()
def get_policy(session: Session, stack_id: str) -> AutoUpdate | None:
return session.exec(select(AutoUpdate).where(AutoUpdate.stack_id == stack_id)).first()
def upsert_policy(
session: Session, stack_id: str, enabled: bool, redeploy: bool, agent_id: int | None = None
) -> AutoUpdate:
policy = get_policy(session, stack_id, agent_id)
def upsert_policy(session: Session, stack_id: str, enabled: bool, redeploy: bool) -> AutoUpdate:
policy = get_policy(session, stack_id)
if policy is None:
policy = AutoUpdate(stack_id=stack_id, agent_id=agent_id)
policy = AutoUpdate(stack_id=stack_id)
policy.enabled = enabled
policy.redeploy = redeploy
session.add(policy)
@@ -58,22 +56,19 @@ def upsert_policy(
return policy
def to_read(session: Session, policy: AutoUpdate | None, stack_id: str, agent_id: int | None = None) -> dict:
def to_read(policy: AutoUpdate | None, stack_id: str) -> dict:
"""Build an AutoUpdateRead-shaped dict, defaulting to disabled when absent."""
agent_name = None
if agent_id is not None:
agent = session.get(Agent, agent_id)
agent_name = agent.name if agent else None
if policy is None:
return {
"id": None, "stack_id": stack_id, "agent_id": agent_id, "agent_name": agent_name,
"id": None, "stack_id": stack_id,
"enabled": False, "redeploy": True,
"last_run": None, "last_status": None, "last_result": None,
}
return {
"id": policy.id, "stack_id": policy.stack_id, "agent_id": policy.agent_id,
"agent_name": agent_name, "enabled": policy.enabled, "redeploy": policy.redeploy,
"last_run": policy.last_run, "last_status": policy.last_status, "last_result": policy.last_result,
"id": policy.id, "stack_id": policy.stack_id,
"enabled": policy.enabled, "redeploy": policy.redeploy,
"last_run": policy.last_run, "last_status": policy.last_status,
"last_result": policy.last_result,
}
@@ -104,12 +99,20 @@ async def _run_local(session: Session, policy: AutoUpdate) -> None:
prev = policy.last_status
if policy.redeploy:
try:
await compose_service.pull(stack_id)
await compose_service.up(stack_id)
# Never redeploy underneath somebody: if a user is mid-deploy on
# this stack, skip and pick it up next cycle rather than racing
# them over the same containers.
with stack_lock_service.hold(session, stack_id, "auto-update", "auto-update"):
await compose_service.pull(stack_id)
await compose_service.up(stack_id)
except stack_lock_service.StackBusy as exc:
_record(session, policy, "skipped", str(exc))
return
except Exception as exc: # noqa: BLE001
_record(session, policy, "error", str(exc))
await _safe_notify(EVENT_PULL_FAILED, f"Auto-update of '{stack_id}' failed", str(exc), session)
return
update_service.refresh_stack_local(stack_id)
_record(session, policy, "updated", stale)
await _safe_notify(
EVENT_STACK_AUTO_UPDATED, f"Stack '{stack_id}' auto-updated",
@@ -124,48 +127,6 @@ async def _run_local(session: Session, policy: AutoUpdate) -> None:
)
async def _run_remote(session: Session, policy: AutoUpdate) -> None:
agent = session.get(Agent, policy.agent_id)
if not agent:
_record(session, policy, "error", "agent not found")
return
stack_id = policy.stack_id
try:
summary = await agent_service.call(
session, agent, "GET", f"/agent/stacks/{stack_id}/updates",
params={"refresh": "true"},
)
except Exception as exc: # noqa: BLE001
_record(session, policy, "error", f"agent check failed: {exc}")
return
if not summary or not summary.get("update_available"):
_record(session, policy, "up-to-date")
return
stale = ", ".join(summary.get("stale_images", []))
label = f"{agent.name}/{stack_id}"
prev = policy.last_status
if policy.redeploy:
try:
await agent_service.call(session, agent, "POST", f"/agent/stacks/{stack_id}/update")
except Exception as exc: # noqa: BLE001
_record(session, policy, "error", str(exc))
await _safe_notify(EVENT_PULL_FAILED, f"Auto-update of '{label}' failed", str(exc), session)
return
_record(session, policy, "updated", stale)
await _safe_notify(
EVENT_STACK_AUTO_UPDATED, f"Stack '{label}' auto-updated",
f"Pulled and redeployed: {stale}.", session,
)
else:
_record(session, policy, "update-available", stale)
if prev != "update-available":
await _safe_notify(
EVENT_STACK_AUTO_UPDATED, f"Update available for '{label}'",
f"Newer images: {stale} (auto-redeploy is off).", session,
)
async def _safe_notify(event: str, title: str, message: str, session: Session) -> None:
try:
await notify_service.notify(event, title, message, session)
@@ -174,10 +135,7 @@ async def _safe_notify(event: str, title: str, message: str, session: Session) -
async def run_policy(session: Session, policy: AutoUpdate) -> None:
if policy.agent_id is None:
await _run_local(session, policy)
else:
await _run_remote(session, policy)
await _run_local(session, policy)
async def run_due() -> None:
backend/services/backup_destination_service.py
+281 -7
View File
@@ -1,20 +1,25 @@
"""Push/pull stack backups to remote destinations (SFTP or S3-compatible).
"""Push/pull stack backups to remote destinations (SFTP, S3-compatible or NFS).
All operations are synchronous (paramiko / boto3); async callers should wrap
them with ``asyncio.to_thread``. Destination config is a plain dict parsed from
the ``BackupDestination.config`` JSON column.
All operations are synchronous (paramiko / boto3 / docker); async callers
should wrap them with ``asyncio.to_thread``. Destination config is a dict
stored in the ``BackupDestination.config`` column as JSON, encrypted at rest
(:mod:`services.crypto_service`) because it carries SFTP passwords, SSH keys
and S3 secret keys. Always go through :func:`parse_config` / :func:`dump_config`
— never touch the column directly.
"""
from __future__ import annotations
import io
import json
import logging
import os
import posixpath
import stat
import tarfile
import tempfile
from typing import Any
from models.backup_destination import BackupDestination
from services import crypto_service
logger = logging.getLogger("stackpilot.backup_dest")
@@ -24,12 +29,48 @@ class DestinationError(Exception):
def parse_config(dest: BackupDestination) -> dict:
"""Decrypt and parse a destination's config.
Tolerates plaintext (pre-encryption rows) and returns ``{}`` rather than
raising if the value can't be decrypted — a destination whose key is gone
should show up as unconfigured in the UI, not take the whole list down with
a 500. The failure is logged with the destination name so it's findable.
"""
try:
return json.loads(dest.config or "{}")
raw = crypto_service.decrypt(dest.config or "{}")
except crypto_service.DecryptError as exc:
logger.error("Destination '%s': %s", dest.name, exc)
return {}
try:
return json.loads(raw or "{}")
except json.JSONDecodeError:
return {}
def dump_config(config: dict) -> str:
"""Serialise and encrypt a config dict for storage."""
return crypto_service.encrypt(json.dumps(config or {}))
def migrate_plaintext_configs(session) -> int:
"""Encrypt destination configs written before encryption existed.
Runs once at startup. Returns how many rows were rewritten.
"""
from sqlmodel import select
migrated = 0
for dest in session.exec(select(BackupDestination)).all():
if crypto_service.is_encrypted(dest.config):
continue
dest.config = crypto_service.encrypt(dest.config or "{}")
session.add(dest)
migrated += 1
if migrated:
session.commit()
return migrated
# --------------------------------------------------------------------------- #
# SFTP (paramiko)
# --------------------------------------------------------------------------- #
@@ -209,6 +250,229 @@ def _s3_delete(cfg: dict, name: str) -> None:
client.delete_object(Bucket=cfg["bucket"], Key=_s3_key(cfg, name))
# --------------------------------------------------------------------------- #
# NFS — the Docker daemon mounts the export as a named volume; file I/O runs
# through a throwaway helper container (same pattern as volume backups), so
# the backend itself needs no mount privileges.
# --------------------------------------------------------------------------- #
_NFS_VOLUME_PREFIX = "stackpilot-nfs-dest-"
def _nfs_check_name(name: str) -> None:
import re
# Same safe charset as the subdir parts: the name is interpolated into the
# helper container's shell commands.
if not name or not re.fullmatch(r"[A-Za-z0-9._-]+", name) or name.startswith("."):
raise DestinationError(f"Invalid backup file name '{name}'")
def _nfs_subdir(cfg: dict) -> str:
"""Sanitized relative directory inside the export ('' = export root).
Parts are restricted to a safe charset because the path is interpolated
into helper-container shell commands.
"""
import re
raw = (cfg.get("subdir") or "").strip().strip("/")
if not raw:
return ""
parts = [p for p in raw.split("/") if p]
for p in parts:
if p == ".." or not re.fullmatch(r"[A-Za-z0-9._-]+", p):
raise DestinationError(
"Subdirectory may only contain letters, digits, '.', '_' and '-'"
)
return "/".join(parts)
def _nfs_volume(dest: BackupDestination, cfg: dict) -> str:
"""Ensure the named volume describing this NFS mount exists; recreate it
when the destination's server/path/options changed (opts are immutable)."""
from docker_client import DockerError, get_client, safe_call
server = (cfg.get("server") or "").strip()
path = (cfg.get("path") or "").strip()
if not server:
raise DestinationError("NFS server is required")
if not path.startswith("/"):
raise DestinationError("NFS export path must be absolute (start with /)")
options = (cfg.get("options") or "rw").strip().strip(",")
driver_opts = {"type": "nfs", "o": f"addr={server},{options}", "device": f":{path}"}
name = f"{_NFS_VOLUME_PREFIX}{dest.id}"
client = get_client()
try:
vol = safe_call(client.volumes.get, name)
if (vol.attrs.get("Options") or {}) != driver_opts:
safe_call(vol.remove)
raise DockerError("recreate", "options changed")
except DockerError:
safe_call(
client.volumes.create,
name=name,
driver="local",
driver_opts=driver_opts,
labels={"stackpilot.nfs-destination": str(dest.id)},
)
return name
def _nfs_target(cfg: dict) -> str:
sub = _nfs_subdir(cfg)
return f"/nfs/{sub}" if sub else "/nfs"
def _nfs_run(volume: str, command: list[str]) -> str:
"""Run a helper container with the NFS volume at /nfs; return stdout."""
import docker.errors
from config import settings
from docker_client import DockerError, get_client
from services.stack_assets_service import ensure_helper_image
client = get_client()
ensure_helper_image(client)
try:
out = client.containers.run(
settings.BACKUP_HELPER_IMAGE,
command,
volumes={volume: {"bind": "/nfs", "mode": "rw"}},
remove=True,
)
return (out or b"").decode("utf-8", "replace")
except docker.errors.ContainerError as exc:
stderr = (exc.stderr or b"").decode("utf-8", "replace").strip()
raise DestinationError(f"NFS operation failed: {stderr or exc}") from exc
except (docker.errors.APIError, DockerError) as exc:
# Mount errors surface here (unreachable server, bad export, ...).
raise DestinationError(f"NFS mount failed: {exc}") from exc
def _nfs_helper(volume: str, command: list[str] | str = "true"):
"""A created (not started) helper container for archive I/O on /nfs."""
import docker.errors
from config import settings
from docker_client import DockerError, get_client, safe_call
from services.stack_assets_service import ensure_helper_image
client = get_client()
ensure_helper_image(client)
try:
return safe_call(
client.containers.create,
settings.BACKUP_HELPER_IMAGE,
command=command,
volumes={volume: {"bind": "/nfs", "mode": "rw"}},
)
except (docker.errors.APIError, DockerError) as exc:
raise DestinationError(f"NFS mount failed: {exc}") from exc
def _nfs_upload(dest: BackupDestination, cfg: dict, local_path: str, filename: str) -> str:
import docker.errors
_nfs_check_name(filename)
volume = _nfs_volume(dest, cfg)
target = _nfs_target(cfg)
# Creates the subdir if needed AND fails early with a clear mount error.
_nfs_run(volume, ["mkdir", "-p", target])
# Unpack into the container's own filesystem, then copy the file across:
# extracting straight into the NFS mount makes the daemon chown the file,
# which a root_squash export refuses ("failed to Lchown ... for UID 0").
container = _nfs_helper(
volume, ["sh", "-c", f"cat '/tmp/{filename}' > '{target}/{filename}'"]
)
try:
with tempfile.TemporaryFile() as tmp:
with tarfile.open(fileobj=tmp, mode="w") as tar:
tar.add(local_path, arcname=filename)
tmp.seek(0)
container.put_archive("/tmp", tmp)
container.start()
status = container.wait(timeout=3600).get("StatusCode", 1)
if status != 0:
err = (container.logs(stdout=True, stderr=True) or b"").decode("utf-8", "replace")
raise DestinationError(f"NFS upload failed: {err.strip() or f'exit {status}'}")
except docker.errors.APIError as exc:
raise DestinationError(f"NFS upload failed: {exc}") from exc
finally:
try:
container.remove(force=True)
except Exception: # noqa: BLE001
pass
sub = _nfs_subdir(cfg)
return posixpath.join(sub, filename) if sub else filename
def _nfs_list(dest: BackupDestination, cfg: dict) -> list[dict]:
volume = _nfs_volume(dest, cfg)
target = _nfs_target(cfg)
out = _nfs_run(
volume,
["sh", "-c", f"cd {target} 2>/dev/null && stat -c '%n|%s|%Y' *.tar.gz 2>/dev/null; true"],
)
entries = []
for line in out.splitlines():
parts = line.strip().split("|")
if len(parts) != 3 or parts[0] == "*.tar.gz":
continue
try:
entries.append({"name": parts[0], "size": int(parts[1]), "modified": int(parts[2])})
except ValueError:
continue
return sorted(entries, key=lambda x: x["modified"] or 0, reverse=True)
def _nfs_download(dest: BackupDestination, cfg: dict, name: str, local_path: str) -> None:
import docker.errors
_nfs_check_name(name)
volume = _nfs_volume(dest, cfg)
target = _nfs_target(cfg)
container = _nfs_helper(volume)
try:
bits, _ = container.get_archive(f"{target}/{name}")
with tempfile.TemporaryFile() as tmp:
for chunk in bits:
tmp.write(chunk)
tmp.seek(0)
with tarfile.open(fileobj=tmp) as tar:
member = next((m for m in tar.getmembers() if m.isreg()), None)
fh = tar.extractfile(member) if member else None
if fh is None:
raise DestinationError(f"'{name}' not found on NFS destination")
with open(local_path, "wb") as out:
while chunk := fh.read(1024 * 1024):
out.write(chunk)
except docker.errors.APIError as exc:
raise DestinationError(f"NFS download failed (does '{name}' exist?): {exc}") from exc
finally:
try:
container.remove(force=True)
except Exception: # noqa: BLE001
pass
def _nfs_delete(dest: BackupDestination, cfg: dict, name: str) -> None:
_nfs_check_name(name)
volume = _nfs_volume(dest, cfg)
_nfs_run(volume, ["rm", "-f", f"{_nfs_target(cfg)}/{name}"])
def _nfs_test(dest: BackupDestination, cfg: dict) -> bool:
volume = _nfs_volume(dest, cfg)
target = _nfs_target(cfg)
_nfs_run(
volume,
["sh", "-c", f"mkdir -p {target} && touch {target}/.stackpilot-test && rm -f {target}/.stackpilot-test"],
)
return True
# --------------------------------------------------------------------------- #
# Dispatch
# --------------------------------------------------------------------------- #
@@ -220,6 +484,8 @@ def upload(dest: BackupDestination, local_path: str, filename: str) -> str:
return _sftp_upload(cfg, local_path, filename)
if dest.type == "s3":
return _s3_upload(cfg, local_path, filename)
if dest.type == "nfs":
return _nfs_upload(dest, cfg, local_path, filename)
raise DestinationError(f"Unknown destination type '{dest.type}'")
@@ -229,6 +495,8 @@ def list_backups(dest: BackupDestination) -> list[dict]:
return _sftp_list(cfg)
if dest.type == "s3":
return _s3_list(cfg)
if dest.type == "nfs":
return _nfs_list(dest, cfg)
raise DestinationError(f"Unknown destination type '{dest.type}'")
@@ -238,6 +506,8 @@ def download(dest: BackupDestination, name: str, local_path: str) -> None:
_sftp_download(cfg, name, local_path)
elif dest.type == "s3":
_s3_download(cfg, name, local_path)
elif dest.type == "nfs":
_nfs_download(dest, cfg, name, local_path)
else:
raise DestinationError(f"Unknown destination type '{dest.type}'")
@@ -248,11 +518,15 @@ def delete(dest: BackupDestination, name: str) -> None:
_sftp_delete(cfg, name)
elif dest.type == "s3":
_s3_delete(cfg, name)
elif dest.type == "nfs":
_nfs_delete(dest, cfg, name)
else:
raise DestinationError(f"Unknown destination type '{dest.type}'")
def test(dest: BackupDestination) -> bool:
"""Connectivity check — lists the target (cheap, validates auth + path)."""
"""Connectivity check — validates reachability, auth and write access."""
if dest.type == "nfs":
return _nfs_test(dest, parse_config(dest))
list_backups(dest)
return True
backend/services/backup_service.py
+340 -135
View File
@@ -1,14 +1,16 @@
"""Stack backup & restore, including named-volume contents.
"""Stack backup & restore — compose files, bind-mount data and named volumes.
A backup is a single ``.tar.gz`` with this layout::
manifest.json metadata + volume/bind inventory
compose/... the full stack directory (compose file, .env, ...)
volumes/<full>.tar raw contents of each compose-managed named volume
manifest.json metadata + full inventory of what was captured
compose/... the stack directory as StackPilot can see it
binds/<n>.tar contents of each captured bind-mount source
volumes/<full>.tar contents of each captured named volume
Named-volume contents are read/written through a throwaway helper container
(``BACKUP_HELPER_IMAGE``) with the volume bind-mounted — this is the portable
way to snapshot a volume regardless of its driver/mountpoint.
Bind sources and volumes are read/written through a throwaway helper container
(see :mod:`services.stack_assets_service`) so that host paths this container
cannot see are still captured — without that, a stack whose data directories
live outside StackPilot's own mount would back up as "just the compose file".
"""
from __future__ import annotations
@@ -22,16 +24,14 @@ import tarfile
import tempfile
from typing import Optional
from config import settings
from docker_client import DockerError, get_client, safe_call
from services import compose_service
from services import compose_service, stack_assets_service as assets
logger = logging.getLogger("stackpilot.backup")
COMPOSE_PROJECT_LABEL = "com.docker.compose.project"
COMPOSE_VOLUME_LABEL = "com.docker.compose.volume"
COMPOSE_PROJECT_LABEL = assets.COMPOSE_PROJECT_LABEL
COMPOSE_VOLUME_LABEL = assets.COMPOSE_VOLUME_LABEL
MANIFEST_NAME = "manifest.json"
BACKUP_FORMAT_VERSION = 1
BACKUP_FORMAT_VERSION = 2
class BackupError(Exception):
@@ -53,92 +53,54 @@ def backup_basename(stack_id: str, prefix: Optional[str] = None) -> str:
def backup_filename(stack_id: str, include_volumes: bool, prefix: Optional[str] = None) -> str:
date = now().strftime("%Y%m%d-%H%M%S")
date = compose_service.now().strftime("%Y%m%d-%H%M%S")
suffix = "full" if include_volumes else "config"
return f"{backup_basename(stack_id, prefix)}-{suffix}-{date}.tar.gz"
# --------------------------------------------------------------------------- #
# Helper container for volume I/O
# Selection
# --------------------------------------------------------------------------- #
def _ensure_helper_image(client) -> None:
image = settings.BACKUP_HELPER_IMAGE
try:
safe_call(client.images.get, image)
except DockerError:
logger.info("Pulling backup helper image %s", image)
safe_call(client.images.pull, image)
def _decide(
items: list[dict], key: str, chosen: Optional[list[str]], enabled: bool
) -> list[dict]:
"""Mark each inventory item ``selected`` (with a reason when it is not).
``chosen`` is an explicit list from the caller; without one the inventory's
own defaults apply (everything except system paths, oversized directories
and remote-backed volumes).
"""
for item in items:
if not enabled:
item["selected"], item["reason"] = False, "not requested"
elif chosen is not None:
selected = item[key] in chosen
item["selected"] = selected
item["reason"] = None if selected else "not selected"
else:
item["selected"] = bool(item.get("include_default"))
item["reason"] = None if item["selected"] else (item.get("reason") or "not selected")
# Never archive plumbing, whatever the caller asked for.
if item["selected"] and (item.get("system") or item.get("kind") in ("special", "unknown")):
item["selected"] = False
item["reason"] = item.get("reason") or "not a regular file or directory"
return items
def _export_volume(full_name: str) -> bytes:
client = get_client()
_ensure_helper_image(client)
container = safe_call(
client.containers.create,
settings.BACKUP_HELPER_IMAGE,
command="true",
volumes={full_name: {"bind": "/v", "mode": "ro"}},
)
try:
# "/v/." copies the *contents* of the volume (no leading "v/" prefix),
# so restore can extract straight back into the volume root.
bits, _ = container.get_archive("/v/.")
buf = io.BytesIO()
for chunk in bits:
buf.write(chunk)
return buf.getvalue()
finally:
try:
container.remove(force=True)
except Exception: # noqa: BLE001
pass
def _restore_volume(full_name: str, labels: dict, tar_bytes: bytes) -> None:
client = get_client()
_ensure_helper_image(client)
try:
safe_call(client.volumes.get, full_name)
except DockerError:
safe_call(client.volumes.create, name=full_name, labels=labels or {})
container = safe_call(
client.containers.create,
settings.BACKUP_HELPER_IMAGE,
command="true",
volumes={full_name: {"bind": "/v", "mode": "rw"}},
)
try:
container.put_archive("/v", tar_bytes)
finally:
try:
container.remove(force=True)
except Exception: # noqa: BLE001
pass
def _compose_volumes(stack_id: str) -> list[dict]:
"""Return [{full, short, labels}] for compose-managed named volumes."""
try:
client = get_client()
vols = safe_call(
client.volumes.list,
filters={"label": f"{COMPOSE_PROJECT_LABEL}={stack_id}"},
)
except DockerError:
return []
out = []
for v in vols:
labels = v.attrs.get("Labels") or {}
out.append(
{
"full": v.name,
"short": labels.get(COMPOSE_VOLUME_LABEL, v.name),
"labels": labels,
}
)
return out
def plan(
stack_id: str,
include_volumes: bool = True,
include_binds: bool = True,
binds: Optional[list[str]] = None,
volumes: Optional[list[str]] = None,
) -> dict:
"""Decide what a backup captures. Returned as-is by the inventory endpoint."""
inv = assets.inventory(stack_id)
_decide(inv["binds"], "source", binds, include_binds)
_decide(inv["volumes"], "name", volumes, include_volumes)
return inv
# --------------------------------------------------------------------------- #
@@ -146,56 +108,170 @@ def _compose_volumes(stack_id: str) -> list[dict]:
# --------------------------------------------------------------------------- #
async def create_backup(
def _compose_filter(excluded_prefixes: list[str]):
"""Drop deselected bind directories from the compose/ tree (keep the mount
point itself, so the stack still starts after a restore)."""
def _filter(info: tarfile.TarInfo) -> Optional[tarfile.TarInfo]:
for prefix in excluded_prefixes:
if info.name.startswith(prefix + "/"):
return None
return info
return _filter
async def create_backup_ex(
stack_id: str,
name: str,
include_volumes: bool = True,
stop_first: bool = True,
) -> str:
"""Create a backup tar.gz and return its path on disk."""
include_binds: bool = True,
binds: Optional[list[str]] = None,
volumes: Optional[list[str]] = None,
) -> tuple[str, dict]:
"""Create a backup tar.gz. Returns (path, report)."""
directory = compose_service.stack_dir(stack_id)
if not os.path.isdir(directory):
raise BackupError("Stack directory missing")
volumes = _compose_volumes(stack_id) if include_volumes else []
selection = await asyncio.to_thread(
plan, stack_id, include_volumes, include_binds, binds, volumes
)
# Selected bind sources this process cannot reach through the filesystem
# need their own archive; the rest already ride along in compose/.
cap_binds = [b for b in selection["binds"] if b["selected"] and b["via"] == "archive"]
cap_volumes = [v for v in selection["volumes"] if v["selected"]]
skipped_binds = [b for b in selection["binds"] if not b["selected"]]
skipped_volumes = [v for v in selection["volumes"] if not v["selected"]]
# For a consistent volume snapshot, stop the stack first.
# Consistent snapshot: stop the stack first — but only if it is actually
# running, so backing up a stopped stack doesn't start it.
stopped = False
if include_volumes and stop_first and volumes:
try:
await compose_service.stop(stack_id)
stopped = True
except Exception as exc: # noqa: BLE001
logger.warning("Could not stop %s before backup: %s", stack_id, exc)
if stop_first and (cap_volumes or cap_binds):
if compose_service.compute_status(stack_id) not in ("stopped", "unknown"):
try:
await compose_service.stop(stack_id)
stopped = True
except Exception as exc: # noqa: BLE001
logger.warning("Could not stop %s before backup: %s", stack_id, exc)
workdir = tempfile.mkdtemp(prefix="sp-backup-")
try:
# Archive bind sources and volumes into the work directory first, so a
# failure on one asset is reported instead of corrupting the tar.
for index, bind in enumerate(cap_binds):
bind["archive"] = f"binds/{index:03d}.tar"
part = os.path.join(workdir, f"bind-{index:03d}.tar")
try:
bind["bytes"] = await asyncio.to_thread(
assets.export_path, bind["source"], bind["kind"], part
)
bind["_part"] = part
except Exception as exc: # noqa: BLE001
logger.warning("Could not archive bind %s: %s", bind["source"], exc)
bind["archive"] = None
bind["error"] = str(exc)
for index, vol in enumerate(cap_volumes):
vol["archive"] = f"volumes/{vol['name']}.tar"
part = os.path.join(workdir, f"vol-{index:03d}.tar")
try:
vol["bytes"] = await asyncio.to_thread(assets.export_volume, vol["name"], part)
vol["_part"] = part
except Exception as exc: # noqa: BLE001
logger.warning("Could not archive volume %s: %s", vol["name"], exc)
vol["archive"] = None
vol["error"] = str(exc)
# Deselected data that sits inside the stack folder must not sneak into
# the archive through compose/ (that is how a 200 GB downloads folder
# ends up in a "config only" backup).
excluded = [
"compose/" + os.path.relpath(b["source"], directory)
for b in skipped_binds
if b.get("inside_stack_dir")
]
manifest = {
"format_version": BACKUP_FORMAT_VERSION,
"stack_id": stack_id,
"name": name,
"created_at": compose_service.now().isoformat(),
"include_volumes": include_volumes,
"volumes": [{"full": v["full"], "short": v["short"], "labels": v["labels"]} for v in volumes],
"include_binds": include_binds,
"stack_dir": directory,
"path_mismatch": selection.get("path_mismatch"),
# Volumes keep the v1 shape (full/short/labels) so older StackPilots
# can still read the manifest they care about.
"volumes": [
{
"full": v["name"],
"short": v["short"],
"labels": v["labels"],
"archive": v.get("archive"),
"remote": v.get("remote", False),
"bytes": v.get("bytes"),
"error": v.get("error"),
}
for v in cap_volumes
if v.get("archive")
],
"binds": [
{
"source": b["source"],
"kind": b["kind"],
"mounts": b["mounts"],
"inside_stack_dir": b["inside_stack_dir"],
"archive": b.get("archive"),
"bytes": b.get("bytes"),
"error": b.get("error"),
}
for b in cap_binds
if b.get("archive")
],
"skipped": [
{"kind": "bind", "source": b["source"], "reason": b.get("reason")}
for b in skipped_binds
]
+ [
{"kind": "volume", "name": v["name"], "reason": v.get("reason")}
for v in skipped_volumes
],
}
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".tar.gz")
tmp.close()
with tarfile.open(tmp.name, "w:gz") as tar:
# manifest
data = json.dumps(manifest, indent=2).encode("utf-8")
info = tarfile.TarInfo(MANIFEST_NAME)
info.size = len(data)
tar.addfile(info, io.BytesIO(data))
# stack directory
tar.add(directory, arcname="compose")
# volume contents
for v in volumes:
vbytes = await asyncio.to_thread(_export_volume, v["full"])
info = tarfile.TarInfo(f"volumes/{v['full']}.tar")
info.size = len(vbytes)
tar.addfile(info, io.BytesIO(vbytes))
return tmp.name
tar.add(directory, arcname="compose", filter=_compose_filter(excluded))
for bind in cap_binds:
if bind.get("_part"):
tar.add(bind["_part"], arcname=bind["archive"])
for vol in cap_volumes:
if vol.get("_part"):
tar.add(vol["_part"], arcname=vol["archive"])
report = {
"file": tmp.name,
"binds": [
{"source": b["source"], "bytes": b.get("bytes"), "error": b.get("error")}
for b in cap_binds
],
"volumes": [
{"name": v["name"], "bytes": v.get("bytes"), "error": v.get("error")}
for v in cap_volumes
],
"skipped": manifest["skipped"],
"path_mismatch": selection.get("path_mismatch"),
"size": os.path.getsize(tmp.name),
}
return tmp.name, report
finally:
shutil.rmtree(workdir, ignore_errors=True)
if stopped:
try:
await compose_service.up(stack_id)
@@ -203,6 +279,21 @@ async def create_backup(
logger.warning("Could not restart %s after backup: %s", stack_id, exc)
async def create_backup(
stack_id: str,
name: str,
include_volumes: bool = True,
stop_first: bool = True,
include_binds: bool = True,
binds: Optional[list[str]] = None,
volumes: Optional[list[str]] = None,
) -> str:
path, _report = await create_backup_ex(
stack_id, name, include_volumes, stop_first, include_binds, binds, volumes
)
return path
# --------------------------------------------------------------------------- #
# Restore
# --------------------------------------------------------------------------- #
@@ -210,33 +301,87 @@ async def create_backup(
def read_manifest(tar_path: str) -> dict:
with tarfile.open(tar_path, "r:gz") as tar:
member = tar.getmember(MANIFEST_NAME)
try:
member = tar.getmember(MANIFEST_NAME)
except KeyError as exc:
raise BackupError("Backup is missing its manifest") from exc
fh = tar.extractfile(member)
if fh is None:
raise BackupError("Backup is missing its manifest")
return json.loads(fh.read().decode("utf-8"))
def _safe_extract_compose(tar: tarfile.TarFile, dest_dir: str) -> None:
"""Extract the ``compose/`` subtree into dest_dir, guarding path traversal."""
def _safe_target(dest_dir: str, rel: str) -> str:
"""Resolve a member path inside dest_dir, refusing traversal *and* writes
through a symlink planted earlier in the same archive."""
root = os.path.abspath(dest_dir)
target = os.path.normpath(os.path.join(root, rel))
if target != root and not target.startswith(root + os.sep):
raise BackupError(f"Refusing unsafe path in backup: {rel}")
parent = os.path.dirname(target)
if os.path.exists(parent):
real_parent = os.path.realpath(parent)
if real_parent != root and not real_parent.startswith(root + os.sep):
raise BackupError(f"Refusing unsafe path in backup: {rel}")
return target
def _apply_meta(path: str, member: tarfile.TarInfo) -> None:
"""Restore mode/ownership/mtime — *arr-style images run as PUID/PGID and
break when their config comes back root-owned with default permissions."""
try:
os.chmod(path, member.mode)
except OSError:
pass
try:
os.chown(path, member.uid, member.gid)
except (OSError, AttributeError):
pass
try:
os.utime(path, (member.mtime, member.mtime))
except OSError:
pass
def _extract_tree(tar: tarfile.TarFile, prefix: str, dest_dir: str) -> None:
"""Extract one subtree of the archive, preserving metadata and symlinks."""
os.makedirs(dest_dir, exist_ok=True)
dirs: list[tuple[str, tarfile.TarInfo]] = []
for member in tar.getmembers():
if not member.name.startswith("compose/"):
if not member.name.startswith(prefix):
continue
rel = member.name[len("compose/") :]
rel = member.name[len(prefix) :].lstrip("/")
if not rel:
continue
target = os.path.normpath(os.path.join(dest_dir, rel))
if not target.startswith(os.path.abspath(dest_dir) + os.sep) and target != os.path.abspath(dest_dir):
raise BackupError(f"Refusing unsafe path in backup: {member.name}")
target = _safe_target(dest_dir, rel)
if member.isdir():
os.makedirs(target, exist_ok=True)
elif member.isreg():
os.makedirs(os.path.dirname(target), exist_ok=True)
src = tar.extractfile(member)
if src is not None:
with open(target, "wb") as out:
shutil.copyfileobj(src, out)
dirs.append((target, member))
continue
os.makedirs(os.path.dirname(target), exist_ok=True)
if member.issym():
if os.path.lexists(target):
os.unlink(target)
os.symlink(member.linkname, target)
continue # chmod/utime would follow the link
if member.islnk():
source = _safe_target(dest_dir, member.linkname[len(prefix) :].lstrip("/"))
if os.path.exists(source):
if os.path.lexists(target):
os.unlink(target)
os.link(source, target)
continue
if not member.isreg():
continue # devices/fifos/sockets are runtime artefacts
src = tar.extractfile(member)
if src is None:
continue
with open(target, "wb") as out:
shutil.copyfileobj(src, out)
_apply_meta(target, member)
# Directory metadata last: a 0500 directory would block writing its files.
for target, member in sorted(dirs, key=lambda d: len(d[0]), reverse=True):
_apply_meta(target, member)
def restore_backup(
@@ -244,27 +389,73 @@ def restore_backup(
target_id: Optional[str] = None,
overwrite: bool = False,
restore_volumes: bool = True,
restore_binds: bool = True,
) -> dict:
"""Restore a backup. Returns {stack_id, name, volumes_restored}."""
"""Restore a backup. Returns a report of what was written."""
manifest = read_manifest(tar_path)
stack_id = target_id or manifest.get("stack_id")
if not stack_id:
raw_id = target_id or manifest.get("stack_id") or ""
if not raw_id.strip():
raise BackupError("Backup manifest has no stack id")
# Slugify whichever id we end up using — the manifest comes from an
# uploaded file, so its stack_id must never be able to escape STACKS_DIR.
stack_id = compose_service.slugify(raw_id)
old_id = manifest.get("stack_id") or stack_id
old_dir = manifest.get("stack_dir") or ""
directory = compose_service.stack_dir(stack_id)
exists = os.path.isdir(directory)
if exists and not overwrite:
raise BackupError(f"Stack '{stack_id}' already exists")
volumes_restored = 0
binds_restored = 0
skipped: list[dict] = []
with tarfile.open(tar_path, "r:gz") as tar:
if exists:
shutil.rmtree(directory)
_safe_extract_compose(tar, directory)
_extract_tree(tar, "compose/", directory)
if restore_binds:
for bind in manifest.get("binds", []):
archive = bind.get("archive")
if not archive:
continue
try:
member = tar.getmember(archive)
except KeyError:
continue
source = bind["source"]
# A renamed stack must not write into the old stack's folder.
if old_dir and bind.get("inside_stack_dir"):
rel = os.path.relpath(source, old_dir)
source = os.path.normpath(os.path.join(directory, rel))
if assets.is_system_path(source):
skipped.append({"kind": "bind", "source": source, "reason": "system path"})
continue
fh = tar.extractfile(member)
if fh is None:
continue
with tempfile.NamedTemporaryFile(delete=False, suffix=".tar") as tmp:
shutil.copyfileobj(fh, tmp)
part = tmp.name
try:
assets.import_path(source, bind.get("kind", "dir"), part)
binds_restored += 1
except Exception as exc: # noqa: BLE001
logger.warning("Could not restore bind %s: %s", source, exc)
skipped.append({"kind": "bind", "source": source, "reason": str(exc)})
finally:
os.unlink(part)
elif manifest.get("binds"):
skipped += [
{"kind": "bind", "source": b["source"], "reason": "not requested"}
for b in manifest["binds"]
]
volumes_restored = 0
if restore_volumes:
for v in manifest.get("volumes", []):
member_name = f"volumes/{v['full']}.tar"
member_name = v.get("archive") or f"volumes/{v['full']}.tar"
try:
member = tar.getmember(member_name)
except KeyError:
@@ -276,13 +467,27 @@ def restore_backup(
labels = dict(v.get("labels") or {})
labels[COMPOSE_PROJECT_LABEL] = stack_id
full = v["full"]
if target_id and manifest.get("stack_id") and full.startswith(manifest["stack_id"] + "_"):
full = stack_id + full[len(manifest["stack_id"]):]
_restore_volume(full, labels, fh.read())
volumes_restored += 1
if target_id and old_id and full.startswith(old_id + "_"):
full = stack_id + full[len(old_id) :]
with tempfile.NamedTemporaryFile(delete=False, suffix=".tar") as tmp:
shutil.copyfileobj(fh, tmp)
part = tmp.name
try:
# Remote-backed volumes (NFS/CIFS) are never wiped: that
# would delete the share the volume points at.
assets.import_volume(full, labels, part, wipe=not v.get("remote", False))
volumes_restored += 1
except Exception as exc: # noqa: BLE001
logger.warning("Could not restore volume %s: %s", full, exc)
skipped.append({"kind": "volume", "name": full, "reason": str(exc)})
finally:
os.unlink(part)
skipped += manifest.get("skipped", [])
return {
"stack_id": stack_id,
"name": manifest.get("name", stack_id),
"volumes_restored": volumes_restored,
"binds_restored": binds_restored,
"skipped": skipped,
}
backend/services/compose_service.py
+92 -48
View File
@@ -16,6 +16,7 @@ from typing import Optional
from config import settings
from docker_client import DockerError, get_client, safe_call
from services import registry_service
COMPOSE_FILENAMES = ("compose.yaml", "compose.yml", "docker-compose.yml", "docker-compose.yaml")
DEFAULT_COMPOSE_NAME = "compose.yaml"
@@ -199,22 +200,6 @@ def containers_for_stack(stack_id: str) -> list[ContainerInfo]:
return result
# in-memory set of stacks currently performing a pull/up
_BUSY: set[str] = set()
def mark_busy(stack_id: str) -> None:
_BUSY.add(stack_id)
def clear_busy(stack_id: str) -> None:
_BUSY.discard(stack_id)
def is_busy(stack_id: str) -> bool:
return stack_id in _BUSY
def _status_from_states(states: list[str]) -> str:
if not states:
return "stopped"
@@ -229,10 +214,13 @@ def _status_from_states(states: list[str]) -> str:
def compute_status(stack_id: str, containers: Optional[list[ContainerInfo]] = None) -> str:
"""Status for one stack. Pass already-fetched ``containers`` to avoid a
redundant Docker round-trip (the detail view already has them)."""
if stack_id in _BUSY:
return "updating"
"""Status for one stack, from its containers alone.
"updating" is not derived here: whether an operation is in flight lives in
``stack_lock_service``, and callers that want to show it overlay the lock on
top of this. Pass already-fetched ``containers`` to avoid a redundant Docker
round-trip (the detail view already has them).
"""
try:
if containers is None:
containers = containers_for_stack(stack_id)
@@ -298,6 +286,7 @@ async def run_compose(
cmd = _compose_base_cmd(stack_id, override) + args
proc = await asyncio.create_subprocess_exec(
*cmd,
env=registry_service.cli_env(),
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
@@ -335,6 +324,7 @@ async def validate_yaml(content: str, env_content: str = "") -> dict:
try:
proc = await asyncio.create_subprocess_exec(
*cmd,
env=registry_service.cli_env(),
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
@@ -352,6 +342,7 @@ async def stream_compose(
cmd = _compose_base_cmd(stack_id, override) + args
proc = await asyncio.create_subprocess_exec(
*cmd,
env=registry_service.cli_env(),
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
)
@@ -361,15 +352,46 @@ async def stream_compose(
await proc.wait()
async def stream_up(stack_id: str, override: Optional[str] = None):
"""Run `compose up -d` streaming combined output, so the deploy console can
show image-pull and container-create progress live.
_json_progress: Optional[bool] = None
Yields ``("log", line)`` for each output line, then ``("done", returncode)``.
async def supports_json_progress() -> bool:
"""Whether this Docker Compose understands ``--progress json``.
The JSON progress stream carries per-layer ``current``/``total`` bytes, which
the deploy console turns into a real progress bar. Older compose releases
reject the value, so probe once (cheap, no side effects) and cache it; on a
negative result callers fall back to the plain text stream.
"""
cmd = _compose_base_cmd(stack_id, override) + ["up", "-d", "--remove-orphans"]
global _json_progress
if _json_progress is None:
try:
proc = await asyncio.create_subprocess_exec(
"docker", "compose", "--progress", "json", "version",
env=registry_service.cli_env(),
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.DEVNULL,
)
rc = await asyncio.wait_for(proc.wait(), timeout=15.0)
_json_progress = rc == 0
except Exception: # noqa: BLE001 - probe failure just disables the feature
_json_progress = False
return _json_progress
async def _stream_phase(
stack_id: str, args: list[str], override: Optional[str], json_progress: bool
):
"""One compose subcommand, streamed. Yields ``("log", line)`` per output
line, then ``("rc", returncode)`` exactly once."""
cmd = _compose_base_cmd(stack_id, override)
if json_progress:
# Global flag, must precede the subcommand.
cmd += ["--progress", "json"]
cmd += args
proc = await asyncio.create_subprocess_exec(
*cmd,
env=registry_service.cli_env(),
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
)
@@ -377,18 +399,48 @@ async def stream_up(stack_id: str, override: Optional[str] = None):
async for raw in proc.stdout:
yield ("log", raw.decode("utf-8", "replace").rstrip("\n"))
await proc.wait()
yield ("done", proc.returncode)
yield ("rc", proc.returncode)
async def stream_up(stack_id: str, override: Optional[str] = None):
"""Run `compose up -d` streaming combined output, so the deploy console can
show image-pull and container-create progress live.
Yields ``("log", line)`` for each output line, then ``("done", returncode)``.
"""
json_progress = await supports_json_progress()
async for kind, payload in _stream_phase(
stack_id, ["up", "-d", "--remove-orphans"], override, json_progress
):
yield ("done", payload) if kind == "rc" else ("log", payload)
async def stream_update(stack_id: str, override: Optional[str] = None):
"""Run `compose pull` then `compose up -d`, streaming both phases, so the
stacks list can show real update progress instead of a spinner.
Yields ``("log", line)`` for each output line of either phase, then
``("done", returncode)`` once. A failed pull short-circuits: recreating
containers on images that never came down would only make things worse.
"""
json_progress = await supports_json_progress()
rc = 0
for args in (["pull"], ["up", "-d", "--remove-orphans"]):
async for kind, payload in _stream_phase(stack_id, args, override, json_progress):
if kind == "log":
yield ("log", payload)
else:
rc = payload
if rc != 0:
break
yield ("done", rc)
# Convenience lifecycle wrappers ------------------------------------------------
async def up(stack_id: str, override: Optional[str] = None) -> dict:
mark_busy(stack_id)
try:
return await run_compose(stack_id, ["up", "-d", "--remove-orphans"], override)
finally:
clear_busy(stack_id)
return await run_compose(stack_id, ["up", "-d", "--remove-orphans"], override)
async def down(stack_id: str, override: Optional[str] = None) -> dict:
@@ -408,27 +460,19 @@ async def restart(stack_id: str, override: Optional[str] = None) -> dict:
async def pull(stack_id: str, override: Optional[str] = None) -> dict:
mark_busy(stack_id)
try:
return await run_compose(stack_id, ["pull"], override)
finally:
clear_busy(stack_id)
return await run_compose(stack_id, ["pull"], override)
async def update(stack_id: str, override: Optional[str] = None) -> dict:
"""Pull then up -d."""
mark_busy(stack_id)
try:
pull_res = await run_compose(stack_id, ["pull"], override)
up_res = await run_compose(stack_id, ["up", "-d", "--remove-orphans"], override)
return {
"returncode": up_res["returncode"],
"stdout": pull_res["stdout"] + "\n" + up_res["stdout"],
"stderr": pull_res["stderr"] + "\n" + up_res["stderr"],
"command": "pull + up -d",
}
finally:
clear_busy(stack_id)
pull_res = await run_compose(stack_id, ["pull"], override)
up_res = await run_compose(stack_id, ["up", "-d", "--remove-orphans"], override)
return {
"returncode": up_res["returncode"],
"stdout": pull_res["stdout"] + "\n" + up_res["stdout"],
"stderr": pull_res["stderr"] + "\n" + up_res["stderr"],
"command": "pull + up -d",
}
async def logs(
backend/services/container_service.py
+1 -1
View File
@@ -1,4 +1,4 @@
"""Single-container inspect + lifecycle — shared by the central app and agent.
"""Single-container inspect + lifecycle.
Only containers that belong to a compose-managed stack (i.e. carry the
``com.docker.compose.project`` label) are exposed, so this never becomes a
backend/services/crypto_service.py
+76
View File
@@ -0,0 +1,76 @@
"""Symmetric encryption for secrets that have to live in the database.
Most of StackPilot's secrets are files on disk (``.env``, ``.secrets/*``) where
filesystem permissions are the right control. A few can't be: backup
destination credentials are needed by background jobs, so they sit in
``stackpilot.db``. This module encrypts those at rest.
The key is derived from ``SECRET_KEY`` rather than being a second thing to
configure — which is exactly why ``SECRET_KEY`` is now persisted (see
``config._ensure_secret``): a key that changed on every restart would take the
ciphertext with it.
Ciphertext is stored with an ``enc:v1:`` prefix so plaintext rows written by
older versions stay recognisable and can be migrated in place.
"""
from __future__ import annotations
import base64
import hashlib
import logging
from typing import Optional
from cryptography.fernet import Fernet, InvalidToken
from config import settings
logger = logging.getLogger("stackpilot.crypto")
PREFIX = "enc:v1:"
_INFO = b"stackpilot-db-field-encryption-v1"
class DecryptError(Exception):
"""Ciphertext could not be decrypted (usually: SECRET_KEY changed)."""
def _fernet() -> Fernet:
"""Fernet built from a 32-byte key derived from SECRET_KEY.
Not cached: SECRET_KEY is fixed for the process lifetime, and building a
Fernet is a hash plus a base64 encode — cheap enough not to bother.
"""
digest = hashlib.blake2b(
settings.SECRET_KEY.encode("utf-8"), key=_INFO, digest_size=32
).digest()
return Fernet(base64.urlsafe_b64encode(digest))
def is_encrypted(value: Optional[str]) -> bool:
return bool(value) and value.startswith(PREFIX)
def encrypt(plaintext: str) -> str:
"""Encrypt a string. Already-encrypted input is returned unchanged."""
if is_encrypted(plaintext):
return plaintext
token = _fernet().encrypt((plaintext or "").encode("utf-8"))
return PREFIX + token.decode("ascii")
def decrypt(value: str) -> str:
"""Decrypt a value written by :func:`encrypt`.
Plaintext (no prefix) is passed straight through, so rows written before
encryption existed keep working until the startup migration rewrites them.
"""
if not is_encrypted(value):
return value or ""
try:
return _fernet().decrypt(value[len(PREFIX):].encode("ascii")).decode("utf-8")
except (InvalidToken, ValueError) as exc:
raise DecryptError(
"Could not decrypt a stored secret. This normally means SECRET_KEY "
"changed since it was saved — restore the old key, or re-enter the "
"affected credentials."
) from exc
backend/services/dashboard_service.py
+176 -162
View File
@@ -1,39 +1,35 @@
"""Aggregated dashboard data: stack-health funnel + summary widgets.
"""Host aggregate for the dashboard cockpit.
Everything here is read-only and cheap by construction: one container
*summary* list (no per-container inspect) feeds the whole funnel, image
One read-only call rolls the host up into a "needs attention" list, headline
KPIs and a resource view. It is cheap by construction: a single container
*summary* list (no per-container inspect) drives the figures, and image
freshness comes from the cache the update-service background loop already
maintains, and the daily uptime sample is appended lazily on read.
maintains.
"""
from __future__ import annotations
import asyncio
import json
import os
import time
from datetime import datetime, timedelta, timezone
from typing import Optional
from sqlmodel import Session, select
from config import settings
from docker_client import DockerError, get_client, safe_call
from models.audit import AuditLog
from models.setting import Webhook
from models.backup_schedule import BackupSchedule
from services import compose_service, update_service
COMPOSE_LABEL = compose_service.COMPOSE_LABEL
DOCKER_TIMEOUT = 5.0 # seconds — a slow daemon must not stall the dashboard
FUNNEL_TTL = 30.0
UPTIME_FILE = os.path.join(settings.DATA_DIR, "uptime.jsonl")
UPTIME_DAYS = 30
# Cached briefly: the dashboard polls this and the rollup is not free.
FLEET_TTL = 25.0
DISK_PRESSURE = 0.85 # disk used fraction above which a host needs attention
MEM_PRESSURE = 0.90 # memory used fraction above which a host needs attention
_funnel_cache: dict = {"data": None, "ts": 0.0}
_fleet_cache: dict = {"data": None, "ts": 0.0}
# --------------------------------------------------------------------------- #
# Container summary (single Docker round-trip)
# Container summary (single Docker round-trip) + shared classifiers
# --------------------------------------------------------------------------- #
@@ -42,10 +38,6 @@ def _list_containers() -> list[dict]:
return safe_call(client.api.containers, all=True)
async def _containers_with_timeout() -> list[dict]:
return await asyncio.wait_for(asyncio.to_thread(_list_containers), timeout=DOCKER_TIMEOUT)
def _group_by_project(raw: list[dict]) -> dict[str, list[dict]]:
by_project: dict[str, list[dict]] = {}
for c in raw:
@@ -77,160 +69,182 @@ def _is_updated(containers: list[dict], cache: dict[str, dict]) -> bool:
return True
def _has_notify_target(session: Session) -> bool:
if settings.NOTIFY_WEBHOOKS:
return True
for wh in session.exec(select(Webhook)).all():
if wh.enabled:
return True
return False
# --------------------------------------------------------------------------- #
# Host aggregate (one call → "needs attention" + KPIs)
#
# The dashboard used to fetch these numbers per stack and recombine them
# client-side. ``compute_fleet`` rolls them up server-side instead, off one
# Docker pass plus the update cache, and holds the result for FLEET_TTL because
# the dashboard polls it.
# --------------------------------------------------------------------------- #
# Stack-status buckets the status bar / KPIs are built from. Anything reporting
# "error" or "dead" containers counts as a problem stack.
_PROBLEM_STATUSES = {"error", "dead"}
async def compute_funnel(session: Session, refresh: bool = False) -> dict:
now = time.time()
if not refresh and _funnel_cache["data"] and now - _funnel_cache["ts"] < FUNNEL_TTL:
return _funnel_cache["data"]
def _bucket_statuses(statuses: list[str]) -> dict[str, int]:
return {
"running": sum(1 for s in statuses if s == "running"),
"partial": sum(1 for s in statuses if s == "partial"),
"stopped": sum(1 for s in statuses if s in ("stopped", "exited")),
"error": sum(1 for s in statuses if s in _PROBLEM_STATUSES),
"total": len(statuses),
}
discovered_ids = compose_service.discover_stacks()
def _attn(severity: str, kind: str, host: str, title: str, detail: str, link: str) -> dict:
return {
"severity": severity,
"kind": kind,
"host": host,
"title": title,
"detail": detail,
"link": link,
}
def _resource_attention(name: str, link: str, mem_used: int, mem_total: int,
disk_used: int, disk_total: int) -> list[dict]:
items: list[dict] = []
if mem_total and mem_used / mem_total >= MEM_PRESSURE:
pct = round(mem_used / mem_total * 100)
items.append(_attn("warn", "mem_pressure", name,
f"{name}: memory at {pct}%", "Free memory or move stacks.", link))
if disk_total and disk_used / disk_total >= DISK_PRESSURE:
pct = round(disk_used / disk_total * 100)
items.append(_attn("warn", "disk_pressure", name,
f"{name}: disk at {pct}%", "Prune images/volumes or add capacity.", link))
return items
def _local_host() -> tuple[dict, list[dict]]:
"""Local host card + attention items from a single Docker container pass."""
discovered = compose_service.discover_stacks()
try:
raw = await _containers_with_timeout()
except (DockerError, asyncio.TimeoutError):
raw = _list_containers()
except DockerError:
raw = []
by_project = _group_by_project(raw)
update_cache = update_service.get_cache()
notify_configured = _has_notify_target(session)
running = healthy = updated = monitored = 0
for stack_id in discovered_ids:
statuses: list[str] = []
unhealthy: list[str] = []
updates = 0
for stack_id in discovered:
containers = by_project.get(stack_id, [])
states = [c.get("State", "") for c in containers]
if not states or any(s != "running" for s in states):
continue
running += 1
if not _is_healthy(containers):
continue
healthy += 1
status = compose_service._status_from_states([c.get("State", "") for c in containers])
statuses.append(status)
if status == "running" and not _is_healthy(containers):
unhealthy.append(stack_id)
if not _is_updated(containers, update_cache):
continue
updated += 1
if notify_configured:
monitored += 1
updates += 1
buckets = _bucket_statuses(statuses)
labelled = [c for c in raw if (c.get("Labels") or {}).get(COMPOSE_LABEL)]
# Local resource figures (lazy import keeps dashboard_service free of a
# router dependency at module load time).
from routers.system import _cpu_count, _disk_usage, _mem_info
mem = _mem_info()
disk = _disk_usage()
host = {
"id": "local",
"name": "local",
"online": True,
"status": "online",
"cpu_cores": _cpu_count(),
"mem_used": mem["used"], "mem_total": mem["total"],
"disk_used": disk["used"], "disk_total": disk["total"],
"stacks": buckets,
"containers_running": sum(1 for c in labelled if c.get("State") == "running"),
"containers_total": len(labelled),
"unhealthy": len(unhealthy),
"updates_available": updates,
}
attention: list[dict] = []
for sid in unhealthy:
attention.append(_attn("error", "unhealthy", "local",
f"{sid} is unhealthy", "A container is failing its healthcheck.",
f"/stacks/{sid}"))
if buckets["error"]:
attention.append(_attn("error", "stack_error", "local",
f"{buckets['error']} stack(s) in error", "Containers are dead.", "/stacks"))
if buckets["partial"]:
attention.append(_attn("warn", "stack_partial", "local",
f"{buckets['partial']} stack(s) partially running",
"Some services are down.", "/stacks"))
if updates:
attention.append(_attn("warn", "updates", "local",
f"{updates} stack(s) have image updates", "Pull the newer images.", "/images"))
attention += _resource_attention("local", "/", mem["used"], mem["total"],
disk["used"], disk["total"])
return host, attention
def _backup_attention(session: Session) -> list[dict]:
"""Flag enabled backup schedules whose last run failed or is overdue."""
now = datetime.now(timezone.utc)
items: list[dict] = []
for sch in session.exec(select(BackupSchedule).where(BackupSchedule.enabled == True)).all(): # noqa: E712
host = "local"
status = (sch.last_status or "").lower()
if status and not status.startswith("ok"):
items.append(_attn("error", "backup_failed", host,
f"Backup of {sch.stack_id} failed", sch.last_status or "",
"/settings"))
elif sch.next_run and _aware(sch.next_run) < now - timedelta(hours=1):
items.append(_attn("warn", "backup_overdue", host,
f"Backup of {sch.stack_id} is overdue",
"Scheduled run did not happen.", "/settings"))
return items
def _aware(dt: datetime) -> datetime:
"""Treat naive DB timestamps as UTC (they're stored that way)."""
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
_SEVERITY_ORDER = {"error": 0, "warn": 1}
async def compute_fleet(session: Session, refresh: bool = False) -> dict:
now = time.time()
if not refresh and _fleet_cache["data"] and now - _fleet_cache["ts"] < FLEET_TTL:
return _fleet_cache["data"]
local_host, attention = await asyncio.to_thread(_local_host)
hosts = [local_host]
attention += _backup_attention(session)
attention.sort(key=lambda a: _SEVERITY_ORDER.get(a["severity"], 9))
kpis = {
"stacks_running": sum(h["stacks"]["running"] for h in hosts),
"stacks_partial": sum(h["stacks"]["partial"] for h in hosts),
"stacks_total": sum(h["stacks"]["total"] for h in hosts),
"containers_running": sum(h["containers_running"] for h in hosts),
"containers_total": sum(h["containers_total"] for h in hosts),
"unhealthy": sum(h["unhealthy"] for h in hosts),
"updates_available": sum(h["updates_available"] for h in hosts),
"backups_failing": sum(1 for a in attention if a["kind"] in ("backup_failed", "backup_overdue")),
}
status_totals = {
"running": kpis["stacks_running"],
"partial": kpis["stacks_partial"],
"stopped": sum(h["stacks"]["stopped"] for h in hosts),
"error": sum(h["stacks"]["error"] for h in hosts),
}
data = {
"discovered": len(discovered_ids),
"running": running,
"healthy": healthy,
"updated": updated,
"monitored": monitored,
"as_of": datetime.now(timezone.utc).isoformat(),
"hosts": hosts,
"kpis": kpis,
"status_totals": status_totals,
"attention": attention,
}
_funnel_cache["data"] = data
_funnel_cache["ts"] = now
_fleet_cache["data"] = data
_fleet_cache["ts"] = now
return data
# --------------------------------------------------------------------------- #
# Uptime series (one sample per day, JSONL on disk)
# --------------------------------------------------------------------------- #
def _read_uptime() -> list[dict]:
if not os.path.isfile(UPTIME_FILE):
return []
entries = []
with open(UPTIME_FILE, "r", encoding="utf-8") as fh:
for line in fh:
line = line.strip()
if not line:
continue
try:
entries.append(json.loads(line))
except json.JSONDecodeError:
continue
return entries
def _append_uptime(entry: dict) -> None:
os.makedirs(settings.DATA_DIR, exist_ok=True)
with open(UPTIME_FILE, "a", encoding="utf-8") as fh:
fh.write(json.dumps(entry) + "\n")
def _sample_uptime(raw: list[dict]) -> Optional[dict]:
"""Append today's sample if not yet recorded. Uptime% = share of compose
containers currently running."""
today = datetime.now(timezone.utc).date().isoformat()
entries = _read_uptime()
if any(e.get("date") == today for e in entries):
return None
labelled = [c for c in raw if (c.get("Labels") or {}).get(COMPOSE_LABEL)]
total = len(labelled)
running = sum(1 for c in labelled if c.get("State") == "running")
value = round(running / total * 100, 1) if total else 100.0
entry = {"date": today, "value": value}
_append_uptime(entry)
return entry
def uptime_series(raw: list[dict]) -> list[dict]:
_sample_uptime(raw)
entries = _read_uptime()
by_date = {e["date"]: e for e in entries if "date" in e}
series = []
today = datetime.now(timezone.utc).date()
last_value: Optional[float] = None
for i in range(UPTIME_DAYS - 1, -1, -1):
day = (today - timedelta(days=i)).isoformat()
e = by_date.get(day)
if e is not None:
last_value = e.get("value")
# Days before monitoring started (or gaps) reuse the last known value
# so the chart doesn't show artificial dips.
series.append({"date": day, "value": last_value})
return series
# --------------------------------------------------------------------------- #
# Ops (audit-log) activity
# --------------------------------------------------------------------------- #
_WEEKDAYS = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
def ops_activity(session: Session) -> tuple[list[dict], Optional[str]]:
today = datetime.now(timezone.utc).date()
cutoff = datetime.combine(today - timedelta(days=UPTIME_DAYS - 1), datetime.min.time(), timezone.utc)
timestamps = session.exec(
select(AuditLog.timestamp).where(AuditLog.timestamp >= cutoff)
).all()
per_day: dict[str, int] = {}
per_weekday = [0] * 7
for ts in timestamps:
per_day[ts.date().isoformat()] = per_day.get(ts.date().isoformat(), 0) + 1
per_weekday[ts.weekday()] += 1
series = []
for i in range(UPTIME_DAYS - 1, -1, -1):
day = (today - timedelta(days=i)).isoformat()
series.append({"date": day, "count": per_day.get(day, 0)})
peak = _WEEKDAYS[per_weekday.index(max(per_weekday))] if any(per_weekday) else None
return series, peak
async def compute_summary(session: Session) -> dict:
try:
raw = await _containers_with_timeout()
except (DockerError, asyncio.TimeoutError):
raw = []
labelled = [c for c in raw if (c.get("Labels") or {}).get(COMPOSE_LABEL)]
ops_series, peak = ops_activity(session)
return {
"total_containers": sum(1 for c in labelled if c.get("State") == "running"),
"containers_total": len(labelled),
"uptime_series": uptime_series(raw),
"ops_last_30d": ops_series,
"ops_peak_day": peak,
"as_of": datetime.now(timezone.utc).isoformat(),
}
backend/services/device_service.py
+19 -9
View File
@@ -4,7 +4,6 @@ from __future__ import annotations
import glob
import os
from dataclasses import asdict, dataclass
from typing import Optional
from config import settings
@@ -87,12 +86,27 @@ def detect_devices() -> dict:
# --------------------------------------------------------------------------- #
class BrowseError(Exception):
pass
def _real_root(path: str) -> str:
"""Map a logical host path into the container view (HOST_ROOT_PREFIX)."""
"""Map a logical host path into the container view (HOST_ROOT_PREFIX).
Refuses anything that resolves inside StackPilot's own ``DATA_DIR``. That
directory holds ``stackpilot.db`` — users, password hashes and
backup-destination credentials — and the API deliberately never hands those
out (destination secrets come back masked). Without this the file browser
would be a way around that, for admins too. Note this only bites when ``HOST_ROOT_PREFIX`` is empty: with a
prefix set, no logical path can reach the container's own ``/data`` at all.
"""
prefix = settings.HOST_ROOT_PREFIX.rstrip("/")
if prefix:
return prefix + path
return path
real = prefix + path if prefix else path
data_dir = os.path.normpath(settings.DATA_DIR)
norm = os.path.normpath(real)
if norm == data_dir or norm.startswith(data_dir + os.sep):
raise BrowseError("Path is inside StackPilot's own data directory")
return real
def _is_allowed(path: str) -> bool:
@@ -104,10 +118,6 @@ def _is_allowed(path: str) -> bool:
return False
class BrowseError(Exception):
pass
def browse(path: str = "/", show_hidden: bool = False) -> dict:
path = os.path.normpath(path or "/")
if not path.startswith("/"):
backend/services/exec_service.py
+1 -2
View File
@@ -14,7 +14,7 @@ import socket as _socket
from starlette.websockets import WebSocketDisconnect
from docker_client import DockerError, get_client, safe_call
from docker_client import get_client, safe_call
from services.container_service import _get_managed
DEFAULT_SHELL = "/bin/sh"
@@ -66,7 +66,6 @@ def exec_exit_code(exec_id: str):
async def pump_exec(websocket, exec_id: str, holder, raw) -> None:
"""Bidirectionally pump an exec socket <-> a WebSocket.
Shared by the central app and the agent (both pass a Starlette WebSocket).
Browser -> container: JSON ``{"type":"data","data":...}`` keystrokes and
``{"type":"resize","rows","cols"}`` control frames (raw text is also
accepted as keystrokes). Container -> browser: ``{"type":"data","data":...}``
backend/services/file_service.py
+87
View File
@@ -10,6 +10,8 @@ from __future__ import annotations
import os
import shutil
import zipfile
from collections.abc import Iterator
from services.device_service import BrowseError, _is_allowed, _real_root
@@ -175,6 +177,91 @@ def resolve_download(path: str) -> tuple[str, str]:
return real, os.path.basename(path)
def is_dir(path: str) -> bool:
"""Whether ``path`` points at a directory inside the sandbox."""
return os.path.isdir(_safe_real(path))
class _ZipBuffer:
"""A writable sink that hands out and clears whatever was written to it.
Lets us drive ``zipfile`` while draining its output incrementally so the
archive can be streamed to the client instead of buffered to disk.
"""
def __init__(self) -> None:
self._buf = bytearray()
def write(self, data: bytes) -> int:
self._buf += data
return len(data)
def flush(self) -> None: # pragma: no cover - zipfile calls this
pass
def take(self) -> bytes:
data = bytes(self._buf)
self._buf.clear()
return data
def open_archive(path: str) -> tuple[str, "Iterator[bytes]"]:
"""Validate a directory and return ``(download_filename, byte_iterator)``.
The iterator zips the directory recursively **on the fly**, yielding bytes
as they are produced so the response starts immediately (no waiting for the
whole archive to build → no gateway timeout) and memory stays bounded.
Only regular files and real subdirectories are archived. Symlinks are
skipped (no sandbox escape / loops); special files (FIFOs, sockets,
devices) are skipped too — opening a FIFO would block forever and a socket
can't be read at all. Files that can't be read (permissions, or that vanish
mid-walk) are skipped individually rather than aborting the whole archive.
"""
real = _safe_real(path)
if not os.path.isdir(real):
raise BrowseError(f"Not a directory: {path}")
name = os.path.basename(path.rstrip("/")) or "root"
return f"{name}.zip", _iter_zip(real, name)
def _iter_zip(real: str, name: str):
sink = _ZipBuffer()
with zipfile.ZipFile(sink, "w", zipfile.ZIP_DEFLATED) as zf:
for root, dirs, files in os.walk(real):
# Don't follow symlinked directories (avoids loops / escapes).
dirs[:] = [d for d in dirs if not os.path.islink(os.path.join(root, d))]
rel_root = os.path.relpath(root, real)
if not files and not dirs and rel_root != ".":
# Preserve otherwise-empty directories.
zf.writestr(os.path.join(name, rel_root) + "/", "")
if chunk := sink.take():
yield chunk
for f in files:
full = os.path.join(root, f)
# os.path.isfile follows symlinks; combined with the islink
# check it admits only real regular files (skips FIFOs, sockets,
# devices and symlinks without ever open()-ing them).
if os.path.islink(full) or not os.path.isfile(full):
continue
arc = (os.path.join(name, rel_root, f) if rel_root != "."
else os.path.join(name, f))
try:
info = zipfile.ZipInfo.from_file(full, arc)
info.compress_type = zipfile.ZIP_DEFLATED
with open(full, "rb") as src, zf.open(info, "w") as dest:
while buf := src.read(1024 * 1024):
dest.write(buf)
if chunk := sink.take():
yield chunk
except OSError:
# Unreadable or vanished mid-walk — skip just this file.
continue
if chunk := sink.take():
yield chunk
yield sink.take()
def upload_target(
dir_path: str,
filename: str,
backend/services/git_service.py
+447
View File
@@ -0,0 +1,447 @@
"""Deploying a stack from a Git repository.
The repository is the source of truth: a sync makes the stack's files match what
the repo says, and optionally runs ``compose up -d`` when that changed anything.
Two things about *how* are worth stating up front, because both are places this
could quietly destroy data.
**The clone does not live in the stack folder.** It is cached under
``${DATA_DIR}/git/<stack_id>`` and the relevant subtree is copied across. A
stack folder holds more than the repo's files — compose creates bind-mount
directories like ``./config`` right there, full of live application data — so a
``git reset --hard`` or ``git clean`` in that folder would be catastrophic. In
the cache directory both are safe, and the copy step is where the care goes.
**Only files the repo has provided are ever deleted.** Each sync records the
paths it wrote (``GitSource.managed_files``); the next sync removes the ones the
repo no longer has, and nothing else. A file that was never in the repository
cannot be touched, no matter what happened to it.
Credentials never reach a command line. A token is passed to git through
``GIT_ASKPASS`` and an environment variable, an SSH key through a 0600 file
outside the working tree — so neither shows up in ``ps``, in the repo's own
config, or in an error message this module passes on.
"""
from __future__ import annotations
import asyncio
import filecmp
import hmac
import json
import logging
import os
import re
import secrets
import shutil
import stat
from datetime import datetime, timezone
from typing import Optional
from sqlmodel import Session
from config import settings
from models.git_source import GitSource, SyncResult
from services import compose_service, crypto_service, stack_lock_service
logger = logging.getLogger("stackpilot.git")
GIT_TIMEOUT = 300.0
_SAFE_ID_RE = re.compile(r"^[A-Za-z0-9._-]{1,128}$")
class GitError(Exception):
"""A repository that cannot be reached, or a sync that cannot be completed."""
# --------------------------------------------------------------------------- #
# Paths
# --------------------------------------------------------------------------- #
def cache_root() -> str:
return os.path.join(settings.DATA_DIR, "git")
def repo_dir(stack_id: str) -> str:
if not _SAFE_ID_RE.match(stack_id or "") or stack_id in (".", ".."):
raise GitError(f"Invalid stack id '{stack_id}'")
return os.path.join(cache_root(), stack_id)
def _key_path(stack_id: str) -> str:
# Beside the clone, never inside it — a working tree gets reset and cleaned.
return os.path.join(cache_root(), f"{stack_id}.key")
def _askpass_path(stack_id: str) -> str:
return os.path.join(cache_root(), f"{stack_id}.askpass")
def new_webhook_secret() -> str:
return secrets.token_urlsafe(24)
# --------------------------------------------------------------------------- #
# Running git
# --------------------------------------------------------------------------- #
def _redact(text: str, *secrets_: Optional[str]) -> str:
"""Strip anything secret out of git's output before it is shown or stored."""
for value in secrets_:
if value:
text = text.replace(value, "••••••")
# A URL that carries credentials, in case one ever reaches git's output.
return re.sub(r"(https?://)[^/\s:@]+:[^/\s@]+@", r"\1••••••@", text)
async def _git(args: list[str], env: dict, secret: Optional[str] = None) -> str:
"""Run git, returning stdout. Raises GitError with a redacted message."""
proc = await asyncio.create_subprocess_exec(
"git",
*args,
env=env,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
try:
out_b, err_b = await asyncio.wait_for(proc.communicate(), timeout=GIT_TIMEOUT)
except asyncio.TimeoutError as exc:
proc.kill()
raise GitError("git timed out") from exc
out = out_b.decode("utf-8", "replace")
if proc.returncode != 0:
err = _redact(err_b.decode("utf-8", "replace").strip(), secret)
raise GitError(err or f"git {args[0]} failed (exit {proc.returncode})")
return out
def _auth_env(source: GitSource) -> tuple[dict, Optional[str]]:
"""Environment for git, plus the plaintext secret so output can be redacted.
Credentials go in the environment, never in argv: ``ps`` is readable by
every process on the host, and StackPilot runs in a container people share
with their whole stack.
"""
env = {
**os.environ,
# No interactive prompting: a private repo without credentials must fail
# fast rather than hang forever waiting on a terminal that is not there.
"GIT_TERMINAL_PROMPT": "0",
"GIT_CONFIG_NOSYSTEM": "1",
"HOME": cache_root(),
}
if source.auth_type == "none" or not source.secret:
return env, None
plaintext = crypto_service.decrypt(source.secret)
os.makedirs(cache_root(), mode=0o700, exist_ok=True)
if source.auth_type == "ssh":
key_file = _key_path(source.stack_id)
fd = os.open(key_file, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(plaintext.rstrip("\n") + "\n")
env["GIT_SSH_COMMAND"] = (
f"ssh -i {key_file} -o IdentitiesOnly=yes "
# accept-new pins the host key on first contact and refuses it if it
# ever changes, which is the strongest option that does not require
# the operator to paste a fingerprint by hand.
"-o StrictHostKeyChecking=accept-new "
f"-o UserKnownHostsFile={os.path.join(cache_root(), 'known_hosts')}"
)
return env, plaintext
# token: HTTPS basic auth, handed over through an askpass helper.
askpass = _askpass_path(source.stack_id)
fd = os.open(askpass, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o700)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write('#!/bin/sh\ncase "$1" in Username*) echo "$GIT_USER";; *) echo "$GIT_TOKEN";; esac\n')
env["GIT_ASKPASS"] = askpass
env["GIT_USER"] = source.username or "git"
env["GIT_TOKEN"] = plaintext
return env, plaintext
def _cleanup_auth(source: GitSource) -> None:
for path in (_key_path(source.stack_id), _askpass_path(source.stack_id)):
try:
os.remove(path)
except OSError:
pass
# --------------------------------------------------------------------------- #
# Fetching
# --------------------------------------------------------------------------- #
async def _fetch(source: GitSource, env: dict, secret: Optional[str]) -> str:
"""Bring the cached clone to the tip of the configured branch. Returns the commit."""
directory = repo_dir(source.stack_id)
branch = source.branch or "main"
os.makedirs(cache_root(), mode=0o700, exist_ok=True)
if os.path.isdir(os.path.join(directory, ".git")):
try:
remote = (await _git(["-C", directory, "remote", "get-url", "origin"], env, secret)).strip()
except GitError:
remote = ""
if remote != source.url:
# Repointed at a different repository: start clean rather than try
# to reconcile two unrelated histories.
shutil.rmtree(directory, ignore_errors=True)
if not os.path.isdir(os.path.join(directory, ".git")):
await _git(
["clone", "--depth", "1", "--branch", branch, source.url, directory], env, secret
)
else:
await _git(["-C", directory, "fetch", "--depth", "1", "origin", branch], env, secret)
await _git(["-C", directory, "checkout", "-B", branch, "FETCH_HEAD"], env, secret)
await _git(["-C", directory, "reset", "--hard", "FETCH_HEAD"], env, secret)
# Safe here and only here: this directory holds nothing but the clone.
await _git(["-C", directory, "clean", "-fdx"], env, secret)
return (await _git(["-C", directory, "rev-parse", "HEAD"], env, secret)).strip()
# --------------------------------------------------------------------------- #
# Copying the repo's files into the stack
# --------------------------------------------------------------------------- #
def _inside(path: str, parent: str) -> bool:
return os.path.realpath(path).startswith(os.path.realpath(parent).rstrip("/") + "/")
def _source_tree(source: GitSource) -> str:
directory = repo_dir(source.stack_id)
subdir = (source.subdir or "").strip().strip("/")
if not subdir:
return directory
tree = os.path.join(directory, subdir)
# The subdirectory comes from user input and is about to be walked.
if not _inside(tree, directory):
raise GitError(f"Subdirectory '{source.subdir}' leaves the repository")
if not os.path.isdir(tree):
raise GitError(f"'{source.subdir}' does not exist in the repository")
return tree
def _materialise(source: GitSource, previous: list[str]) -> tuple[list[str], list[str], list[str]]:
"""Copy the repo subtree into the stack folder.
Returns (current, written, removed): everything the repo provides, the
subset that actually changed on disk, and the files dropped because the repo
no longer has them.
"""
tree = _source_tree(source)
stack_dir = compose_service.stack_dir(source.stack_id)
os.makedirs(stack_dir, exist_ok=True)
current: list[str] = []
written: list[str] = []
for root, dirs, files in os.walk(tree):
dirs[:] = [d for d in dirs if d != ".git"]
for name in files:
src = os.path.join(root, name)
rel = os.path.relpath(src, tree)
dest = os.path.join(stack_dir, rel)
if not _inside(dest, stack_dir):
continue # a symlinked path trying to escape the stack folder
current.append(rel)
# shallow=False: compare contents, not just size and mtime, or a
# revert to a same-sized earlier version would look like no change.
if os.path.isfile(dest) and filecmp.cmp(src, dest, shallow=False):
continue
os.makedirs(os.path.dirname(dest), exist_ok=True)
shutil.copy2(src, dest)
written.append(rel)
removed: list[str] = []
for rel in previous:
if rel in current:
continue
dest = os.path.join(stack_dir, rel)
if not _inside(dest, stack_dir) or not os.path.isfile(dest):
continue
try:
os.remove(dest)
removed.append(rel)
except OSError as exc:
logger.warning("Could not remove %s: %s", dest, exc)
_prune_empty_dirs(stack_dir, removed)
return sorted(current), sorted(written), sorted(removed)
def _prune_empty_dirs(stack_dir: str, removed: list[str]) -> None:
"""Drop directories left empty by removed files, never the stack folder."""
for rel in removed:
directory = os.path.dirname(os.path.join(stack_dir, rel))
while _inside(directory, stack_dir):
try:
os.rmdir(directory) # fails unless empty, which is what we want
except OSError:
break
directory = os.path.dirname(directory)
# --------------------------------------------------------------------------- #
# Syncing
# --------------------------------------------------------------------------- #
async def sync(session: Session, source: GitSource, actor: str = "system") -> SyncResult:
"""Fetch, copy into the stack, and deploy when something changed."""
env, secret = _auth_env(source)
try:
commit = await _fetch(source, env, secret)
previous = _managed(source)
current, written, removed = _materialise(source, previous)
except GitError as exc:
source.last_error = _redact(str(exc), secret)[:1000]
source.updated_at = datetime.now(timezone.utc)
session.add(source)
session.commit()
raise
finally:
_cleanup_auth(source)
changed = bool(written or removed)
source.managed_files = json.dumps(current)
source.last_commit = commit
source.last_synced_at = datetime.now(timezone.utc)
source.updated_at = source.last_synced_at
source.last_error = None
session.add(source)
session.commit()
result = SyncResult(changed=changed, commit=commit, written=written, removed=removed)
if changed and source.auto_deploy:
result.deployed, result.detail = await _deploy(session, source.stack_id, actor)
return result
async def _deploy(session: Session, stack_id: str, actor: str) -> tuple[bool, Optional[str]]:
"""`compose up -d`, under the same lock every other lifecycle action takes."""
from services import audit_service
try:
with stack_lock_service.hold(session, stack_id, "git-deploy", actor):
outcome = await compose_service.up(stack_id)
except stack_lock_service.StackBusy as exc:
# Somebody is already deploying. The files are updated; say so rather
# than queue a second compose run at the same project.
return False, f"stack is busy ({exc.action}); files synced but not deployed"
except Exception as exc: # noqa: BLE001 - a failed deploy must not lose the sync
return False, str(exc)[:500]
ok = outcome.get("returncode") in (0, None)
audit_service.record(
session, user=actor, action="stack.git-deploy", target=stack_id,
detail=f"rc={outcome.get('returncode')}",
)
return ok, (outcome.get("stderr") or "").strip()[-1000:] or None
def _managed(source: GitSource) -> list[str]:
try:
value = json.loads(source.managed_files or "[]")
except json.JSONDecodeError:
return []
return [str(v) for v in value] if isinstance(value, list) else []
def forget(stack_id: str) -> None:
"""Drop the cached clone and any credential files for a stack."""
try:
shutil.rmtree(repo_dir(stack_id), ignore_errors=True)
except GitError:
return
for path in (_key_path(stack_id), _askpass_path(stack_id)):
try:
os.remove(path)
except OSError:
pass
# --------------------------------------------------------------------------- #
# Webhooks
# --------------------------------------------------------------------------- #
def verify_webhook(source: GitSource, body: bytes, headers) -> bool:
"""Is this webhook really from the forge that holds our secret?
Supports the two schemes between them covered by GitHub, Gitea, Forgejo and
GitLab. Both comparisons are constant-time.
"""
expected = source.webhook_secret or ""
if not expected:
return False
signature = headers.get("X-Hub-Signature-256") or ""
if signature.startswith("sha256="):
digest = hmac.new(expected.encode(), body, "sha256").hexdigest()
return hmac.compare_digest(signature[len("sha256=") :], digest)
gitlab = headers.get("X-Gitlab-Token") or ""
if gitlab:
return hmac.compare_digest(gitlab, expected)
return False
# --------------------------------------------------------------------------- #
# Polling
# --------------------------------------------------------------------------- #
def due(source: GitSource, now: Optional[datetime] = None) -> bool:
"""Is this source's polling interval up?"""
if not source.poll_interval_minutes or source.poll_interval_minutes < 1:
return False
if source.last_synced_at is None:
return True
last = source.last_synced_at
if last.tzinfo is None:
last = last.replace(tzinfo=timezone.utc)
elapsed = (now or datetime.now(timezone.utc)) - last
return elapsed.total_seconds() >= source.poll_interval_minutes * 60
async def poll_loop(interval: float = 60.0) -> None:
"""Sync every repository whose interval is up. Never dies on one failure."""
from sqlmodel import select
from database import engine
while True:
await asyncio.sleep(interval)
try:
with Session(engine) as session:
sources = session.exec(select(GitSource)).all()
for source in sources:
if not due(source):
continue
try:
result = await sync(session, source, actor="poll")
if result.changed:
logger.info(
"Git sync updated %s to %s", source.stack_id,
(result.commit or "")[:8],
)
except Exception as exc: # noqa: BLE001 - one repo, not all
logger.warning("Git sync failed for %s: %s", source.stack_id, exc)
except Exception as exc: # noqa: BLE001 - the loop outlives everything
logger.warning("Git polling pass failed: %s", exc)
def ensure_cache_root() -> None:
"""The cache doubles as git's HOME, so it must exist and stay private."""
try:
os.makedirs(cache_root(), mode=0o700, exist_ok=True)
os.chmod(cache_root(), stat.S_IRWXU)
except OSError as exc:
logger.warning("Could not create the Git cache directory: %s", exc)
backend/services/icon_service.py
+221
View File
@@ -0,0 +1,221 @@
"""Stack icons — validation of the stored choice and custom-image storage.
A stack's ``icon`` column holds one of three things:
``None``
Automatic: the app logo the stack's name resolves to, and failing that the
glyph the frontend derives from the name (``frontend/src/lib/stackIcons.ts``).
Nothing is stored, so every stack that existed before this feature lands
here and an upgraded install shows sensible icons without a data migration.
``logo:<slug>``
The real logo of a known app, from the selfh.st catalog (see
``services/logo_service.py``). The file is cached per *slug*, not per stack,
so every Postgres stack shares one download.
``lucide:<name>``
A built-in icon the user picked explicitly. The catalog of names lives in
the frontend because that is the only place that can actually *render* one;
keeping a second copy here would only add a list to drift out of sync. An
unknown name is therefore not an error — the UI falls back to the automatic
icon for it.
``custom:<ext>:<version>``
An uploaded image at ``${DATA_DIR}/stack-icons/<stack_id>.<ext>``.
``<version>`` is the upload's unix timestamp. It carries no meaning beyond
changing the column value on every re-upload, which is what makes the
frontend's cache key change and the new image appear instead of the one the
browser already has.
"""
from __future__ import annotations
import logging
import os
import re
import shutil
import time
from typing import Optional
from config import settings
logger = logging.getLogger("stackpilot.icons")
#: Uploads are meant to be small app logos. The cap is deliberately generous
#: for a logo and still far too small to make the data directory grow.
MAX_ICON_BYTES = 512 * 1024
#: Extension per accepted image type. The extension is derived from the bytes
#: (see :func:`_sniff`), never from the upload's filename or Content-Type — a
#: client is free to lie about both.
_EXTENSIONS = {"png", "jpg", "gif", "webp", "svg"}
_BUILTIN_RE = re.compile(r"^lucide:[a-z0-9-]{1,48}$")
_LOGO_RE = re.compile(r"^logo:[a-z0-9][a-z0-9-]{0,63}$")
_CUSTOM_RE = re.compile(r"^custom:(png|jpg|gif|webp|svg):(\d{1,12})$")
_SAFE_ID_RE = re.compile(r"^[A-Za-z0-9._-]{1,128}$")
#: ``<img src>`` never executes script, but a custom icon is also reachable
#: directly under /api/..., where an SVG *would* run in the browser's own
#: context. Serving it as a download-only attachment keeps that door shut.
_CONTENT_TYPES = {
"png": "image/png",
"jpg": "image/jpeg",
"gif": "image/gif",
"webp": "image/webp",
"svg": "image/svg+xml",
}
class IconError(Exception):
"""An icon value or upload the server refuses."""
# --------------------------------------------------------------------------- #
# Paths
# --------------------------------------------------------------------------- #
def icon_dir() -> str:
return os.path.join(settings.DATA_DIR, "stack-icons")
def _safe_id(stack_id: str) -> str:
"""Reject anything that could escape the icon directory.
Stack ids are slugs, so this never fires in practice — it is here because
the id arrives from the URL and is about to be pasted into a filesystem
path.
"""
if not _SAFE_ID_RE.match(stack_id) or stack_id in (".", ".."):
raise IconError(f"Invalid stack id '{stack_id}'")
return stack_id
def custom_path(stack_id: str, ext: str) -> str:
return os.path.join(icon_dir(), f"{_safe_id(stack_id)}.{ext}")
def custom_ext(value: Optional[str]) -> Optional[str]:
"""The file extension of a ``custom:`` icon value, or None for the rest."""
match = _CUSTOM_RE.match(value or "")
return match.group(1) if match else None
def content_type(ext: str) -> str:
return _CONTENT_TYPES.get(ext, "application/octet-stream")
def file_for(stack_id: str, value: Optional[str]) -> Optional[str]:
"""Existing file backing a ``custom:`` icon value, or None."""
ext = custom_ext(value)
if not ext:
return None
path = custom_path(stack_id, ext)
return path if os.path.isfile(path) else None
# --------------------------------------------------------------------------- #
# Stored value
# --------------------------------------------------------------------------- #
def normalize_choice(value: str) -> Optional[str]:
"""Validate an icon chosen through the API.
An empty string means "back to automatic" and maps to ``None``. A built-in
glyph and an app logo are both fine to take from a client — they name a
catalog entry, not a file this stack owns. A ``custom:`` value is not: it is
minted by :func:`store_upload`, or a stack could be pointed at another
stack's uploaded image.
"""
value = (value or "").strip()
if not value:
return None
if _BUILTIN_RE.match(value) or _LOGO_RE.match(value):
return value
raise IconError(
"Icon must be empty (automatic), 'lucide:<name>' or 'logo:<slug>'; "
"upload custom images through POST /api/stacks/{id}/icon"
)
def logo_slug(value: Optional[str]) -> Optional[str]:
"""The catalog slug of a ``logo:`` icon value, or None for the rest."""
return value[len("logo:"):] if _LOGO_RE.match(value or "") else None
# --------------------------------------------------------------------------- #
# Uploads
# --------------------------------------------------------------------------- #
def _sniff(data: bytes) -> str:
"""Extension for the image these bytes actually are.
Trusting the declared Content-Type would mean storing (and later serving)
whatever a client cares to send under an image's name.
"""
if data.startswith(b"\x89PNG\r\n\x1a\n"):
return "png"
if data.startswith(b"\xff\xd8\xff"):
return "jpg"
if data.startswith(b"GIF87a") or data.startswith(b"GIF89a"):
return "gif"
if data[:4] == b"RIFF" and data[8:12] == b"WEBP":
return "webp"
head = data[:512].lstrip()
if head.startswith(b"<?xml") or head.startswith(b"<svg") or b"<svg" in head:
return "svg"
raise IconError("Unsupported image type — use PNG, JPEG, GIF, WebP or SVG")
def store_upload(stack_id: str, data: bytes) -> str:
"""Write an uploaded icon and return the value for ``Stack.icon``."""
if not data:
raise IconError("The uploaded file is empty")
if len(data) > MAX_ICON_BYTES:
raise IconError(
f"Icon is too large ({len(data) // 1024} KiB); "
f"the limit is {MAX_ICON_BYTES // 1024} KiB"
)
ext = _sniff(data)
os.makedirs(icon_dir(), exist_ok=True)
# A re-upload in a different format would otherwise leave the old file
# behind as an orphan nothing ever cleans up.
remove(stack_id)
path = custom_path(stack_id, ext)
with open(path, "wb") as fh:
fh.write(data)
return f"custom:{ext}:{int(time.time())}"
def remove(stack_id: str) -> None:
"""Delete every custom icon file belonging to a stack. Best effort."""
for ext in _EXTENSIONS:
try:
os.remove(custom_path(stack_id, ext))
except FileNotFoundError:
continue
except OSError as exc: # noqa: PERF203 - one bad file must not block the rest
logger.warning("Could not remove icon %s.%s: %s", stack_id, ext, exc)
def copy(src_id: str, dst_id: str, value: Optional[str]) -> Optional[str]:
"""Copy a stack's custom icon to another stack (used when cloning).
Returns the icon value for the new stack: the copied ``custom:`` value, the
unchanged built-in choice, or None when there is nothing to carry over.
"""
ext = custom_ext(value)
if not ext:
return value
source = file_for(src_id, value)
if not source:
return None
os.makedirs(icon_dir(), exist_ok=True)
try:
shutil.copyfile(source, custom_path(dst_id, ext))
except OSError as exc:
logger.warning("Could not copy icon %s -> %s: %s", src_id, dst_id, exc)
return None
return f"custom:{ext}:{int(time.time())}"
backend/services/image_service.py
+1 -1
View File
@@ -1,4 +1,4 @@
"""Image listing — shared by the central images router and the agent."""
"""Image listing for the images router."""
from __future__ import annotations
from docker_client import DockerError, get_client, safe_call
backend/services/image_status_store.py
+80
View File
@@ -0,0 +1,80 @@
"""Persistence for the image update cache.
``update_service`` holds the registry-digest results in a module dict and knows
nothing about storage — it is pure registry logic and stays unit-testable
without a database. This module is its persistence half: it seeds that dict at
startup and mirrors every write back into SQLite, wired up in ``main.lifespan``.
What it buys: after a restart the update badges are there immediately instead
of blank until the next background sweep (up to an hour), and the "already
notified" marks come back with them, so a restart no longer re-announces
updates the user has already seen.
"""
from __future__ import annotations
import logging
from sqlmodel import Session, select
from database import engine
from models.runtime_state import ImageStatus
from services import update_service
logger = logging.getLogger("stackpilot.image_status")
def _to_status(row: ImageStatus) -> update_service.UpdateStatus:
return update_service.UpdateStatus(
image=row.image,
update_available=row.update_available,
current_digest=row.current_digest,
remote_digest=row.remote_digest,
checked_at=row.checked_at,
error=row.error,
)
def save(status: update_service.UpdateStatus, notified: bool) -> None:
"""Upsert one image's status. Opens its own session — the caller is the
background loop, which has none."""
with Session(engine) as session:
row = session.get(ImageStatus, status.image)
if row is None:
row = ImageStatus(image=status.image)
row.update_available = status.update_available
row.current_digest = status.current_digest
row.remote_digest = status.remote_digest
row.checked_at = status.checked_at
row.error = status.error
row.notified = notified
session.add(row)
session.commit()
def install() -> int:
"""Seed the in-memory cache from the database and start mirroring writes.
Returns how many entries were restored.
"""
with Session(engine) as session:
rows = session.exec(select(ImageStatus)).all()
update_service.restore_cache([(_to_status(r), r.notified) for r in rows])
update_service.set_persist_callback(save, prune)
return len(rows)
def prune(keep: set[str]) -> int:
"""Drop rows for images that are no longer used by any stack.
Without this the table grows for the life of the install, one row per image
tag that was ever running.
"""
removed = 0
with Session(engine) as session:
for row in session.exec(select(ImageStatus)).all():
if row.image not in keep:
session.delete(row)
removed += 1
if removed:
session.commit()
return removed
backend/services/logo_service.py
+428
View File
@@ -0,0 +1,428 @@
"""Real app logos for stacks, fetched once and then served from disk.
A stack called ``jellyfin`` should show *the Jellyfin logo*, not a generic
clapperboard. The logos come from the selfh.st icon set (~2900 self-hosted
apps), which is the same catalog Homarr, Homepage and Dashy draw on.
Everything crosses the network exactly once and on the server:
* the **catalog** (a JSON index of slugs and display names) is downloaded on
startup and refreshed weekly into ``${DATA_DIR}/stack-icons/catalog.json``,
* a **logo** is downloaded the first time something asks for it and cached at
``${DATA_DIR}/stack-icons/logos/<slug>.png``, keyed by slug rather than by
stack so ten Postgres stacks share one file.
Browsers therefore never talk to the CDN: they fetch logos from StackPilot's
own authenticated icon endpoint, like an uploaded image. After the first fetch
the whole feature works offline, and an installation with no outbound internet
degrades to the built-in glyphs rather than breaking — every entry point here
returns None instead of raising when the network is not there.
Matching a stack to a slug is deliberately conservative: an exact name, then the
name with punctuation rearranged, then the longest run of words inside it, then
the images its compose file pulls (``lscr.io/linuxserver/jellyfin:latest`` →
``jellyfin``). A stack whose name means nothing to the catalog gets no logo and
falls back to the keyword-derived glyph in the frontend.
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import re
import time
from typing import Iterable, Optional
import httpx
import yaml
from config import settings
from services import compose_service
logger = logging.getLogger("stackpilot.logos")
CATALOG_URL = "https://cdn.jsdelivr.net/gh/selfhst/icons/index.json"
LOGO_URL = "https://cdn.jsdelivr.net/gh/selfhst/icons/png/{slug}.png"
#: The catalog gains a handful of apps a week; there is nothing to gain from
#: checking more often, and a failed refresh simply keeps the previous copy.
CATALOG_TTL = 7 * 24 * 3600
CATALOG_MAX_BYTES = 8 * 1024 * 1024
LOGO_MAX_BYTES = 2 * 1024 * 1024
TIMEOUT = httpx.Timeout(15.0)
_SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9-]{0,63}$")
#: Docker image names that are not what the catalog calls the app. Kept short on
#: purpose — this is for the cases the matcher genuinely cannot reach, not a
#: second catalog.
ALIASES = {
"postgres": "postgresql",
"pgsql": "postgresql",
"mongo": "mongodb",
"trilium": "trilium-notes",
"wg-easy": "wireguard",
"wg": "wireguard",
"homeassistant": "home-assistant",
"hass": "home-assistant",
"pihole": "pi-hole",
"openwebui": "open-webui",
"nextcloud-aio": "nextcloud",
"paperless": "paperless-ngx",
"paperless-ng": "paperless-ngx",
"code-server": "coder",
"filebrowser": "file-browser",
"qbit": "qbittorrent",
"sab": "sabnzbd",
}
#: Image name components that say nothing about the app.
_IMAGE_NOISE = {
"latest", "linuxserver", "lscr", "ghcr", "docker", "io", "com", "library",
"hotio", "alpine", "amd64", "arm64v8", "bitnami", "official",
}
# --------------------------------------------------------------------------- #
# Paths
# --------------------------------------------------------------------------- #
def _root() -> str:
return os.path.join(settings.DATA_DIR, "stack-icons")
def catalog_path() -> str:
return os.path.join(_root(), "catalog.json")
def logo_dir() -> str:
return os.path.join(_root(), "logos")
def logo_path(slug: str) -> Optional[str]:
"""Where a slug's logo is cached, or None if the slug is malformed.
The slug reaches this from the database and from query strings, and it is
about to become a filename.
"""
if not _SLUG_RE.match(slug or ""):
return None
return os.path.join(logo_dir(), f"{slug}.png")
# --------------------------------------------------------------------------- #
# The catalog
# --------------------------------------------------------------------------- #
class Catalog:
"""Slug lookups built once per catalog file."""
def __init__(self, entries: list[dict]):
self.entries = entries
self.slugs: set[str] = set()
self.by_name: dict[str, str] = {}
self.compact: dict[str, str] = {}
for entry in entries:
slug = (entry.get("Reference") or "").strip().lower()
if not _SLUG_RE.match(slug):
continue
self.slugs.add(slug)
# "AdGuard Home" → "adguard home", so a stack named that matches
# even though the slug is hyphenated.
name = _normalize(entry.get("Name") or "")
self.by_name.setdefault(name, slug)
# "pihole" → "pi-hole": people drop the punctuation the catalog keeps.
self.compact.setdefault(slug.replace("-", ""), slug)
self.compact.setdefault(name.replace(" ", ""), slug)
def display_name(self, slug: str) -> str:
for entry in self.entries:
if (entry.get("Reference") or "").lower() == slug:
return entry.get("Name") or slug
return slug
def __len__(self) -> int:
return len(self.slugs)
_catalog: Optional[Catalog] = None
_catalog_mtime: float = 0.0
def load_catalog() -> Optional[Catalog]:
"""The cached catalog, re-read only when the file on disk changed."""
global _catalog, _catalog_mtime
path = catalog_path()
try:
mtime = os.path.getmtime(path)
except OSError:
return _catalog # never downloaded, or removed under us
if _catalog is not None and mtime == _catalog_mtime:
return _catalog
try:
with open(path, "r", encoding="utf-8") as fh:
entries = json.load(fh)
except (OSError, json.JSONDecodeError) as exc:
logger.warning("Icon catalog is unreadable (%s); ignoring it", exc)
return _catalog
if not isinstance(entries, list):
return _catalog
_catalog = Catalog(entries)
_catalog_mtime = mtime
logger.info("Loaded %d app logos from the icon catalog", len(_catalog))
return _catalog
def catalog_age() -> Optional[float]:
try:
return time.time() - os.path.getmtime(catalog_path())
except OSError:
return None
async def refresh_catalog(force: bool = False) -> bool:
"""Download the catalog unless the copy on disk is still fresh."""
age = catalog_age()
if not force and age is not None and age < CATALOG_TTL:
return False
try:
async with httpx.AsyncClient(follow_redirects=True, timeout=TIMEOUT) as client:
response = await client.get(CATALOG_URL)
response.raise_for_status()
if len(response.content) > CATALOG_MAX_BYTES:
raise ValueError("catalog is implausibly large")
entries = response.json()
if not isinstance(entries, list) or not entries:
raise ValueError("catalog is not a non-empty list")
except (httpx.HTTPError, ValueError, json.JSONDecodeError) as exc:
# No internet is a normal state for a self-hosted box. Say so once and
# carry on with the built-in glyphs.
logger.info("Could not refresh the app icon catalog: %s", exc)
return False
os.makedirs(_root(), exist_ok=True)
tmp = catalog_path() + ".tmp"
with open(tmp, "w", encoding="utf-8") as fh:
json.dump(entries, fh)
os.replace(tmp, catalog_path())
load_catalog()
return True
async def catalog_loop() -> None:
"""Keep the catalog fresh for as long as the app runs."""
while True:
try:
await refresh_catalog()
except Exception as exc: # noqa: BLE001 - a background loop may not die
logger.warning("Icon catalog refresh failed: %s", exc)
await asyncio.sleep(CATALOG_TTL)
# --------------------------------------------------------------------------- #
# Matching a stack to a slug
# --------------------------------------------------------------------------- #
def _normalize(text: str) -> str:
return re.sub(r"[^a-z0-9]+", " ", text.lower()).strip()
def _lookup(catalog: Catalog, phrase: str) -> Optional[str]:
"""A slug for one normalized phrase, trying every spelling of it."""
if not phrase:
return None
hyphenated = phrase.replace(" ", "-")
squashed = phrase.replace(" ", "")
if (alias := ALIASES.get(hyphenated)) and alias in catalog.slugs:
return alias
if hyphenated in catalog.slugs:
return hyphenated
if phrase in catalog.by_name:
return catalog.by_name[phrase]
if squashed in catalog.compact:
return catalog.compact[squashed]
return None
def match_slug(name: str, images: Iterable[str] = ()) -> Optional[str]:
"""The catalog slug a stack's name (then its images) points at."""
catalog = load_catalog()
if not catalog:
return None
phrase = _normalize(name)
if (slug := _lookup(catalog, phrase)):
return slug
# The app's name inside a longer one ("my jellyfin stack", "medien plex").
# Longest run of words first, so "home assistant" beats "home".
tokens = phrase.split()
for size in range(len(tokens), 0, -1):
for start in range(len(tokens) - size + 1):
gram = tokens[start : start + size]
# A single short word is far more likely to be a coincidence than
# an app ("app", "web", "db" are all slugs somewhere).
if size == 1 and len(gram[0]) < 4:
continue
if (slug := _lookup(catalog, " ".join(gram))):
return slug
# Nothing in the name: ask what the stack actually runs.
for image in images:
if (slug := _lookup(catalog, _normalize(image))):
return slug
return None
def images_for(stack_id: str) -> list[str]:
"""Image names a stack's compose file pulls, most specific part first.
``lscr.io/linuxserver/jellyfin:latest`` contributes ``jellyfin``: the tag,
the registry and the vendor namespace say nothing about which app it is.
"""
directory = compose_service.stack_dir(stack_id)
compose_file = compose_service.find_compose_file(directory)
if not compose_file:
return []
try:
with open(compose_file, "r", encoding="utf-8", errors="replace") as fh:
data = yaml.safe_load(fh) or {}
except (OSError, yaml.YAMLError):
return []
out: list[str] = []
for spec in (data.get("services") or {}).values():
if not isinstance(spec, dict):
continue
image = spec.get("image")
if not isinstance(image, str) or not image:
continue
# Strip the tag/digest, then take the last path segment.
base = image.split("@")[0].rsplit(":", 1)[0]
candidate = base.rstrip("/").split("/")[-1]
if candidate and candidate not in _IMAGE_NOISE and candidate not in out:
out.append(candidate)
return out
#: Per-stack results, keyed by what they were computed from. Matching is pure
#: string work, but it reads the compose file, and the stacks list runs it for
#: every row on every poll.
_resolved: dict[str, tuple[tuple, Optional[str]]] = {}
def _signature(stack_id: str, name: str) -> tuple:
try:
mtime = os.path.getmtime(
compose_service.find_compose_file(compose_service.stack_dir(stack_id)) or ""
)
except OSError:
mtime = 0.0
return (name, mtime, _catalog_mtime)
def auto_slug(stack_id: str, name: str) -> Optional[str]:
"""The logo a stack gets with nothing configured, or None for no match.
Cached against the stack's name and its compose file's mtime, so a rename or
an edited compose re-matches and everything else is a dictionary hit.
"""
signature = _signature(stack_id, name)
cached = _resolved.get(stack_id)
if cached and cached[0] == signature:
return cached[1]
slug = match_slug(name, images_for(stack_id))
_resolved[stack_id] = (signature, slug)
return slug
def forget(stack_id: str) -> None:
"""Drop a stack's memoized match (it was deleted, or renamed by clone)."""
_resolved.pop(stack_id, None)
# --------------------------------------------------------------------------- #
# The logo files
# --------------------------------------------------------------------------- #
#: Slugs currently being downloaded, so N rows asking at once fetch once.
_inflight: dict[str, asyncio.Task] = {}
async def ensure_logo(slug: str) -> Optional[str]:
"""Path to a slug's cached logo, downloading it the first time."""
path = logo_path(slug)
if not path:
return None
if os.path.isfile(path):
return path
catalog = load_catalog()
if catalog and slug not in catalog.slugs:
return None
if (task := _inflight.get(slug)) is None:
task = asyncio.create_task(_download(slug, path))
_inflight[slug] = task
task.add_done_callback(lambda _t, s=slug: _inflight.pop(s, None))
try:
return await asyncio.shield(task)
except asyncio.CancelledError:
# The *caller* went away (client disconnected); the download itself is
# shielded and still finishes for whoever asks next.
raise
except Exception: # noqa: BLE001 - a missing logo is not an error
return None
async def _download(slug: str, path: str) -> Optional[str]:
try:
async with httpx.AsyncClient(follow_redirects=True, timeout=TIMEOUT) as client:
response = await client.get(LOGO_URL.format(slug=slug))
response.raise_for_status()
data = response.content
except httpx.HTTPError as exc:
logger.info("Could not fetch the logo for '%s': %s", slug, exc)
return None
if not data.startswith(b"\x89PNG\r\n\x1a\n") or len(data) > LOGO_MAX_BYTES:
logger.info("Ignoring the logo for '%s': not a plausible PNG", slug)
return None
os.makedirs(logo_dir(), exist_ok=True)
tmp = f"{path}.{os.getpid()}.tmp"
try:
with open(tmp, "wb") as fh:
fh.write(data)
os.replace(tmp, path)
except OSError as exc:
logger.warning("Could not cache the logo for '%s': %s", slug, exc)
return None
return path
def search(query: str, limit: int = 60) -> list[dict]:
"""Catalog entries matching a search term, best match first."""
catalog = load_catalog()
if not catalog:
return []
needle = _normalize(query)
results: list[tuple[int, str, dict]] = []
for entry in catalog.entries:
slug = (entry.get("Reference") or "").lower()
name = entry.get("Name") or slug
if not _SLUG_RE.match(slug):
continue
haystack = _normalize(name)
if not needle:
rank = 2
elif haystack == needle or slug == needle.replace(" ", "-"):
rank = 0
elif haystack.startswith(needle) or slug.startswith(needle.replace(" ", "-")):
rank = 1
elif needle in haystack or needle.replace(" ", "-") in slug:
rank = 2
else:
continue
results.append((rank, haystack, {"slug": slug, "name": name}))
results.sort(key=lambda row: (row[0], row[1]))
return [row[2] for row in results[:limit]]
backend/services/registry_service.py
+249
View File
@@ -0,0 +1,249 @@
"""Credentials for private container registries.
Two very different consumers need these, which is why this module exists rather
than the credentials living next to either of them:
* **StackPilot's own update checker** (``services/update_service.py``) talks to
the registry v2 API over HTTP itself. Without credentials a private repository
answers 401, the check gave up, and the UI said nothing at all — a stack could
sit on a stale image for months and look up to date. That was the actual bug
here, not a missing feature.
* **The Docker CLI**, which runs ``docker compose pull``. It reads its own
``config.json``, so this module writes one into ``${DATA_DIR}/docker`` and
compose runs with ``DOCKER_CONFIG`` pointed at it.
Lookups happen inside async registry calls that have no database session, so
the rows are mirrored into a small in-memory cache. :func:`reload` refills it and
rewrites the CLI config; every write path calls it, and so does startup.
Passwords are encrypted at rest and only ever decrypted into this cache and the
CLI config file (0600, in the data volume). They are never returned by the API.
"""
from __future__ import annotations
import base64
import json
import logging
import os
import threading
from typing import Optional
import httpx
from sqlmodel import Session, select
from config import settings
from models.registry import Registry
from services import crypto_service
logger = logging.getLogger("stackpilot.registries")
#: What `parse_ref` calls Docker Hub, and what the CLI calls it. Every spelling
#: users type — docker.io, index.docker.io, the v1 URL — normalizes to the
#: first; the second is what has to appear in config.json for `docker pull`.
DOCKER_HUB = "registry-1.docker.io"
DOCKER_HUB_CONFIG_KEY = "https://index.docker.io/v1/"
_DOCKER_HUB_ALIASES = {
"docker.io",
"index.docker.io",
"registry.docker.io",
"registry-1.docker.io",
"https://index.docker.io/v1/",
"index.docker.io/v1/",
}
TIMEOUT = httpx.Timeout(15.0)
_lock = threading.Lock()
_credentials: dict[str, tuple[str, str]] = {}
class RegistryError(Exception):
"""A registry that cannot be reached, or credentials it rejects."""
# --------------------------------------------------------------------------- #
# Host normalization
# --------------------------------------------------------------------------- #
def canonical_host(value: str) -> str:
"""The registry host as an image reference would name it.
Accepts what people actually paste: a bare host, a URL with a scheme, a
trailing slash, Docker Hub under any of its names. Without this, credentials
entered as ``docker.io`` would never be found for an image that parses as
``registry-1.docker.io``.
"""
host = (value or "").strip().lower()
if not host:
raise RegistryError("Registry host is required")
if host in _DOCKER_HUB_ALIASES:
return DOCKER_HUB
for scheme in ("https://", "http://"):
if host.startswith(scheme):
host = host[len(scheme) :]
break
host = host.split("/", 1)[0].rstrip("/")
if host in _DOCKER_HUB_ALIASES:
return DOCKER_HUB
if not host:
raise RegistryError("Registry host is required")
return host
def is_docker_hub(host: str) -> bool:
return canonical_host(host) == DOCKER_HUB
# --------------------------------------------------------------------------- #
# The cache the async paths read
# --------------------------------------------------------------------------- #
def reload(session: Session) -> int:
"""Refill the credential cache from the database and rewrite config.json.
Called at startup and after every write. Returns how many registries are
configured.
"""
fresh: dict[str, tuple[str, str]] = {}
for row in session.exec(select(Registry)).all():
try:
password = crypto_service.decrypt(row.password)
except crypto_service.DecryptError as exc:
# One unreadable row must not take the others down with it.
logger.warning("Ignoring credentials for %s: %s", row.host, exc)
continue
fresh[row.host] = (row.username, password)
with _lock:
_credentials.clear()
_credentials.update(fresh)
_write_docker_config(fresh)
return len(fresh)
def credentials_for(host: str) -> Optional[tuple[str, str]]:
"""(username, password) for a registry host, or None."""
try:
key = canonical_host(host)
except RegistryError:
return None
with _lock:
return _credentials.get(key)
def credentials_for_image(image: str) -> Optional[tuple[str, str]]:
"""Credentials for whichever registry an image reference points at."""
from services import update_service
registry, _repo, _tag = update_service.parse_ref(image)
return credentials_for(registry)
def configured_hosts() -> list[str]:
with _lock:
return sorted(_credentials)
# --------------------------------------------------------------------------- #
# The Docker CLI's config.json
# --------------------------------------------------------------------------- #
def docker_config_dir() -> str:
return os.path.join(settings.DATA_DIR, "docker")
def _write_docker_config(creds: dict[str, tuple[str, str]]) -> None:
"""Write the auths file ``docker compose pull`` reads.
The file is rewritten from the database every time, so removing a registry
in the UI actually revokes the CLI's access rather than leaving a stale
login behind.
"""
directory = docker_config_dir()
path = os.path.join(directory, "config.json")
auths = {}
for host, (username, password) in creds.items():
key = DOCKER_HUB_CONFIG_KEY if host == DOCKER_HUB else host
token = base64.b64encode(f"{username}:{password}".encode("utf-8")).decode("ascii")
auths[key] = {"auth": token}
try:
os.makedirs(directory, mode=0o700, exist_ok=True)
tmp = f"{path}.tmp"
# Written 0600 before it is put in place, so the credentials are never
# briefly world-readable.
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
json.dump({"auths": auths}, fh)
os.replace(tmp, path)
except OSError as exc:
logger.warning("Could not write the Docker CLI credentials file: %s", exc)
def cli_env() -> dict:
"""Environment for a ``docker``/``docker compose`` subprocess.
Points DOCKER_CONFIG at our generated file rather than writing into
``~/.docker``, so what StackPilot manages stays separate from anything the
image ships or an operator put there by hand.
"""
return {**os.environ, "DOCKER_CONFIG": docker_config_dir()}
# --------------------------------------------------------------------------- #
# Verifying credentials
# --------------------------------------------------------------------------- #
async def verify(host: str, username: str, password: str) -> None:
"""Check that a registry accepts these credentials. Raises RegistryError.
Asks for a pull-scoped token the way a client would, and treats only an
outright 401 as "wrong credentials" — a registry that answers anything else
is reachable and talking, which is as much as a credentials check can
honestly claim.
"""
registry = canonical_host(host)
url = f"https://{registry}/v2/"
auth = (username, password)
try:
async with httpx.AsyncClient(follow_redirects=True, timeout=TIMEOUT) as client:
response = await client.get(url, auth=auth)
if response.status_code == 401:
challenge = response.headers.get("WWW-Authenticate", "")
if challenge.lower().startswith("bearer"):
token = await _token(client, challenge, auth)
if not token:
raise RegistryError("The registry rejected these credentials")
return
raise RegistryError("The registry rejected these credentials")
if response.status_code >= 500:
raise RegistryError(
f"The registry answered {response.status_code}; try again later"
)
except httpx.HTTPError as exc:
raise RegistryError(f"Could not reach {registry}: {exc}") from exc
async def _token(
client: httpx.AsyncClient, challenge: str, auth: tuple[str, str]
) -> Optional[str]:
"""Follow a Bearer challenge with credentials attached."""
params = {}
for part in challenge[len("Bearer ") :].split(","):
if "=" in part:
key, value = part.split("=", 1)
params[key.strip()] = value.strip().strip('"')
realm = params.pop("realm", None)
if not realm:
return None
try:
response = await client.get(realm, params=params, auth=auth, timeout=TIMEOUT)
if response.status_code == 401:
return None
response.raise_for_status()
data = response.json()
return data.get("token") or data.get("access_token")
except (httpx.HTTPError, ValueError):
return None
backend/services/schedule_service.py
+9 -28
View File
@@ -11,22 +11,18 @@ from __future__ import annotations
import asyncio
import logging
import os
import tempfile
from datetime import datetime, timedelta, timezone
from sqlmodel import Session, select
from database import engine
from models.agent import Agent
from models.backup_destination import BackupDestination
from models.backup_schedule import BackupSchedule
from models.setting import EVENT_BACKUP_FAILED
from models.stack import Stack
from services import (
agent_service,
backup_destination_service as dest_service,
backup_service,
compose_service,
notify_service,
)
@@ -87,31 +83,16 @@ async def run_schedule(session: Session, schedule: BackupSchedule) -> dict:
if not dest:
raise RuntimeError(f"destination {schedule.destination_id} not found")
# Produce the backup archive — locally or by streaming it from an agent.
if schedule.agent_id is not None:
agent = session.get(Agent, schedule.agent_id)
if not agent:
raise RuntimeError(f"agent {schedule.agent_id} not found")
prefix = compose_service.slugify(agent.name)
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".tar.gz")
tmp.close()
path = tmp.name
await agent_service.download_to_file(
session, agent, f"/agent/stacks/{schedule.stack_id}/backup", path,
params={"include_volumes": schedule.include_volumes, "stop_first": schedule.stop_first},
)
else:
stack = session.get(Stack, schedule.stack_id)
if not stack:
raise RuntimeError(f"stack '{schedule.stack_id}' not found")
prefix = None
path = await backup_service.create_backup(
schedule.stack_id, stack.name,
include_volumes=schedule.include_volumes, stop_first=schedule.stop_first,
)
stack = session.get(Stack, schedule.stack_id)
if not stack:
raise RuntimeError(f"stack '{schedule.stack_id}' not found")
path = await backup_service.create_backup(
schedule.stack_id, stack.name,
include_volumes=schedule.include_volumes, stop_first=schedule.stop_first,
)
filename = backup_service.backup_filename(schedule.stack_id, schedule.include_volumes, prefix=prefix)
basename = backup_service.backup_basename(schedule.stack_id, prefix)
filename = backup_service.backup_filename(schedule.stack_id, schedule.include_volumes)
basename = backup_service.backup_basename(schedule.stack_id)
await asyncio.to_thread(dest_service.upload, dest, path, filename)
pruned = await asyncio.to_thread(_prune, dest, basename, schedule.keep)
schedule.last_status = "ok"
backend/services/self_update_service.py
+198
View File
@@ -0,0 +1,198 @@
"""Self-update: is a newer StackPilot release in the registry, and apply it.
The check reads the version tags of the backend's *own* image repository
(anonymous v2 token flow, https with http fallback for insecure registries)
and compares the highest semver tag against the running APP_VERSION.
Applying the update spawns a detached **helper container** (from the current
backend image — it ships the docker CLI + compose plugin) that runs
``docker compose pull && up -d`` against the compose project this backend
belongs to, resolved from its own container labels. The helper outlives the
backend container being recreated, which is what makes self-update possible.
"""
from __future__ import annotations
import logging
import re
import socket
import time
from typing import Optional
import httpx
from docker_client import DockerError, get_client, safe_call
from services import update_service
from version import APP_VERSION
logger = logging.getLogger("stackpilot.selfupdate")
STATUS_TTL = 600.0 # seconds between registry checks
_VERSION_RE = re.compile(r"^\d+(\.\d+)*$")
_LABEL_PROJECT = "com.docker.compose.project"
_LABEL_WORKING_DIR = "com.docker.compose.project.working_dir"
_LABEL_CONFIG_FILES = "com.docker.compose.project.config_files"
_status_cache: dict = {"data": None, "ts": 0.0}
class SelfUpdateError(Exception):
pass
# --------------------------------------------------------------------------- #
# Own container / image discovery
# --------------------------------------------------------------------------- #
def _own_container():
"""The container this backend runs in (None outside a container)."""
client = get_client()
hostname = socket.gethostname()
try:
return safe_call(client.containers.get, hostname)
except DockerError:
pass
# Fallback (custom hostname set): match by image name.
try:
for c in safe_call(client.containers.list):
if "stackpilot-backend" in (c.attrs.get("Config", {}).get("Image") or ""):
return c
except DockerError:
pass
return None
def _compose_info(container) -> Optional[dict]:
labels = container.attrs.get("Config", {}).get("Labels") or {}
project = labels.get(_LABEL_PROJECT)
working_dir = labels.get(_LABEL_WORKING_DIR)
config_files = [f for f in (labels.get(_LABEL_CONFIG_FILES) or "").split(",") if f]
if not project or not working_dir or not config_files:
return None
return {"project": project, "working_dir": working_dir, "config_files": config_files}
# --------------------------------------------------------------------------- #
# Registry version lookup
# --------------------------------------------------------------------------- #
def _version_key(v: str) -> tuple[int, ...]:
return tuple(int(p) for p in v.split("."))
async def _fetch_tags(registry: str, repo: str) -> list[str]:
"""Tag list via the v2 API; anonymous token flow; http fallback for
insecure registries (plain-IP registries usually aren't behind TLS)."""
last_exc: Optional[Exception] = None
for scheme in ("https", "http"):
url = f"{scheme}://{registry}/v2/{repo}/tags/list"
try:
async with httpx.AsyncClient(follow_redirects=True) as client:
resp = await client.get(url, timeout=10)
if resp.status_code == 401:
token = await update_service._get_token(
client, resp.headers.get("WWW-Authenticate", "")
)
if not token:
raise SelfUpdateError("Registry requires authentication")
resp = await client.get(
url, headers={"Authorization": f"Bearer {token}"}, timeout=10
)
resp.raise_for_status()
return resp.json().get("tags") or []
except (httpx.HTTPError, ValueError) as exc:
last_exc = exc
continue
raise SelfUpdateError(f"Cannot reach registry {registry}: {last_exc}")
async def get_status(refresh: bool = False) -> dict:
now = time.time()
if not refresh and _status_cache["data"] and now - _status_cache["ts"] < STATUS_TTL:
return _status_cache["data"]
data = {
"current_version": APP_VERSION,
"latest_version": None,
"update_available": False,
"update_supported": False,
"image": None,
"error": None,
}
container = _own_container()
if container is None:
data["error"] = "Not running in a container"
_status_cache.update(data=data, ts=now)
return data
image = container.attrs.get("Config", {}).get("Image") or ""
data["image"] = image
data["update_supported"] = _compose_info(container) is not None
try:
registry, repo, _tag = update_service.parse_ref(image)
tags = await _fetch_tags(registry, repo)
versions = sorted(
(t for t in tags if _VERSION_RE.match(t)), key=_version_key
)
if versions:
latest = versions[-1]
data["latest_version"] = latest
data["update_available"] = _version_key(latest) > _version_key(APP_VERSION)
else:
data["error"] = "No version tags found in the registry"
except SelfUpdateError as exc:
data["error"] = str(exc)
_status_cache.update(data=data, ts=now)
return data
# --------------------------------------------------------------------------- #
# Apply: helper container runs compose pull + up on our own project
# --------------------------------------------------------------------------- #
def apply_update() -> dict:
container = _own_container()
if container is None:
raise SelfUpdateError("Not running in a container — update manually")
info = _compose_info(container)
if info is None:
raise SelfUpdateError(
"This StackPilot is not compose-managed — update it the way it was deployed"
)
image = container.attrs.get("Config", {}).get("Image") or ""
compose = f"docker compose --project-name {info['project']} --project-directory {info['working_dir']}"
for f in info["config_files"]:
compose += f" -f {f}"
command = f"{compose} pull --quiet && {compose} up -d --remove-orphans"
# Bind the project dir (and any config file living outside it) read-only
# at its host path so relative paths and .env resolve exactly as on host.
volumes = {
"/var/run/docker.sock": {"bind": "/var/run/docker.sock", "mode": "rw"},
info["working_dir"]: {"bind": info["working_dir"], "mode": "ro"},
}
for f in info["config_files"]:
parent = f.rsplit("/", 1)[0] or "/"
if parent != info["working_dir"] and not parent.startswith(info["working_dir"] + "/"):
volumes.setdefault(parent, {"bind": parent, "mode": "ro"})
client = get_client()
helper = safe_call(
client.containers.run,
image,
["sh", "-c", command],
detach=True,
auto_remove=True,
name=f"stackpilot-self-update-{int(time.time())}",
labels={"stackpilot.helper": "self-update"},
volumes=volumes,
working_dir=info["working_dir"],
environment={"DOCKER_CONFIG": "/tmp/.docker"}, # don't expect host creds
)
logger.info("Self-update helper %s started: %s", helper.short_id, command)
return {"status": "updating", "helper": helper.short_id, "command": command}
backend/services/settings_service.py
+1 -1
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
import json
from typing import Any, Optional
from sqlmodel import Session, select
from sqlmodel import Session
from config import settings as env_settings
from database import engine
backend/services/stack_assets_service.py
+548
View File
@@ -0,0 +1,548 @@
"""Inventory of everything a stack's data actually lives in.
A stack is more than its compose file: bind-mounted directories (``./config``)
and named volumes hold the real state. StackPilot itself runs in a container, so
it can only *see* what is mounted into it — a bind source like
``/opt/stacks/arr-stack/gluetun`` may exist on the host and still be invisible
here (that happens whenever the stacks directory is mounted under a different
host path than ``STACKS_DIR``, because compose resolves ``./gluetun`` against the
path *inside* this container and the daemon then creates it at that same path on
the **host**).
Everything in this module therefore reads and writes host paths through a
throwaway helper container: the daemon does the mounting, so the data is
reachable regardless of what StackPilot has mounted. That is what makes backups
complete instead of "just the compose file".
"""
from __future__ import annotations
import logging
import os
import re
from typing import Optional
import yaml
from config import settings
from docker_client import DockerError, get_client, safe_call
from services import compose_service
logger = logging.getLogger("stackpilot.assets")
COMPOSE_PROJECT_LABEL = "com.docker.compose.project"
COMPOSE_SERVICE_LABEL = "com.docker.compose.service"
COMPOSE_VOLUME_LABEL = "com.docker.compose.volume"
# Paths that are plumbing, never stack data.
SYSTEM_PATHS = {
"/var/run/docker.sock",
"/run/docker.sock",
"/etc/localtime",
"/etc/timezone",
"/etc/hosts",
"/etc/resolv.conf",
}
SYSTEM_PREFIXES = ("/dev", "/proc", "/sys", "/run", "/var/run", "/var/lib/docker")
# Volume driver_opts types that point at storage which lives somewhere else
# entirely (a NAS). Pulling a media library through a tar.gz is never what the
# user wants, and *restoring* one would overwrite the share.
REMOTE_VOLUME_TYPES = {"nfs", "nfs4", "cifs", "smb", "smb3", "smbfs", "sshfs", "glusterfs"}
# Bind directories larger than this are listed but not selected by default.
DEFAULT_MAX_BIND_BYTES = 2 * 1024**3
_ENV_RE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)(?::?-([^}]*))?\}|\$([A-Za-z_][A-Za-z0-9_]*)")
class AssetError(Exception):
pass
# --------------------------------------------------------------------------- #
# Helper container primitives (host-path I/O)
# --------------------------------------------------------------------------- #
def ensure_helper_image(client) -> None:
image = settings.BACKUP_HELPER_IMAGE
try:
safe_call(client.images.get, image)
except DockerError:
logger.info("Pulling helper image %s", image)
safe_call(client.images.pull, image)
def _create_helper(client, volumes: dict):
return safe_call(
client.containers.create,
settings.BACKUP_HELPER_IMAGE,
command="true",
volumes=volumes,
)
def _remove(container) -> None:
try:
container.remove(force=True)
except Exception: # noqa: BLE001 - cleanup is best effort
pass
def _split(path: str) -> tuple[str, str]:
clean = path.rstrip("/") or "/"
return os.path.dirname(clean) or "/", os.path.basename(clean)
def _is_chown_error(exc: Exception) -> bool:
text = str(exc)
return "chown" in text.lower() and "not permitted" in text.lower()
def _put_archive(container, dest: str, src_file: str, volumes: dict) -> None:
"""Unpack an archive into a container path, coping with squashed mounts.
The daemon restores ownership while extracting, which an NFS/CIFS export
with ``root_squash`` refuses. In that case the archive is unpacked into a
throwaway container's own filesystem first and the files are then copied
across — ownership cannot be preserved there, but the restore completes
instead of failing outright.
"""
with open(src_file, "rb") as fh:
try:
container.put_archive(dest, fh)
return
except Exception as exc: # noqa: BLE001
if not _is_chown_error(exc):
raise
logger.warning("%s rejects ownership changes; restoring without it", dest)
client = get_client()
staging = _create_helper_with(
client, volumes, ["sh", "-c", f"cp -R /tmp/. '{dest}/'"]
)
try:
with open(src_file, "rb") as fh:
staging.put_archive("/tmp", fh)
staging.start()
status = staging.wait(timeout=3600).get("StatusCode", 1)
if status != 0:
err = (staging.logs(stdout=True, stderr=True) or b"").decode("utf-8", "replace")
raise AssetError(err.strip() or f"copy failed (exit {status})")
finally:
_remove(staging)
def _create_helper_with(client, volumes: dict, command: list[str]):
ensure_helper_image(client)
return safe_call(
client.containers.create,
settings.BACKUP_HELPER_IMAGE,
command=command,
volumes=volumes,
)
def inspect_paths(paths: list[str]) -> dict[str, dict]:
"""Classify host paths ({path: {"kind", "size"}}) via one helper container.
``kind`` is dir / file / special; ``size`` is bytes (best effort, the walk is
capped so a huge media share can't stall the request).
"""
unique = [p for p in dict.fromkeys(paths) if p]
if not unique:
return {}
client = get_client()
ensure_helper_image(client)
mounts = {p: {"bind": f"/m/{i}", "mode": "ro"} for i, p in enumerate(unique)}
script_parts = []
for i in range(len(unique)):
script_parts.append(
f'd=/m/{i}; '
f'if [ -d "$d" ]; then s=$(timeout 20 du -sk "$d" 2>/dev/null | cut -f1); '
f'echo "{i} dir ${{s:-}}"; '
f'elif [ -f "$d" ]; then echo "{i} file $(stat -c %s "$d" 2>/dev/null)"; '
f'else echo "{i} special"; fi'
)
script = "; ".join(script_parts)
try:
out = safe_call(
client.containers.run,
settings.BACKUP_HELPER_IMAGE,
["sh", "-c", script],
volumes=mounts,
remove=True,
stdout=True,
stderr=False,
)
except DockerError as exc:
logger.warning("Path inspection failed: %s", exc)
return {p: {"kind": "unknown", "size": None} for p in unique}
result: dict[str, dict] = {p: {"kind": "unknown", "size": None} for p in unique}
for line in (out or b"").decode("utf-8", "replace").splitlines():
parts = line.strip().split()
if len(parts) < 2 or not parts[0].isdigit():
continue
idx = int(parts[0])
if idx >= len(unique):
continue
kind = parts[1]
size: Optional[int] = None
if len(parts) > 2 and parts[2].isdigit():
size = int(parts[2]) * 1024 if kind == "dir" else int(parts[2])
result[unique[idx]] = {"kind": kind, "size": size}
return result
def export_path(source: str, kind: str, dest_file: str) -> int:
"""Tar a host path (dir contents, or a single file) into ``dest_file``."""
client = get_client()
ensure_helper_image(client)
if kind == "file":
parent, base = _split(source)
if not base:
raise AssetError(f"Cannot archive {source}")
container = _create_helper(client, {parent: {"bind": "/src", "mode": "ro"}})
member = f"/src/{base}"
else:
container = _create_helper(client, {source: {"bind": "/src", "mode": "ro"}})
# "/src/." archives the *contents*, so restore can unpack straight back
# into the directory without a stray prefix.
member = "/src/."
written = 0
try:
bits, _ = container.get_archive(member)
with open(dest_file, "wb") as fh:
for chunk in bits:
fh.write(chunk)
written += len(chunk)
finally:
_remove(container)
return written
def import_path(source: str, kind: str, src_file: str) -> None:
"""Unpack an archive produced by :func:`export_path` back to its host path."""
client = get_client()
ensure_helper_image(client)
if kind == "file":
parent, _base = _split(source)
mounts = {parent: {"bind": "/dst", "mode": "rw"}}
else:
mounts = {source: {"bind": "/dst", "mode": "rw"}}
container = _create_helper(client, mounts)
try:
_put_archive(container, "/dst", src_file, mounts)
finally:
_remove(container)
def export_volume(full_name: str, dest_file: str) -> int:
"""Stream a named volume's contents into ``dest_file`` (never into RAM)."""
client = get_client()
ensure_helper_image(client)
container = _create_helper(client, {full_name: {"bind": "/v", "mode": "ro"}})
written = 0
try:
bits, _ = container.get_archive("/v/.")
with open(dest_file, "wb") as fh:
for chunk in bits:
fh.write(chunk)
written += len(chunk)
finally:
_remove(container)
return written
def import_volume(full_name: str, labels: dict, src_file: str, wipe: bool = True) -> None:
"""Restore a volume from an archive, optionally clearing it first."""
client = get_client()
ensure_helper_image(client)
existed = True
try:
safe_call(client.volumes.get, full_name)
except DockerError:
existed = False
safe_call(client.volumes.create, name=full_name, labels=labels or {})
if existed and wipe:
# Restore means "back to the snapshot": drop files created since.
safe_call(
client.containers.run,
settings.BACKUP_HELPER_IMAGE,
["sh", "-c", "find /v -mindepth 1 -delete"],
volumes={full_name: {"bind": "/v", "mode": "rw"}},
remove=True,
)
mounts = {full_name: {"bind": "/v", "mode": "rw"}}
container = _create_helper(client, mounts)
try:
_put_archive(container, "/v", src_file, mounts)
finally:
_remove(container)
# --------------------------------------------------------------------------- #
# Where does STACKS_DIR really live on the host?
# --------------------------------------------------------------------------- #
def host_stacks_dir() -> Optional[str]:
"""Host path backing ``STACKS_DIR`` inside this container, if detectable.
Read from /proc/self/mountinfo (field 4 is the source subtree on the host
filesystem). Returns None when not running in a container / not bind-mounted.
"""
target = settings.STACKS_DIR.rstrip("/") or "/"
try:
with open("/proc/self/mountinfo", "r", encoding="utf-8") as fh:
for line in fh:
parts = line.split()
if len(parts) < 5:
continue
if parts[4].rstrip("/") == target:
return parts[3]
except OSError:
return None
return None
def stacks_path_mismatch() -> Optional[dict]:
"""Report a host/container path mismatch for the stacks directory.
When they differ, compose resolves a stack's relative bind mounts against
the *container* path, so the daemon creates the data directories at that
path on the host — invisible to StackPilot. Backups then only find the
compose file unless bind sources are captured through a helper container.
"""
host = host_stacks_dir()
container = settings.STACKS_DIR.rstrip("/")
if not host or host.rstrip("/") == container:
return None
return {"host": host, "container": container}
# --------------------------------------------------------------------------- #
# Compose / container mount discovery
# --------------------------------------------------------------------------- #
def _env_for_stack(stack_id: str) -> dict:
env: dict[str, str] = {}
path = os.path.join(compose_service.stack_dir(stack_id), ".env")
try:
with open(path, "r", encoding="utf-8", errors="replace") as fh:
for raw in fh:
line = raw.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, value = line.partition("=")
env[key.strip()] = value.strip().strip('"').strip("'")
except OSError:
pass
return env
def _interpolate(text: str, env: dict) -> str:
def repl(m: re.Match) -> str:
name = m.group(1) or m.group(3)
default = m.group(2) or ""
return env.get(name, default)
return _ENV_RE.sub(repl, text)
def _bind_specs_from_compose(stack_id: str) -> list[dict]:
"""Bind sources declared in the compose file (used when no containers exist)."""
directory = compose_service.stack_dir(stack_id)
compose_file = compose_service.find_compose_file(directory)
if not compose_file:
return []
try:
with open(compose_file, "r", encoding="utf-8", errors="replace") as fh:
data = yaml.safe_load(fh) or {}
except (OSError, yaml.YAMLError):
return []
env = _env_for_stack(stack_id)
out: list[dict] = []
for service, spec in (data.get("services") or {}).items():
if not isinstance(spec, dict):
continue
for entry in spec.get("volumes") or []:
source = target = None
if isinstance(entry, str):
parts = _interpolate(entry, env).split(":")
if len(parts) >= 2:
source, target = parts[0], parts[1]
elif isinstance(entry, dict):
if entry.get("type") not in (None, "bind"):
continue
source = _interpolate(str(entry.get("source") or ""), env)
target = _interpolate(str(entry.get("target") or ""), env)
if not source or not target:
continue
if not (source.startswith("/") or source.startswith(".") or source.startswith("~")):
continue # named volume
if source.startswith("~"):
continue # home-relative: resolved by the daemon's user, skip
resolved = source if source.startswith("/") else os.path.normpath(
os.path.join(directory, source)
)
out.append({"source": resolved, "service": str(service), "target": target})
return out
def _bind_specs_from_containers(stack_id: str) -> list[dict]:
"""Bind sources as the daemon actually mounted them (authoritative)."""
try:
client = get_client()
containers = safe_call(
client.containers.list,
all=True,
filters={"label": f"{COMPOSE_PROJECT_LABEL}={stack_id}"},
)
except DockerError:
return []
out: list[dict] = []
for c in containers:
service = (c.labels or {}).get(COMPOSE_SERVICE_LABEL, c.name)
for mount in c.attrs.get("Mounts") or []:
if mount.get("Type") != "bind" or not mount.get("Source"):
continue
out.append(
{
"source": mount["Source"],
"service": service,
"target": mount.get("Destination") or "",
}
)
return out
def is_system_path(path: str) -> bool:
if path in SYSTEM_PATHS:
return True
return any(path == p or path.startswith(p + "/") for p in SYSTEM_PREFIXES)
def _inside(path: str, parent: str) -> bool:
parent = parent.rstrip("/")
return path == parent or path.startswith(parent + "/")
def compose_volumes(stack_id: str) -> list[dict]:
"""Compose-managed named volumes, with remote-storage detection."""
try:
client = get_client()
vols = safe_call(
client.volumes.list,
filters={"label": f"{COMPOSE_PROJECT_LABEL}={stack_id}"},
)
except DockerError:
return []
out = []
for v in vols:
attrs = v.attrs or {}
labels = attrs.get("Labels") or {}
options = attrs.get("Options") or {}
driver = attrs.get("Driver", "local")
vtype = str(options.get("type") or "").lower()
device = str(options.get("device") or "")
remote = (
vtype in REMOTE_VOLUME_TYPES
or driver != "local"
or device.startswith("//")
or device.startswith(":")
)
out.append(
{
"name": v.name,
"short": labels.get(COMPOSE_VOLUME_LABEL, v.name),
"labels": labels,
"driver": driver,
"options": options,
"remote": remote,
"remote_type": vtype or (driver if driver != "local" else None),
}
)
return out
def inventory(stack_id: str, max_bind_bytes: int = DEFAULT_MAX_BIND_BYTES) -> dict:
"""What a backup of this stack would (and would not) capture.
Bind sources are merged from the running containers (authoritative) and the
compose file (covers stacks that were never started), classified through a
helper container so host-only paths are seen too.
"""
directory = compose_service.stack_dir(stack_id)
specs = _bind_specs_from_containers(stack_id) or []
seen = {(s["source"], s["service"], s["target"]) for s in specs}
for spec in _bind_specs_from_compose(stack_id):
if (spec["source"], spec["service"], spec["target"]) not in seen:
specs.append(spec)
grouped: dict[str, dict] = {}
for spec in specs:
entry = grouped.setdefault(spec["source"], {"source": spec["source"], "mounts": []})
mount = {"service": spec["service"], "target": spec["target"]}
if mount not in entry["mounts"]:
entry["mounts"].append(mount)
real_paths = [p for p in grouped if not is_system_path(p)]
stats = inspect_paths(real_paths)
binds = []
for path, entry in sorted(grouped.items()):
system = is_system_path(path)
info = stats.get(path, {"kind": "unknown", "size": None})
kind, size = info["kind"], info["size"]
inside = _inside(path, directory)
# A path inside the stack directory that this process can actually read
# is already covered by the compose/ tree in the archive.
visible = inside and os.path.exists(path)
include = True
reason = None
if system:
include, reason = False, "system path"
elif kind == "special":
include, reason = False, "not a regular file or directory"
elif kind == "unknown":
include, reason = False, "could not inspect path"
elif size is not None and size > max_bind_bytes:
include, reason = False, f"larger than {max_bind_bytes // 1024**3} GiB"
binds.append(
{
"source": path,
"mounts": entry["mounts"],
"kind": kind,
"size": size,
"inside_stack_dir": inside,
# Readable from here and inside the stack folder → the compose/
# tree already carries it, no separate archive needed.
"covered_by_compose": visible,
"via": "compose" if visible else "archive",
"system": system,
"include_default": include,
"reason": reason,
}
)
volumes = []
for vol in compose_volumes(stack_id):
include = not vol["remote"]
volumes.append(
{
**vol,
"include_default": include,
"reason": None if include else f"remote storage ({vol['remote_type']})",
}
)
return {
"stack_id": stack_id,
"stack_dir": directory,
"stack_dir_visible": os.path.isdir(directory),
"path_mismatch": stacks_path_mismatch(),
"binds": binds,
"volumes": volumes,
}
backend/services/stack_lock_service.py
+144
View File
@@ -0,0 +1,144 @@
"""One compose operation per stack at a time.
``docker compose`` does no locking. Two ``update`` calls against the same
project — two open browser tabs, or the auto-update pass landing on a stack
somebody just clicked — both run ``pull`` and then ``up -d``, and race each
other recreating the same containers.
There *was* a busy flag in ``compose_service``, but it only ever fed the status
column: no lifecycle handler consulted it before acting. This module is the
actual guard, and it lives in the database so it holds across workers and
across a restart. ``compose_service.compute_status`` therefore reports only
what the containers say; callers overlay the lock to show "updating".
"""
from __future__ import annotations
import logging
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from typing import Optional
from sqlalchemy.exc import IntegrityError
from sqlmodel import Session, delete, select
from models.runtime_state import StackLock
logger = logging.getLogger("stackpilot.stack_lock")
#: Long enough to outlast the slowest legitimate operation (compose commands
#: time out at 600s, a full pull of a large stack can chain several), short
#: enough that a lock orphaned by a killed worker clears itself within an hour.
DEFAULT_TTL = timedelta(minutes=30)
class StackBusy(Exception):
"""The stack is already running an operation."""
def __init__(self, stack_id: str, action: str):
self.stack_id = stack_id
self.action = action
super().__init__(f"Stack '{stack_id}' is busy: {action} in progress")
def _now() -> datetime:
return datetime.now(timezone.utc)
def _aware(value: Optional[datetime]) -> Optional[datetime]:
"""SQLite hands datetimes back naive; compare them as UTC."""
if value is not None and value.tzinfo is None:
return value.replace(tzinfo=timezone.utc)
return value
def acquire(
session: Session,
stack_id: str,
action: str,
owner: str = "",
ttl: timedelta = DEFAULT_TTL,
) -> None:
"""Take the lock for ``stack_id`` or raise :class:`StackBusy`.
An expired lock is taken over — that is the recovery path for a worker that
died mid-deploy, which would otherwise leave the stack unusable.
"""
now = _now()
existing = session.get(StackLock, stack_id)
if existing is not None:
if (_aware(existing.expires_at) or now) > now:
raise StackBusy(stack_id, existing.action)
logger.warning(
"Taking over an expired %s lock on '%s' (held by %r since %s)",
existing.action, stack_id, existing.owner, existing.acquired_at,
)
session.delete(existing)
session.commit()
session.add(
StackLock(
stack_id=stack_id,
action=action,
owner=owner,
acquired_at=now,
expires_at=now + ttl,
)
)
try:
session.commit()
except IntegrityError as exc:
# Another worker inserted between our check and our commit. The primary
# key is what actually makes this safe; the read above is only there to
# give a useful error and to clear stale rows.
session.rollback()
raise StackBusy(stack_id, action) from exc
def release(session: Session, stack_id: str) -> None:
"""Drop the lock. Safe to call when it is not held."""
existing = session.get(StackLock, stack_id)
if existing is not None:
session.delete(existing)
session.commit()
@contextmanager
def hold(session: Session, stack_id: str, action: str, owner: str = ""):
"""Hold the lock for the duration of the block.
Raises :class:`StackBusy` if somebody else has it. Always releases, so a
failed deploy does not leave the stack locked.
"""
acquire(session, stack_id, action, owner)
try:
yield
finally:
try:
release(session, stack_id)
except Exception: # noqa: BLE001 - never mask the original error
logger.exception("Failed to release the lock on '%s'", stack_id)
def active(session: Session) -> dict[str, str]:
"""``{stack_id: action}`` for every lock still in force.
One query for the whole stacks list, rather than a lookup per row.
"""
now = _now()
return {
lock.stack_id: lock.action
for lock in session.exec(select(StackLock)).all()
if (_aware(lock.expires_at) or now) > now
}
def is_busy(session: Session, stack_id: str) -> bool:
lock = session.get(StackLock, stack_id)
return lock is not None and (_aware(lock.expires_at) or _now()) > _now()
def prune_expired(session: Session) -> int:
"""Drop locks that have timed out. Called at startup and by the scheduler."""
result = session.exec(delete(StackLock).where(StackLock.expires_at < _now()))
session.commit()
return result.rowcount or 0
backend/services/stats_service.py
+38 -2
View File
@@ -3,15 +3,33 @@
Reads a one-shot ``docker stats`` sample per running container (the daemon
includes ``precpu_stats`` so a single read yields a usable CPU delta) and sums
them by ``com.docker.compose.project`` label, which equals the stack id.
Sampling is not free: it is one blocking call to the daemon *per running
container*, and the dashboard and the stacks list both poll this every five
seconds. Two open tabs on a 40-container host meant a sustained ~16 samples a
second. Results are therefore cached for :data:`CACHE_TTL`, the same shape
``dashboard_service`` already uses for its fleet aggregate — one sweep serves
every reader in the window, and the numbers stay well inside what a
five-second poll can show.
"""
from __future__ import annotations
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from docker_client import DockerError, get_client, safe_call
COMPOSE_PROJECT_LABEL = "com.docker.compose.project"
#: Slightly under the frontend's 5s poll, so a refresh usually gets fresh
#: numbers while concurrent readers still share one sweep.
CACHE_TTL = 4.0
_cache: dict = {"data": None, "ts": 0.0}
# Held across the sample so N simultaneous callers trigger one sweep, not N.
_lock = threading.Lock()
def _container_stats(container) -> dict | None:
try:
@@ -60,12 +78,30 @@ def _container_stats(container) -> dict | None:
}
def stack_stats() -> dict:
def stack_stats(refresh: bool = False) -> dict:
"""Return {stack_id: {cpu_used, cpu_limit, mem_used, mem_limit, containers}}.
Limits are the summed assigned limits across the stack's containers, or null
when none of them have that limit set.
when none of them have that limit set. Served from a short-lived cache
unless ``refresh`` is set.
"""
if not refresh and _cache["data"] is not None:
if time.monotonic() - _cache["ts"] < CACHE_TTL:
return _cache["data"]
with _lock:
# Somebody may have refreshed it while we waited for the lock.
if not refresh and _cache["data"] is not None:
if time.monotonic() - _cache["ts"] < CACHE_TTL:
return _cache["data"]
data = _sample()
_cache["data"] = data
_cache["ts"] = time.monotonic()
return data
def _sample() -> dict:
"""One full sweep across every running container."""
try:
client = get_client()
containers = safe_call(client.containers.list) # running only
backend/services/template_service.py
+220 -125
View File
@@ -1,32 +1,44 @@
"""Template library — bundled (on-disk) + custom (DB)."""
"""Template library — stack-shaped folders.
A template is just a directory laid out like a real stack (``compose.yaml`` plus
optional ``.env.example`` and any extra files), accompanied by a small
``template.json`` describing it. "Pulling" a template copies the whole folder
into a new stack, which is then editable like any other stack.
Two roots are scanned:
* **bundled** — ``backend/templates/`` ships in the image / git repo (read-only).
* **custom** — ``${DATA_DIR}/templates/`` is writable and persists on the data
volume; this is where "save stack as template" writes to.
"""
from __future__ import annotations
import json
import os
import re
from functools import lru_cache
import shutil
from typing import Optional
from sqlmodel import Session, select
from config import settings
from services import compose_service
from models.template import Template
BUNDLED_DIR = os.path.join(os.path.dirname(os.path.dirname(__file__)), "templates")
_TEMPLATES_DIR = os.path.join(os.path.dirname(os.path.dirname(__file__)), "templates")
_VAR_RE = re.compile(r"\{\{\s*([A-Za-z0-9_]+)\s*\}\}")
_META_NAME = "template.json"
_ENV_EXAMPLE = ".env.example"
_CUSTOM_PREFIX = "custom:"
@lru_cache(maxsize=1)
def _manifest() -> list[dict]:
path = os.path.join(_TEMPLATES_DIR, "manifest.json")
try:
with open(path, "r", encoding="utf-8") as fh:
return json.load(fh)
except (OSError, json.JSONDecodeError):
return []
def custom_dir() -> str:
return os.path.join(settings.DATA_DIR, "templates")
def _read_template_file(filename: str) -> str:
path = os.path.join(_TEMPLATES_DIR, filename)
# --------------------------------------------------------------------------- #
# Low-level helpers
# --------------------------------------------------------------------------- #
def _read_file(path: str) -> str:
try:
with open(path, "r", encoding="utf-8") as fh:
return fh.read()
@@ -34,134 +46,217 @@ def _read_template_file(filename: str) -> str:
return ""
def extract_variables(yaml_str: str) -> list[str]:
seen: list[str] = []
for m in _VAR_RE.finditer(yaml_str):
if m.group(1) not in seen:
seen.append(m.group(1))
return seen
def _load_meta(folder: str, slug: str) -> dict:
"""Metadata from template.json, with sensible fallbacks."""
meta: dict = {}
raw = _read_file(os.path.join(folder, _META_NAME))
if raw:
try:
meta = json.loads(raw)
except json.JSONDecodeError:
meta = {}
return {
"name": meta.get("name") or slug.replace("-", " ").title(),
"description": meta.get("description"),
"tags": meta.get("tags") or [],
"gpu": meta.get("gpu"),
}
def render(yaml_str: str, values: dict[str, str]) -> str:
def repl(m: re.Match) -> str:
key = m.group(1)
return str(values.get(key, m.group(0)))
return _VAR_RE.sub(repl, yaml_str)
def _is_template(folder: str) -> bool:
return os.path.isdir(folder) and compose_service.find_compose_file(folder) is not None
# --------------------------------------------------------------------------- #
# Listing
# --------------------------------------------------------------------------- #
def _list_files(folder: str) -> list[str]:
"""Relative paths shipped by the template (excludes the metadata file)."""
out: list[str] = []
for root, _dirs, files in os.walk(folder):
for fn in sorted(files):
rel = os.path.relpath(os.path.join(root, fn), folder)
if rel == _META_NAME:
continue
out.append(rel)
return sorted(out)
def list_templates(session: Session) -> list[dict]:
def _resolve_dir(template_id: str) -> Optional[str]:
"""Map a template id to its on-disk folder (guards against traversal)."""
if template_id.startswith(_CUSTOM_PREFIX):
slug = template_id[len(_CUSTOM_PREFIX):]
base = custom_dir()
else:
slug = template_id
base = BUNDLED_DIR
slug = os.path.basename(slug.strip())
if not slug:
return None
path = os.path.join(base, slug)
return path if _is_template(path) else None
def _scan(base: str, source: str) -> list[dict]:
out: list[dict] = []
for entry in _manifest():
if not os.path.isdir(base):
return out
for slug in sorted(os.listdir(base)):
folder = os.path.join(base, slug)
if not _is_template(folder):
continue
meta = _load_meta(folder, slug)
out.append(
{
"id": entry["id"],
"name": entry["name"],
"description": entry.get("description"),
"tags": entry.get("tags", []),
"gpu": entry.get("gpu"),
"source": "bundled",
}
)
for tpl in session.exec(select(Template)).all():
out.append(
{
"id": f"custom:{tpl.slug}",
"name": tpl.name,
"description": tpl.description,
"tags": [t for t in tpl.tags.split(",") if t],
"gpu": None,
"source": "custom",
"id": f"{_CUSTOM_PREFIX}{slug}" if source == "custom" else slug,
"name": meta["name"],
"description": meta["description"],
"tags": meta["tags"],
"gpu": meta["gpu"],
"source": source,
}
)
return out
def get_template(session: Session, template_id: str) -> Optional[dict]:
if template_id.startswith("custom:"):
slug = template_id.split(":", 1)[1]
tpl = session.exec(select(Template).where(Template.slug == slug)).first()
if not tpl:
return None
variables = [
{"name": v, "description": "", "default": ""}
for v in extract_variables(tpl.yaml)
]
return {
"id": template_id,
"name": tpl.name,
"description": tpl.description,
"tags": [t for t in tpl.tags.split(",") if t],
"gpu": None,
"source": "custom",
"yaml": tpl.yaml,
"variables": variables,
}
# --------------------------------------------------------------------------- #
# Listing / detail
# --------------------------------------------------------------------------- #
for entry in _manifest():
if entry["id"] == template_id:
yaml_str = _read_template_file(entry["file"])
declared = {v["name"]: v for v in entry.get("variables", [])}
# Merge declared metadata with any vars actually present.
variables = []
for name in extract_variables(yaml_str):
meta = declared.get(name, {})
variables.append(
{
"name": name,
"description": meta.get("description", ""),
"default": meta.get("default", ""),
}
)
return {
"id": entry["id"],
"name": entry["name"],
"description": entry.get("description"),
"tags": entry.get("tags", []),
"gpu": entry.get("gpu"),
"source": "bundled",
"yaml": yaml_str,
"variables": variables,
}
return None
def list_templates() -> list[dict]:
return _scan(BUNDLED_DIR, "bundled") + _scan(custom_dir(), "custom")
def get_template(template_id: str) -> Optional[dict]:
folder = _resolve_dir(template_id)
if not folder:
return None
slug = os.path.basename(folder)
meta = _load_meta(folder, slug)
compose_file = compose_service.find_compose_file(folder)
return {
"id": template_id,
"name": meta["name"],
"description": meta["description"],
"tags": meta["tags"],
"gpu": meta["gpu"],
"source": "custom" if template_id.startswith(_CUSTOM_PREFIX) else "bundled",
"compose": _read_file(compose_file) if compose_file else "",
"env": _read_file(os.path.join(folder, _ENV_EXAMPLE)),
"files": _list_files(folder),
}
# --------------------------------------------------------------------------- #
# Pull (instantiate) — copy the whole folder into a new stack
# --------------------------------------------------------------------------- #
def copy_into_stack(template_id: str, stack_id: str, override: Optional[str] = None) -> None:
"""Copy a template folder into a fresh stack directory.
The ``template.json`` is left behind and any ``.env.example`` is promoted to
a real ``.env`` so the pulled stack is immediately runnable + editable.
"""
src = _resolve_dir(template_id)
if not src:
raise FileNotFoundError(f"Template '{template_id}' not found")
dst = compose_service.stack_dir(stack_id, override)
if os.path.exists(dst):
raise FileExistsError(f"Stack '{stack_id}' already exists")
shutil.copytree(src, dst, ignore=shutil.ignore_patterns(_META_NAME))
example = os.path.join(dst, _ENV_EXAMPLE)
env = os.path.join(dst, ".env")
if os.path.isfile(example) and not os.path.isfile(env):
os.replace(example, env)
# --------------------------------------------------------------------------- #
# Save / delete custom templates (folder-based, persisted on the data volume)
# --------------------------------------------------------------------------- #
def save_custom(
session: Session, name: str, yaml_str: str, description: str = "", tags: list[str] | None = None
) -> Template:
slug = re.sub(r"[^a-z0-9_-]+", "-", name.strip().lower()).strip("-") or "template"
existing = session.exec(select(Template).where(Template.slug == slug)).first()
if existing:
existing.name = name
existing.description = description
existing.tags = ",".join(tags or [])
existing.yaml = yaml_str
session.add(existing)
session.commit()
session.refresh(existing)
return existing
tpl = Template(
slug=slug,
name=name,
description=description,
tags=",".join(tags or []),
yaml=yaml_str,
)
session.add(tpl)
session.commit()
session.refresh(tpl)
return tpl
name: str,
compose: str,
env: str = "",
description: str = "",
tags: list[str] | None = None,
gpu: str | None = None,
) -> str:
"""Write a custom template folder; returns its slug. Overwrites if it exists."""
slug = compose_service.slugify(name)
folder = os.path.join(custom_dir(), slug)
os.makedirs(folder, exist_ok=True)
meta = {
"name": name,
"description": description or None,
"tags": tags or [],
"gpu": gpu,
}
with open(os.path.join(folder, _META_NAME), "w", encoding="utf-8") as fh:
json.dump(meta, fh, indent=2)
fh.write("\n")
with open(os.path.join(folder, compose_service.DEFAULT_COMPOSE_NAME), "w", encoding="utf-8") as fh:
fh.write(compose or "services:\n")
example = os.path.join(folder, _ENV_EXAMPLE)
if env.strip():
with open(example, "w", encoding="utf-8") as fh:
fh.write(env)
elif os.path.isfile(example):
os.remove(example)
return slug
def delete_custom(session: Session, slug: str) -> bool:
tpl = session.exec(select(Template).where(Template.slug == slug)).first()
if not tpl:
def save_from_stack(stack_id: str, name: str, description: str = "") -> str:
"""Snapshot an existing stack's compose + env into a custom template."""
compose = compose_service.read_compose(stack_id)
env = compose_service.read_env(stack_id)
return save_custom(name, compose, env, description=description)
def delete_custom(slug: str) -> bool:
folder = os.path.join(custom_dir(), os.path.basename(slug.strip()))
if not _is_template(folder):
return False
session.delete(tpl)
session.commit()
shutil.rmtree(folder)
return True
# --------------------------------------------------------------------------- #
# Legacy migration (pre-0.31 custom templates lived in the database)
# --------------------------------------------------------------------------- #
_LEGACY_VAR_RE = re.compile(r"\{\{\s*([A-Za-z0-9_]+)\s*\}\}")
def migrate_legacy_db_templates() -> int:
"""One-time: move custom templates out of the dropped ``template`` table.
Old templates used ``{{VAR}}`` placeholders; compose interpolates ``${VAR}``
from ``.env``, so placeholders are rewritten and the variable names land in
the template's ``.env.example``. Returns the number of templates moved.
"""
from sqlalchemy import inspect, text
from database import engine
if not inspect(engine).has_table("template"):
return 0
moved = 0
with engine.begin() as conn:
rows = conn.execute(
text("SELECT name, description, tags, yaml FROM template")
).all()
for name, description, tags, yaml_str in rows:
compose = _LEGACY_VAR_RE.sub(r"${\1}", yaml_str or "")
variables = dict.fromkeys(_LEGACY_VAR_RE.findall(yaml_str or ""))
env = "".join(f"{v}=\n" for v in variables)
save_custom(
name or "template",
compose,
env,
description=description or "",
tags=[t for t in (tags or "").split(",") if t],
)
moved += 1
conn.execute(text("DROP TABLE template"))
return moved
backend/services/update_service.py
+174 -16
View File
@@ -10,14 +10,14 @@ import asyncio
import logging
import time
from dataclasses import asdict, dataclass
from collections.abc import Callable
from typing import Optional
import httpx
from config import settings
from docker_client import DockerError, get_client, safe_call
from models.setting import EVENT_UPDATE_AVAILABLE
from services import notify_service, settings_service
from services import notify_service, registry_service, settings_service
logger = logging.getLogger("stackpilot.update")
@@ -51,6 +51,49 @@ _CACHE: dict[str, UpdateStatus] = {}
# background loop doesn't re-notify on every cycle.
_NOTIFIED: set[str] = set()
#: Optional sink for cache writes.
#:
#: This module is pure registry logic and knows nothing about storage, which
#: keeps it unit-testable without a database. ``main.lifespan`` registers a
#: callback that mirrors each entry into SQLite (see
#: ``services/image_status_store.py``) and seeds the cache from it at startup.
#: Without it a restart blanked every update badge until the next background
#: sweep — up to an hour — and re-announced updates it had already notified
#: about.
_persist_cb: Optional[Callable[[UpdateStatus, bool], None]] = None
#: Optional sink for "these images are still in use", same opt-in shape as
#: _persist_cb. Keeps both the dict and the table from growing one entry per
#: image tag that was ever running, for the life of the install.
_prune_cb: Optional[Callable[[set], int]] = None
def set_persist_callback(
callback: Optional[Callable[[UpdateStatus, bool], None]],
prune: Optional[Callable[[set], int]] = None,
) -> None:
global _persist_cb, _prune_cb
_persist_cb = callback
_prune_cb = prune
def restore_cache(entries: list[tuple[UpdateStatus, bool]]) -> None:
"""Seed the in-memory cache from persisted rows at startup."""
for status, notified in entries:
_CACHE[status.image] = status
if notified:
_NOTIFIED.add(status.image)
def _store(status: UpdateStatus, notified: bool) -> None:
_CACHE[status.image] = status
if _persist_cb is not None:
try:
_persist_cb(status, notified)
except Exception as exc: # noqa: BLE001 - persistence is best-effort
logger.debug("Could not persist update status for %s: %s", status.image, exc)
# --------------------------------------------------------------------------- #
# Image reference parsing
@@ -102,7 +145,16 @@ def _local_digest(image: str) -> Optional[str]:
# --------------------------------------------------------------------------- #
async def _get_token(client: httpx.AsyncClient, www_auth: str) -> Optional[str]:
async def _get_token(
client: httpx.AsyncClient,
www_auth: str,
auth: Optional[tuple[str, str]] = None,
) -> Optional[str]:
"""Follow a Bearer challenge, with credentials when we have them.
A public image gets an anonymous token; a private one only gets a token at
all if the request to the token realm is authenticated.
"""
# Parse: Bearer realm="...",service="...",scope="..."
params = {}
if not www_auth.lower().startswith("bearer"):
@@ -115,7 +167,7 @@ async def _get_token(client: httpx.AsyncClient, www_auth: str) -> Optional[str]:
if not realm:
return None
try:
resp = await client.get(realm, params=params, timeout=10)
resp = await client.get(realm, params=params, auth=auth, timeout=10)
resp.raise_for_status()
data = resp.json()
return data.get("token") or data.get("access_token")
@@ -123,25 +175,54 @@ async def _get_token(client: httpx.AsyncClient, www_auth: str) -> Optional[str]:
return None
class AuthRequired(Exception):
"""The registry wants credentials we do not have (or rejected ours).
Distinct from "could not reach the registry" on purpose: a private image
with no configured credentials used to be indistinguishable from a network
blip, so the UI said nothing and the stack looked up to date forever.
"""
async def remote_digest(image: str) -> Optional[str]:
"""The digest the registry currently serves for this tag.
Raises :class:`AuthRequired` when the registry refuses us; returns None when
it could not be reached or answered without a digest.
"""
registry, repo, tag = parse_ref(image)
if tag.startswith("sha256:"):
return tag
scheme = "https"
url = f"{scheme}://{registry}/v2/{repo}/manifests/{tag}"
headers = {"Accept": _MANIFEST_ACCEPT}
auth = registry_service.credentials_for(registry)
async with httpx.AsyncClient(follow_redirects=True) as client:
try:
resp = await client.head(url, headers=headers, timeout=10)
if resp.status_code == 401:
token = await _get_token(client, resp.headers.get("WWW-Authenticate", ""))
if not token:
return None
headers["Authorization"] = f"Bearer {token}"
resp = await client.head(url, headers=headers, timeout=10)
challenge = resp.headers.get("WWW-Authenticate", "")
if challenge.lower().startswith("basic"):
# A plain htpasswd-protected registry: no token dance.
if not auth:
raise AuthRequired(registry)
resp = await client.head(url, headers=headers, auth=auth, timeout=10)
else:
token = await _get_token(client, challenge, auth)
if not token:
raise AuthRequired(registry)
headers["Authorization"] = f"Bearer {token}"
resp = await client.head(url, headers=headers, timeout=10)
if resp.status_code in (401, 403):
raise AuthRequired(registry)
if resp.status_code == 405 or "Docker-Content-Digest" not in resp.headers:
# Some registries don't support HEAD; fall back to GET.
resp = await client.get(url, headers=headers, timeout=10)
resp = await client.get(
url, headers=headers, auth=auth if "Authorization" not in headers else None,
timeout=10,
)
if resp.status_code in (401, 403):
raise AuthRequired(registry)
digest = resp.headers.get("Docker-Content-Digest")
return digest
except httpx.HTTPError as exc:
@@ -156,9 +237,18 @@ async def remote_digest(image: str) -> Optional[str]:
async def check_image(image: str) -> UpdateStatus:
local = _local_digest(image)
remote = await remote_digest(image)
error = None
if remote is None:
try:
remote = await remote_digest(image)
except AuthRequired as exc:
# Say which registry, because the fix is to add credentials for it.
remote = None
error = (
f"{exc} needs credentials"
if not registry_service.credentials_for(str(exc))
else f"{exc} rejected the stored credentials"
)
if remote is None and error is None:
error = "could not reach registry"
update_available = bool(local and remote and local != remote)
status = UpdateStatus(
@@ -169,9 +259,11 @@ async def check_image(image: str) -> UpdateStatus:
checked_at=time.time(),
error=error,
)
_CACHE[image] = status
if update_available and image not in _NOTIFIED:
# Marked before the attempt, not after: a notifier that is down should
# not make every cycle re-announce the same update.
_NOTIFIED.add(image)
_store(status, True)
try:
await notify_service.notify(
EVENT_UPDATE_AVAILABLE,
@@ -180,8 +272,10 @@ async def check_image(image: str) -> UpdateStatus:
)
except Exception as exc: # noqa: BLE001 - notifications are best-effort
logger.debug("update notify failed for %s: %s", image, exc)
elif not update_available:
_NOTIFIED.discard(image)
else:
if not update_available:
_NOTIFIED.discard(image)
_store(status, image in _NOTIFIED)
return status
@@ -215,12 +309,44 @@ def stack_images(stack_id: str) -> set[str]:
return images
def stacks_update_summary() -> dict[str, dict]:
"""Per-stack image-update status for every running compose project, read
from the digest cache the background loop maintains — no registry calls, so
it's cheap enough for the stacks list to poll. Stacks with no cached image
yet are simply absent (treated as "no update" by the UI)."""
by_stack: dict[str, set[str]] = {}
try:
client = get_client()
for c in safe_call(client.containers.list, all=True):
project = (c.labels or {}).get("com.docker.compose.project")
if not project:
continue
cfg_image = c.attrs.get("Config", {}).get("Image")
if cfg_image:
by_stack.setdefault(project, set()).add(cfg_image)
except DockerError:
return {}
summary: dict[str, dict] = {}
for stack_id, images in by_stack.items():
stale = [
img
for img in images
if (st := _CACHE.get(img)) is not None and st.update_available
]
summary[stack_id] = {
"update_available": bool(stale),
"stale_images": stale,
}
return summary
async def stack_updates(stack_id: str, refresh: bool = True) -> dict:
"""Update status for one stack's images.
``refresh=True`` queries the registry now; ``False`` reads the cache the
background loop already populated (so the auto-update pass adds no extra
registry round-trips). DB-free, so the agent can reuse it verbatim.
registry round-trips).
"""
images = stack_images(stack_id)
result: dict[str, dict] = {}
@@ -237,13 +363,45 @@ async def stack_updates(stack_id: str, refresh: bool = True) -> dict:
}
def refresh_stack_local(stack_id: str) -> None:
"""Re-read the local digests of one stack's images and reconcile them with
the cached remote digests (no registry calls). Called right after a manual
pull/update so the amber indicator clears immediately instead of lingering
until the next background pass."""
for image in stack_images(stack_id):
status = _CACHE.get(image)
if status is None:
continue
local = _local_digest(image)
status.current_digest = local
status.update_available = bool(
local and status.remote_digest and local != status.remote_digest
)
status.checked_at = time.time()
if not status.update_available:
_NOTIFIED.discard(image)
async def check_all() -> dict[str, dict]:
images = _all_running_images()
for image in images:
await check_image(image)
_forget_unused(set(images))
return {k: v.to_dict() for k, v in _CACHE.items()}
def _forget_unused(keep: set) -> None:
"""Drop images no running container references any more."""
for image in [i for i in _CACHE if i not in keep]:
del _CACHE[image]
_NOTIFIED.discard(image)
if _prune_cb is not None:
try:
_prune_cb(keep)
except Exception as exc: # noqa: BLE001 - housekeeping is best-effort
logger.debug("Could not prune persisted update statuses: %s", exc)
def get_cache() -> dict[str, dict]:
return {k: v.to_dict() for k, v in _CACHE.items()}
backend/templates/actual-budget/.env.example
+3
View File
@@ -0,0 +1,3 @@
TZ=Europe/Berlin
DATA_PATH=/srv/actual
HTTP_PORT=5006
backend/templates/actual-budget/compose.yaml
+11
View File
@@ -0,0 +1,11 @@
services:
actual:
image: ghcr.io/actualbudget/actual-server:latest
container_name: actual-budget
restart: unless-stopped
environment:
- TZ=${TZ:-Europe/Berlin}
ports:
- "${HTTP_PORT:-5006}:5006"
volumes:
- ${DATA_PATH:-/srv/actual}:/data
backend/templates/actual-budget/template.json
+9
View File
@@ -0,0 +1,9 @@
{
"name": "Actual Budget",
"description": "Local-first envelope budgeting with end-to-end encrypted sync across devices.",
"tags": [
"finance",
"productivity"
],
"gpu": null
}
backend/templates/adguard-home/.env.example
+4
View File
@@ -0,0 +1,4 @@
DATA_PATH=/srv/adguardhome
DNS_PORT=53
SETUP_PORT=3300
HTTP_PORT=8082
backend/templates/adguard-home/compose.yaml
+14
View File
@@ -0,0 +1,14 @@
services:
adguardhome:
image: adguard/adguardhome:latest
container_name: adguardhome
restart: unless-stopped
ports:
- "${DNS_PORT:-53}:53/tcp"
- "${DNS_PORT:-53}:53/udp"
# Setup wizard on first run; the UI moves to HTTP_PORT afterwards.
- "${SETUP_PORT:-3300}:3000/tcp"
- "${HTTP_PORT:-8082}:80/tcp"
volumes:
- ${DATA_PATH:-/srv/adguardhome}/work:/opt/adguardhome/work
- ${DATA_PATH:-/srv/adguardhome}/conf:/opt/adguardhome/conf
backend/templates/adguard-home/template.json
+10
View File
@@ -0,0 +1,10 @@
{
"name": "AdGuard Home",
"description": "DNS server with ad and tracker blocking, DoH/DoT and per-client rules.",
"tags": [
"network",
"dns",
"ad-blocking"
],
"gpu": null
}
backend/templates/audiobookshelf/.env.example
+5
View File
@@ -0,0 +1,5 @@
TZ=Europe/Berlin
DATA_PATH=/srv/audiobookshelf
AUDIOBOOKS_PATH=/srv/media/audiobooks
PODCASTS_PATH=/srv/media/podcasts
HTTP_PORT=13378
backend/templates/audiobookshelf/compose.yaml
+14
View File
@@ -0,0 +1,14 @@
services:
audiobookshelf:
image: ghcr.io/advplyr/audiobookshelf:latest
container_name: audiobookshelf
restart: unless-stopped
environment:
- TZ=${TZ:-Europe/Berlin}
ports:
- "${HTTP_PORT:-13378}:80"
volumes:
- ${DATA_PATH:-/srv/audiobookshelf}/config:/config
- ${DATA_PATH:-/srv/audiobookshelf}/metadata:/metadata
- ${AUDIOBOOKS_PATH:-/srv/media/audiobooks}:/audiobooks
- ${PODCASTS_PATH:-/srv/media/podcasts}:/podcasts
backend/templates/audiobookshelf/template.json
+10
View File
@@ -0,0 +1,10 @@
{
"name": "Audiobookshelf",
"description": "Audiobook and podcast server that keeps progress in sync across devices.",
"tags": [
"media",
"books",
"streaming"
],
"gpu": null
}
backend/templates/authelia/.env.example
+3
View File
@@ -0,0 +1,3 @@
TZ=Europe/Berlin
DATA_PATH=/srv/authelia
HTTP_PORT=9091
backend/templates/authelia/compose.yaml
+13
View File
@@ -0,0 +1,13 @@
services:
authelia:
image: authelia/authelia:latest
container_name: authelia
restart: unless-stopped
environment:
- TZ=${TZ:-Europe/Berlin}
ports:
- "${HTTP_PORT:-9091}:9091"
volumes:
- ./configuration.yml:/config/configuration.yml:ro
- ./users_database.yml:/config/users_database.yml
- ${DATA_PATH:-/srv/authelia}:/config/db
backend/templates/authelia/configuration.yml
+34
View File
@@ -0,0 +1,34 @@
# Minimal Authelia config. Replace every "change-me" and the example domains.
theme: dark
identity_validation:
reset_password:
jwt_secret: change-me-jwt-secret
server:
address: tcp://0.0.0.0:9091
authentication_backend:
file:
path: /config/users_database.yml
access_control:
default_policy: deny
rules:
- domain: "*.example.com"
policy: two_factor
session:
secret: change-me-session-secret
cookies:
- domain: example.com
authelia_url: https://auth.example.com
storage:
encryption_key: change-me-encryption-key-at-least-20-chars
local:
path: /config/db/db.sqlite3
notifier:
filesystem:
filename: /config/db/notification.txt
backend/templates/authelia/template.json
+10
View File
@@ -0,0 +1,10 @@
{
"name": "Authelia",
"description": "Lightweight authentication and 2FA portal for reverse-proxy forward auth.",
"tags": [
"security",
"identity",
"sso"
],
"gpu": null
}
backend/templates/authelia/users_database.yml
+10
View File
@@ -0,0 +1,10 @@
# Generate a password hash with:
# docker run --rm authelia/authelia:latest authelia crypto hash generate argon2 --password 'yourpassword'
users:
admin:
disabled: false
displayname: "Admin"
password: "$argon2id$v=19$m=65536,t=3,p=4$REPLACE_ME"
email: admin@example.com
groups:
- admins
backend/templates/authentik/.env.example
+8
View File
@@ -0,0 +1,8 @@
DATA_PATH=/srv/authentik
AUTHENTIK_TAG=2026.8.0
HTTP_PORT=9200
HTTPS_PORT=9243
# Both required — the stack refuses to start until they are set.
# Generate each with: openssl rand -base64 36
PG_PASS=
AUTHENTIK_SECRET_KEY=
backend/templates/authentik/compose.yaml
+61
View File
@@ -0,0 +1,61 @@
services:
postgresql:
image: postgres:16-alpine
container_name: authentik-db
restart: unless-stopped
environment:
- POSTGRES_DB=authentik
- POSTGRES_USER=authentik
- POSTGRES_PASSWORD=${PG_PASS:?database password required}
healthcheck:
test: ["CMD-SHELL", "pg_isready -d authentik -U authentik"]
interval: 30s
timeout: 5s
retries: 5
start_period: 20s
volumes:
- ${DATA_PATH:-/srv/authentik}/database:/var/lib/postgresql/data
server:
image: ghcr.io/goauthentik/server:${AUTHENTIK_TAG:-2026.8.0}
container_name: authentik-server
command: server
restart: unless-stopped
shm_size: 512mb
depends_on:
postgresql:
condition: service_healthy
environment:
- AUTHENTIK_SECRET_KEY=${AUTHENTIK_SECRET_KEY:?secret key required}
- AUTHENTIK_POSTGRESQL__HOST=postgresql
- AUTHENTIK_POSTGRESQL__NAME=authentik
- AUTHENTIK_POSTGRESQL__USER=authentik
- AUTHENTIK_POSTGRESQL__PASSWORD=${PG_PASS}
ports:
- "${HTTP_PORT:-9200}:9000"
- "${HTTPS_PORT:-9243}:9443"
volumes:
- ${DATA_PATH:-/srv/authentik}/data:/data
- ${DATA_PATH:-/srv/authentik}/custom-templates:/templates
worker:
image: ghcr.io/goauthentik/server:${AUTHENTIK_TAG:-2026.8.0}
container_name: authentik-worker
command: worker
restart: unless-stopped
shm_size: 512mb
user: root
depends_on:
postgresql:
condition: service_healthy
environment:
- AUTHENTIK_SECRET_KEY=${AUTHENTIK_SECRET_KEY}
- AUTHENTIK_POSTGRESQL__HOST=postgresql
- AUTHENTIK_POSTGRESQL__NAME=authentik
- AUTHENTIK_POSTGRESQL__USER=authentik
- AUTHENTIK_POSTGRESQL__PASSWORD=${PG_PASS}
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ${DATA_PATH:-/srv/authentik}/data:/data
- ${DATA_PATH:-/srv/authentik}/certs:/certs
- ${DATA_PATH:-/srv/authentik}/custom-templates:/templates
backend/templates/authentik/template.json
+10
View File
@@ -0,0 +1,10 @@
{
"name": "authentik",
"description": "Identity provider and SSO gateway (OAuth2, SAML, LDAP, forward auth). Finish setup at /if/flow/initial-setup/.",
"tags": [
"security",
"identity",
"sso"
],
"gpu": null
}
backend/templates/bazarr/.env.example
+6
View File
@@ -0,0 +1,6 @@
PUID=1000
PGID=1000
TZ=Europe/Berlin
DATA_PATH=/srv/bazarr
MEDIA_PATH=/srv/media
HTTP_PORT=6767
backend/templates/bazarr/compose.yaml
+14
View File
@@ -0,0 +1,14 @@
services:
bazarr:
image: lscr.io/linuxserver/bazarr:latest
container_name: bazarr
restart: unless-stopped
environment:
- PUID=${PUID:-1000}
- PGID=${PGID:-1000}
- TZ=${TZ:-Europe/Berlin}
ports:
- "${HTTP_PORT:-6767}:6767"
volumes:
- ${DATA_PATH:-/srv/bazarr}:/config
- ${MEDIA_PATH:-/srv/media}:/media
backend/templates/bazarr/template.json
+10
View File
@@ -0,0 +1,10 @@
{
"name": "Bazarr",
"description": "Companion to Sonarr and Radarr that downloads matching subtitles.",
"tags": [
"media",
"automation",
"arr"
],
"gpu": null
}
backend/templates/beszel/.env.example
+4
View File
@@ -0,0 +1,4 @@
DATA_PATH=/srv/beszel
HTTP_PORT=8090
AGENT_PORT=45876
AGENT_KEY=
backend/templates/beszel/compose.yaml
+22
View File
@@ -0,0 +1,22 @@
services:
beszel:
image: henrygd/beszel:latest
container_name: beszel
restart: unless-stopped
ports:
- "${HTTP_PORT:-8090}:8090"
volumes:
- ${DATA_PATH:-/srv/beszel}/data:/beszel_data
beszel-agent:
image: henrygd/beszel-agent:latest
container_name: beszel-agent
restart: unless-stopped
network_mode: host
environment:
- LISTEN=${AGENT_PORT:-45876}
# Copy this from the "add system" dialog in the Beszel UI.
- KEY=${AGENT_KEY:-}
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ${DATA_PATH:-/srv/beszel}/agent:/var/lib/beszel-agent
backend/templates/beszel/template.json
+10
View File
@@ -0,0 +1,10 @@
{
"name": "Beszel",
"description": "Lightweight server monitoring with historical charts, alerts and Docker stats.",
"tags": [
"monitoring",
"metrics",
"docker"
],
"gpu": null
}
backend/templates/bookstack/.env.example
+10
View File
@@ -0,0 +1,10 @@
PUID=1000
PGID=1000
TZ=Europe/Berlin
DATA_PATH=/srv/bookstack
HTTP_PORT=6875
APP_URL=http://localhost:6875
# Required, must be "base64:..." — generate with: echo "base64:$(openssl rand -base64 32)"
APP_KEY=
DB_PASSWORD=change-me
DB_ROOT_PASSWORD=change-me
backend/templates/bookstack/compose.yaml
+38
View File
@@ -0,0 +1,38 @@
services:
bookstack:
image: lscr.io/linuxserver/bookstack:latest
container_name: bookstack
restart: unless-stopped
depends_on:
- bookstack-db
environment:
- PUID=${PUID:-1000}
- PGID=${PGID:-1000}
- TZ=${TZ:-Europe/Berlin}
- APP_URL=${APP_URL:-http://localhost:6875}
# Required. Generate with: openssl rand -base64 32
- APP_KEY=${APP_KEY:?app key required}
- DB_HOST=bookstack-db
- DB_PORT=3306
- DB_DATABASE=bookstackapp
- DB_USERNAME=bookstack
- DB_PASSWORD=${DB_PASSWORD:-bookstack}
ports:
- "${HTTP_PORT:-6875}:80"
volumes:
- ${DATA_PATH:-/srv/bookstack}/config:/config
bookstack-db:
image: lscr.io/linuxserver/mariadb:latest
container_name: bookstack-db
restart: unless-stopped
environment:
- PUID=${PUID:-1000}
- PGID=${PGID:-1000}
- TZ=${TZ:-Europe/Berlin}
- MYSQL_ROOT_PASSWORD=${DB_ROOT_PASSWORD:-bookstack}
- MYSQL_DATABASE=bookstackapp
- MYSQL_USER=bookstack
- MYSQL_PASSWORD=${DB_PASSWORD:-bookstack}
volumes:
- ${DATA_PATH:-/srv/bookstack}/db:/config
backend/templates/bookstack/template.json
+10
View File
@@ -0,0 +1,10 @@
{
"name": "BookStack",
"description": "Documentation platform organised into shelves, books, chapters and pages. First login: admin@admin.com / password.",
"tags": [
"documents",
"wiki",
"productivity"
],
"gpu": null
}
backend/templates/caddy/.env.example
+3
View File
@@ -0,0 +1,3 @@
DATA_PATH=/srv/caddy
HTTP_PORT=80
HTTPS_PORT=443
backend/templates/caddy/Caddyfile
+11
View File
@@ -0,0 +1,11 @@
# Caddy issues and renews TLS certificates automatically for any real hostname.
# Replace the examples below with your own, then restart the stack.
app.example.com {
reverse_proxy host.docker.internal:8080
}
# Local-only site on plain HTTP (no certificate needed):
# http://nas.lan {
# reverse_proxy 192.168.1.10:5000
# }

Some files were not shown because too many files have changed in this diff Show More