Lock stacks during compose runs, cache stats, persist runtime state (0.47.0)
CI / check (push) Successful in 7m4s
CI / build-and-push (push) Successful in 1m47s

F7 — Nothing stopped two compose operations landing on the same stack. There
was a busy flag, but is_busy() was only ever read to colour the status column;
no lifecycle handler consulted it before acting. Two tabs, or auto-update
picking up a stack somebody had just clicked, both ran pull + up -d against the
same project and raced over recreating containers.

Lifecycle calls, the two deploy WebSockets and the auto-update pass now take a
real lock; a second caller gets 409 (or an error frame and close 4409) and
auto-update skips and retries next cycle. The lock is a row rather than a set
in one worker's memory, so it holds across workers and across a restart, and it
carries an expiry — a worker killed mid-deploy would otherwise strand the stack
with no fix short of editing the database.

F10 — /api/stacks/stats sampled every running container on every call, one
blocking daemon request each, and both the dashboard and the stacks list poll
it every five seconds. Two tabs on a 40-container host meant a sustained ~16
samples a second. Cached for 4s behind a lock so concurrent callers share one
sweep, the same shape dashboard_service already used for its fleet aggregate.

F11 — Three module dicts assumed exactly one uvicorn worker without saying so
and were lost on restart. The busy set is the lock above. The image update
cache is now mirrored to SQLite, so a restart shows the badges immediately
instead of blanking them for up to an hour, and the already-notified marks come
back with them rather than re-announcing the same updates. The login rate
limiter is a table, so it cannot be cleared by getting the process to restart
and no longer multiplies by the worker count.

The constraint that shaped this: compose_service and update_service are shared
with the agent, which has no database. Neither may import one. So the lock is a
separate service the central app enforces at its own entry points, and update
persistence is an opt-in callback the central app registers in its lifespan —
the agent registers nothing and behaves exactly as before. A test asserts
update_service never imports the database, since that is the kind of thing a
later change breaks silently.

Both new nets were checked by reverting the fix: dropping the lock from
_lifecycle fails six tests, removing the stats cache fails the one that names
the behaviour.

Also wires up cache pruning in the same sweep — without it both the dict and
the table grew one entry per image tag ever run, for the life of the install.

31 new tests (729 total).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
This commit is contained in:
menzelj
2026-08-31 13:43:39 +02:00
co-authored by Claude Opus 5
parent 41a21b5a25
commit 09bed274eb
16 changed files with 1053 additions and 32 deletions
+34 -1
View File
@@ -15,6 +15,24 @@ as intuitive as Dockge, as capable as Portainer for Compose workflows.
> (Auto-update) + Phase 23 (Secrets & configs) + Phase 24 (Design System v2)
> complete.
## Upgrading to 0.47.0 — nothing to do
Three pieces of state moved out of process memory and into the database, and
the live-stats endpoint got a cache. No configuration changes, no migration
steps; the new tables are created on first start.
- **Stacks can only run one compose operation at a time.** A second `start` /
`update` / `down` on a busy stack answers `409` instead of racing the first
one over the same containers. Auto-update skips a stack you are already
deploying and picks it up next cycle.
- **`GET /api/stacks/stats` is cached for four seconds.** It sampled every
running container on every call, and the dashboard and the stacks list both
poll it every five seconds — two open tabs on a 40-container host meant a
sustained ~16 daemon calls a second.
- **The update cache and the login rate limiter persist.** Both used to reset on
restart; the rate limiter also used to multiply by the worker count, so
neither behaved as documented with `--workers` set.
## Upgrading to 0.46.0 — everyone is signed out once
Sessions now hold their access token in memory and the refresh token in an
@@ -86,6 +104,10 @@ it is what your saved destination credentials are encrypted with.
stop / restart / pull / update` via `docker compose`.
- **Live status** — running / partial / stopped / error / updating, computed from
Docker container labels.
- **One operation per stack** — a lifecycle call takes a lock (a row, so it
holds across workers and across a restart) and a second one gets `409` while
it is held; auto-update skips a stack somebody is already deploying. Locks
carry an expiry, so a worker killed mid-deploy does not strand a stack.
- **Real-time logs** — streamed over WebSocket, color-coded per service.
- **Live deploy console** — deploying from the editor streams `compose up`
output (image pulls, container creation) over a WebSocket in real time instead
@@ -133,6 +155,9 @@ it is what your saved destination credentials are encrypted with.
- **Image update checker**: background task compares the local manifest digest with
the registry (Docker Hub / ghcr / lscr / private v2 with token auth); update
badges on the Images page + an "updates available" banner on the dashboard.
Results are cached in the database, so a restart shows the badges immediately
instead of blanking them until the next sweep — and does not re-announce
updates it already notified about.
- **Port conflict detector**: pre-deploy check against host-bound ports
(`/proc/net/tcp[6]`) and running container bindings, with a confirm dialog.
- **Resource limits**: CPU/memory sliders in the editor → `deploy.resources.limits`.
@@ -538,7 +563,7 @@ Same three commands the CI runs — `build-and-push` only starts once they pass.
```bash
cd backend
pip install -r requirements-dev.txt
pytest # 698 tests, no Docker daemon needed
pytest # 729 tests, no Docker daemon needed
ruff check .
cd ../frontend && npx tsc --noEmit -p tsconfig.json
```
@@ -564,6 +589,14 @@ HTTP. `tests/test_schema_migration.py` builds a database with the *old* user
table and asserts the added column is backfilled rather than left NULL, which is
what would otherwise have signed out every user on every install.
`tests/test_stack_locking.py` and `tests/test_runtime_state.py` cover the state
that moved into the database: that a busy stack answers 409 without ever
reaching Docker, that an expired lock is taken over rather than stranding the
stack, that the stats cache serves repeat callers from one sweep, and that the
update cache and rate limiter survive a restart. One of them asserts that
`update_service` never imports the database — it is shared with the agent,
which has none, so persistence has to stay opt-in.
`tests/test_bundled_templates.py` covers the 83 shipped templates: each must
parse, name an image per service, keep `.env.example` in sync with the variables
compose actually reads, ship every file it bind-mounts, and never come with a
+16
View File
@@ -38,7 +38,9 @@ from routers import (
)
from services import (
backup_destination_service,
image_status_store,
schedule_service,
stack_lock_service,
template_service,
update_service,
)
@@ -71,6 +73,20 @@ async def lifespan(app: FastAPI):
logger.info("Migrated %d custom template(s) from the database to folders", moved)
except Exception as exc: # noqa: BLE001
logger.warning("Legacy template migration failed: %s", exc)
# Runtime state that used to live in module dicts and was lost on restart.
try:
with Session(engine) as session:
stale = stack_lock_service.prune_expired(session)
if stale:
logger.info("Cleared %d stale stack lock(s) from a previous run", stale)
except Exception as exc: # noqa: BLE001
logger.warning("Could not prune stack locks: %s", exc)
try:
restored = image_status_store.install()
logger.info("Restored %d cached image update status(es)", restored)
except Exception as exc: # noqa: BLE001
logger.warning("Could not restore the image update cache: %s", exc)
update_task = asyncio.create_task(update_service.background_loop())
schedule_task = asyncio.create_task(schedule_service.scheduler_loop())
logger.info("StackPilot backend ready on port %s", settings.PORT)
+2
View File
@@ -4,6 +4,7 @@ from models.audit import AuditLog
from models.auto_update import AutoUpdate
from models.backup_destination import BackupDestination
from models.backup_schedule import BackupSchedule
from models.runtime_state import ImageStatus, LoginAttempt, StackLock
from models.setting import Setting, Webhook
from models.stack import Stack
from models.user import User
@@ -11,4 +12,5 @@ from models.user import User
__all__ = [
"User", "Stack", "AuditLog", "Setting", "Webhook", "Agent",
"BackupDestination", "BackupSchedule", "AutoUpdate",
"StackLock", "ImageStatus", "LoginAttempt",
]
+76
View File
@@ -0,0 +1,76 @@
"""Runtime state that used to live in module-level dicts.
Three things were kept in process memory: which stacks are mid-deploy, the
registry digests behind the "update available" badges, and the login rate
limiter's counters. All three assumed exactly one uvicorn worker — nothing said
so, and ``--workers 2`` would have silently given each worker its own copy —
and all three were lost on restart.
They are tables now. SQLite is already here; this needs no new dependency.
Note that ``services/compose_service.py`` and ``services/update_service.py``
are shared with the agent, which has no database at all — so these tables are
only ever touched from the central app's own routers and background loops.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Optional
from sqlmodel import Field, SQLModel
def _now() -> datetime:
return datetime.now(timezone.utc)
class StackLock(SQLModel, table=True):
"""A stack is mid-operation and must not be touched concurrently.
``docker compose`` has no locking of its own, so two simultaneous ``update``
calls — two browser tabs, or auto-update racing a manual click — would both
run ``pull`` and ``up`` against the same project and fight over recreating
containers.
``expires_at`` is what keeps a crashed worker from locking a stack forever:
an expired row is simply taken over by the next caller.
"""
stack_id: str = Field(primary_key=True)
action: str # "update", "start", "backup", …
#: Free-form owner, for the log when a lock is stolen. Not a security control.
owner: str = ""
acquired_at: datetime = Field(default_factory=_now)
expires_at: datetime
class ImageStatus(SQLModel, table=True):
"""Cached result of one image's registry digest check.
Persisted so a restart does not blank every update badge until the next
background sweep (up to an hour), and so ``notified`` survives with it —
otherwise every restart re-announced the same pending updates.
"""
image: str = Field(primary_key=True)
update_available: bool = False
current_digest: Optional[str] = None
remote_digest: Optional[str] = None
checked_at: float = 0.0
error: Optional[str] = None
#: Whether an "update available" notification already went out for this
#: image at its current state.
notified: bool = False
class LoginAttempt(SQLModel, table=True):
"""One login attempt, for the rate limiter.
In memory this reset on every restart, so an attacker could clear their own
budget by getting the process to restart — and with more than one worker the
limit multiplied by the worker count. Rows are pruned as they age out.
"""
id: Optional[int] = Field(default=None, primary_key=True)
ip: str = Field(index=True)
at: datetime = Field(default_factory=_now, index=True)
+29 -14
View File
@@ -1,14 +1,14 @@
"""Authentication routes + first-launch setup wizard."""
from __future__ import annotations
import time
from collections import defaultdict, deque
from datetime import datetime, timedelta, timezone
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
from sqlmodel import Session, select
from sqlmodel import Session, delete, select
import auth as auth_mod
from database import get_session
from models.runtime_state import LoginAttempt
from models.user import (
LoginRequest,
RefreshRequest,
@@ -23,23 +23,38 @@ from services import audit_service
router = APIRouter(prefix="/api/auth", tags=["auth"])
# Simple in-memory rate limiter for login (max 10 / minute / IP).
_LOGIN_HITS: dict[str, deque] = defaultdict(deque)
# Login rate limit: max 10 attempts per minute per client IP.
#
# Kept in the database rather than a module dict. In memory it reset on every
# restart — so an attacker could clear their own budget by getting the process
# to restart — and with more than one uvicorn worker each worker enforced its
# own limit, multiplying the real allowance by the worker count.
#
# The IP is only meaningful because uvicorn runs with --proxy-headers; without
# that every request looks like it comes from the frontend container and this
# would throttle all users together.
_RATE_LIMIT = 10
_RATE_WINDOW = 60.0
_RATE_WINDOW = timedelta(seconds=60)
#: Attempts older than this are deleted while we are in the table anyway.
_RATE_RETENTION = timedelta(hours=1)
def _check_rate_limit(ip: str) -> None:
now = time.monotonic()
hits = _LOGIN_HITS[ip]
while hits and now - hits[0] > _RATE_WINDOW:
hits.popleft()
if len(hits) >= _RATE_LIMIT:
def _check_rate_limit(session: Session, ip: str) -> None:
now = datetime.now(timezone.utc)
session.exec(delete(LoginAttempt).where(LoginAttempt.at < now - _RATE_RETENTION))
recent = session.exec(
select(LoginAttempt).where(
LoginAttempt.ip == ip, LoginAttempt.at >= now - _RATE_WINDOW
)
).all()
if len(recent) >= _RATE_LIMIT:
session.commit()
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="Too many login attempts, slow down.",
)
hits.append(now)
session.add(LoginAttempt(ip=ip, at=now))
session.commit()
#: The refresh cookie is scoped to the two endpoints that consume it, so it is
@@ -122,7 +137,7 @@ def login(
session: Session = Depends(get_session),
) -> TokenPair:
ip = request.client.host if request.client else "unknown"
_check_rate_limit(ip)
_check_rate_limit(session, ip)
user = auth_mod.authenticate(session, body.username, body.password)
if not user:
raise HTTPException(
+30 -7
View File
@@ -27,7 +27,15 @@ from models.setting import (
)
from models.auto_update import AutoUpdateRead, AutoUpdateWrite
from models.user import User
from services import audit_service, auto_update_service, compose_service, notify_service, stats_service, update_service
from services import (
audit_service,
auto_update_service,
compose_service,
notify_service,
stack_lock_service,
stats_service,
update_service,
)
from services.convert_service import convert_docker_run
router = APIRouter(prefix="/api/stacks", tags=["stacks"])
@@ -59,13 +67,17 @@ def _get_stack_or_404(session: Session, stack_id: str) -> Stack:
return stack
def _stack_summary(stack: Stack, summaries: dict | None = None) -> dict:
def _stack_summary(
stack: Stack, summaries: dict | None = None, busy: dict[str, str] | None = None
) -> dict:
"""Build a list-row summary.
Pass ``summaries`` (from :func:`compose_service.stack_status_summaries`) to
serve the whole stacks list from a single Docker call. Without it (single
Pass ``summaries`` (from :func:`compose_service.stack_status_summaries`) and
``busy`` (from :func:`stack_lock_service.active`) to serve the whole stacks
list from one Docker call and one query. Without them (single
create/update/clone responses), fall back to one direct query for this stack.
"""
busy = busy or {}
if summaries is None:
try:
containers = compose_service.containers_for_stack(stack.id)
@@ -79,7 +91,7 @@ def _stack_summary(stack: Stack, summaries: dict | None = None) -> dict:
info = summaries.get(stack.id)
total = info["total"] if info else 0
running = info["running"] if info else 0
if compose_service.is_busy(stack.id):
if stack.id in busy:
status = "updating"
else:
status = info["status"] if info else "stopped"
@@ -111,7 +123,8 @@ def list_stacks(
summaries = compose_service.stack_status_summaries()
except DockerError:
summaries = {}
return [_stack_summary(s, summaries) for s in stacks]
busy = stack_lock_service.active(session)
return [_stack_summary(s, summaries, busy) for s in stacks]
@router.post("", status_code=201)
@@ -296,7 +309,17 @@ async def _notify_lifecycle(action_name: str, stack_id: str, ok: bool, detail: s
async def _lifecycle(action_fn, action_name, stack_id, request, session, user):
_get_stack_or_404(session, stack_id)
result = await action_fn(stack_id)
# One compose operation per stack. Without this two tabs (or auto-update
# landing on a stack somebody just clicked) both run pull + up -d against
# the same project and race over recreating containers.
try:
with stack_lock_service.hold(session, stack_id, action_name, user.username):
result = await action_fn(stack_id)
except stack_lock_service.StackBusy as exc:
raise HTTPException(
status_code=409,
detail=f"Stack '{stack_id}' is busy: {exc.action} in progress",
) from exc
audit_service.record(
session, user=user.username, action=f"stack.{action_name}", target=stack_id,
detail=f"rc={result.get('returncode')}", ip=_client_ip(request),
+28
View File
@@ -22,6 +22,7 @@ from services import (
compose_service,
exec_service,
notify_service,
stack_lock_service,
update_service,
)
@@ -145,6 +146,18 @@ async def ws_deploy(
rc: int | None = None
disconnected = False
# Same guard the REST lifecycle uses — the deploy console runs the very
# same `compose up`, so it has to queue behind an in-flight operation
# rather than race it.
lock_session = Session(engine)
try:
stack_lock_service.acquire(lock_session, stack_id, "start", username)
except stack_lock_service.StackBusy as exc:
lock_session.close()
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
await websocket.close(code=4409)
return
compose_service.mark_busy(stack_id)
try:
async for kind, payload in compose_service.stream_up(stack_id):
@@ -162,6 +175,9 @@ async def ws_deploy(
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
finally:
compose_service.clear_busy(stack_id)
with contextlib.suppress(Exception):
stack_lock_service.release(lock_session, stack_id)
lock_session.close()
ok = rc in (0, None)
try:
@@ -275,6 +291,15 @@ async def ws_update(
rc: int | None = None
disconnected = False
lock_session = Session(engine)
try:
stack_lock_service.acquire(lock_session, stack_id, "update", username)
except stack_lock_service.StackBusy as exc:
lock_session.close()
with contextlib.suppress(Exception):
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
await websocket.close(code=4409)
return
compose_service.mark_busy(stack_id)
try:
async for kind, payload in compose_service.stream_update(stack_id):
@@ -291,6 +316,9 @@ async def ws_update(
await websocket.send_text(json.dumps({"type": "error", "detail": str(exc)}))
finally:
compose_service.clear_busy(stack_id)
with contextlib.suppress(Exception):
stack_lock_service.release(lock_session, stack_id)
lock_session.close()
ok = rc in (0, None)
try:
+16 -3
View File
@@ -20,7 +20,13 @@ from database import engine
from models.agent import Agent
from models.auto_update import AutoUpdate
from models.setting import EVENT_PULL_FAILED, EVENT_STACK_AUTO_UPDATED
from services import agent_service, compose_service, notify_service, update_service
from services import (
agent_service,
compose_service,
notify_service,
stack_lock_service,
update_service,
)
logger = logging.getLogger("stackpilot.autoupdate")
@@ -104,8 +110,15 @@ async def _run_local(session: Session, policy: AutoUpdate) -> None:
prev = policy.last_status
if policy.redeploy:
try:
await compose_service.pull(stack_id)
await compose_service.up(stack_id)
# Never redeploy underneath somebody: if a user is mid-deploy on
# this stack, skip and pick it up next cycle rather than racing
# them over the same containers.
with stack_lock_service.hold(session, stack_id, "auto-update", "auto-update"):
await compose_service.pull(stack_id)
await compose_service.up(stack_id)
except stack_lock_service.StackBusy as exc:
_record(session, policy, "skipped", str(exc))
return
except Exception as exc: # noqa: BLE001
_record(session, policy, "error", str(exc))
await _safe_notify(EVENT_PULL_FAILED, f"Auto-update of '{stack_id}' failed", str(exc), session)
+80
View File
@@ -0,0 +1,80 @@
"""Persistence for the image update cache.
``update_service`` holds the registry-digest results in a module dict because
it is shared with the agent, which has no database. This module is the central
app's half: it seeds that dict at startup and mirrors every write back into
SQLite, wired up in ``main.lifespan``.
What it buys: after a restart the update badges are there immediately instead
of blank until the next background sweep (up to an hour), and the "already
notified" marks come back with them, so a restart no longer re-announces
updates the user has already seen.
"""
from __future__ import annotations
import logging
from sqlmodel import Session, select
from database import engine
from models.runtime_state import ImageStatus
from services import update_service
logger = logging.getLogger("stackpilot.image_status")
def _to_status(row: ImageStatus) -> update_service.UpdateStatus:
return update_service.UpdateStatus(
image=row.image,
update_available=row.update_available,
current_digest=row.current_digest,
remote_digest=row.remote_digest,
checked_at=row.checked_at,
error=row.error,
)
def save(status: update_service.UpdateStatus, notified: bool) -> None:
"""Upsert one image's status. Opens its own session — the caller is the
background loop, which has none."""
with Session(engine) as session:
row = session.get(ImageStatus, status.image)
if row is None:
row = ImageStatus(image=status.image)
row.update_available = status.update_available
row.current_digest = status.current_digest
row.remote_digest = status.remote_digest
row.checked_at = status.checked_at
row.error = status.error
row.notified = notified
session.add(row)
session.commit()
def install() -> int:
"""Seed the in-memory cache from the database and start mirroring writes.
Returns how many entries were restored.
"""
with Session(engine) as session:
rows = session.exec(select(ImageStatus)).all()
update_service.restore_cache([(_to_status(r), r.notified) for r in rows])
update_service.set_persist_callback(save, prune)
return len(rows)
def prune(keep: set[str]) -> int:
"""Drop rows for images that are no longer used by any stack.
Without this the table grows for the life of the install, one row per image
tag that was ever running.
"""
removed = 0
with Session(engine) as session:
for row in session.exec(select(ImageStatus)).all():
if row.image not in keep:
session.delete(row)
removed += 1
if removed:
session.commit()
return removed
+148
View File
@@ -0,0 +1,148 @@
"""One compose operation per stack at a time.
``docker compose`` does no locking. Two ``update`` calls against the same
project — two open browser tabs, or the auto-update pass landing on a stack
somebody just clicked — both run ``pull`` and then ``up -d``, and race each
other recreating the same containers.
There *was* a busy flag (``compose_service._BUSY``), but it only ever fed the
status column: no lifecycle handler consulted it before acting. This module is
the actual guard, and it lives in the database so it holds across workers and
across a restart.
``compose_service`` keeps its in-process set because it is shared with the
agent, which has no database. The agent is a single process managing one host,
and the central app holds this lock before calling it, so the two do not
conflict.
"""
from __future__ import annotations
import logging
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from typing import Optional
from sqlalchemy.exc import IntegrityError
from sqlmodel import Session, delete, select
from models.runtime_state import StackLock
logger = logging.getLogger("stackpilot.stack_lock")
#: Long enough to outlast the slowest legitimate operation (compose commands
#: time out at 600s, a full pull of a large stack can chain several), short
#: enough that a lock orphaned by a killed worker clears itself within an hour.
DEFAULT_TTL = timedelta(minutes=30)
class StackBusy(Exception):
"""The stack is already running an operation."""
def __init__(self, stack_id: str, action: str):
self.stack_id = stack_id
self.action = action
super().__init__(f"Stack '{stack_id}' is busy: {action} in progress")
def _now() -> datetime:
return datetime.now(timezone.utc)
def _aware(value: Optional[datetime]) -> Optional[datetime]:
"""SQLite hands datetimes back naive; compare them as UTC."""
if value is not None and value.tzinfo is None:
return value.replace(tzinfo=timezone.utc)
return value
def acquire(
session: Session,
stack_id: str,
action: str,
owner: str = "",
ttl: timedelta = DEFAULT_TTL,
) -> None:
"""Take the lock for ``stack_id`` or raise :class:`StackBusy`.
An expired lock is taken over — that is the recovery path for a worker that
died mid-deploy, which would otherwise leave the stack unusable.
"""
now = _now()
existing = session.get(StackLock, stack_id)
if existing is not None:
if (_aware(existing.expires_at) or now) > now:
raise StackBusy(stack_id, existing.action)
logger.warning(
"Taking over an expired %s lock on '%s' (held by %r since %s)",
existing.action, stack_id, existing.owner, existing.acquired_at,
)
session.delete(existing)
session.commit()
session.add(
StackLock(
stack_id=stack_id,
action=action,
owner=owner,
acquired_at=now,
expires_at=now + ttl,
)
)
try:
session.commit()
except IntegrityError as exc:
# Another worker inserted between our check and our commit. The primary
# key is what actually makes this safe; the read above is only there to
# give a useful error and to clear stale rows.
session.rollback()
raise StackBusy(stack_id, action) from exc
def release(session: Session, stack_id: str) -> None:
"""Drop the lock. Safe to call when it is not held."""
existing = session.get(StackLock, stack_id)
if existing is not None:
session.delete(existing)
session.commit()
@contextmanager
def hold(session: Session, stack_id: str, action: str, owner: str = ""):
"""Hold the lock for the duration of the block.
Raises :class:`StackBusy` if somebody else has it. Always releases, so a
failed deploy does not leave the stack locked.
"""
acquire(session, stack_id, action, owner)
try:
yield
finally:
try:
release(session, stack_id)
except Exception: # noqa: BLE001 - never mask the original error
logger.exception("Failed to release the lock on '%s'", stack_id)
def active(session: Session) -> dict[str, str]:
"""``{stack_id: action}`` for every lock still in force.
One query for the whole stacks list, rather than a lookup per row.
"""
now = _now()
return {
lock.stack_id: lock.action
for lock in session.exec(select(StackLock)).all()
if (_aware(lock.expires_at) or now) > now
}
def is_busy(session: Session, stack_id: str) -> bool:
lock = session.get(StackLock, stack_id)
return lock is not None and (_aware(lock.expires_at) or _now()) > _now()
def prune_expired(session: Session) -> int:
"""Drop locks that have timed out. Called at startup and by the scheduler."""
result = session.exec(delete(StackLock).where(StackLock.expires_at < _now()))
session.commit()
return result.rowcount or 0
+38 -2
View File
@@ -3,15 +3,33 @@
Reads a one-shot ``docker stats`` sample per running container (the daemon
includes ``precpu_stats`` so a single read yields a usable CPU delta) and sums
them by ``com.docker.compose.project`` label, which equals the stack id.
Sampling is not free: it is one blocking call to the daemon *per running
container*, and the dashboard and the stacks list both poll this every five
seconds. Two open tabs on a 40-container host meant a sustained ~16 samples a
second. Results are therefore cached for :data:`CACHE_TTL`, the same shape
``dashboard_service`` already uses for its fleet aggregate — one sweep serves
every reader in the window, and the numbers stay well inside what a
five-second poll can show.
"""
from __future__ import annotations
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from docker_client import DockerError, get_client, safe_call
COMPOSE_PROJECT_LABEL = "com.docker.compose.project"
#: Slightly under the frontend's 5s poll, so a refresh usually gets fresh
#: numbers while concurrent readers still share one sweep.
CACHE_TTL = 4.0
_cache: dict = {"data": None, "ts": 0.0}
# Held across the sample so N simultaneous callers trigger one sweep, not N.
_lock = threading.Lock()
def _container_stats(container) -> dict | None:
try:
@@ -60,12 +78,30 @@ def _container_stats(container) -> dict | None:
}
def stack_stats() -> dict:
def stack_stats(refresh: bool = False) -> dict:
"""Return {stack_id: {cpu_used, cpu_limit, mem_used, mem_limit, containers}}.
Limits are the summed assigned limits across the stack's containers, or null
when none of them have that limit set.
when none of them have that limit set. Served from a short-lived cache
unless ``refresh`` is set.
"""
if not refresh and _cache["data"] is not None:
if time.monotonic() - _cache["ts"] < CACHE_TTL:
return _cache["data"]
with _lock:
# Somebody may have refreshed it while we waited for the lock.
if not refresh and _cache["data"] is not None:
if time.monotonic() - _cache["ts"] < CACHE_TTL:
return _cache["data"]
data = _sample()
_cache["data"] = data
_cache["ts"] = time.monotonic()
return data
def _sample() -> dict:
"""One full sweep across every running container."""
try:
client = get_client()
containers = safe_call(client.containers.list) # running only
+64 -3
View File
@@ -10,6 +10,7 @@ import asyncio
import logging
import time
from dataclasses import asdict, dataclass
from collections.abc import Callable
from typing import Optional
import httpx
@@ -50,6 +51,49 @@ _CACHE: dict[str, UpdateStatus] = {}
# background loop doesn't re-notify on every cycle.
_NOTIFIED: set[str] = set()
#: Optional sink for cache writes.
#:
#: This module is shared with the agent, which has no database, so persistence
#: cannot live here. The central app registers a callback that mirrors each
#: entry into SQLite (see ``services/image_status_store.py``) and seeds the
#: cache from it at startup; the agent registers nothing and behaves exactly as
#: before. Without it a restart blanked every update badge until the next
#: background sweep — up to an hour — and re-announced updates it had already
#: notified about.
_persist_cb: Optional[Callable[[UpdateStatus, bool], None]] = None
#: Optional sink for "these images are still in use", same opt-in shape as
#: _persist_cb. Keeps both the dict and the table from growing one entry per
#: image tag that was ever running, for the life of the install.
_prune_cb: Optional[Callable[[set], int]] = None
def set_persist_callback(
callback: Optional[Callable[[UpdateStatus, bool], None]],
prune: Optional[Callable[[set], int]] = None,
) -> None:
global _persist_cb, _prune_cb
_persist_cb = callback
_prune_cb = prune
def restore_cache(entries: list[tuple[UpdateStatus, bool]]) -> None:
"""Seed the in-memory cache from persisted rows at startup."""
for status, notified in entries:
_CACHE[status.image] = status
if notified:
_NOTIFIED.add(status.image)
def _store(status: UpdateStatus, notified: bool) -> None:
_CACHE[status.image] = status
if _persist_cb is not None:
try:
_persist_cb(status, notified)
except Exception as exc: # noqa: BLE001 - persistence is best-effort
logger.debug("Could not persist update status for %s: %s", status.image, exc)
# --------------------------------------------------------------------------- #
# Image reference parsing
@@ -168,9 +212,11 @@ async def check_image(image: str) -> UpdateStatus:
checked_at=time.time(),
error=error,
)
_CACHE[image] = status
if update_available and image not in _NOTIFIED:
# Marked before the attempt, not after: a notifier that is down should
# not make every cycle re-announce the same update.
_NOTIFIED.add(image)
_store(status, True)
try:
await notify_service.notify(
EVENT_UPDATE_AVAILABLE,
@@ -179,8 +225,10 @@ async def check_image(image: str) -> UpdateStatus:
)
except Exception as exc: # noqa: BLE001 - notifications are best-effort
logger.debug("update notify failed for %s: %s", image, exc)
elif not update_available:
_NOTIFIED.discard(image)
else:
if not update_available:
_NOTIFIED.discard(image)
_store(status, image in _NOTIFIED)
return status
@@ -291,9 +339,22 @@ async def check_all() -> dict[str, dict]:
images = _all_running_images()
for image in images:
await check_image(image)
_forget_unused(set(images))
return {k: v.to_dict() for k, v in _CACHE.items()}
def _forget_unused(keep: set) -> None:
"""Drop images no running container references any more."""
for image in [i for i in _CACHE if i not in keep]:
del _CACHE[image]
_NOTIFIED.discard(image)
if _prune_cb is not None:
try:
_prune_cb(keep)
except Exception as exc: # noqa: BLE001 - housekeeping is best-effort
logger.debug("Could not prune persisted update statuses: %s", exc)
def get_cache() -> dict[str, dict]:
return {k: v.to_dict() for k, v in _CACHE.items()}
+284
View File
@@ -0,0 +1,284 @@
"""State that used to live in module dicts (F10, F11).
Three things were kept in process memory, all of them assuming exactly one
uvicorn worker without saying so, and all of them lost on restart:
* the live stats sweep, which was not cached at all and got polled every five
seconds by two pages at once;
* the registry digests behind the update badges, so a restart blanked every
badge for up to an hour and re-announced updates already notified about;
* the login rate limiter, which an attacker could reset by getting the process
to restart, and which multiplied by the worker count.
"""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
import pytest
from sqlmodel import Session, delete, select
# --------------------------------------------------------------------------- #
# F10 — the stats cache
# --------------------------------------------------------------------------- #
@pytest.fixture
def stats(monkeypatch):
from services import stats_service
stats_service._cache["data"] = None
stats_service._cache["ts"] = 0.0
return stats_service
def test_repeat_calls_inside_the_window_reuse_one_sweep(stats, monkeypatch):
"""The dashboard and the stacks list both poll this every 5s.
Each sweep is one blocking daemon call per running container, so serving
them from one sample is the whole point.
"""
calls = []
monkeypatch.setattr(stats, "_sample", lambda: calls.append(1) or {"a": {}})
assert stats.stack_stats() == {"a": {}}
assert stats.stack_stats() == {"a": {}}
assert stats.stack_stats() == {"a": {}}
assert len(calls) == 1
def test_the_cache_expires(stats, monkeypatch):
calls = []
monkeypatch.setattr(stats, "_sample", lambda: calls.append(1) or {})
stats.stack_stats()
# Pretend the window has passed rather than sleeping through it.
stats._cache["ts"] -= stats.CACHE_TTL + 1
stats.stack_stats()
assert len(calls) == 2
def test_refresh_bypasses_the_cache(stats, monkeypatch):
calls = []
monkeypatch.setattr(stats, "_sample", lambda: calls.append(1) or {})
stats.stack_stats()
stats.stack_stats(refresh=True)
assert len(calls) == 2
def test_a_failing_sample_is_not_cached_as_a_success(stats, monkeypatch):
"""A Docker outage returns {} — it must not stick for the whole window
once the daemon is back."""
monkeypatch.setattr(stats, "_sample", dict)
assert stats.stack_stats() == {}
monkeypatch.setattr(stats, "_sample", lambda: {"a": {"containers": 1}})
assert stats.stack_stats(refresh=True) == {"a": {"containers": 1}}
# --------------------------------------------------------------------------- #
# F11 — the image update cache survives a restart
# --------------------------------------------------------------------------- #
@pytest.fixture
def clean_image_status(db):
from database import engine
from models.runtime_state import ImageStatus
from services import update_service
with Session(engine) as session:
session.exec(delete(ImageStatus))
session.commit()
update_service._CACHE.clear()
update_service._NOTIFIED.clear()
update_service.set_persist_callback(None)
yield
update_service.set_persist_callback(None)
update_service._CACHE.clear()
update_service._NOTIFIED.clear()
def test_a_saved_status_comes_back_after_a_restart(clean_image_status):
"""The badge is there immediately instead of blank until the next sweep."""
from services import image_status_store, update_service
status = update_service.UpdateStatus(
image="nginx:latest",
update_available=True,
current_digest="sha256:old",
remote_digest="sha256:new",
checked_at=123.0,
)
image_status_store.save(status, notified=True)
# Simulate a restart: memory is empty, then install() seeds it.
update_service._CACHE.clear()
update_service._NOTIFIED.clear()
assert image_status_store.install() == 1
cached = update_service.get_cache()["nginx:latest"]
assert cached["update_available"] is True
assert cached["remote_digest"] == "sha256:new"
# And the "already told the user" mark comes back with it, so the restart
# does not re-announce the same update.
assert "nginx:latest" in update_service._NOTIFIED
def test_installing_starts_mirroring_writes(clean_image_status):
from database import engine
from models.runtime_state import ImageStatus
from services import image_status_store, update_service
image_status_store.install()
update_service._store(
update_service.UpdateStatus(
image="redis:7", update_available=False, current_digest="sha256:a",
remote_digest="sha256:a", checked_at=1.0,
),
notified=False,
)
with Session(engine) as session:
row = session.get(ImageStatus, "redis:7")
assert row is not None and row.update_available is False
def test_saving_the_same_image_twice_updates_it(clean_image_status):
from database import engine
from models.runtime_state import ImageStatus
from services import image_status_store, update_service
def status(remote):
return update_service.UpdateStatus(
image="app:1", update_available=True, current_digest="sha256:x",
remote_digest=remote, checked_at=1.0,
)
image_status_store.save(status("sha256:one"), notified=False)
image_status_store.save(status("sha256:two"), notified=True)
with Session(engine) as session:
rows = session.exec(select(ImageStatus).where(ImageStatus.image == "app:1")).all()
assert len(rows) == 1
assert rows[0].remote_digest == "sha256:two"
assert rows[0].notified is True
def test_pruning_drops_images_no_stack_uses_any_more(clean_image_status):
from database import engine
from models.runtime_state import ImageStatus
from services import image_status_store, update_service
for image in ("kept:1", "gone:1"):
image_status_store.save(
update_service.UpdateStatus(
image=image, update_available=False, current_digest=None,
remote_digest=None, checked_at=0.0,
),
notified=False,
)
assert image_status_store.prune(keep={"kept:1"}) == 1
with Session(engine) as session:
assert [r.image for r in session.exec(select(ImageStatus)).all()] == ["kept:1"]
def test_the_agent_has_no_persistence_wired_up(clean_image_status):
"""``update_service`` is shared with the agent, which has no database.
Persistence therefore has to be opt-in, registered by the central app —
if this module ever imports the database directly, the agent breaks.
"""
import inspect
from services import update_service
source = inspect.getsource(update_service)
assert "from database import" not in source
assert "import database" not in source
assert update_service._persist_cb is None
# --------------------------------------------------------------------------- #
# F11 — the login rate limiter
# --------------------------------------------------------------------------- #
@pytest.fixture
def clean_attempts(db):
from database import engine
from models.runtime_state import LoginAttempt
with Session(engine) as session:
session.exec(delete(LoginAttempt))
session.commit()
yield
def test_the_eleventh_attempt_in_a_minute_is_refused(client, clean_attempts):
body = {"username": "nobody", "password": "wrong"}
for _ in range(10):
assert client.post("/api/auth/login", json=body).status_code == 401
assert client.post("/api/auth/login", json=body).status_code == 429
def test_the_limit_survives_a_restart(db, clean_attempts):
"""In memory this reset on every restart, so an attacker could clear their
own budget by getting the process to fall over."""
from database import engine
from fastapi import HTTPException
from routers import auth as auth_router
with Session(engine) as session:
for _ in range(10):
auth_router._check_rate_limit(session, "10.0.0.9")
# A fresh session stands in for a fresh process — the counter is a table.
with Session(engine) as session:
with pytest.raises(HTTPException) as excinfo:
auth_router._check_rate_limit(session, "10.0.0.9")
assert excinfo.value.status_code == 429
def test_different_clients_have_separate_budgets(db, clean_attempts):
"""Only meaningful because uvicorn runs with --proxy-headers; without it
every request carries the frontend container's IP and one client would
throttle everyone."""
from database import engine
from fastapi import HTTPException
from routers import auth as auth_router
with Session(engine) as session:
for _ in range(10):
auth_router._check_rate_limit(session, "10.0.0.1")
auth_router._check_rate_limit(session, "10.0.0.2") # must not raise
with pytest.raises(HTTPException):
auth_router._check_rate_limit(session, "10.0.0.1")
def test_attempts_age_out_of_the_window(db, clean_attempts):
from database import engine
from models.runtime_state import LoginAttempt
from routers import auth as auth_router
old = datetime.now(timezone.utc) - timedelta(minutes=5)
with Session(engine) as session:
for _ in range(10):
session.add(LoginAttempt(ip="10.0.0.3", at=old))
session.commit()
auth_router._check_rate_limit(session, "10.0.0.3") # window has moved on
def test_stale_rows_are_pruned(db, clean_attempts):
from database import engine
from models.runtime_state import LoginAttempt
from routers import auth as auth_router
ancient = datetime.now(timezone.utc) - timedelta(days=2)
with Session(engine) as session:
session.add(LoginAttempt(ip="10.0.0.4", at=ancient))
session.commit()
auth_router._check_rate_limit(session, "10.0.0.5")
remaining = session.exec(select(LoginAttempt)).all()
assert all(r.ip != "10.0.0.4" for r in remaining), "the table must not grow forever"
+206
View File
@@ -0,0 +1,206 @@
"""One compose operation per stack (F7), and the lock surviving a restart (F11).
``docker compose`` does no locking of its own. Before 0.47.0 nothing stopped two
``update`` calls landing on the same project at once — two browser tabs, or the
auto-update pass picking up a stack somebody had just clicked — and both would
run ``pull`` then ``up -d`` and fight over recreating the same containers.
There was a busy flag, but it only fed the status column: no handler consulted
it before acting. The lock is a table now, so it also holds across workers and
across a restart.
"""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
import pytest
from sqlmodel import Session
@pytest.fixture
def lock_session(db):
from database import engine
with Session(engine) as session:
yield session
@pytest.fixture(autouse=True)
def clean_locks(db):
"""Locks are global state; don't let one test leak into the next."""
from database import engine
from models.runtime_state import StackLock
from sqlmodel import delete
with Session(engine) as session:
session.exec(delete(StackLock))
session.commit()
yield
@pytest.fixture
def svc():
from services import stack_lock_service
return stack_lock_service
# --------------------------------------------------------------------------- #
# The lock itself
# --------------------------------------------------------------------------- #
def test_a_second_caller_is_refused(svc, lock_session):
svc.acquire(lock_session, "demo", "update", "alice")
with pytest.raises(svc.StackBusy) as excinfo:
svc.acquire(lock_session, "demo", "start", "bob")
assert excinfo.value.action == "update", "the error names the operation in flight"
def test_releasing_frees_it(svc, lock_session):
svc.acquire(lock_session, "demo", "update")
svc.release(lock_session, "demo")
svc.acquire(lock_session, "demo", "start") # must not raise
def test_releasing_a_lock_nobody_holds_is_harmless(svc, lock_session):
svc.release(lock_session, "never-locked")
def test_different_stacks_do_not_block_each_other(svc, lock_session):
svc.acquire(lock_session, "one", "update")
svc.acquire(lock_session, "two", "update")
def test_hold_releases_even_when_the_operation_fails(svc, lock_session):
"""A failed deploy must not leave the stack locked."""
with pytest.raises(RuntimeError):
with svc.hold(lock_session, "demo", "update"):
raise RuntimeError("compose blew up")
assert not svc.is_busy(lock_session, "demo")
def test_an_expired_lock_is_taken_over(svc, lock_session):
"""The recovery path for a worker killed mid-deploy.
Without it the stack would stay locked forever and the only fix would be
editing the database by hand.
"""
from models.runtime_state import StackLock
past = datetime.now(timezone.utc) - timedelta(minutes=5)
lock_session.add(
StackLock(
stack_id="orphaned",
action="update",
owner="a worker that died",
acquired_at=past - timedelta(minutes=30),
expires_at=past,
)
)
lock_session.commit()
assert not svc.is_busy(lock_session, "orphaned")
svc.acquire(lock_session, "orphaned", "start") # must not raise
def test_active_lists_current_locks_and_skips_expired(svc, lock_session):
from models.runtime_state import StackLock
svc.acquire(lock_session, "live", "update")
lock_session.add(
StackLock(
stack_id="dead",
action="start",
expires_at=datetime.now(timezone.utc) - timedelta(seconds=1),
)
)
lock_session.commit()
assert svc.active(lock_session) == {"live": "update"}
def test_prune_removes_only_expired_locks(svc, lock_session):
from models.runtime_state import StackLock
svc.acquire(lock_session, "live", "update")
lock_session.add(
StackLock(
stack_id="dead",
action="start",
expires_at=datetime.now(timezone.utc) - timedelta(seconds=1),
)
)
lock_session.commit()
assert svc.prune_expired(lock_session) == 1
assert svc.active(lock_session) == {"live": "update"}
def test_the_lock_outlives_the_session_that_took_it(svc, db):
"""It is a row, not a set in one worker's memory — that is the point."""
from database import engine
with Session(engine) as first:
svc.acquire(first, "persisted", "update")
with Session(engine) as second:
assert svc.is_busy(second, "persisted")
with pytest.raises(svc.StackBusy):
svc.acquire(second, "persisted", "start")
# --------------------------------------------------------------------------- #
# Enforcement at the HTTP entry point
# --------------------------------------------------------------------------- #
@pytest.fixture
def locked_stack(db):
"""A registered stack that is already mid-operation."""
from database import engine
from models.stack import Stack
from services import compose_service, stack_lock_service
stack_id = "lock-fixture"
compose_service.write_compose(stack_id, "services:\n a:\n image: alpine\n")
with Session(engine) as session:
if not session.get(Stack, stack_id):
session.add(Stack(id=stack_id, name="lock fixture"))
session.commit()
stack_lock_service.acquire(session, stack_id, "update", "someone-else")
return stack_id
@pytest.mark.parametrize("action", ["start", "stop", "restart", "pull", "update", "down"])
def test_lifecycle_calls_are_refused_while_a_stack_is_busy(as_admin, locked_stack, action):
"""409, and crucially *without* reaching Docker — the guard is in front."""
response = as_admin.post(f"/api/stacks/{locked_stack}/{action}")
assert response.status_code == 409
assert "busy" in response.json()["detail"].lower()
assert "update in progress" in response.json()["detail"]
def test_the_stacks_list_shows_a_busy_stack_as_updating(as_admin, locked_stack):
rows = {row["id"]: row for row in as_admin.get("/api/stacks").json()}
assert rows[locked_stack]["status"] == "updating"
def test_an_unlocked_stack_is_not_refused(as_admin, db):
"""The guard must not block ordinary use.
Without a Docker daemon the call fails further in — anything but 409 means
it got past the lock, which is what this asserts.
"""
from database import engine
from models.stack import Stack
from services import compose_service
stack_id = "unlocked-fixture"
compose_service.write_compose(stack_id, "services:\n a:\n image: alpine\n")
with Session(engine) as session:
if not session.get(Stack, stack_id):
session.add(Stack(id=stack_id, name="unlocked fixture"))
session.commit()
assert as_admin.post(f"/api/stacks/{stack_id}/start").status_code != 409
+1 -1
View File
@@ -1,3 +1,3 @@
"""Single source of truth for the StackPilot release version."""
APP_VERSION = "0.46.0"
APP_VERSION = "0.47.0"
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "stackpilot-frontend",
"private": true,
"version": "0.46.0",
"version": "0.47.0",
"type": "module",
"scripts": {
"dev": "vite",