Remove the remote-host (agent) integration (0.48.0)
StackPilot now manages exactly one Docker host: the one it runs on. The
stackpilot-agent sidecar and everything that proxied to it are gone — 4721
lines deleted against 657 added.
Deleted outright: agent/ (image, compose, env), agent_app.py, models/agent.py,
routers/agents.py (1200 lines), services/agent_service.py, the agent API client,
RemoteStackDetail, the host components and AgentStacksSection. That removes 57
API routes and the three /ws/agent-* proxies.
Threaded out everywhere else, which was the bulk of the work. Every API module
carried an optional agentId that switched the base path; every page that listed
Docker objects rendered one section per host behind a HostHeader; Files had a
host switcher; the New Stack editor and the template dialog had host selectors;
schedules, auto-update policies and stack summaries carried agent_id. All of it
is gone, and the typechecker drove the sweep — 85 files touched, tsc and the
build clean.
Two things the removal exposed as dead weight rather than merely unused:
compose_service kept an in-process busy set purely because the agent needed a
lock and has no database. With the agent gone that was a second source of truth
next to the real DB lock, so it is deleted; compute_status now reports only what
the containers say and the two callers that want "updating" overlay the lock.
StacksTable's linkBase prop only ever existed to point at /hosts/{id}/stacks.
The dashboard's "Hosts 1/1 online" KPI can no longer say anything else, so the
tile and the KPIs behind it are gone and the row is five wide.
Upgrading matters here. An existing install still has an agent table holding
each remote host's URL and bearer token — full Docker control of that host,
sitting in the database with nothing left to use it. _drop_removed_schema drops
it on first start, and drops the agent_id columns where the SQLite build
supports DROP COLUMN. Each statement runs in its own transaction on purpose: a
failed DDL poisons the transaction it is in, so sharing one would let an
unsupported column drop take the table drop down with it. test_agent_removal
covers both branches plus the fresh-install and idempotent cases, and an
end-to-end run against a seeded pre-0.48 database confirms the table is gone and
every /api/agents route answers 404.
Docstrings that justified a design by "shared with the agent, which has no
database" were rewritten rather than left lying: update_service's persistence
callback and image_status_store are still the right split (registry logic stays
testable without a database), but for that reason now, not the old one. The
README's multi-host sections are removed and an upgrade note explains what to do
with running agent containers; ROADMAP keeps its history behind a note saying
the feature it describes no longer exists.
CI no longer builds or pushes stackpilot-agent.
735 tests pass, ruff and tsc clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dk43rmEeRfYi5wsLDbmfyG
This commit is contained in:
@@ -1,211 +0,0 @@
|
||||
"""Talk to remote stackpilot-agent hosts over HTTP.
|
||||
|
||||
The central app stores an ``Agent`` row per remote host and proxies stack /
|
||||
system calls to it using the agent's shared token. Connectivity state
|
||||
(``status``, ``hostname``, ``last_seen``) is refreshed on every successful or
|
||||
failed call so the UI can show a live dot per host.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Optional
|
||||
|
||||
import httpx
|
||||
from sqlmodel import Session
|
||||
|
||||
from models.agent import Agent
|
||||
|
||||
logger = logging.getLogger("stackpilot.agent_proxy")
|
||||
|
||||
_TIMEOUT = 30.0
|
||||
|
||||
|
||||
class AgentError(Exception):
|
||||
def __init__(self, status: int, error: str, detail: str = ""):
|
||||
self.status = status
|
||||
self.error = error
|
||||
self.detail = detail
|
||||
super().__init__(f"{error}: {detail}" if detail else error)
|
||||
|
||||
|
||||
def _now() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _mark(session: Session, agent: Agent, status: str, hostname: Optional[str] = None) -> None:
|
||||
agent.status = status
|
||||
if status == "online":
|
||||
agent.last_seen = _now()
|
||||
if hostname:
|
||||
agent.hostname = hostname
|
||||
session.add(agent)
|
||||
session.commit()
|
||||
session.refresh(agent)
|
||||
|
||||
|
||||
async def _request(
|
||||
agent: Agent,
|
||||
method: str,
|
||||
path: str,
|
||||
*,
|
||||
params: Optional[dict] = None,
|
||||
json: Any = None,
|
||||
) -> httpx.Response:
|
||||
url = agent.url.rstrip("/") + path
|
||||
headers = {"Authorization": f"Bearer {agent.token}"}
|
||||
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||
return await client.request(
|
||||
method, url, headers=headers, params=params, json=json, timeout=_TIMEOUT
|
||||
)
|
||||
|
||||
|
||||
async def call(
|
||||
session: Session,
|
||||
agent: Agent,
|
||||
method: str,
|
||||
path: str,
|
||||
*,
|
||||
params: Optional[dict] = None,
|
||||
json: Any = None,
|
||||
) -> Any:
|
||||
"""Proxy a request to the agent, updating its status, returning parsed JSON."""
|
||||
try:
|
||||
resp = await _request(agent, method, path, params=params, json=json)
|
||||
except httpx.HTTPError as exc:
|
||||
_mark(session, agent, "offline")
|
||||
raise AgentError(502, "agent_unreachable", str(exc)) from exc
|
||||
|
||||
if resp.status_code in (401, 403):
|
||||
_mark(session, agent, "unauthorized")
|
||||
raise AgentError(resp.status_code, "agent_unauthorized", "Invalid agent token")
|
||||
|
||||
_mark(session, agent, "online")
|
||||
|
||||
if resp.status_code >= 400:
|
||||
detail = ""
|
||||
try:
|
||||
body = resp.json()
|
||||
detail = body.get("detail") if isinstance(body, dict) else str(body)
|
||||
if isinstance(detail, dict):
|
||||
detail = detail.get("detail") or detail.get("error") or str(detail)
|
||||
except ValueError:
|
||||
detail = resp.text[:500]
|
||||
raise AgentError(resp.status_code, "agent_error", str(detail))
|
||||
|
||||
if resp.content:
|
||||
try:
|
||||
return resp.json()
|
||||
except ValueError:
|
||||
return resp.text
|
||||
return None
|
||||
|
||||
|
||||
def _handle_status(session: Session, agent: Agent, status_code: int, body_text: str = "") -> None:
|
||||
"""Update agent status from a response code; raise AgentError on failure."""
|
||||
if status_code in (401, 403):
|
||||
_mark(session, agent, "unauthorized")
|
||||
raise AgentError(status_code, "agent_unauthorized", "Invalid agent token")
|
||||
_mark(session, agent, "online")
|
||||
if status_code >= 400:
|
||||
raise AgentError(status_code, "agent_error", body_text[:500])
|
||||
|
||||
|
||||
async def download_to_file(
|
||||
session: Session,
|
||||
agent: Agent,
|
||||
path: str,
|
||||
dest_path: str,
|
||||
*,
|
||||
params: Optional[dict] = None,
|
||||
) -> None:
|
||||
"""Stream a GET from the agent into ``dest_path``."""
|
||||
url = agent.url.rstrip("/") + path
|
||||
headers = {"Authorization": f"Bearer {agent.token}"}
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||
async with client.stream("GET", url, headers=headers, params=params, timeout=None) as resp:
|
||||
if resp.status_code >= 400:
|
||||
text = (await resp.aread()).decode("utf-8", "replace")
|
||||
_handle_status(session, agent, resp.status_code, text)
|
||||
_handle_status(session, agent, resp.status_code)
|
||||
with open(dest_path, "wb") as fh:
|
||||
async for chunk in resp.aiter_bytes(1024 * 256):
|
||||
fh.write(chunk)
|
||||
except httpx.HTTPError as exc:
|
||||
_mark(session, agent, "offline")
|
||||
raise AgentError(502, "agent_unreachable", str(exc)) from exc
|
||||
|
||||
|
||||
async def stream_download(
|
||||
session: Session,
|
||||
agent: Agent,
|
||||
path: str,
|
||||
*,
|
||||
params: Optional[dict] = None,
|
||||
):
|
||||
"""Stream a GET from the agent straight through, yielding chunks.
|
||||
|
||||
Unlike :func:`download_to_file` this never buffers to disk, so a large
|
||||
response (e.g. a folder zip the agent builds on the fly) starts flowing to
|
||||
the browser immediately instead of being staged first.
|
||||
"""
|
||||
url = agent.url.rstrip("/") + path
|
||||
headers = {"Authorization": f"Bearer {agent.token}"}
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||
async with client.stream("GET", url, headers=headers, params=params, timeout=None) as resp:
|
||||
if resp.status_code >= 400:
|
||||
text = (await resp.aread()).decode("utf-8", "replace")
|
||||
_handle_status(session, agent, resp.status_code, text)
|
||||
_handle_status(session, agent, resp.status_code)
|
||||
async for chunk in resp.aiter_bytes(1024 * 256):
|
||||
yield chunk
|
||||
except httpx.HTTPError as exc:
|
||||
_mark(session, agent, "offline")
|
||||
raise AgentError(502, "agent_unreachable", str(exc)) from exc
|
||||
|
||||
|
||||
async def upload_file(
|
||||
session: Session,
|
||||
agent: Agent,
|
||||
path: str,
|
||||
file_path: str,
|
||||
filename: str,
|
||||
data: dict,
|
||||
) -> Any:
|
||||
"""Stream a multipart POST (file + form fields) to the agent, returning JSON."""
|
||||
url = agent.url.rstrip("/") + path
|
||||
headers = {"Authorization": f"Bearer {agent.token}"}
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||
with open(file_path, "rb") as fh:
|
||||
files = {"file": (filename, fh, "application/gzip")}
|
||||
resp = await client.post(url, headers=headers, files=files, data=data, timeout=None)
|
||||
except httpx.HTTPError as exc:
|
||||
_mark(session, agent, "offline")
|
||||
raise AgentError(502, "agent_unreachable", str(exc)) from exc
|
||||
|
||||
detail = ""
|
||||
if resp.status_code >= 400:
|
||||
try:
|
||||
body = resp.json()
|
||||
detail = body.get("detail") if isinstance(body, dict) else str(body)
|
||||
except ValueError:
|
||||
detail = resp.text[:500]
|
||||
_handle_status(session, agent, resp.status_code, str(detail))
|
||||
return resp.json() if resp.content else None
|
||||
|
||||
|
||||
async def ping(session: Session, agent: Agent) -> dict:
|
||||
"""Health-check an agent and refresh its status + hostname. Never raises."""
|
||||
try:
|
||||
data = await call(session, agent, "GET", "/agent/ping")
|
||||
if isinstance(data, dict) and data.get("hostname"):
|
||||
agent.hostname = data["hostname"]
|
||||
session.add(agent)
|
||||
session.commit()
|
||||
session.refresh(agent)
|
||||
return {"status": agent.status, "hostname": agent.hostname, "data": data}
|
||||
except AgentError:
|
||||
return {"status": agent.status, "hostname": agent.hostname, "data": None}
|
||||
@@ -5,9 +5,8 @@ Runs once per image-update-check cycle (called from
|
||||
cache). For each enabled policy whose stack has a newer image available, either
|
||||
pulls + redeploys the stack or just notifies, recording the outcome.
|
||||
|
||||
Central-only / DB-aware. Image resolution + digest comparison live in the
|
||||
DB-free ``update_service`` so the agent can answer ``/agent/stacks/{id}/updates``
|
||||
with the same logic.
|
||||
Image resolution and digest comparison live in ``update_service``; this module
|
||||
adds the policy layer on top.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -17,11 +16,9 @@ from datetime import datetime, timezone
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from database import engine
|
||||
from models.agent import Agent
|
||||
from models.auto_update import AutoUpdate
|
||||
from models.setting import EVENT_PULL_FAILED, EVENT_STACK_AUTO_UPDATED
|
||||
from services import (
|
||||
agent_service,
|
||||
compose_service,
|
||||
notify_service,
|
||||
stack_lock_service,
|
||||
@@ -39,23 +36,18 @@ def _now() -> datetime:
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Policy CRUD helpers (shared by the stacks + agents routers)
|
||||
# Policy CRUD helpers
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def get_policy(session: Session, stack_id: str, agent_id: int | None = None) -> AutoUpdate | None:
|
||||
stmt = select(AutoUpdate).where(AutoUpdate.stack_id == stack_id)
|
||||
stmt = stmt.where(AutoUpdate.agent_id == agent_id) if agent_id is not None \
|
||||
else stmt.where(AutoUpdate.agent_id.is_(None))
|
||||
return session.exec(stmt).first()
|
||||
def get_policy(session: Session, stack_id: str) -> AutoUpdate | None:
|
||||
return session.exec(select(AutoUpdate).where(AutoUpdate.stack_id == stack_id)).first()
|
||||
|
||||
|
||||
def upsert_policy(
|
||||
session: Session, stack_id: str, enabled: bool, redeploy: bool, agent_id: int | None = None
|
||||
) -> AutoUpdate:
|
||||
policy = get_policy(session, stack_id, agent_id)
|
||||
def upsert_policy(session: Session, stack_id: str, enabled: bool, redeploy: bool) -> AutoUpdate:
|
||||
policy = get_policy(session, stack_id)
|
||||
if policy is None:
|
||||
policy = AutoUpdate(stack_id=stack_id, agent_id=agent_id)
|
||||
policy = AutoUpdate(stack_id=stack_id)
|
||||
policy.enabled = enabled
|
||||
policy.redeploy = redeploy
|
||||
session.add(policy)
|
||||
@@ -64,22 +56,19 @@ def upsert_policy(
|
||||
return policy
|
||||
|
||||
|
||||
def to_read(session: Session, policy: AutoUpdate | None, stack_id: str, agent_id: int | None = None) -> dict:
|
||||
def to_read(policy: AutoUpdate | None, stack_id: str) -> dict:
|
||||
"""Build an AutoUpdateRead-shaped dict, defaulting to disabled when absent."""
|
||||
agent_name = None
|
||||
if agent_id is not None:
|
||||
agent = session.get(Agent, agent_id)
|
||||
agent_name = agent.name if agent else None
|
||||
if policy is None:
|
||||
return {
|
||||
"id": None, "stack_id": stack_id, "agent_id": agent_id, "agent_name": agent_name,
|
||||
"id": None, "stack_id": stack_id,
|
||||
"enabled": False, "redeploy": True,
|
||||
"last_run": None, "last_status": None, "last_result": None,
|
||||
}
|
||||
return {
|
||||
"id": policy.id, "stack_id": policy.stack_id, "agent_id": policy.agent_id,
|
||||
"agent_name": agent_name, "enabled": policy.enabled, "redeploy": policy.redeploy,
|
||||
"last_run": policy.last_run, "last_status": policy.last_status, "last_result": policy.last_result,
|
||||
"id": policy.id, "stack_id": policy.stack_id,
|
||||
"enabled": policy.enabled, "redeploy": policy.redeploy,
|
||||
"last_run": policy.last_run, "last_status": policy.last_status,
|
||||
"last_result": policy.last_result,
|
||||
}
|
||||
|
||||
|
||||
@@ -138,48 +127,6 @@ async def _run_local(session: Session, policy: AutoUpdate) -> None:
|
||||
)
|
||||
|
||||
|
||||
async def _run_remote(session: Session, policy: AutoUpdate) -> None:
|
||||
agent = session.get(Agent, policy.agent_id)
|
||||
if not agent:
|
||||
_record(session, policy, "error", "agent not found")
|
||||
return
|
||||
stack_id = policy.stack_id
|
||||
try:
|
||||
summary = await agent_service.call(
|
||||
session, agent, "GET", f"/agent/stacks/{stack_id}/updates",
|
||||
params={"refresh": "true"},
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
_record(session, policy, "error", f"agent check failed: {exc}")
|
||||
return
|
||||
if not summary or not summary.get("update_available"):
|
||||
_record(session, policy, "up-to-date")
|
||||
return
|
||||
|
||||
stale = ", ".join(summary.get("stale_images", []))
|
||||
label = f"{agent.name}/{stack_id}"
|
||||
prev = policy.last_status
|
||||
if policy.redeploy:
|
||||
try:
|
||||
await agent_service.call(session, agent, "POST", f"/agent/stacks/{stack_id}/update")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
_record(session, policy, "error", str(exc))
|
||||
await _safe_notify(EVENT_PULL_FAILED, f"Auto-update of '{label}' failed", str(exc), session)
|
||||
return
|
||||
_record(session, policy, "updated", stale)
|
||||
await _safe_notify(
|
||||
EVENT_STACK_AUTO_UPDATED, f"Stack '{label}' auto-updated",
|
||||
f"Pulled and redeployed: {stale}.", session,
|
||||
)
|
||||
else:
|
||||
_record(session, policy, "update-available", stale)
|
||||
if prev != "update-available":
|
||||
await _safe_notify(
|
||||
EVENT_STACK_AUTO_UPDATED, f"Update available for '{label}'",
|
||||
f"Newer images: {stale} (auto-redeploy is off).", session,
|
||||
)
|
||||
|
||||
|
||||
async def _safe_notify(event: str, title: str, message: str, session: Session) -> None:
|
||||
try:
|
||||
await notify_service.notify(event, title, message, session)
|
||||
@@ -188,10 +135,7 @@ async def _safe_notify(event: str, title: str, message: str, session: Session) -
|
||||
|
||||
|
||||
async def run_policy(session: Session, policy: AutoUpdate) -> None:
|
||||
if policy.agent_id is None:
|
||||
await _run_local(session, policy)
|
||||
else:
|
||||
await _run_remote(session, policy)
|
||||
await _run_local(session, policy)
|
||||
|
||||
|
||||
async def run_due() -> None:
|
||||
|
||||
@@ -199,22 +199,6 @@ def containers_for_stack(stack_id: str) -> list[ContainerInfo]:
|
||||
return result
|
||||
|
||||
|
||||
# in-memory set of stacks currently performing a pull/up
|
||||
_BUSY: set[str] = set()
|
||||
|
||||
|
||||
def mark_busy(stack_id: str) -> None:
|
||||
_BUSY.add(stack_id)
|
||||
|
||||
|
||||
def clear_busy(stack_id: str) -> None:
|
||||
_BUSY.discard(stack_id)
|
||||
|
||||
|
||||
def is_busy(stack_id: str) -> bool:
|
||||
return stack_id in _BUSY
|
||||
|
||||
|
||||
def _status_from_states(states: list[str]) -> str:
|
||||
if not states:
|
||||
return "stopped"
|
||||
@@ -229,10 +213,13 @@ def _status_from_states(states: list[str]) -> str:
|
||||
|
||||
|
||||
def compute_status(stack_id: str, containers: Optional[list[ContainerInfo]] = None) -> str:
|
||||
"""Status for one stack. Pass already-fetched ``containers`` to avoid a
|
||||
redundant Docker round-trip (the detail view already has them)."""
|
||||
if stack_id in _BUSY:
|
||||
return "updating"
|
||||
"""Status for one stack, from its containers alone.
|
||||
|
||||
"updating" is not derived here: whether an operation is in flight lives in
|
||||
``stack_lock_service``, and callers that want to show it overlay the lock on
|
||||
top of this. Pass already-fetched ``containers`` to avoid a redundant Docker
|
||||
round-trip (the detail view already has them).
|
||||
"""
|
||||
try:
|
||||
if containers is None:
|
||||
containers = containers_for_stack(stack_id)
|
||||
@@ -447,11 +434,7 @@ async def stream_update(stack_id: str, override: Optional[str] = None):
|
||||
|
||||
|
||||
async def up(stack_id: str, override: Optional[str] = None) -> dict:
|
||||
mark_busy(stack_id)
|
||||
try:
|
||||
return await run_compose(stack_id, ["up", "-d", "--remove-orphans"], override)
|
||||
finally:
|
||||
clear_busy(stack_id)
|
||||
return await run_compose(stack_id, ["up", "-d", "--remove-orphans"], override)
|
||||
|
||||
|
||||
async def down(stack_id: str, override: Optional[str] = None) -> dict:
|
||||
@@ -471,27 +454,19 @@ async def restart(stack_id: str, override: Optional[str] = None) -> dict:
|
||||
|
||||
|
||||
async def pull(stack_id: str, override: Optional[str] = None) -> dict:
|
||||
mark_busy(stack_id)
|
||||
try:
|
||||
return await run_compose(stack_id, ["pull"], override)
|
||||
finally:
|
||||
clear_busy(stack_id)
|
||||
return await run_compose(stack_id, ["pull"], override)
|
||||
|
||||
|
||||
async def update(stack_id: str, override: Optional[str] = None) -> dict:
|
||||
"""Pull then up -d."""
|
||||
mark_busy(stack_id)
|
||||
try:
|
||||
pull_res = await run_compose(stack_id, ["pull"], override)
|
||||
up_res = await run_compose(stack_id, ["up", "-d", "--remove-orphans"], override)
|
||||
return {
|
||||
"returncode": up_res["returncode"],
|
||||
"stdout": pull_res["stdout"] + "\n" + up_res["stdout"],
|
||||
"stderr": pull_res["stderr"] + "\n" + up_res["stderr"],
|
||||
"command": "pull + up -d",
|
||||
}
|
||||
finally:
|
||||
clear_busy(stack_id)
|
||||
pull_res = await run_compose(stack_id, ["pull"], override)
|
||||
up_res = await run_compose(stack_id, ["up", "-d", "--remove-orphans"], override)
|
||||
return {
|
||||
"returncode": up_res["returncode"],
|
||||
"stdout": pull_res["stdout"] + "\n" + up_res["stdout"],
|
||||
"stderr": pull_res["stderr"] + "\n" + up_res["stderr"],
|
||||
"command": "pull + up -d",
|
||||
}
|
||||
|
||||
|
||||
async def logs(
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Single-container inspect + lifecycle — shared by the central app and agent.
|
||||
"""Single-container inspect + lifecycle.
|
||||
|
||||
Only containers that belong to a compose-managed stack (i.e. carry the
|
||||
``com.docker.compose.project`` label) are exposed, so this never becomes a
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
Most of StackPilot's secrets are files on disk (``.env``, ``.secrets/*``) where
|
||||
filesystem permissions are the right control. A few can't be: backup
|
||||
destination credentials and agent tokens are needed by background jobs, so they
|
||||
sit in ``stackpilot.db``. This module encrypts those at rest.
|
||||
destination credentials are needed by background jobs, so they sit in
|
||||
``stackpilot.db``. This module encrypts those at rest.
|
||||
|
||||
The key is derived from ``SECRET_KEY`` rather than being a second thing to
|
||||
configure — which is exactly why ``SECRET_KEY`` is now persisted (see
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
"""Fleet aggregate for the dashboard cockpit.
|
||||
"""Host aggregate for the dashboard cockpit.
|
||||
|
||||
One read-only call rolls up every host (local + agents) into a "needs
|
||||
attention" list, headline KPIs and a per-host resource view. It is cheap by
|
||||
construction: a single container *summary* list (no per-container inspect)
|
||||
drives the local figures, image freshness comes from the cache the
|
||||
update-service background loop already maintains, and per agent it makes a
|
||||
small bounded set of HTTP fetches that degrade gracefully on failure.
|
||||
One read-only call rolls the host up into a "needs attention" list, headline
|
||||
KPIs and a resource view. It is cheap by construction: a single container
|
||||
*summary* list (no per-container inspect) drives the figures, and image
|
||||
freshness comes from the cache the update-service background loop already
|
||||
maintains.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -15,15 +14,13 @@ from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from database import engine
|
||||
from docker_client import DockerError, get_client, safe_call
|
||||
from models.agent import Agent
|
||||
from models.backup_schedule import BackupSchedule
|
||||
from services import agent_service, compose_service, update_service
|
||||
from services import compose_service, update_service
|
||||
|
||||
COMPOSE_LABEL = compose_service.COMPOSE_LABEL
|
||||
|
||||
# Fleet aggregate: cached briefly because each call may fan out to every agent.
|
||||
# Cached briefly: the dashboard polls this and the rollup is not free.
|
||||
FLEET_TTL = 25.0
|
||||
DISK_PRESSURE = 0.85 # disk used fraction above which a host needs attention
|
||||
MEM_PRESSURE = 0.90 # memory used fraction above which a host needs attention
|
||||
@@ -73,14 +70,12 @@ def _is_updated(containers: list[dict], cache: dict[str, dict]) -> bool:
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Fleet aggregate (one call → "needs attention" + KPIs across every host)
|
||||
# Host aggregate (one call → "needs attention" + KPIs)
|
||||
#
|
||||
# The dashboard used to poll each agent individually and recombine the numbers
|
||||
# client-side. ``compute_fleet`` does the fan-out server-side instead: one local
|
||||
# Docker pass plus, per online agent, a small set of system/stacks/updates
|
||||
# fetches — all wrapped so a slow or broken agent degrades to "offline" rather
|
||||
# than stalling the whole view. Cached for FLEET_TTL because the fan-out is not
|
||||
# free.
|
||||
# The dashboard used to fetch these numbers per stack and recombine them
|
||||
# client-side. ``compute_fleet`` rolls them up server-side instead, off one
|
||||
# Docker pass plus the update cache, and holds the result for FLEET_TTL because
|
||||
# the dashboard polls it.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
# Stack-status buckets the status bar / KPIs are built from. Anything reporting
|
||||
@@ -189,98 +184,12 @@ def _local_host() -> tuple[dict, list[dict]]:
|
||||
return host, attention
|
||||
|
||||
|
||||
def _offline_host(agent_id: int, name: str, status: str) -> dict:
|
||||
return {"id": agent_id, "name": name, "online": False, "status": status,
|
||||
"cpu_cores": 0, "mem_used": 0, "mem_total": 0, "disk_used": 0, "disk_total": 0,
|
||||
"stacks": _bucket_statuses([]), "containers_running": 0, "containers_total": 0,
|
||||
"unhealthy": 0, "updates_available": 0}
|
||||
|
||||
|
||||
async def _agent_host(agent_id: int) -> tuple[dict, list[dict]]:
|
||||
"""One agent's host card + attention items, tolerant of partial failure.
|
||||
|
||||
Opens its own Session so the fan-out across agents stays concurrency-safe
|
||||
(a shared SQLModel session is not), and fetches sequentially within the
|
||||
agent because :func:`agent_service.call` commits a status update each time.
|
||||
"""
|
||||
with Session(engine) as session:
|
||||
agent = session.get(Agent, agent_id)
|
||||
if agent is None:
|
||||
return _offline_host(agent_id, str(agent_id), "unknown"), []
|
||||
name = agent.name
|
||||
# Agent stacks live in the host section of the dashboard, not a route
|
||||
# of their own, so aggregate agent items deep-link back to it.
|
||||
link = "/"
|
||||
|
||||
if agent.status != "online":
|
||||
return _offline_host(agent_id, name, agent.status), [
|
||||
_attn("error", "agent_offline", name,
|
||||
f"{name} is {agent.status}", "Check it under Settings → Remote hosts.",
|
||||
"/settings")]
|
||||
|
||||
async def _fetch(path: str):
|
||||
try:
|
||||
return await agent_service.call(session, agent, "GET", path)
|
||||
except Exception: # AgentError or transport — degrade gracefully
|
||||
return None
|
||||
|
||||
sys_data = await _fetch("/agent/system")
|
||||
stacks = await _fetch("/agent/stacks")
|
||||
updates = await _fetch("/agent/stacks/updates")
|
||||
|
||||
if sys_data is None and stacks is None:
|
||||
# Couldn't reach it at all — agent_service.call already marked it offline.
|
||||
return _offline_host(agent_id, name, "offline"), [
|
||||
_attn("error", "agent_offline", name,
|
||||
f"{name} is unreachable", "Check it under Settings → Remote hosts.",
|
||||
"/settings")]
|
||||
|
||||
sys_data = sys_data or {}
|
||||
statuses = [s.get("status", "") for s in (stacks or [])]
|
||||
buckets = _bucket_statuses(statuses)
|
||||
update_count = sum(1 for v in (updates or {}).values()
|
||||
if isinstance(v, dict) and v.get("update_available"))
|
||||
|
||||
host = {
|
||||
"id": agent_id,
|
||||
"name": name,
|
||||
"online": True,
|
||||
"status": "online",
|
||||
"cpu_cores": sys_data.get("cpu_cores", 0),
|
||||
"mem_used": sys_data.get("mem_used", 0), "mem_total": sys_data.get("mem_total", 0),
|
||||
"disk_used": sys_data.get("disk_used", 0), "disk_total": sys_data.get("disk_total", 0),
|
||||
"stacks": buckets,
|
||||
"containers_running": sys_data.get("compose_running", sys_data.get("containers_running", 0)),
|
||||
"containers_total": sys_data.get("containers_total", 0),
|
||||
"unhealthy": buckets["error"], # agents expose no healthcheck rollup; error is the proxy
|
||||
"updates_available": update_count,
|
||||
}
|
||||
|
||||
attention: list[dict] = []
|
||||
if buckets["error"]:
|
||||
attention.append(_attn("error", "stack_error", name,
|
||||
f"{name}: {buckets['error']} stack(s) in error",
|
||||
"Containers are dead.", link))
|
||||
if buckets["partial"]:
|
||||
attention.append(_attn("warn", "stack_partial", name,
|
||||
f"{name}: {buckets['partial']} stack(s) partially running",
|
||||
"Some services are down.", link))
|
||||
if update_count:
|
||||
attention.append(_attn("warn", "updates", name,
|
||||
f"{name}: {update_count} stack(s) have image updates",
|
||||
"Pull the newer images.", link))
|
||||
attention += _resource_attention(name, link,
|
||||
host["mem_used"], host["mem_total"],
|
||||
host["disk_used"], host["disk_total"])
|
||||
return host, attention
|
||||
|
||||
|
||||
def _backup_attention(session: Session, agent_names: dict[int, str]) -> list[dict]:
|
||||
def _backup_attention(session: Session) -> list[dict]:
|
||||
"""Flag enabled backup schedules whose last run failed or is overdue."""
|
||||
now = datetime.now(timezone.utc)
|
||||
items: list[dict] = []
|
||||
for sch in session.exec(select(BackupSchedule).where(BackupSchedule.enabled == True)).all(): # noqa: E712
|
||||
host = agent_names.get(sch.agent_id, "local") if sch.agent_id else "local"
|
||||
host = "local"
|
||||
status = (sch.last_status or "").lower()
|
||||
if status and not status.startswith("ok"):
|
||||
items.append(_attn("error", "backup_failed", host,
|
||||
@@ -307,23 +216,12 @@ async def compute_fleet(session: Session, refresh: bool = False) -> dict:
|
||||
return _fleet_cache["data"]
|
||||
|
||||
local_host, attention = await asyncio.to_thread(_local_host)
|
||||
|
||||
agents = session.exec(select(Agent)).all()
|
||||
agent_names = {a.id: a.name for a in agents}
|
||||
agent_ids = [a.id for a in agents]
|
||||
agent_results = await asyncio.gather(*[_agent_host(aid) for aid in agent_ids])
|
||||
|
||||
hosts = [local_host]
|
||||
for host, items in agent_results:
|
||||
hosts.append(host)
|
||||
attention += items
|
||||
attention += _backup_attention(session, agent_names)
|
||||
attention += _backup_attention(session)
|
||||
|
||||
attention.sort(key=lambda a: _SEVERITY_ORDER.get(a["severity"], 9))
|
||||
|
||||
kpis = {
|
||||
"hosts_online": sum(1 for h in hosts if h["online"]),
|
||||
"hosts_total": len(hosts),
|
||||
"stacks_running": sum(h["stacks"]["running"] for h in hosts),
|
||||
"stacks_partial": sum(h["stacks"]["partial"] for h in hosts),
|
||||
"stacks_total": sum(h["stacks"]["total"] for h in hosts),
|
||||
|
||||
@@ -94,11 +94,10 @@ def _real_root(path: str) -> str:
|
||||
"""Map a logical host path into the container view (HOST_ROOT_PREFIX).
|
||||
|
||||
Refuses anything that resolves inside StackPilot's own ``DATA_DIR``. That
|
||||
directory holds ``stackpilot.db`` — users, password hashes, agent tokens and
|
||||
directory holds ``stackpilot.db`` — users, password hashes and
|
||||
backup-destination credentials — and the API deliberately never hands those
|
||||
out (``AgentRead.token_set`` is a bool, destination secrets come back
|
||||
masked). Without this the file browser would be a way around that, for
|
||||
admins too. Note this only bites when ``HOST_ROOT_PREFIX`` is empty: with a
|
||||
out (destination secrets come back masked). Without this the file browser
|
||||
would be a way around that, for admins too. Note this only bites when ``HOST_ROOT_PREFIX`` is empty: with a
|
||||
prefix set, no logical path can reach the container's own ``/data`` at all.
|
||||
"""
|
||||
prefix = settings.HOST_ROOT_PREFIX.rstrip("/")
|
||||
|
||||
@@ -66,7 +66,6 @@ def exec_exit_code(exec_id: str):
|
||||
async def pump_exec(websocket, exec_id: str, holder, raw) -> None:
|
||||
"""Bidirectionally pump an exec socket <-> a WebSocket.
|
||||
|
||||
Shared by the central app and the agent (both pass a Starlette WebSocket).
|
||||
Browser -> container: JSON ``{"type":"data","data":...}`` keystrokes and
|
||||
``{"type":"resize","rows","cols"}`` control frames (raw text is also
|
||||
accepted as keystrokes). Container -> browser: ``{"type":"data","data":...}``
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Image listing — shared by the central images router and the agent."""
|
||||
"""Image listing for the images router."""
|
||||
from __future__ import annotations
|
||||
|
||||
from docker_client import DockerError, get_client, safe_call
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
"""Persistence for the image update cache.
|
||||
|
||||
``update_service`` holds the registry-digest results in a module dict because
|
||||
it is shared with the agent, which has no database. This module is the central
|
||||
app's half: it seeds that dict at startup and mirrors every write back into
|
||||
SQLite, wired up in ``main.lifespan``.
|
||||
``update_service`` holds the registry-digest results in a module dict and knows
|
||||
nothing about storage — it is pure registry logic and stays unit-testable
|
||||
without a database. This module is its persistence half: it seeds that dict at
|
||||
startup and mirrors every write back into SQLite, wired up in ``main.lifespan``.
|
||||
|
||||
What it buys: after a restart the update badges are there immediately instead
|
||||
of blank until the next background sweep (up to an hour), and the "already
|
||||
|
||||
@@ -11,22 +11,18 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from database import engine
|
||||
from models.agent import Agent
|
||||
from models.backup_destination import BackupDestination
|
||||
from models.backup_schedule import BackupSchedule
|
||||
from models.setting import EVENT_BACKUP_FAILED
|
||||
from models.stack import Stack
|
||||
from services import (
|
||||
agent_service,
|
||||
backup_destination_service as dest_service,
|
||||
backup_service,
|
||||
compose_service,
|
||||
notify_service,
|
||||
)
|
||||
|
||||
@@ -87,31 +83,16 @@ async def run_schedule(session: Session, schedule: BackupSchedule) -> dict:
|
||||
if not dest:
|
||||
raise RuntimeError(f"destination {schedule.destination_id} not found")
|
||||
|
||||
# Produce the backup archive — locally or by streaming it from an agent.
|
||||
if schedule.agent_id is not None:
|
||||
agent = session.get(Agent, schedule.agent_id)
|
||||
if not agent:
|
||||
raise RuntimeError(f"agent {schedule.agent_id} not found")
|
||||
prefix = compose_service.slugify(agent.name)
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".tar.gz")
|
||||
tmp.close()
|
||||
path = tmp.name
|
||||
await agent_service.download_to_file(
|
||||
session, agent, f"/agent/stacks/{schedule.stack_id}/backup", path,
|
||||
params={"include_volumes": schedule.include_volumes, "stop_first": schedule.stop_first},
|
||||
)
|
||||
else:
|
||||
stack = session.get(Stack, schedule.stack_id)
|
||||
if not stack:
|
||||
raise RuntimeError(f"stack '{schedule.stack_id}' not found")
|
||||
prefix = None
|
||||
path = await backup_service.create_backup(
|
||||
schedule.stack_id, stack.name,
|
||||
include_volumes=schedule.include_volumes, stop_first=schedule.stop_first,
|
||||
)
|
||||
stack = session.get(Stack, schedule.stack_id)
|
||||
if not stack:
|
||||
raise RuntimeError(f"stack '{schedule.stack_id}' not found")
|
||||
path = await backup_service.create_backup(
|
||||
schedule.stack_id, stack.name,
|
||||
include_volumes=schedule.include_volumes, stop_first=schedule.stop_first,
|
||||
)
|
||||
|
||||
filename = backup_service.backup_filename(schedule.stack_id, schedule.include_volumes, prefix=prefix)
|
||||
basename = backup_service.backup_basename(schedule.stack_id, prefix)
|
||||
filename = backup_service.backup_filename(schedule.stack_id, schedule.include_volumes)
|
||||
basename = backup_service.backup_basename(schedule.stack_id)
|
||||
await asyncio.to_thread(dest_service.upload, dest, path, filename)
|
||||
pruned = await asyncio.to_thread(_prune, dest, basename, schedule.keep)
|
||||
schedule.last_status = "ok"
|
||||
|
||||
@@ -5,15 +5,11 @@ project — two open browser tabs, or the auto-update pass landing on a stack
|
||||
somebody just clicked — both run ``pull`` and then ``up -d``, and race each
|
||||
other recreating the same containers.
|
||||
|
||||
There *was* a busy flag (``compose_service._BUSY``), but it only ever fed the
|
||||
status column: no lifecycle handler consulted it before acting. This module is
|
||||
the actual guard, and it lives in the database so it holds across workers and
|
||||
across a restart.
|
||||
|
||||
``compose_service`` keeps its in-process set because it is shared with the
|
||||
agent, which has no database. The agent is a single process managing one host,
|
||||
and the central app holds this lock before calling it, so the two do not
|
||||
conflict.
|
||||
There *was* a busy flag in ``compose_service``, but it only ever fed the status
|
||||
column: no lifecycle handler consulted it before acting. This module is the
|
||||
actual guard, and it lives in the database so it holds across workers and
|
||||
across a restart. ``compose_service.compute_status`` therefore reports only
|
||||
what the containers say; callers overlay the lock to show "updating".
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
@@ -53,13 +53,13 @@ _NOTIFIED: set[str] = set()
|
||||
|
||||
#: Optional sink for cache writes.
|
||||
#:
|
||||
#: This module is shared with the agent, which has no database, so persistence
|
||||
#: cannot live here. The central app registers a callback that mirrors each
|
||||
#: entry into SQLite (see ``services/image_status_store.py``) and seeds the
|
||||
#: cache from it at startup; the agent registers nothing and behaves exactly as
|
||||
#: before. Without it a restart blanked every update badge until the next
|
||||
#: background sweep — up to an hour — and re-announced updates it had already
|
||||
#: notified about.
|
||||
#: This module is pure registry logic and knows nothing about storage, which
|
||||
#: keeps it unit-testable without a database. ``main.lifespan`` registers a
|
||||
#: callback that mirrors each entry into SQLite (see
|
||||
#: ``services/image_status_store.py``) and seeds the cache from it at startup.
|
||||
#: Without it a restart blanked every update badge until the next background
|
||||
#: sweep — up to an hour — and re-announced updates it had already notified
|
||||
#: about.
|
||||
_persist_cb: Optional[Callable[[UpdateStatus, bool], None]] = None
|
||||
|
||||
|
||||
@@ -299,7 +299,7 @@ async def stack_updates(stack_id: str, refresh: bool = True) -> dict:
|
||||
|
||||
``refresh=True`` queries the registry now; ``False`` reads the cache the
|
||||
background loop already populated (so the auto-update pass adds no extra
|
||||
registry round-trips). DB-free, so the agent can reuse it verbatim.
|
||||
registry round-trips).
|
||||
"""
|
||||
images = stack_images(stack_id)
|
||||
result: dict[str, dict] = {}
|
||||
|
||||
Reference in New Issue
Block a user