menzeljonas/stackpilot
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make backup and restore reliable, and keep NFS out of them (0.64.0)
CI / check (push) Successful in 14m50s
Details
CI / build-and-push (push) Successful in 2m20s
Details

Browse Source 
A review of the backup path found several ways a backup was not one, and a
restore could destroy data it did not contain.

Data loss:
- Restore with overwrite rmtree'd the whole stack directory, including
  folders the backup had left out (over 2 GiB, deselected, an NFS share
  mounted inside). It now clears only what the backup holds, keeps the
  excluded paths (recorded in the manifest) and never crosses into another
  filesystem.
- The stack was only stopped when named volumes were selected, so data
  folders inside the stack directory were archived while in use. Any
  selected data now stops it.
- Retention matched by prefix: pruning "app" deleted "app-db"'s backups.
  It now matches backup-<id>-(full|config)-<date>.tar.gz exactly.
- A restore under a new name wrote into the original's volumes when they
  had an explicit name, and into its folders outside the stack directory.
  Both are skipped now; a volume owned by another stack or mounted by a
  running container is never overwritten.
- A truncated archive was only noticed after the old stack was gone. The
  whole archive is verified first, and the stack directory is unpacked
  next to the old one and swapped in.

Remote storage:
- Bind folders on a mounted share (fstab/autofs), bind-type volumes
  pointing at one and shares mounted inside the stack folder are detected
  by filesystem type and not selected; a folder du cannot measure in 20s
  is no longer selected by default either.

Reliability:
- After a backup exactly the containers that ran are started again with
  compose start (falling back to the Docker API), instead of up -d, which
  started services stopped on purpose and applied undeployed edits. A
  stack that cannot be stopped is not backed up.
- Backup and restore hold the stack lock; a scheduled backup that finds
  the stack busy retries after 10 minutes. Restore stops the stack while
  replacing its data and brings it back up.
- external: true volumes are backed up; anonymous ones are listed, not
  selected. Restored volumes get their original driver and options.
- A backup with uncaptured data or a stack that did not restart is
  reported as incomplete: in the dialog, and for schedules as a warning
  with a notification and without rotating older backups out.
- Archive writing and restore run off the event loop; downloads delete
  their temp file; helper containers use the long-timeout client; S3
  listing is paginated; a file changing mid-read no longer corrupts the
  archive.

Manifest format 3 records compose_binds/excluded and volume
driver/options/external; format 1/2 backups still restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
menzeljandClaude Opus 5 committed 2026-09-18 11:33:54 +02:00
1 parent ee98d0da6d
commit 8f0a01c4f5
12 files changed
+1938 -276
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

backend/services/stack_assets_service.py
+224 -59
View File
@@ -24,7 +24,7 @@ from typing import Optional
import yaml
from config import settings
from docker_client import DockerError, get_client, safe_call
from docker_client import DockerError, get_client, get_long_client, safe_call
from services import compose_service
logger = logging.getLogger("stackpilot.assets")
@@ -49,6 +49,24 @@ SYSTEM_PREFIXES = ("/dev", "/proc", "/sys", "/run", "/var/run", "/var/lib/docker
# user wants, and *restoring* one would overwrite the share.
REMOTE_VOLUME_TYPES = {"nfs", "nfs4", "cifs", "smb", "smb3", "smbfs", "sshfs", "glusterfs"}
# The same for bind mounts and bind-type volumes, which only say *where* they
# are, not what: a host directory can be an NFS mount (fstab/autofs) just as
# well. Recognised by the filesystem magic `stat -f` reports for the path.
REMOTE_FS_MAGIC = {
0x6969: "nfs",
0x517B: "smb",
0xFF534D42: "cifs",
0xFE534D42: "smb2",
0x00C36400: "ceph",
}
# Mount types in /proc/self/mountinfo that are remote storage.
REMOTE_MOUNT_TYPES = {"nfs", "nfs4", "cifs", "smb3", "smbfs", "ceph", "glusterfs",
"fuse.sshfs", "fuse.rclone", "fuse.glusterfs"}
# Volumes Docker names itself (64 hex chars): recreated with every container, so
# a restore could never hand them back to the service.
_ANONYMOUS_RE = re.compile(r"^[0-9a-f]{64}$")
# Bind directories larger than this are listed but not selected by default.
DEFAULT_MAX_BIND_BYTES = 2 * 1024**3
@@ -116,7 +134,7 @@ def _put_archive(container, dest: str, src_file: str, volumes: dict) -> None:
if not _is_chown_error(exc):
raise
logger.warning("%s rejects ownership changes; restoring without it", dest)
client = get_client()
client = get_long_client()
staging = _create_helper_with(
client, volumes, ["sh", "-c", f"cp -R /tmp/. '{dest}/'"]
)
@@ -142,26 +160,35 @@ def _create_helper_with(client, volumes: dict, command: list[str]):
)
def inspect_paths(paths: list[str]) -> dict[str, dict]:
"""Classify host paths ({path: {"kind", "size"}}) via one helper container.
def _unknown() -> dict:
return {"kind": "unknown", "size": None, "fs": None, "remote": None, "measured": False}
``kind`` is dir / file / special; ``size`` is bytes (best effort, the walk is
capped so a huge media share can't stall the request).
def inspect_paths(paths: list[str]) -> dict[str, dict]:
"""Classify host paths via one helper container.
``{path: {"kind", "size", "fs", "remote", "measured"}}``: ``kind`` is dir /
file / special, ``size`` bytes. ``remote`` names the network filesystem the
path sits on (nfs, cifs, ...), whose contents are not walked at all.
Directory sizes are capped at 20s of ``du``; ``measured`` is False when that
ran out, which on its own says the directory is big.
"""
unique = [p for p in dict.fromkeys(paths) if p]
if not unique:
return {}
client = get_client()
client = get_long_client()
ensure_helper_image(client)
mounts = {p: {"bind": f"/m/{i}", "mode": "ro"} for i, p in enumerate(unique)}
remote_hex = "|".join(f"{m:x}" for m in REMOTE_FS_MAGIC)
script_parts = []
for i in range(len(unique)):
script_parts.append(
f'd=/m/{i}; '
f'if [ -d "$d" ]; then s=$(timeout 20 du -sk "$d" 2>/dev/null | cut -f1); '
f'echo "{i} dir ${{s:-}}"; '
f'elif [ -f "$d" ]; then echo "{i} file $(stat -c %s "$d" 2>/dev/null)"; '
f'else echo "{i} special"; fi'
f'd=/m/{i}; t=$(stat -f -c %t "$d" 2>/dev/null); t=${{t:--}}; '
f'if [ -d "$d" ]; then '
f'case "$t" in {remote_hex}) echo "{i} dir - $t";; '
f'*) s=$(timeout 20 du -sk "$d" 2>/dev/null | cut -f1); echo "{i} dir ${{s:--}} $t";; esac; '
f'elif [ -f "$d" ]; then echo "{i} file $(stat -c %s "$d" 2>/dev/null) $t"; '
f'else echo "{i} special - $t"; fi'
)
script = "; ".join(script_parts)
try:
@@ -176,9 +203,9 @@ def inspect_paths(paths: list[str]) -> dict[str, dict]:
)
except DockerError as exc:
logger.warning("Path inspection failed: %s", exc)
return {p: {"kind": "unknown", "size": None} for p in unique}
return {p: _unknown() for p in unique}
result: dict[str, dict] = {p: {"kind": "unknown", "size": None} for p in unique}
result: dict[str, dict] = {p: _unknown() for p in unique}
for line in (out or b"").decode("utf-8", "replace").splitlines():
parts = line.strip().split()
if len(parts) < 2 or not parts[0].isdigit():
@@ -190,13 +217,24 @@ def inspect_paths(paths: list[str]) -> dict[str, dict]:
size: Optional[int] = None
if len(parts) > 2 and parts[2].isdigit():
size = int(parts[2]) * 1024 if kind == "dir" else int(parts[2])
result[unique[idx]] = {"kind": kind, "size": size}
fs = parts[3] if len(parts) > 3 and parts[3] != "-" else None
try:
remote = REMOTE_FS_MAGIC.get(int(fs, 16)) if fs else None
except ValueError:
remote = None
result[unique[idx]] = {
"kind": kind,
"size": size,
"fs": fs,
"remote": remote,
"measured": kind != "dir" or remote is not None or size is not None,
}
return result
def export_path(source: str, kind: str, dest_file: str) -> int:
"""Tar a host path (dir contents, or a single file) into ``dest_file``."""
client = get_client()
client = get_long_client()
ensure_helper_image(client)
if kind == "file":
parent, base = _split(source)
@@ -223,7 +261,7 @@ def export_path(source: str, kind: str, dest_file: str) -> int:
def import_path(source: str, kind: str, src_file: str) -> None:
"""Unpack an archive produced by :func:`export_path` back to its host path."""
client = get_client()
client = get_long_client()
ensure_helper_image(client)
if kind == "file":
parent, _base = _split(source)
@@ -239,7 +277,7 @@ def import_path(source: str, kind: str, src_file: str) -> None:
def export_volume(full_name: str, dest_file: str) -> int:
"""Stream a named volume's contents into ``dest_file`` (never into RAM)."""
client = get_client()
client = get_long_client()
ensure_helper_image(client)
container = _create_helper(client, {full_name: {"bind": "/v", "mode": "ro"}})
written = 0
@@ -254,16 +292,53 @@ def export_volume(full_name: str, dest_file: str) -> int:
return written
def import_volume(full_name: str, labels: dict, src_file: str, wipe: bool = True) -> None:
"""Restore a volume from an archive, optionally clearing it first."""
client = get_client()
def volume_attrs(full_name: str) -> Optional[dict]:
"""A volume's inspect data, or None if it does not exist."""
try:
return safe_call(get_client().volumes.get, full_name).attrs or {}
except DockerError:
return None
def running_users(full_name: str) -> list[str]:
"""Names of running containers that have the volume mounted."""
try:
containers = safe_call(
get_client().containers.list, filters={"volume": full_name, "status": "running"}
)
except DockerError:
return []
return sorted(c.name for c in containers)
def import_volume(
full_name: str,
labels: dict,
src_file: str,
wipe: bool = True,
driver: Optional[str] = None,
driver_opts: Optional[dict] = None,
) -> None:
"""Restore a volume from an archive, optionally clearing it first.
A missing volume is recreated with the driver and options it had, not as a
plain local volume: compose refuses to use an existing volume whose
configuration differs from the compose file.
"""
client = get_long_client()
ensure_helper_image(client)
existed = True
try:
safe_call(client.volumes.get, full_name)
except DockerError:
existed = False
safe_call(client.volumes.create, name=full_name, labels=labels or {})
safe_call(
client.volumes.create,
name=full_name,
driver=driver or "local",
driver_opts=driver_opts or None,
labels=labels or {},
)
if existed and wipe:
# Restore means "back to the snapshot": drop files created since.
safe_call(
@@ -429,8 +504,77 @@ def _inside(path: str, parent: str) -> bool:
return path == parent or path.startswith(parent + "/")
def _volume_entry(name: str, attrs: dict, external: bool) -> dict:
labels = attrs.get("Labels") or {}
options = attrs.get("Options") or {}
driver = attrs.get("Driver", "local")
vtype = str(options.get("type") or "").lower()
device = str(options.get("device") or "")
remote = (
vtype in REMOTE_VOLUME_TYPES
or driver != "local"
or device.startswith("//")
or device.startswith(":")
)
# A local volume that binds a host directory (type none, o=bind) is only as
# local as that directory; inventory() checks what it sits on.
bind_device = (
device
if not remote and device.startswith("/") and "bind" in str(options.get("o") or "")
else None
)
return {
"name": name,
"short": labels.get(COMPOSE_VOLUME_LABEL, name),
"labels": labels,
"driver": driver,
"options": options,
"remote": remote,
"remote_type": vtype or (driver if driver != "local" else None),
"external": external,
"bind_device": bind_device,
}
def _external_volume_names(stack_id: str) -> set[str]:
"""Named volumes the stack uses but compose did not create for it:
``external: true`` in the compose file, or mounted by its containers."""
names: set[str] = set()
try:
client = get_client()
containers = safe_call(
client.containers.list,
all=True,
filters={"label": f"{COMPOSE_PROJECT_LABEL}={stack_id}"},
)
for c in containers:
for mount in c.attrs.get("Mounts") or []:
if mount.get("Type") == "volume" and mount.get("Name"):
names.add(mount["Name"])
except DockerError:
pass
compose_file = compose_service.find_compose_file(compose_service.stack_dir(stack_id))
if compose_file:
try:
with open(compose_file, "r", encoding="utf-8", errors="replace") as fh:
data = yaml.safe_load(fh) or {}
except (OSError, yaml.YAMLError):
data = {}
env = _env_for_stack(stack_id)
for key, spec in (data.get("volumes") or {}).items():
if isinstance(spec, dict) and spec.get("external"):
names.add(_interpolate(str(spec.get("name") or key), env))
return names
def compose_volumes(stack_id: str) -> list[dict]:
"""Compose-managed named volumes, with remote-storage detection."""
"""The stack's named volumes, with remote-storage detection.
Compose-created ones carry the project label. Volumes the stack only uses
(``external: true``, or created by hand) do not, and were silently missing
from backups before; they are listed with ``external`` set. Anonymous
volumes are listed too, so it is visible that data lives in one.
"""
try:
client = get_client()
vols = safe_call(
@@ -439,34 +583,43 @@ def compose_volumes(stack_id: str) -> list[dict]:
)
except DockerError:
return []
out = []
for v in vols:
attrs = v.attrs or {}
labels = attrs.get("Labels") or {}
options = attrs.get("Options") or {}
driver = attrs.get("Driver", "local")
vtype = str(options.get("type") or "").lower()
device = str(options.get("device") or "")
remote = (
vtype in REMOTE_VOLUME_TYPES
or driver != "local"
or device.startswith("//")
or device.startswith(":")
)
out.append(
{
"name": v.name,
"short": labels.get(COMPOSE_VOLUME_LABEL, v.name),
"labels": labels,
"driver": driver,
"options": options,
"remote": remote,
"remote_type": vtype or (driver if driver != "local" else None),
}
)
out = [_volume_entry(v.name, v.attrs or {}, external=False) for v in vols]
known = {v["name"] for v in out}
for name in sorted(_external_volume_names(stack_id) - known):
attrs = volume_attrs(name)
if attrs is None:
continue # declared external but not created yet
out.append(_volume_entry(name, attrs, external=True))
return out
def _mounts() -> list[tuple[str, str]]:
"""(mount point, filesystem type) of every mount this process sees."""
out = []
try:
with open("/proc/self/mountinfo", "r", encoding="utf-8") as fh:
for line in fh:
left, _, right = line.partition(" - ")
fields, rfields = left.split(), right.split()
if len(fields) >= 5 and rfields:
# Spaces etc. are octal-escaped in mountinfo.
point = fields[4].encode().decode("unicode_escape")
out.append((point, rfields[0]))
except OSError:
pass
return out
def remote_mount_type(path: str) -> Optional[str]:
"""The remote filesystem type ``path`` lives on as this process sees it,
e.g. an NFS share mounted inside the stacks directory. None if local."""
best, best_type = "", None
for point, fstype in _mounts():
if _inside(path, point) and len(point) >= len(best):
best, best_type = point, fstype
return best_type if best_type in REMOTE_MOUNT_TYPES else None
def inventory(stack_id: str, max_bind_bytes: int = DEFAULT_MAX_BIND_BYTES) -> dict:
"""What a backup of this stack would (and would not) capture.
@@ -488,14 +641,18 @@ def inventory(stack_id: str, max_bind_bytes: int = DEFAULT_MAX_BIND_BYTES) -> di
if mount not in entry["mounts"]:
entry["mounts"].append(mount)
volumes_raw = compose_volumes(stack_id)
real_paths = [p for p in grouped if not is_system_path(p)]
stats = inspect_paths(real_paths)
stats = inspect_paths(
real_paths + [v["bind_device"] for v in volumes_raw if v["bind_device"]]
)
binds = []
for path, entry in sorted(grouped.items()):
system = is_system_path(path)
info = stats.get(path, {"kind": "unknown", "size": None})
info = stats.get(path) or _unknown()
kind, size = info["kind"], info["size"]
remote = info.get("remote") or (remote_mount_type(path) if not system else None)
inside = _inside(path, directory)
# A path inside the stack directory that this process can actually read
# is already covered by the compose/ tree in the archive.
@@ -504,12 +661,17 @@ def inventory(stack_id: str, max_bind_bytes: int = DEFAULT_MAX_BIND_BYTES) -> di
reason = None
if system:
include, reason = False, "system path"
elif remote:
include, reason = False, f"remote storage ({remote})"
elif kind == "special":
include, reason = False, "not a regular file or directory"
elif kind == "unknown":
include, reason = False, "could not inspect path"
elif size is not None and size > max_bind_bytes:
include, reason = False, f"larger than {max_bind_bytes // 1024**3} GiB"
elif not info.get("measured", True):
# du ran out of time: too big to have been walked in 20s.
include, reason = False, "too large to measure"
binds.append(
{
"source": path,
@@ -522,21 +684,24 @@ def inventory(stack_id: str, max_bind_bytes: int = DEFAULT_MAX_BIND_BYTES) -> di
"covered_by_compose": visible,
"via": "compose" if visible else "archive",
"system": system,
"remote": remote,
"include_default": include,
"reason": reason,
}
)
volumes = []
for vol in compose_volumes(stack_id):
include = not vol["remote"]
volumes.append(
{
**vol,
"include_default": include,
"reason": None if include else f"remote storage ({vol['remote_type']})",
}
)
for vol in volumes_raw:
if vol["bind_device"]:
device_remote = (stats.get(vol["bind_device"]) or {}).get("remote")
if device_remote:
vol = {**vol, "remote": True, "remote_type": device_remote}
include, reason = True, None
if vol["remote"]:
include, reason = False, f"remote storage ({vol['remote_type']})"
elif _ANONYMOUS_RE.match(vol["name"]):
include, reason = False, "anonymous volume (Docker recreates it with the container)"
volumes.append({**vol, "include_default": include, "reason": reason})
return {
"stack_id": stack_id,