Make backup and restore reliable, and keep NFS out of them (0.64.0)

A review of the backup path found several ways a backup was not one, and a
restore could destroy data it did not contain.

Data loss:
- Restore with overwrite rmtree'd the whole stack directory, including
  folders the backup had left out (over 2 GiB, deselected, an NFS share
  mounted inside). It now clears only what the backup holds, keeps the
  excluded paths (recorded in the manifest) and never crosses into another
  filesystem.
- The stack was only stopped when named volumes were selected, so data
  folders inside the stack directory were archived while in use. Any
  selected data now stops it.
- Retention matched by prefix: pruning "app" deleted "app-db"'s backups.
  It now matches backup-<id>-(full|config)-<date>.tar.gz exactly.
- A restore under a new name wrote into the original's volumes when they
  had an explicit name, and into its folders outside the stack directory.
  Both are skipped now; a volume owned by another stack or mounted by a
  running container is never overwritten.
- A truncated archive was only noticed after the old stack was gone. The
  whole archive is verified first, and the stack directory is unpacked
  next to the old one and swapped in.

Remote storage:
- Bind folders on a mounted share (fstab/autofs), bind-type volumes
  pointing at one and shares mounted inside the stack folder are detected
  by filesystem type and not selected; a folder du cannot measure in 20s
  is no longer selected by default either.

Reliability:
- After a backup exactly the containers that ran are started again with
  compose start (falling back to the Docker API), instead of up -d, which
  started services stopped on purpose and applied undeployed edits. A
  stack that cannot be stopped is not backed up.
- Backup and restore hold the stack lock; a scheduled backup that finds
  the stack busy retries after 10 minutes. Restore stops the stack while
  replacing its data and brings it back up.
- external: true volumes are backed up; anonymous ones are listed, not
  selected. Restored volumes get their original driver and options.
- A backup with uncaptured data or a stack that did not restart is
  reported as incomplete: in the dialog, and for schedules as a warning
  with a notification and without rotating older backups out.
- Archive writing and restore run off the event loop; downloads delete
  their temp file; helper containers use the long-timeout client; S3
  listing is paginated; a file changing mid-read no longer corrupts the
  archive.

Manifest format 3 records compose_binds/excluded and volume
driver/options/external; format 1/2 backups still restore.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

backend/services/schedule_service.py

@@ -28,6 +28,10 @@ from services import (
 
 logger = logging.getLogger("stackpilot.schedule")
 
+#: A stack that is busy (deploying, updating) when its backup is due is tried
+#: again this much later instead of waiting for the next regular run.
+BUSY_RETRY = timedelta(minutes=10)
+
 
 def _now() -> datetime:
     return datetime.now(timezone.utc).replace(tzinfo=None)
@@ -57,11 +61,12 @@ def compute_next_run(
     return base
 
 
-def _prune(dest: BackupDestination, basename: str, keep: int) -> int:
+def _prune(dest: BackupDestination, stack_id: str, keep: int) -> int:
     if keep <= 0:
         return 0
     items = dest_service.list_backups(dest)
-    mine = [i for i in items if i["name"].startswith(f"{basename}-")]
+    pattern = backup_service.backup_name_pattern(stack_id)
+    mine = [i for i in items if pattern.match(i["name"])]
     mine.sort(key=lambda x: x.get("modified") or 0, reverse=True)
     removed = 0
     for old in mine[keep:]:
@@ -73,12 +78,28 @@ def _prune(dest: BackupDestination, basename: str, keep: int) -> int:
     return removed
 
 
+def _problems(report: dict) -> list[str]:
+    """What makes a backup that was written less than a success."""
+    problems = [f"not captured: {e}" for e in report.get("errors") or []]
+    if report.get("restart_error"):
+        problems.append(f"stack did not start again: {report['restart_error']}")
+    return problems
+
+
+async def _alert(session: Session, title: str, body: str) -> None:
+    try:
+        await notify_service.notify(EVENT_BACKUP_FAILED, title, body, session)
+    except Exception:  # noqa: BLE001
+        pass
+
+
 async def run_schedule(session: Session, schedule: BackupSchedule) -> dict:
     """Execute one schedule now. Updates status + next_run. Returns a summary."""
     now = _now()
     dest = session.get(BackupDestination, schedule.destination_id)
     result: dict = {"ok": False}
     path = None
+    next_run = None
     try:
         if not dest:
             raise RuntimeError(f"destination {schedule.destination_id} not found")
@@ -86,36 +107,50 @@ async def run_schedule(session: Session, schedule: BackupSchedule) -> dict:
         stack = session.get(Stack, schedule.stack_id)
         if not stack:
             raise RuntimeError(f"stack '{schedule.stack_id}' not found")
-        path = await backup_service.create_backup(
+        path, report = await backup_service.create_backup_ex(
             schedule.stack_id, stack.name,
             include_volumes=schedule.include_volumes, stop_first=schedule.stop_first,
         )
 
         filename = backup_service.backup_filename(schedule.stack_id, schedule.include_volumes)
-        basename = backup_service.backup_basename(schedule.stack_id)
         await asyncio.to_thread(dest_service.upload, dest, path, filename)
-        pruned = await asyncio.to_thread(_prune, dest, basename, schedule.keep)
-        schedule.last_status = "ok"
-        result = {"ok": True, "name": filename, "destination": dest.name, "pruned": pruned}
-        logger.info("Scheduled backup %s → %s ok (pruned %d)", schedule.stack_id, dest.name, pruned)
+        problems = _problems(report)
+        # Retention only after a complete backup: an incomplete one must not
+        # push the last good backup out.
+        pruned = 0 if problems else await asyncio.to_thread(
+            _prune, dest, schedule.stack_id, schedule.keep
+        )
+        result = {"ok": not problems, "name": filename, "destination": dest.name, "pruned": pruned}
+        if problems:
+            schedule.last_status = f"warning: {'; '.join(problems)}"[:300]
+            result["error"] = "; ".join(problems)
+            logger.warning("Scheduled backup %s incomplete: %s", schedule.stack_id, problems)
+            await _alert(
+                session,
+                f"Scheduled backup of '{schedule.stack_id}' is incomplete",
+                "\n".join(problems),
+            )
+        else:
+            schedule.last_status = "ok"
+            logger.info(
+                "Scheduled backup %s → %s ok (pruned %d)", schedule.stack_id, dest.name, pruned
+            )
+    except backup_service.StackBusyError as exc:
+        # Deploys and updates are short: try again soon rather than skipping a day.
+        schedule.last_status = f"retrying: {exc}"[:300]
+        result = {"ok": False, "error": str(exc)}
+        next_run = now + BUSY_RETRY
+        logger.info("Scheduled backup %s postponed: %s", schedule.stack_id, exc)
     except Exception as exc:  # noqa: BLE001
         schedule.last_status = f"error: {exc}"[:300]
         result = {"ok": False, "error": str(exc)}
         logger.warning("Scheduled backup %s failed: %s", schedule.stack_id, exc)
-        try:
-            await notify_service.notify(
-                EVENT_BACKUP_FAILED,
-                f"Scheduled backup of '{schedule.stack_id}' failed",
-                str(exc),
-                session,
-            )
-        except Exception:  # noqa: BLE001
-            pass
+        await _alert(session, f"Scheduled backup of '{schedule.stack_id}' failed", str(exc))
     finally:
         if path and os.path.exists(path):
             os.unlink(path)
         schedule.last_run = now
-        schedule.next_run = compute_next_run(
+        schedule.next_run = next_run or compute_next_run(
             schedule.frequency, schedule.hour, schedule.minute, schedule.weekday, now
         )
         session.add(schedule)