0.37.2: stream folder zip-downloads to fix 504 on large folders

A big folder hit a 504 Gateway Timeout: the zip was built into a temp
file *before* any response was sent, so for large folders the backend
stayed silent past nginx's proxy_read_timeout.

Now the zip is streamed as it's built, end to end:
- file_service.open_archive() returns (filename, byte iterator); _iter_zip
  walks the dir and yields zip bytes incrementally via a small drain
  buffer, writing each file in 1 MiB chunks (bounded memory, valid CRCs).
  Same hardening as before — only real regular files; FIFOs/sockets/
  devices/symlinks skipped without open(); per-file read errors skipped.
- /api/files/download and /agent/files/download return a StreamingResponse
  (no temp file). The agent proxy streams the agent response straight
  through (agent_service.stream_download), pulling the first chunk eagerly
  so an offline/bad-token agent still yields a clean status before 200.
- Files page: streamed downloads have no Content-Length, so the progress
  bar shows the running downloaded byte count ("Downloading … 12.3 MB")
  instead of a percentage, after the initial "Preparing …".

Verified end to end via TestClient (200, application/zip, valid zip,
2 MiB file intact, FIFO skipped, no hang).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
menzelj
2026-06-21 20:28:49 +00:00
co-authored by Claude Opus 4.8
parent 0dc430bb2a
commit 5ac9f15de4
8 changed files with 149 additions and 66 deletions
+13 -6
View File
@@ -19,8 +19,7 @@ from fastapi import (
Request,
UploadFile,
)
from fastapi.responses import FileResponse
from starlette.background import BackgroundTask
from fastapi.responses import FileResponse, StreamingResponse
from pydantic import BaseModel
from sqlmodel import Session
@@ -32,6 +31,12 @@ from services import audit_service, device_service, file_service
router = APIRouter(prefix="/api/files", tags=["files"])
def _attachment(filename: str) -> str:
"""A safe ``Content-Disposition`` value for an arbitrary filename."""
safe = filename.replace("\\", "_").replace('"', "_")
return f'attachment; filename="{safe}"'
def _ip(request: Request) -> str:
return request.client.host if request.client else "unknown"
@@ -71,10 +76,12 @@ def download(
_user: User = Depends(get_current_user),
):
if _guard(file_service.is_dir, path):
tmp, filename = _guard(file_service.archive_dir, path)
return FileResponse(
tmp, filename=filename, media_type="application/zip",
background=BackgroundTask(os.unlink, tmp),
filename, chunks = _guard(file_service.open_archive, path)
# Stream the zip as it's built so the response starts immediately
# (large folders no longer hit the proxy's read timeout).
return StreamingResponse(
chunks, media_type="application/zip",
headers={"Content-Disposition": _attachment(filename)},
)
real, filename = _guard(file_service.resolve_download, path)
return FileResponse(real, filename=filename, media_type="application/octet-stream")