0.37.2: stream folder zip-downloads to fix 504 on large folders

A big folder hit a 504 Gateway Timeout: the zip was built into a temp
file *before* any response was sent, so for large folders the backend
stayed silent past nginx's proxy_read_timeout.

Now the zip is streamed as it's built, end to end:
- file_service.open_archive() returns (filename, byte iterator); _iter_zip
  walks the dir and yields zip bytes incrementally via a small drain
  buffer, writing each file in 1 MiB chunks (bounded memory, valid CRCs).
  Same hardening as before — only real regular files; FIFOs/sockets/
  devices/symlinks skipped without open(); per-file read errors skipped.
- /api/files/download and /agent/files/download return a StreamingResponse
  (no temp file). The agent proxy streams the agent response straight
  through (agent_service.stream_download), pulling the first chunk eagerly
  so an offline/bad-token agent still yields a clean status before 200.
- Files page: streamed downloads have no Content-Length, so the progress
  bar shows the running downloaded byte count ("Downloading … 12.3 MB")
  instead of a percentage, after the initial "Preparing …".

Verified end to end via TestClient (200, application/zip, valid zip,
2 MiB file intact, FIFO skipped, no hang).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
menzelj
2026-06-21 20:28:49 +00:00
co-authored by Claude Opus 4.8
parent 0dc430bb2a
commit 5ac9f15de4
8 changed files with 149 additions and 66 deletions
+6 -6
View File
@@ -32,8 +32,7 @@ from fastapi import (
WebSocket,
WebSocketDisconnect,
)
from fastapi.responses import FileResponse, JSONResponse
from starlette.background import BackgroundTask
from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
from pydantic import BaseModel
from config import settings
@@ -668,10 +667,11 @@ def files_read(path: str = Query(...)) -> dict:
@app.get("/agent/files/download", dependencies=[Depends(verify_token)])
def files_download(path: str = Query(...)):
if _file_guard(file_service.is_dir, path):
tmp, filename = _file_guard(file_service.archive_dir, path)
return FileResponse(
tmp, filename=filename, media_type="application/zip",
background=BackgroundTask(os.unlink, tmp),
filename, chunks = _file_guard(file_service.open_archive, path)
# Stream the zip as it's built (no temp file, starts immediately).
return StreamingResponse(
chunks, media_type="application/zip",
headers={"Content-Disposition": f'attachment; filename="{filename}"'},
)
real, filename = _file_guard(file_service.resolve_download, path)
return FileResponse(real, filename=filename, media_type="application/octet-stream")