0.37.2: stream folder zip-downloads to fix 504 on large folders

A big folder hit a 504 Gateway Timeout: the zip was built into a temp
file *before* any response was sent, so for large folders the backend
stayed silent past nginx's proxy_read_timeout.

Now the zip is streamed as it's built, end to end:
- file_service.open_archive() returns (filename, byte iterator); _iter_zip
  walks the dir and yields zip bytes incrementally via a small drain
  buffer, writing each file in 1 MiB chunks (bounded memory, valid CRCs).
  Same hardening as before — only real regular files; FIFOs/sockets/
  devices/symlinks skipped without open(); per-file read errors skipped.
- /api/files/download and /agent/files/download return a StreamingResponse
  (no temp file). The agent proxy streams the agent response straight
  through (agent_service.stream_download), pulling the first chunk eagerly
  so an offline/bad-token agent still yields a clean status before 200.
- Files page: streamed downloads have no Content-Length, so the progress
  bar shows the running downloaded byte count ("Downloading … 12.3 MB")
  instead of a percentage, after the initial "Preparing …".

Verified end to end via TestClient (200, application/zip, valid zip,
2 MiB file intact, FIFO skipped, no hang).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
menzelj
2026-06-21 20:28:49 +00:00
co-authored by Claude Opus 4.8
parent 0dc430bb2a
commit 5ac9f15de4
8 changed files with 149 additions and 66 deletions
+29
View File
@@ -137,6 +137,35 @@ async def download_to_file(
raise AgentError(502, "agent_unreachable", str(exc)) from exc
async def stream_download(
session: Session,
agent: Agent,
path: str,
*,
params: Optional[dict] = None,
):
"""Stream a GET from the agent straight through, yielding chunks.
Unlike :func:`download_to_file` this never buffers to disk, so a large
response (e.g. a folder zip the agent builds on the fly) starts flowing to
the browser immediately instead of being staged first.
"""
url = agent.url.rstrip("/") + path
headers = {"Authorization": f"Bearer {agent.token}"}
try:
async with httpx.AsyncClient(follow_redirects=True) as client:
async with client.stream("GET", url, headers=headers, params=params, timeout=None) as resp:
if resp.status_code >= 400:
text = (await resp.aread()).decode("utf-8", "replace")
_handle_status(session, agent, resp.status_code, text)
_handle_status(session, agent, resp.status_code)
async for chunk in resp.aiter_bytes(1024 * 256):
yield chunk
except httpx.HTTPError as exc:
_mark(session, agent, "offline")
raise AgentError(502, "agent_unreachable", str(exc)) from exc
async def upload_file(
session: Session,
agent: Agent,