0.37.2: stream folder zip-downloads to fix 504 on large folders
A big folder hit a 504 Gateway Timeout: the zip was built into a temp
file *before* any response was sent, so for large folders the backend
stayed silent past nginx's proxy_read_timeout.
Now the zip is streamed as it's built, end to end:
- file_service.open_archive() returns (filename, byte iterator); _iter_zip
walks the dir and yields zip bytes incrementally via a small drain
buffer, writing each file in 1 MiB chunks (bounded memory, valid CRCs).
Same hardening as before — only real regular files; FIFOs/sockets/
devices/symlinks skipped without open(); per-file read errors skipped.
- /api/files/download and /agent/files/download return a StreamingResponse
(no temp file). The agent proxy streams the agent response straight
through (agent_service.stream_download), pulling the first chunk eagerly
so an offline/bad-token agent still yields a clean status before 200.
- Files page: streamed downloads have no Content-Length, so the progress
bar shows the running downloaded byte count ("Downloading … 12.3 MB")
instead of a percentage, after the initial "Preparing …".
Verified end to end via TestClient (200, application/zip, valid zip,
2 MiB file intact, FIFO skipped, no hang).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
0dc430bb2a
commit
5ac9f15de4
@@ -137,6 +137,35 @@ async def download_to_file(
|
||||
raise AgentError(502, "agent_unreachable", str(exc)) from exc
|
||||
|
||||
|
||||
async def stream_download(
|
||||
session: Session,
|
||||
agent: Agent,
|
||||
path: str,
|
||||
*,
|
||||
params: Optional[dict] = None,
|
||||
):
|
||||
"""Stream a GET from the agent straight through, yielding chunks.
|
||||
|
||||
Unlike :func:`download_to_file` this never buffers to disk, so a large
|
||||
response (e.g. a folder zip the agent builds on the fly) starts flowing to
|
||||
the browser immediately instead of being staged first.
|
||||
"""
|
||||
url = agent.url.rstrip("/") + path
|
||||
headers = {"Authorization": f"Bearer {agent.token}"}
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True) as client:
|
||||
async with client.stream("GET", url, headers=headers, params=params, timeout=None) as resp:
|
||||
if resp.status_code >= 400:
|
||||
text = (await resp.aread()).decode("utf-8", "replace")
|
||||
_handle_status(session, agent, resp.status_code, text)
|
||||
_handle_status(session, agent, resp.status_code)
|
||||
async for chunk in resp.aiter_bytes(1024 * 256):
|
||||
yield chunk
|
||||
except httpx.HTTPError as exc:
|
||||
_mark(session, agent, "offline")
|
||||
raise AgentError(502, "agent_unreachable", str(exc)) from exc
|
||||
|
||||
|
||||
async def upload_file(
|
||||
session: Session,
|
||||
agent: Agent,
|
||||
|
||||
Reference in New Issue
Block a user