DEV Community

Ace Meridian
Ace Meridian

Posted on Originally published at Medium

OpenAI timeout=60 Never Fired on a Stream With Keep-Alives

The openai Python SDK's timeout=60 limits each socket read, not the request, so a streamed completion that receives an SSE keep-alive comment every 15 seconds never times out, even though your loop never sees an event. Put a wall clock around the whole attempt instead: with AsyncOpenAI, asyncio.timeout(30) ended the same stalled stream at 30.0 to 30.1 seconds in every run.

This story was written with the assistance of an AI writing program. The numbers come from openai 2.43.0 and httpx 0.28.1 on Python 3.14.4, run against a local fake OpenAI-compatible endpoint over plain HTTP and TLS. I did not test OpenAI's own API, and I do not know whether it sends comment lines.

The Server Sends One Token, Then Only Heartbeats

The fake endpoint answers POST /v1/chat/completions with text/event-stream, sends one normal data: chunk containing "Hello", and then stops producing tokens. In the silent variant it sends nothing more and holds the connection open. In the heartbeat variant it writes : keep-alive every 15 seconds and never sends [DONE].

The client copies the settings of a real pipeline I reviewed this week: OpenAI(timeout=60.0, max_retries=0), stream=True, and a for chunk in stream loop. Each scenario ran as its own process under a 150-second timeout command, so exit code 124 means it was still blocked when the cap hit.

Silence Times Out at 60 Seconds; Heartbeats Never Do

Result lines from one plain-HTTP run, all scenarios in parallel:

{"result": "exception", "type": "httpx.ReadTimeout", "msg": "timed out", "is_openai_APITimeoutError": false, "t": 60.6, "scenario": "silent", "tls": false}
{"scenario": "comments", "exit": 124}
{"result": "exception", "type": "builtins.TimeoutError", "msg": "", "t": 30.1, "scenario": "comments-async", "tls": false}
Enter fullscreen mode Exit fullscreen mode

The silent stream failed at 60.4 to 60.6 seconds across three runs, as expected. The heartbeat stream was still waiting at 150 seconds in all three, including the TLS run. Nothing in the loop body ran after the first chunk.

Look at the type in the silent case: httpx.ReadTimeout, not openai.APITimeoutError. The SDK converts timeouts into its own exception while it sends the request; once you are iterating, the raw httpx exception comes through. An except openai.APITimeoutError around a stream misses it.

Why the Loop Sees Nothing but the Timer Resets

Two pieces combine. First, the SDK's SSE decoder discards comment lines: openai/_streaming.py has if line.startswith(":"): return None, which is correct for the format, so your code never learns that bytes arrived. Second, timeout=60 becomes httpx.Timeout(60), which sets the same 60 seconds for connect, read, write and pool acquisition. None of the four is a total. The read limit restarts on every successful read, and each comment is one.

So the 60 seconds measure silence on the socket, not progress on your completion. A heartbeat every 15 seconds means the socket is never silent for 60.

I hit the same shape with requests last week, when a server dripping one byte every three seconds kept timeout=5 busy for 57 seconds. The difference here is that the bytes are invisible: no partial content, no log line, nothing that tells you the call is still alive.

Who Actually Sends Comment Lines

This is not exotic. The HTML Standard's section on server-sent events suggests sending a comment line every 15 seconds or so, because some legacy proxies drop connections that go quiet. OpenRouter's streaming docs say it occasionally sends comments such as : OPENROUTER PROCESSING to prevent connection timeouts. Any gateway, proxy or self-hosted server that follows that advice turns a per-read timeout into no timeout while the model is stuck.

A Wall-Clock Budget Ended It at 30 Seconds

With AsyncOpenAI, I put asyncio.timeout around the request and the iteration together. This is the exact function I ran this morning, pointed at the fake endpoint through OPENAI_BASE_URL:

import asyncio
import openai
client = openai.AsyncOpenAI(timeout=60.0, max_retries=0)
async def ask(prompt: str, budget_s: float = 30.0) -> str:
    parts = []
    async with asyncio.timeout(budget_s):  # wall clock for the whole attempt
        stream = await client.chat.completions.create(
            model="gpt-4o-mini", stream=True,
            messages=[{"role": "user", "content": prompt}])
        async for chunk in stream:
            if chunk.choices and chunk.choices[0].delta.content:
                parts.append(chunk.choices[0].delta.content)
    return "".join(parts)
Enter fullscreen mode Exit fullscreen mode

It raised TimeoutError at 30.0 seconds in both runs this morning; an earlier version of the test ended at 30.0 to 30.1 seconds in three runs. I also checked the socket. Right after the timeout, ss showed no established connection from the client, and the server's next keep-alive write, at 45 seconds, failed with BrokenPipeError. Wrapping the loop in async with stream: made no difference in the same two runs; the cancellation had already closed the connection.

In Sync Code, close() Was 15 Seconds Late

Sync code has no cancellation, so I tried a threading.Timer that fires at 30 seconds. Calling stream.close() from the timer did not interrupt the blocked read. The loop only failed when the next keep-alive arrived, with httpx.ReadError: [Errno 9] Bad file descriptor at 45.4 to 45.5 seconds. So a 30-second deadline ended at 45 seconds. I did not test close() against a silent stream.

Shutting down the socket worked. The timer took the raw socket from stream.response.extensions["network_stream"].get_extra_info("socket") and called shutdown(socket.SHUT_RDWR), and the loop ended at 30.4 to 30.5 seconds over plain HTTP and TLS. That reaches into a low-level httpx response extension, so I would rather move the call into async code or a worker process that can be killed.

Takeaways

  • Treat timeout= on a streamed call as an idle limit. It does not bound how long the attempt runs.
  • Put one wall-clock budget around the request and the iteration together: asyncio.timeout() with AsyncOpenAI.
  • In sync code, stream.close() from another thread waits for the next read. Shut the socket down instead.
  • Around iteration, catch httpx.TimeoutException as well as openai.APITimeoutError.
  • Test against a server that sends one chunk and then only : keep-alive. Mine is 77 lines of standard-library Python, and one run shows whether your deadline holds.

If you run a gateway in front of a model API, does it send comment lines while the model thinks?

Top comments (5)

Collapse
 
recallrundev profile image
RecallRun •

The httpx.ReadTimeout vs openai.APITimeoutError detail is the one I'd have missed. I just wrote up the SDK's error handling and the except chain there catches openai.APITimeoutError, which by your runs lets a mid-stream stall straight through. For streaming calls that branch needs httpx.TimeoutException as well.

Did you try a per-chunk deadline (reset a timer on every yielded chunk) instead of one wall clock for the whole attempt? A 30 s total cap would also kill a healthy long generation, while "no data chunk for N seconds" only fires on the stall you describe. Keep-alive comments wouldn't reset it either, since the decoder drops them before your loop sees anything.

Collapse
 
aiapikey profile image
于侃 •

Agreed, and the distinction that matters is which layer the timer watches. A byte-level idle timer is reset by keep-alive comments; an event-level idle timer, reset on each parsed data event, is not, because those comments never become events. They answer different questions: transport silence versus model silence. You want both, plus a total cap that is a product decision per endpoint rather than a safety net.

That is also why the except chain matters. openai.APITimeoutError wraps the transport case, so a stall that only trips the event-level timer surfaces as whatever your code raises - and if nothing raises, the stream ends "successfully" with no content, which is the version that reaches users.

Per-chunk is the better primitive. The wall clock is the one I would keep only for ending the request, not for deciding it is broken.

Collapse
 
acemeridian profile image
Ace Meridian •

I hadn't, so I ran it today (openai 2.43.0, httpx 0.28.1, Python 3.14, local fake endpoint), with an idle timer re-armed on every parsed chunk:

async def ask_idle(prompt: str, idle_s: float = 10.0) -> str:
    parts = []
    loop = asyncio.get_running_loop()
    async with asyncio.timeout(idle_s) as cm:  # re-armed on every data chunk
        stream = await client.chat.completions.create(
            model=MODEL, stream=True,
            messages=[{"role": "user", "content": prompt}])
        async with stream:
            async for chunk in stream:
                cm.reschedule(loop.time() + idle_s)
                if chunk.choices and chunk.choices[0].delta.content:
                    parts.append(chunk.choices[0].delta.content)
    return "".join(parts)
Enter fullscreen mode Exit fullscreen mode
  • One data chunk, then : keep-alive every 5 s, idle_s=10: TimeoutError at 10.4 s, 10 s after the data chunk. The keep-alive at +5 s did not re-arm it, as you said.
  • A slow but healthy stream, one data chunk every 4 s for 11 chunks: the idle timer let it finish, all 11 at 40.4 s.
  • The post's 30 s wall clock on that same stream: TimeoutError at 30.0 s, after 8 of the 11 chunks.

So the total cap does kill a healthy long generation. I'd let the per-chunk timer decide that a stream is broken and keep a total cap only as a much larger product limit. One caveat: this timer also covers the wait for the first chunk, so idle_s has to sit above the slowest time-to-first-token you expect. And it raises the builtin TimeoutError, which except openai.APITimeoutError doesn't catch.

Collapse
 
aiapikey profile image
于侃 •

That is a better answer than mine - you measured all three arms, and the third is the one that settles it: the 30 s wall clock dropped a healthy stream at 8 of 11 chunks, so it destroyed roughly a quarter of a generation that was going to succeed. The keep-alive arm is the other finding worth keeping: it fired at 10.4 s, ten seconds after the data chunk, so the comments genuinely never re-armed anything.

One implication for the except chain: with cm.reschedule the stall surfaces as TimeoutError from the outer timeout context, not APITimeoutError. A handler that only catches the SDK error lets it escape as unhandled, which in a streaming endpoint usually means the response closes without a terminator and the client sees a truncated success instead of an error.

Collapse
 
aiapikey profile image
于侃 •

That is the classic gap between a read timeout and a deadline. Keep-alive frames are bytes arriving, so every frame resets the read clock - with a 10s keep-alive and timeout=60, the timeout can never fire, and a wedged stream keeps looking like a busy one.

Two budget numbers worth separating. An idle budget, set shorter than the keep-alive interval, so true silence is caught before the next frame would have masked it. And a total budget for the whole stream, which is the only thing that bounds a stream that keeps saying "still here".

The second is what protects your caller's latency, because it does not care how healthy the stream claims to be.