Skip to content

fix: read-idle timeout on SSE so a stalled stream can't hang events()#12

Merged
wei-hai merged 2 commits into
mainfrom
fix/sse-idle-timeout
Jul 22, 2026
Merged

fix: read-idle timeout on SSE so a stalled stream can't hang events()#12
wei-hai merged 2 commits into
mainfrom
fix/sse-idle-timeout

Conversation

@wei-hai

@wei-hai wei-hai commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Found running a real >10-minute job against prod: the SSE stream went silent at ~3 min (no frames, no keepalive, no close — a "zombie" connection), and job.events() hung — still blocked minutes after the job had already completed. poll/wait/run were unaffected (the job itself finished fine).

Root cause

ComfyLow.get_job_events (sync + async) opened the SSE stream with timeout=None, which disables httpx's read timeout — so iter_lines() blocks forever on a stalled-but-open connection. The high-level Job.events() reconnect/poll-fallback only runs when the stream ends (the generator returns or raises), which never happens for a zombie → events() hangs.

Fix

Give get_job_events a read-idle timeout (_SSE_IDLE_TIMEOUT, well above the server's keepalive interval) by default. A stalled stream now raises httpx.ReadTimeout, which Job.events() already catches and turns into a poll (terminal?) / reconnect. timeout=None still available to opt out.

Tests

Added a threaded stub stall mode (sends a couple frames, then holds the socket open and silent) and a regression test asserting events() recovers via the poll fallback in ~0.3s instead of waiting for close / hanging. Full suite: 74 passed; ruff/format/mypy/hygiene clean.

🤖 Generated with Claude Code

The live-events SSE reader used timeout=None, so a connection that goes
silent-but-open (no frames, no keepalive, no close — e.g. a gateway idle
timeout on a long job) blocked iter_lines() forever. The high-level events()
reconnect/poll-fallback only runs when the stream *ends*, so it never
recovered — events() hung indefinitely on jobs longer than a few minutes,
even though the job itself completed fine (poll/wait/run stayed correct).

Give get_job_events (sync + async) a read-idle timeout well above the
server's keepalive interval: a stalled stream now raises ReadTimeout, which
events() already catches and turns into a poll/reconnect. Adds a threaded
stub "stall" mode + a regression test proving recovery instead of a hang.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@wei-hai
wei-hai requested review from a team as code owners July 22, 2026 04:49
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@wei-hai, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 48 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 351b356a-1330-45c7-b95b-8132191b9c1f

📥 Commits

Reviewing files that changed from the base of the PR and between 157765b and 8d217ec.

📒 Files selected for processing (1)
  • tests/test_sse_idle.py

Warning

.coderabbit.yaml has a parsing error

The CodeRabbit configuration file in this repository has a parsing error and default settings were used instead. Please fix the error(s) in the configuration file. You can initialize chat with CodeRabbit to get help with the configuration file.

💥 Parsing errors (1)
Validation error: Too big: expected string to have <=250 characters at "tone_instructions"
⚙️ Configuration instructions
  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
📝 Walkthrough

Walkthrough

Changes

SSE idle-timeout recovery

Layer / File(s) Summary
SSE timeout integration
src/comfy_low/transport.py
Sync and async get_job_events methods now apply _SSE_IDLE_TIMEOUT by default, while timeout=None remains an explicit opt-out.
Stalled SSE fixture
tests/conftest.py
The threaded test server supports stalled SSE connections that emit non-terminal frames and delay before returning.
Recovery regression test
tests/test_sse_idle.py
The test verifies timeout-driven recovery, progress delivery, successful terminal status, and bounded completion time.
🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/sse-idle-timeout
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch fix/sse-idle-timeout

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_sse_idle.py`:
- Around line 19-39: Extend test_events_recovers_from_zombie_stream coverage to
verify the async get_job_events path also recovers from a stalled SSE stream and
reaches the terminal StatusChange without hanging. Add a case using timeout=None
to confirm the explicit no-timeout contract, while preserving the existing
default-timeout recovery assertion and terminal-status checks.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a37df146-b945-4f32-a970-4e49905d5d6c

📥 Commits

Reviewing files that changed from the base of the PR and between e53affd and 157765b.

📒 Files selected for processing (3)
  • src/comfy_low/transport.py
  • tests/conftest.py
  • tests/test_sse_idle.py

Comment thread tests/test_sse_idle.py
Extend the zombie-stream regression to the async get_job_events path (same
idle timeout + poll fallback) and add a test proving timeout=None disables
the idle timeout (rides out the stall until close instead of raising).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@wei-hai
wei-hai merged commit c02a255 into main Jul 22, 2026
10 checks passed
@wei-hai
wei-hai deleted the fix/sse-idle-timeout branch July 22, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant