Skip to content

Retry GitHub calls that drop the connection in the Spot interruption notices - #75829

Open
potiuk wants to merge 1 commit into
apache:mainfrom
potiuk:spot-notices-retry-dropped-connections
Open

potiuk wants to merge 1 commit into
apache:mainfrom
potiuk:spot-notices-retry-dropped-connections

Conversation

@potiuk

@potiuk potiuk commented Oct 27, 2026

Copy link
Copy Markdown
Member

The scheduled Spot interruption notice run failed when GitHub dropped a connection mid-request: https://github.com/apache/airflow/actions/runs/40590296911/job/122492480657

Could not handle run https://github.com/apache/airflow/actions/runs/40178639592: stream error: stream ID 3; CANCEL; received from peer

Only (HTTP 5xx) counted as transient, so this HTTP/2 stream reset was neither retried nor skipped with a warning like a server error, and the run exited 1 although the next scheduled run does the same work again. Dropped connections (stream resets, connection resets, unexpected EOF, TLS handshake and I/O timeouts) now count as transient too. A POST is still not retried, since one that lost its connection may have created the comment.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Claude Opus 5)

Generated-by: Claude Code (Claude Opus 5) following the guidelines

🤖 Generated with Claude Code

…notices

Only server errors counted as transient, so a connection GitHub dropped
mid-request (an HTTP/2 stream reset) was neither retried nor skipped like
one, and failed the scheduled run although the next run would do the same
work again.

Generated-by: Claude Opus 5
[
pytest.param(("repos/x",), [SERVER_ERROR, SERVER_ERROR], 3, None, id="retries-server-errors"),
pytest.param(("repos/x",), [SERVER_ERROR] * 3, 3, SERVER_ERROR, id="gives-up"),
pytest.param(("repos/x",), [STREAM_ERROR], 2, None, id="retries-dropped-connections"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like only stream error is covered here. How about adding a test for is_transient like:

@pytest.mark.parametrize(
    ("stderr", "expected"),
    [
        pytest.param("gh: Server Error (HTTP 502)\n", True, id="server-error"),
        pytest.param("stream error: stream ID 3; CANCEL; received from peer\n", True, id="stream-reset"),
        pytest.param('Get "https://api.github.com/repos/x": read tcp …: read: connection reset by peer\n', True, id="connection-reset"),
        pytest.param('Get "https://api.github.com/repos/x": unexpected EOF\n', True, id="unexpected-eof"),
        pytest.param('Get "https://api.github.com/repos/x": net/http: TLS handshake timeout\n', True, id="tls-timeout"),
        pytest.param('Get "https://api.github.com/repos/x": dial tcp 140.82.112.6:443: i/o timeout\n', True, id="dial-timeout"),
        pytest.param("gh: Not Found (HTTP 404)\n", False, id="client-error"),
    ],
)
def test_is_transient(spot, stderr, expected):
    assert spot.is_transient(subprocess.CalledProcessError(1, ["gh"], stderr=stderr)) is expected

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants