datahub icon indicating copy to clipboard operation
datahub copied to clipboard

fix(actions): increase retries

Open chakru-r opened this issue 4 weeks ago • 3 comments

When there are transient kafka errors, the consumer on the externalEventsSource need to reconnect -- and there is a pool of such consumers that get allocated LIFO thus potentially requiring upto 10 retries to get to a consumer that has gone through a reconnect.

chakru-r avatar Nov 26 '25 13:11 chakru-r

✅ Meticulous spotted 0 visual differences across 998 screens tested: view results.

Meticulous evaluated ~8 hours of user flows against your PR.

Expected differences? Click here. Last updated for commit 92fbcb7. This comment will update as new commits are pushed.

alwaysmeticulous[bot] avatar Nov 26 '25 13:11 alwaysmeticulous[bot]

:x: 4 Tests Failed:

Tests completed Failed Passed Skipped
5400 4 5396 34
View the top 3 failed test(s) by shortest run time
tests.unit.plugin.source.acryl.test_datahub_cloud_events_consumer::test_poll_events_timeout
Stack Traces | 544s run time
mock_graph = <MagicMock spec='DataHubGraph' id='139731538278336'>

    def test_poll_events_timeout(mock_graph: DataHubGraph) -> None:
        """
        Test that poll_events retries and raises a Timeout if requests.get times out.
        """
        consumer = DataHubEventsConsumer(
            graph=mock_graph,
            consumer_id="test-consumer",
            offset_id="initial-offset",
        )
    
        with patch("requests.get", side_effect=Timeout("Request Timeout")) as mock_get:
            with pytest.raises(Timeout):
                consumer.poll_events(topic="TestTopic")
    
            # requests.get should be called multiple times due to retry
>           assert mock_get.call_count == 3
E           AssertionError: assert 15 == 3
E            +  where 15 = <MagicMock name='get' id='139731533288624'>.call_count

.../source/acryl/test_datahub_cloud_events_consumer.py:266: AssertionError
tests.unit.plugin.source.acryl.test_datahub_cloud_events_consumer::test_poll_events_http_error
Stack Traces | 544s run time
mock_graph = <MagicMock spec='DataHubGraph' id='139731538730416'>

    def test_poll_events_http_error(mock_graph: DataHubGraph) -> None:
        """
        Test that poll_events retries and raises an HTTPError if requests.get fails.
        """
        consumer = DataHubEventsConsumer(
            graph=mock_graph,
            consumer_id="test-consumer",
            offset_id="initial-offset",
        )
        dummy_response = Response()  # Create a dummy Response object
        with patch(
            "requests.get", side_effect=HTTPError(response=dummy_response)
        ) as mock_get:
            # The default Tenacity stop_after_attempt=3
            with pytest.raises(HTTPError):
                consumer.poll_events(topic="TestTopic")
    
            # requests.get should be called multiple times due to retry
>           assert mock_get.call_count == 3
E           AssertionError: assert 15 == 3
E            +  where 15 = <MagicMock name='get' id='139731526764192'>.call_count

.../source/acryl/test_datahub_cloud_events_consumer.py:208: AssertionError
tests.unit.plugin.source.acryl.test_datahub_cloud_events_consumer::test_poll_events_chunked_encoding_error
Stack Traces | 545s run time
mock_graph = <MagicMock spec='DataHubGraph' id='139731536255856'>

    def test_poll_events_chunked_encoding_error(mock_graph: DataHubGraph) -> None:
        """
        Test that poll_events retries and raises a ChunkedEncodingError if requests.get fails.
        """
        consumer = DataHubEventsConsumer(
            graph=mock_graph,
            consumer_id="test-consumer",
            offset_id="initial-offset",
        )
    
        with patch(
            "requests.get", side_effect=ChunkedEncodingError("Chunked Encoding Error")
        ) as mock_get:
            with pytest.raises(ChunkedEncodingError):
                consumer.poll_events(topic="TestTopic")
    
            # requests.get should be called multiple times due to retry
>           assert mock_get.call_count == 3
E           AssertionError: assert 15 == 3
E            +  where 15 = <MagicMock name='get' id='139731534724544'>.call_count

.../source/acryl/test_datahub_cloud_events_consumer.py:248: AssertionError
tests.unit.plugin.source.acryl.test_datahub_cloud_events_consumer::test_poll_events_connection_error
Stack Traces | 545s run time
mock_graph = <MagicMock spec='DataHubGraph' id='139731535050688'>

    def test_poll_events_connection_error(mock_graph: DataHubGraph) -> None:
        """
        Test that poll_events retries and raises a ConnectionError if requests.get fails.
        """
        consumer = DataHubEventsConsumer(
            graph=mock_graph,
            consumer_id="test-consumer",
            offset_id="initial-offset",
        )
    
        with patch(
            "requests.get", side_effect=ConnectionError("Connection Error")
        ) as mock_get:
            with pytest.raises(ConnectionError):
                consumer.poll_events(topic="TestTopic")
    
            # requests.get should be called multiple times due to retry
>           assert mock_get.call_count == 3
E           AssertionError: assert 15 == 3
E            +  where 15 = <MagicMock name='get' id='139731533290160'>.call_count

.../source/acryl/test_datahub_cloud_events_consumer.py:228: AssertionError

To view more test analytics, go to the Test Analytics Dashboard 📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

codecov[bot] avatar Nov 26 '25 13:11 codecov[bot]

Bundle Report

Bundle size has no change :white_check_mark:

codecov[bot] avatar Nov 26 '25 13:11 codecov[bot]