Retry uploads on (connection) errors

[blueyed]

Example: https://ci.appveyor.com/project/pytestbot/pytest/build/1.0.5141/job/ycpyxb9yvrckho21#L402 (RemoteDisconnected)

codecov-bash tries 3 times already, sleeping for 30s inbetween.

[blueyed]

Another example:

      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.0.15
==> Detecting CI provider
    AppVeyor Detected
    Fixing merge commit SHA
==> Preparing upload
    Error running `cd 'C:\projects\pytest' && git ls-files`: None
XX> Skip processing gcov
==> Collecting reports
XX> Searching for reports disabled
    Targeting specific files
    + coverage.xml bytes=1071184
==> Appending environment variables
    + TOXENV
==> Uploading
    .url https://codecov.io
    .query commit=3acfa3abdc7577f97bb40b9790c10d43c0e486e4&branch=master&job=pytestbot%2Fpytest%2F1.0.5975&pr=4409&service=appveyor&build=vku9wlsxtbjwwk0j&slug=pytest-dev%2Fpytest&flags=py27%2Cwindows&package=py2.0.15
    Pinging Codecov...
Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
Tip: See all example repositories: https://github.com/codecov?query=example
Support channels:
  Email:   [email protected]
  IRC:     #codecov
  Gitter:  https://gitter.im/codecov/support
  Twitter: @codecov

[blueyed]

Any update on this?

Another example (from AppVeyor):

==> Uploading
    .url https://codecov.io
    .query commit=7e9f35e6cba84c386d29fc45e28d60b64927f753&branch=master&job=pytestbot%2Fpytest%2F1.0.6650&pr=4799&service=appveyor&build=r1cx4er7tmmd98ul&slug=pytest-dev%2Fpytest&package=py2.0.15
    Pinging Codecov...
Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /upload/v4?commit=7e9f35e6cba84c386d29fc45e28d60b64927f753&branch=master&job=pytestbot%2Fpytest%2F1.0.6650&pr=4799&service=appveyor&build=r1cx4er7tmmd98ul&slug=pytest-dev%2Fpytest&package=py2.0.15 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x02DFC030>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))

Tip: See all example repositories: https://github.com/codecov?query=example
Support channels:
  Email:   [email protected]
  IRC:     #codecov
  Gitter:  https://gitter.im/codecov/support
  Twitter: @codecov

Sorry, we tried running command for 3 times and all attempts were unsuccessful!

[quantus]

I've also seen quite a few similar connection errors from our CircleCI builds. My current solution for the issue has been running codecov like this:

codecov --required || (sleep 5 && codecov --required) || (sleep 5 && codecov --required) || (sleep 5 && codecov --required) || (sleep 5 && codecov --required)

I know it isn't pretty but I haven't seen it failing yet.

[timandy]

I got the same issue Could add a parameter to specify the number of retries?

1748      _____          _
1749     / ____|        | |
1750    | |     ___   __| | ___  ___ _____   __
1751    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
1752    | |___| (_) | (_| |  __/ (_| (_) \ V /
1753     \_____\___/ \____|\___|\___\___/ \_/
1754                                    v2.0.15
1755
1756==> Detecting CI provider
1757    Travis Detected
1758==> Preparing upload
1759==> Processing gcov (disable by -X gcov)
1760    Executing gcov (find /home/travis/build/timandy/linq -not -path './bower_components/**' -not -path './node_modules/**' -not -path './vendor/**' -type f -name '*.gcno'  -exec gcov -pb  {} +)
1761==> Collecting reports
1762    + /home/travis/build/timandy/linq/target/site/jacoco/jacoco.xml bytes=1354762
1763==> Appending environment variables
1764    + TRAVIS_JDK_VERSION
1765    + TRAVIS_OS_NAME
1766==> Uploading
1767    .url https://codecov.io
1768    .query pr=false&service=travis&package=py2.0.15&job=573661452&build=230.1&branch=master&commit=f768b4f9b4b1cd2acdf5152d0f7be269a22793c0&slug=timandy%2Flinq
1769    Pinging Codecov...
1770    Uploading to S3...
1771Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /codecov/v4/raw/2019-08-19/BD907C23C2C05F757F59BF33F38DF67D/f768b4f9b4b1cd2acdf5152d0f7be269a22793c0/7d85c4bc-4456-43cd-9289-81a22b1c23b1.txt?AWSAccessKeyId=AKIAIHLZSCQCS4WIHD4A&Expires=1566195120&Signature=Hj%2BcQ2c4lj7zLAMWUejjidNN6DU%3D (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb360244390>: Failed to establish a new connection: [Errno 110] Connection timed out',))
1772
1773Tip: See an example jdk repo: https://github.com/codecov/example-jdk
1774Support channels:
1775  Email:   [email protected]
1776  IRC:     #codecov
1777  Gitter:  https://gitter.im/codecov/support
1778  Twitter: @codecov
1779
1780
1781

[ikostan]

Hi all. Same issue, using GitLab CI:

Uploading to S3... Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /codecov/v4/raw/2020-04-20/102BD5852C808ECE77423D85935DC840/fc50b3d848de669df5679d7f36fbd3b14e460e2d/88c1c006-6219-41d0-8a6e-8dc3709b6235.txt?AWSAccessKeyId=AKIAIHLZSCQCS4WIHD4A&Expires=1587375167&Signature=ycNO0ZmrV%2FVhr2jmGd3dIi7Juc4%3D (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b00bfb6d8>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Tip: See all example repositories: https://github.com/codecov?query=example Support channels: Email: [email protected] IRC: #codecov Gitter: https://gitter.im/codecov/support Twitter: @codecov

[jquast]

Dear maintainer(s): Would a PR to use a 3rd party "retryable" decorator, https://tenacity.readthedocs.io/en/latest/ be accepted?

[nmoinvaz]

I implemented a retry for requests here and retry number is configurable via args:

https://github.com/nmoinvaz/codecov-python/commit/9632dc655ed378ece67bf54861a300dd1e732aef https://github.com/nmoinvaz/codecov-python/commit/30ff5e3140291b1a11bea9f2973efae72d8dd2aa

If this is something that would be accepted I can submit a PR.

[xoxys]

Looks like this is a long-running issue. What about the solution from @nmoinvaz? Could it be added or not? What is needed to finally get a solution into codecov-python?

[blueyed]

Retries have been added in https://github.com/codecov/codecov-python/commit/07583f324c194d05ff32bc7ba20cc34cd1e12cc0 by @thomasrockhu, thanks!

However I've still seen this now:

      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.1.9
==> Detecting CI provider
    Gitlab CI Detected
==> Preparing upload
XX> Skip processing gcov
==> Collecting reports
    Targeting specific files
    + coverage.xml bytes=1272194
==> Appending environment variables
    + TOXENV
==> Uploading
    .url https://codecov.io
    .query commit=XXX
    Gzipping contents..
    Compressed contents to 86377 bytes
    Pinging Codecov...
Error: HTTPSConnectionPool(host='codecov.io', port=443): Max retries exceeded with url: /upload/v4?commit=XXX (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000026EC415FE48>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

From this it is not clear if there have been any retries at all (should get logged probably), and it might have worked better with a delay between retries (#293, #295).

[nmoinvaz]

codecov-bash tries 3 times already, sleeping for 30s inbetween.

Looking at codecov bash uploader it appears to now try 5 times, sleeping for 2s in between.

[nmoinvaz]

I believe this issue is resolved now. It does delay between retries and also displays each time it retries.