feedback
feedback copied to clipboard
codecov
Problem uploading reports to docker-based single-instance codecov install
Describe the bug I am running codecov self-hosted on a single EC2 instance using docker-compose.yml. Logged in via Github and set up the test repo per tutorial. However, trying to upload reports fails.
Environment (please complete the following information):)
- System: codecov on EC2, CI is Github
CI log
/home/runner/work/_actions/codecov/codecov-action/v4.2.0/dist/codecov --enterprise-url https://codecov.bbbb.ccc/ -v do-upload -Z --git-service github
info - 2024-05-14 18:58:33,195 -- ci service found: github-actions
debug - 2024-05-14 18:58:33,198 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-05-14 18:58:33,201 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
warning - 2024-05-14 18:58:33,203 -- No config file could be found. Ignoring config.
debug - 2024-05-14 18:58:33,203 -- No codecov_yaml found
debug - 2024-05-14 18:58:33,205 -- Starting upload processing --- {"branch": "step2", "build_code": "9084726456", "build_url": "https://github.com/aaaaa/testrepo/actions/runs/9084726456", "commit_sha": "1d711a6f6b0a141f1499341aed98469db16bf83d", "disable_file_fixes": false, "disable_search": false, "enterprise_url": "https://codecov.bbbb.ccc/", "env_vars": {}, "files_search_exclude_folders": [], "files_search_explicitly_listed_files": [], "files_search_root_folder": "/home/runner/work/testrepo/testrepo", "flags": [], "git_service": "github", "handle_no_reports_found": false, "job_code": "API workflow", "name": "9084726456", "network_filter": null, "network_prefix": null, "network_root_folder": "/home/runner/work/testrepo/testrepo", "plugin_names": ["xcode", "gcov", "pycoverage"], "pull_request_number": null, "report_code": "default", "slug": "bbbb/testrepo", "token": "f******************", "upload_file_type": "coverage"}
debug - 2024-05-14 18:58:33,205 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"]}
debug - 2024-05-14 18:58:33,205 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-05-14 18:58:33,205 -- Running xcode plugin...
warning - 2024-05-14 18:58:33,205 -- xcrun is not installed or can't be found.
debug - 2024-05-14 18:58:33,205 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-05-14 18:58:33,205 -- Running gcov plugin...
warning - 2024-05-14 18:58:33,208 -- No gcov data found.
debug - 2024-05-14 18:58:33,208 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
info - 2024-05-14 18:58:33,209 -- Generating coverage.xml report in /home/runner/work/testrepo/testrepo
info - 2024-05-14 18:58:33,305 -- Wrote XML report to coverage.xml
debug - 2024-05-14 18:58:33,306 -- Collecting relevant files
info - 2024-05-14 18:58:33,315 -- Found 1 coverage files to report
info - 2024-05-14 18:58:33,315 -- > /home/runner/work/testrepo/testrepo/coverage.xml
debug - 2024-05-14 18:58:33,319 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-05-14 18:58:33,319 -- Sending upload request to Codecov
info - 2024-05-14 18:58:33,457 -- Your upload is now processing. When finished, results will be available at: https://codecov.bbbb.ccc/github/bbbb/testrepo/commit/1d711a6f6b0a141f1499341aed98469db16bf83d
debug - 2024-05-14 18:58:33,457 -- Upload request to Codecov complete. --- {"response": {"external_id": "0f7b6457-cd4f-44ad-bcac-c9f9b171621d", "created_at": "2024-05-14T18:58:34.208832Z", "raw_upload_location": "http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=codecov-default-key%2F20240514%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240514T185834Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ddd12c90eb7b6c20dda5f33f7256b87d2998d4ffb7424168fe64de4d24852ab7", "state": "", "provider": null, "upload_type": "uploaded", "url": "https://codecov.bbbb.ccc/github/bbbb/testrepo/commit/1d711a6f6b0a141f1499341aed98469db16bf83d", "ci_url": "https://github.com/bbbb/testrepo/actions/runs/9084726456", "flags": [], "env": {}, "name": "9084726456", "job_code": "API workflow"}}
debug - 2024-05-14 18:58:33,458 -- Sending upload to storage
warning - 2024-05-14 18:58:33,465 -- Request failed. Retrying --- {"retry": 0}
warning - 2024-05-14 18:58:33,972 -- Request failed. Retrying --- {"retry": 1}
warning - 2024-05-14 18:58:34,980 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
File "codecov_cli/main.py", line 83, in <module>
File "codecov_cli/main.py", line 79, in run
File "click/core.py", line 1157, in __call__
File "click/core.py", line 1078, in main
File "click/core.py", line 1688, in invoke
File "click/core.py", line 1434, in invoke
File "click/core.py", line 783, in invoke
File "click/decorators.py", line 33, in new_func
File "codecov_cli/commands/upload.py", line 257, in do_upload
File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic
File "codecov_cli/services/upload/upload_sender.py", line 104, in send_upload_data
File "codecov_cli/helpers/request.py", line 78, in wrapper
Exception: Request failed after too many retries
[1628] Failed to execute script 'main' due to unhandled exception!
Error: Codecov:
Failed to properly upload report: The process '/home/runner/work/_actions/codecov/codecov-action/v4.2.0/dist/codecov' failed with exit code 1
Additional context Add any other context about the problem here.
@andreiz
Is http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txt reachable from the uploader?
Since it's a local docker name, and you are uploading from GitHub, I'm guessing it is not.
@andreiz
Is
http://minio:9000/archive/v4/raw/2024-05-14/8B2BBCBD873D6CDADDAC6E1444671A7C/1d711a6f6b0a141f1499341aed98469db16bf83d/9fa1e0a2-cd45-4dfe-ad28-183f3e4e544f/0f7b6457-cd4f-44ad-bcac-c9f9b171621d.txtreachable from the uploader?Since it's a local docker name, and you are uploading from GitHub, I'm guessing it is not.
If it is supposed to be reachable from the outside, it's not something I saw mentioned in the docs. I literally just cloned the https://github.com/codecov/self-hosted repo and then followed the Self-Hosted Configuration Guide and nothing there mentioned minio.
I was under the impression that the codecov API uses minio locally as the storage backend API, but I guess I am mistaken?
Minio uses whatever the host is set for the storage service
https://github.com/codecov/codecov-api/blob/main/services/storage.py#L26
The value needs to be reachable both in the container and externally, as the report is uploaded directed to storage, bypassing Codecov.
I got it running by opening up port 9000 in the firewall and configuring minio host to be the same as the main codecov one. But can codecov-gateway proxy external traffic to minio container instead and do SSL termination? If not, then how do I set up minio to run with SSL enabled?
I'm also receiving the same error. Is there any solution or work around?
warning - 2024-06-19 21:46:27,782 -- Request failed. Retrying --- {"retry": 0}
warning - 2024-06-19 21:48:38,854 -- Request failed. Retrying --- {"retry": 1}
warning - 2024-06-19 21:50:49,926 -- Request failed. Retrying --- {"retry": 2}
Traceback (most recent call last):
File "codecov_cli/main.py", line 83, in <module>
File "codecov_cli/main.py", line 79, in run
File "click/core.py", line 1157, in __call__
File "click/core.py", line 1078, in main
File "click/core.py", line 1688, in invoke
File "click/core.py", line 1434, in invoke
File "click/core.py", line 783, in invoke
File "click/decorators.py", line 33, in new_func
File "codecov_cli/commands/upload.py", line 257, in do_upload
File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic
File "codecov_cli/services/upload/upload_sender.py", line [104](https://github.com/XXXXXX/some-new-service/actions/runs/95884677415/job/264412345062#step:16:105), in send_upload_data
File "codecov_cli/helpers/request.py", line 78, in wrapper
Exception: Request failed after too many retries
[936] Failed to execute script 'main' due to unhandled exception!
Warning: Codecov:
Failed to properly upload report: The process '/runner/_work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1```
I'm also receiving the same error. Is there any solution or work around?
warning - 2024-06-19 21:46:27,782 -- Request failed. Retrying --- {"retry": 0} warning - 2024-06-19 21:48:38,854 -- Request failed. Retrying --- {"retry": 1} warning - 2024-06-19 21:50:49,926 -- Request failed. Retrying --- {"retry": 2} Traceback (most recent call last): File "codecov_cli/main.py", line 83, in <module> File "codecov_cli/main.py", line 79, in run File "click/core.py", line 1157, in __call__ File "click/core.py", line 1078, in main File "click/core.py", line 1688, in invoke File "click/core.py", line 1434, in invoke File "click/core.py", line 783, in invoke File "click/decorators.py", line 33, in new_func File "codecov_cli/commands/upload.py", line 257, in do_upload File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic File "codecov_cli/services/upload/upload_sender.py", line [104](https://github.com/XXXXXX/some-new-service/actions/runs/95884677415/job/264412345062#step:16:105), in send_upload_data File "codecov_cli/helpers/request.py", line 78, in wrapper Exception: Request failed after too many retries [936] Failed to execute script 'main' due to unhandled exception! Warning: Codecov: Failed to properly upload report: The process '/runner/_work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1```
I solved this by placing nginx in front (of docker compose resources) and serving minio on a separate subdomain.
Thanks for sharing your solution , @jonas-brauer ! Are you also using SSL?
@drazisil-codecov Yes we are. Depending on your setup you may want to set this in your codecov.yml
services:
minio:
host: your.host.name
port: 443
verify_ssl: true
@andreiz Please see if the instructions above help you.
I will give it a shot
@drazisil-codecov Yes we are. Depending on your setup you may want to set this in your codecov.yml
services: minio: host: your.host.name port: 443 verify_ssl: true
This didn't work for me. Also I'm able to access both Codecov UI and Minio using SSL.
For minio SSL I've added the following:
volumes:
- archive-volume:/export
- ./cert/data/live/something.com/fullchain.pem:/root/.minio/certs/public.crt:ro
- ./cert/data/live/something.com/privkey.pem:/root/.minio/certs/private.key:ro
I'm using certbot for SSL
I'm also receiving the same error. Is there any solution or work around?
warning - 2024-06-19 21:46:27,782 -- Request failed. Retrying --- {"retry": 0} warning - 2024-06-19 21:48:38,854 -- Request failed. Retrying --- {"retry": 1} warning - 2024-06-19 21:50:49,926 -- Request failed. Retrying --- {"retry": 2} Traceback (most recent call last): File "codecov_cli/main.py", line 83, in <module> File "codecov_cli/main.py", line 79, in run File "click/core.py", line 1157, in __call__ File "click/core.py", line 1078, in main File "click/core.py", line 1688, in invoke File "click/core.py", line 1434, in invoke File "click/core.py", line 783, in invoke File "click/decorators.py", line 33, in new_func File "codecov_cli/commands/upload.py", line 257, in do_upload File "codecov_cli/services/upload/__init__.py", line 112, in do_upload_logic File "codecov_cli/services/upload/upload_sender.py", line [104](https://github.com/XXXXXX/some-new-service/actions/runs/95884677415/job/264412345062#step:16:105), in send_upload_data File "codecov_cli/helpers/request.py", line 78, in wrapper Exception: Request failed after too many retries [936] Failed to execute script 'main' due to unhandled exception! Warning: Codecov: Failed to properly upload report: The process '/runner/_work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1```I solved this by placing nginx in front (of docker compose resources) and serving minio on a separate subdomain.
@jonas-brauer thanks for your response. A simple/small code snippet would have been very helpful.
Thanks.
@rohit5roy I'm not sure I can boil down nginx into a small snippet, but this link should be a good place to start to learn about reverse proxy if it's new to you 😃 https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/
Just ran into the same issue, it seems like the gateway was once set up to proxy requests to <host>/archive (where archive is the default bucket name) to the minio storage. This was broken due to a missing file rename, just opened a PR to resolve this: https://github.com/codecov/codecov-gateway/pull/97
Thank you @derklaro for the contribution, we'll review and move discussion there. In the meantime I'll close this and direct any further discussion to the PR itself: https://github.com/codecov/codecov-gateway/pull/97
