neon
neon copied to clipboard
neondatabase
Fix CI for external contributors
"Tests fail because they rely on infra you have no access to. Sorry about that, we'll work around it somehow to merge this PR and solve the issue for the next external contributor. Thanks for contributing."
Originally posted by @bojanserafimov in https://github.com/neondatabase/neon/pull/2195#pullrequestreview-1063218757
Also see https://github.com/neondatabase/neon/pull/2560#issuecomment-1269512883
- Regression tests fail:
FAILED test_runner/regress/test_remote_storage.py::test_remote_storage_backup_and_restore[RemoteStorageKind.REAL_S3]
[18087](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850177#step:4:18106)
FAILED test_runner/regress/test_tenants.py::test_pageserver_with_empty_tenants[RemoteStorageKind.REAL_S3]
[18088](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850177#step:4:18107)
FAILED test_runner/regress/test_tenants_with_remote_storage.py::test_tenants_many[RemoteStorageKind.REAL_S3]
[18089](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850177#step:4:18108)
FAILED test_runner/regress/test_wal_acceptor.py::test_wal_backup[RemoteStorageKind.REAL_S3]
[18090](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850177#step:4:18109)
FAILED test_runner/regress/test_wal_acceptor.py::test_s3_wal_replay[RemoteStorageKind.REAL_S3]
[18091](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850177#step:4:18110)
- e2e tests trigger fails:
Enter host password for user '':
[36](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850445#step:3:37)
% Total % Received % Xferd Average Speed Time Time Time Current
[37](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850445#step:3:38)
Dload Upload Total Spent Left Speed
[38](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850445#step:3:39)
[39](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850445#step:3:40)
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
[40](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850445#step:3:41)
100 136 0 0 100 136 0 925 --:--:-- --:--:-- --:--:-- 925
[41](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850445#step:3:42)
curl: (22) The requested URL returned error: 404
[42](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215850445#step:3:43)
Error: Process completed with exit code 22.
- Allure reporting for tests fails:
/usr/bin/docker exec c33d59d6262df37eb371b8ea3e441270ef6bed37df2f9c6bdfdaf76e42ab2d97 sh -c "cat /etc/*release | grep ^ID"
[292](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5216010955#step:4:302)
RequestError [HttpError]: Resource not accessible by integration
[293](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5216010955#step:4:303)
at /__w/_actions/actions/github-script/v6/dist/index.js:6173:21
[294](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5216010955#step:4:304)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
[295](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5216010955#step:4:305)
at async eval (eval at callAsyncFunction (/__w/_actions/actions/github-script/v6/dist/index.js:13340:16), <anonymous>:5:1)
[296](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5216010955#step:4:306)
Error: Unhandled error: HttpError: Resource not accessible by integration
[297](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5216010955#step:4:307)
- For some reason, we do all our Docker stuff and fail when pushing it
Run crane push neon neondatabase/neon:3191604676
[7](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:8)
2022/10/06 07:31:46 existing blob: sha256:d8ab0372c19aa8a6e8353f8487abfbd3cd7e7938a7a20336fbfb43e3d2eff8cb
[8](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:9)
2022/10/06 07:31:46 existing blob: sha256:d916ebda59e87b7eb1cb820ec70a3fe34f92df0e36f0d2b6314e0371b916edb6
[9](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:10)
2022/10/06 07:31:46 existing blob: sha256:bd159e379b3b1bc0134341e4ffdeab5f966ec422ae04818bb69ecef08a823b05
[10](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:11)
2022/10/06 07:31:46 existing blob: sha256:ceee83c66784c9ba1e4189721f2d482bbdc3cc9a677ef88e1b1627b3ab4f73f4
[11](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:12)
2022/10/06 07:31:46 existing blob: sha256:e32a62e6e9d6a8e570bc2e042cd1e30c56840dd1c427e4d8732592a46db8ed2f
[12](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:13)
2022/10/06 07:31:46 existing blob: sha256:3a82bbaf62cf42e431d5e833d2eb7d8a825b43aefbf1d0ab8e1c76a36b8613cd
[13](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:14)
2022/10/06 07:31:46 existing blob: sha256:8231816d2597cfbd43bd6a59203ae67ff9595d70bdd3ec882dce1989a48f428a
[14](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:15)
2022/10/06 07:31:46 existing blob: sha256:c49b48db45e7a091a9da55c3d937d905ef1999147099063217dad34c7d0f8278
[15](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:16)
2022/10/06 07:31:47 existing blob: sha256:d5d4d43d059e78f7e2f540fc7d826758560593bd826c82be972ff616a2c59930
[16](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:17)
2022/10/06 07:31:47 existing blob: sha256:f703af43f88ad0bab1b9ae0db335879287a61c386c8d1e59db5d25267545f016
[17](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:18)
2022/10/06 07:31:47 existing blob: sha256:13a1e684c6d58532b3d6f96895030bbfc2893bcce0420553a4be1a5216dea91f
[18](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:19)
Error: PUT https://index.docker.io/v2/neondatabase/neon/manifests/3191604676: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:neondatabase/neon Type:repository] map[Action:push Class: Name:neondatabase/neon Type:repository]]
[19](https://github.com/neondatabase/neon/actions/runs/3191604676/jobs/5215851041#step:12:20)
Error: Process completed with exit code 1.
Push compute tools image to Docker Hub
I vaguely remember a bot/workflow script a bit like bors which allowed triggering a workflow run with full permissions with some comment-based commands (post-review for example). This problem affects any repo using secrets so there should be a number of workarounds... Will edit if I can find it.
It seems any job that isn't assuming roles (from the AWS instance it runs on) fails due to secrets not being available
Yes, that's the reason for the most of it.
If we won't be implementing a solution to the creds soon, we can meanwhile detect if that's an external PR and skip all Docker, coverage, e2e and allure steps: then it's a matter of fixing a few Python tests to make the CI look green for external contributors.
@zoete following on your comment in slack "It is not impossible to have more pipeline steps work for external contributors, however that will at least require its own separated environment and other considerations."
Can we automate the part where a neon team member opens a draft pr with a cherrypicked commit just to trigger the CI ?
Recently, one of the contributors has shared this link: https://dev.to/petrsvihlik/using-environment-protection-rules-to-secure-secrets-when-building-external-forks-with-pullrequesttarget-hci
We could use pull_request_target to make PR from external contributors work, but it has huge warnings message (which is kinda expected):
Warning: For workflows that are triggered by the pull_request_target event, the GITHUB_TOKEN is granted read/write repository permission unless the permissions key is specified and the workflow can access secrets, even when it is triggered from a fork. Although the workflow runs in the context of the base of the pull request, you should make sure that you do not check out, build, or run untrusted code from the pull request with this event. Additionally, any caches share the same scope as the base branch. To help prevent cache poisoning, you should not save the cache if there is a possibility that the cache contents were altered. For more information, see "Keeping your GitHub Actions and workflows secure: Preventing pwn requests" on the GitHub Security Lab website.
Ref https://github.com/neondatabase/neon/pull/4188#issuecomment-1543940685
Another PR to test this workflow on https://github.com/neondatabase/neon/pull/5120
I clicked the "Approve and run tests" button but they're failing on permissions