bitbucket-branch-source-plugin
bitbucket-branch-source-plugin copied to clipboard
jenkinsci
401 Errors Occur randomly.
Jenkins and plugins versions report
Environment
Jenkins: 2.346.1 OS: Linux - 4.14.252-131.483.amzn1.x86_64
ace-editor:1.1 amazon-ecs:1.41 ant:475.vf34069fef73c antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 artifact-manager-s3:633.v4813787e78a_9 authentication-tokens:1.4 authorize-project:1.4.0 aws-credentials:191.vcb_f183ce58b_9 aws-global-configuration:100.v4619b_45d2dfc aws-java-sdk:1.12.215-339.vdc07efc5320c aws-java-sdk-cloudformation:1.12.215-339.vdc07efc5320c aws-java-sdk-codebuild:1.12.215-339.vdc07efc5320c aws-java-sdk-ec2:1.12.215-339.vdc07efc5320c aws-java-sdk-ecr:1.12.215-339.vdc07efc5320c aws-java-sdk-ecs:1.12.215-339.vdc07efc5320c aws-java-sdk-elasticbeanstalk:1.12.215-339.vdc07efc5320c aws-java-sdk-iam:1.12.215-339.vdc07efc5320c aws-java-sdk-logs:1.12.215-339.vdc07efc5320c aws-java-sdk-minimal:1.12.215-339.vdc07efc5320c aws-java-sdk-ssm:1.12.215-339.vdc07efc5320c blueocean:1.25.5 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.5 blueocean-commons:1.25.5 blueocean-config:1.25.5 blueocean-core-js:1.25.5 blueocean-dashboard:1.25.5 blueocean-display-url:2.4.1 blueocean-events:1.25.5 blueocean-git-pipeline:1.25.5 blueocean-github-pipeline:1.25.5 blueocean-i18n:1.25.5 blueocean-jwt:1.25.5 blueocean-personalization:1.25.5 blueocean-pipeline-api-impl:1.25.5 blueocean-pipeline-editor:1.25.5 blueocean-pipeline-scm-api:1.25.5 blueocean-rest:1.25.5 blueocean-rest-impl:1.25.5 blueocean-web:1.25.5 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-7 bouncycastle-api:2.25 branch-api:2.1046.v0ca_37783ecc5 build-failure-analyzer:2.3.0 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checkmarx:2022.2.3 checks-api:1.7.4 cloudbees-bitbucket-branch-source:773.v4b_9b_005b_562b_ cloudbees-folder:6.729.v2b_9d1a_74d673 command-launcher:1.2 config-file-provider:3.10.0 configuration-as-code:1429.v09b_044a_c93de configuration-as-code-secret-ssm:1.0.1 copyartifact:1.46.4 credentials:1129.vef26f5df883c credentials-binding:523.vd859a_4b_122e6 custom-tools-plugin:0.8 display-url-api:2.3.6 docker-commons:1.19 docker-workflow:1.28 durable-task:496.va67c6f9eefa7 echarts-api:5.3.3-1 email-ext:2.88 extended-choice-parameter:346.vd87693c5a_86c favorite:2.4.1 ffsnsnotifier:1.1 font-awesome-api:6.1.1-1 git:4.11.3 git-client:3.11.0 git-server:1.11 github:1.34.3 github-api:1.303-400.v35c2d8258028 github-branch-source:1637.vd833b_7ca_7654 google-login:1.6 h2-api:1.4.199 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.30 jackson2-api:2.13.3-285.vc03c0256d517 javax-activation-api:1.2.0-3 javax-mail-api:1.6.2-6 jaxb:2.3.6-1 jdk-tool:1.0 jenkins-design-language:1.25.5 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.79 jquery:1.12.4-1 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1119.1121.vc43d0fc45561 mailer:414.vcc4c33714601 matrix-auth:3.1.2 matrix-project:771.v574584b_39e60 metrics:4.1.6.2 okhttp-api:4.9.3-105.vb96869f8ac3a pipeline-build-step:2.18 pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:593.va_a_fc25d520e9 pipeline-input-step:449.v77f0e8b_845c4 pipeline-maven:3.10.0 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2097.v33db_b_de764b_e pipeline-model-definition:2.2097.v33db_b_de764b_e pipeline-model-extensions:2.2097.v33db_b_de764b_e pipeline-rest-api:2.24 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2097.v33db_b_de764b_e pipeline-utility-steps:2.12.1 plain-credentials:1.8 plugin-util-api:2.17.0 popper-api:1.16.1-2 popper2-api:2.11.5-2 prometheus:2.0.11 pubsub-light:1.16 resource-disposer:0.19 role-strategy:484.v8a_a_e4b_d785fd scm-api:608.vfa_f971c5a_a_e9 script-security:1175.v4b_d517d6db_f0 skip-notifications-trait:1.0.5 slack:608.v19e3b_44b_b_9ff snakeyaml-api:1.30.1 sse-gateway:1.25 ssh-credentials:277.v95c2fec1c047 sshd:3.228.v4c9f9e652c86 stashNotifier:1.28 structs:318.va_f3ccb_729b_71 timestamper:1.17 token-macro:293.v283932a_0a_b_49 trilead-api:1.57.v6e90e07157e1 variant:1.4 workflow-api:1164.v760c223ddb_32 workflow-basic-steps:948.v2c72a_091b_b_68 workflow-cps:2725.v7b_c717eb_12ce workflow-cps-global-lib:581.ve633085a_8a_87 workflow-durable-task-step:1146.v1a_d2e603f929 workflow-job:1186.v8def1a_5f3944 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:625.vd896b_f445a_f8 workflow-support:820.vd1a_6cc65ef33 ws-cleanup:0.42
What Operating System are you using (both controller, and any agents involved in the problem)?
Running using the official Docker images for both controller and agent, running in AWS ECS.
Reproduction steps
- Commit code and wait for build.
Expected Results
Build works and notifies Bitbucket that this has occurred.
Actual Results
Largely works OK.
Less frequently but very regularly receive the following...
Started by upstream project
Anything else?
Looking at the code from the stack trace it appears to be loading all the branches of the repo using a paged Bitbucket API. We do indeed see the occurrence of this error as correlated to the number of branches in a repo (Some of ours have many hundred branches). However from a quick glance at the error it appears to be using OAuth, whereas the credentials we have supplied to the plugin are username/password. Not sure if that is relevant/related?
I am also seeing random 401 errors and the only way I can get it working again is to clear a CAPTCHA request in Bitbucket.
I've installed Jenkins from an RPM on a Fedora OS.
I expect to Jenkins to be able to query Bitbucket and retrieve a list of repositories and branches for the given Bitbucket project and then create the necessary jobs.
Instead, I receive the following error:
Started by user
[Tue Oct 04 12:50:42 UTC 2022] Starting organization scan...
[Tue Oct 04 12:50:42 UTC 2022] Updating actions...
Looking up team details of project...
Connecting to https://bitbucket using user/****** (BitBucket username and PAT for user)
ERROR: [Tue Oct 04 12:50:42 UTC 2022] Could not refresh actions for navigator com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator@19f7d03b
com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .
HttpResponseProxy{HTTP/1.1 401 [Date: Tue, 04 Oct 2022 12:50:42 GMT, Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Connection: keep-alive, Set-Cookie: AWSALB=iEN691L4MYfhWZm03u97DS1xKHizDSQ+Q+1Wh/G4vsSuciTdbG04KWs3YWrNiSKmL8wgiIwHOUlIdvKS953cgA3YRNHl+ivejLapY2vhxRQThOFBt9b8teChZreH; Expires=Tue, 11 Oct 2022 12:50:42 GMT; Path=/, Set-Cookie: AWSALBCORS=iEN691L4MYfhWZm03u97DS1xKHizDSQ+Q+1Wh/G4vsSuciTdbG04KWs3YWrNiSKmL8wgiIwHOUlIdvKS953cgA3YRNHl+ivejLapY2vhxRQThOFBt9b8teChZreH; Expires=Tue, 11 Oct 2022 12:50:42 GMT; Path=/; SameSite=None; Secure, X-AREQUESTID: *6HPJ19x770x45931629x0, WWW-Authenticate: Basic realm="Atlassian Bitbucket", vary: accept-encoding] org.apache.http.client.entity.DecompressingEntity@3a3671e0}
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:884)
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:756)
Caused: java.io.IOException: I/O error when accessing URL: /rest/api/1.0/projects/project
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:761)
at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.retrieveActions(BitbucketSCMNavigator.java:539)
at jenkins.scm.api.SCMNavigator.fetchActions(SCMNavigator.java:332)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:487)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:918)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)
[Tue Oct 04 12:50:42 UTC 2022] Consulting Bitbucket Team/Project
Connecting to https://bitbucket using user/****** (BitBucket username and PAT for user)
ERROR: [Tue Oct 04 12:50:42 UTC 2022] Could not fetch sources from navigator com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator@19f7d03b
[Tue Oct 04 12:50:42 UTC 2022] Finished organization scan. Scan took 55 ms
FATAL: Failed to recompute children of PROJECT
com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .
HttpResponseProxy{HTTP/1.1 401 [Date: Tue, 04 Oct 2022 12:50:42 GMT, Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Connection: keep-alive, Set-Cookie: AWSALB=PN9dRjWNalH7fr3ZSCuB365FTcYDIWRt+/c8Jyt+CdicXYFTbT7DlVqPeZ7BvhqnLp2t8gyNqPZhO2tXDM8GtYtQyh6PrAGbG/SaIUJK5sEaentabRO+k5g+Lrm/; Expires=Tue, 11 Oct 2022 12:50:42 GMT; Path=/, Set-Cookie: AWSALBCORS=PN9dRjWNalH7fr3ZSCuB365FTcYDIWRt+/c8Jyt+CdicXYFTbT7DlVqPeZ7BvhqnLp2t8gyNqPZhO2tXDM8GtYtQyh6PrAGbG/SaIUJK5sEaentabRO+k5g+Lrm/; Expires=Tue, 11 Oct 2022 12:50:42 GMT; Path=/; SameSite=None; Secure, X-AREQUESTID: *6HPJ19x770x45931630x0, WWW-Authenticate: Basic realm="Atlassian Bitbucket", vary: accept-encoding] org.apache.http.client.entity.DecompressingEntity@76528557}
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:884)
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:756)
Caused: java.io.IOException: I/O error when accessing URL: /rest/api/1.0/projects/project
at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getTeam(BitbucketServerAPIClient.java:761)
at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.visitSources(BitbucketSCMNavigator.java:490)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:536)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:918)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)
Finished: FAILURE
This issue also affects builds (not just scanning Bitbucket projects) and the build log reports:
10:53:34 Started by user
10:53:34 com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .
10:53:34 HttpResponseProxy{HTTP/1.1 401 [Date: Tue, 04 Oct 2022 09:53:34 GMT, Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Connection: keep-alive, Set-Cookie: AWSALB=YEUobGH3gNCSWWLlH5sYlFxa7wNhWxDqbH+xzlFH6dwYHIiTKKazmtAQ35xaTiJsXf59fB0JcAi39DcvNSbcskAwUjifJQgVsvDO4xOwgIBWYeFLvc28/ekYH6+u; Expires=Tue, 11 Oct 2022 09:53:34 GMT; Path=/, Set-Cookie: AWSALBCORS=YEUobGH3gNCSWWLlH5sYlFxa7wNhWxDqbH+xzlFH6dwYHIiTKKazmtAQ35xaTiJsXf59fB0JcAi39DcvNSbcskAwUjifJQgVsvDO4xOwgIBWYeFLvc28/ekYH6+u; Expires=Tue, 11 Oct 2022 09:53:34 GMT; Path=/; SameSite=None; Secure, X-AREQUESTID: *Q55LJLx593x46291775x2, WWW-Authenticate: Basic realm="Atlassian Bitbucket", vary: accept-encoding] org.apache.http.client.entity.DecompressingEntity@79f1bdc1}
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:884)
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getResources(BitbucketServerAPIClient.java:836)
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getServerBranches(BitbucketServerAPIClient.java:589)
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getBranches(BitbucketServerAPIClient.java:580)
10:53:34 at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:820)
10:53:34 at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
10:53:34 at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:101)
10:53:34 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:311)
10:53:34 at hudson.model.ResourceController.execute(ResourceController.java:107)
10:53:34 at hudson.model.Executor.run(Executor.java:449)
10:53:34 Finished: FAILURE
This is happening increasingly frequently but stills seems somewhat random. I've not been able to correlate certain actions to repeatably cause this.
It seems that no credentials get sent seemingly randomly (~ 1 per 1000 builds?). As we do several thousand builds a day and it seems to be mostly on the same repo (with the most branches) this is quite annoying for the team impacted. Is this just because that makes more requests to get all the branches?
I updated the BitBucket Branch Source plugin for Jenkins (to version 791.vb_eea_a_476405b) a couple weeks ago and since then I've not experienced the 401 / CAPTCHA request issues I described above. So, for me at least, I think it was fixed by #637 (https://issues.jenkins.io/browse/JENKINS-61200).