github-api
github-api copied to clipboard
hub4j
Unable to clone submodules
Describe the bug
When Github organization scans for repos, it fails if the repo contains submodules with following error:
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of [Lorg.kohsuke.github.GHContent; out of START_OBJECT token
To Reproduce Steps to reproduce the behavior:
- Scan Github Org with repos having sub-modules
Expected behavior Scan and clone the sub-modules
Desktop (please complete the following information):
- OS: Windows Server 2016
- Browser: Chrome
- Version 1.115
Additional context
@saptakniyogi Could you point to a public example of a repository with submodules? Also please provide a full stack trace.
It is unlikely that the issue is purely in this library.
@saptakniyogi This is definitely not something that can be fixed in this library alone. Some work will need to be done in the related Jenkins plugins to make this work. Make sure there as an issue filed in the Jenkins JIRA database and crosslink with this issue.
Still need the full stack trace and a public example of a repository that reproduces this problem.
You can use the following public repository: https://github.com/saptakniyogi/Test.git. Submodule is part of the repo where a sample Jenkinsfile is present
ERROR: [Thu Aug 13 11:44:59 CEST 2020] Could not fetch sources from navigator org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator@4f469272
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `[Lorg.kohsuke.github.GHContent;` out of START_OBJECT token
at [Source: (String)"{"name":"name","path":"path","sha":"c46a5f7...26a4","size":0,"url":"base_url/repos/owners/repo/contents/sub?ref=refs/heads/master","html_url":"url//owner/subRepo/tree/c46a5f7...26a4","git_url":"base_url/repos/owners/subRepo/git/trees/c46a5f7...26a4","download_url":null,"type":"submodule","submodule_git_url":"git@githu"[truncated 380 chars]; line: 1, column: 1]
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1468)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1242)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1148)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.handleNonArray(ObjectArrayDeserializer.java:329)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:175)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:21)
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2057)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1496)
at org.kohsuke.github.GitHubResponse.parseBody(GitHubResponse.java:91)
at org.kohsuke.github.GitHubPageIterator.lambda$fetch$0(GitHubPageIterator.java:145)
at org.kohsuke.github.GitHubClient.createResponse(GitHubClient.java:462)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:413)
Caused: org.kohsuke.github.HttpException: Server returned HTTP response code: 200, message: '200 OK' for URL: base_url3/repos/owner/repo/contents/subRepo?ref=refs%2Fheads%2Fmaster
at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:501)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:421)
at org.kohsuke.github.GitHubPageIterator.fetch(GitHubPageIterator.java:144)
at org.kohsuke.github.GitHubPageIterator.hasNext(GitHubPageIterator.java:91)
at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:106)
at org.kohsuke.github.PagedIterator.nextPageArray(PagedIterator.java:134)
at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:78)
at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:106)
at org.kohsuke.github.PagedIterable.toList(PagedIterable.java:118)
at org.kohsuke.github.GHRepository.getDirectoryContent(GHRepository.java:2308)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.stat(GitHubSCMProbe.java:138)
at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:249)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1000)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263)
at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1405)
at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1420)
at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:254)
at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:204)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:1012)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:488)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:969)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:428)
[Thu Aug 13 11:44:59 CEST 2020] Finished organization scan. Scan took 0.63 sec
FATAL: Failed to recompute children of org-name
org.kohsuke.github.HttpException: Server returned HTTP response code: 200, message: '200 OK' for URL: base_url/repos/owner/repo/contents/subRepo?ref=refs%2Fheads%2Fmaster
at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:501)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:421)
at org.kohsuke.github.GitHubPageIterator.fetch(GitHubPageIterator.java:144)
at org.kohsuke.github.GitHubPageIterator.hasNext(GitHubPageIterator.java:91)
at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:106)
at org.kohsuke.github.PagedIterator.nextPageArray(PagedIterator.java:134)
at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:78)
at org.kohsuke.github.PagedIterable.toArray(PagedIterable.java:106)
at org.kohsuke.github.PagedIterable.toList(PagedIterable.java:118)
at org.kohsuke.github.GHRepository.getDirectoryContent(GHRepository.java:2308)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMProbe.stat(GitHubSCMProbe.java:138)
at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:249)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:1000)
at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
at jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263)
at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1405)
at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1420)
at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:254)
at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:204)
at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:1012)
at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:488)
at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:969)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:428)
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `[Lorg.kohsuke.github.GHContent;` out of START_OBJECT token
at [Source: (String)"{"name":"name","path":"path","sha":"c46a5f7...26a4","size":0,"url":"base_url/repos/owner/repo/contents/subRepo?ref=refs/heads/master","html_url":"url/owner/subRepo/tree/c46a5f7...26a4","git_url":"base_url/repos/owner/subRepo/git/trees/c46a5f7...26a4","download_url":null,"type":"submodule","submodule_git_url":"git@githu"[truncated 380 chars]; line: 1, column: 1]
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1468)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1242)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1148)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.handleNonArray(ObjectArrayDeserializer.java:329)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:175)
at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:21)
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2057)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1496)
at org.kohsuke.github.GitHubResponse.parseBody(GitHubResponse.java:91)
at org.kohsuke.github.GitHubPageIterator.lambda$fetch$0(GitHubPageIterator.java:145)
at org.kohsuke.github.GitHubClient.createResponse(GitHubClient.java:462)
at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:413)
... 27 more
Finished: FAILURE
This is in the same area as #875 with many of the same issues. Basically, what you're trying to do is poorly supported by the GitHub REST API, making it hard to implement a good fix for this issue.
Traversing symlink or submodule folders is expensive, requiring multiple GitHub API calls to determine success or failure. This library could offer a convenience method to make this easier for clients, but clients (such as the Jenkins plugin mentioned here) would need to opt in. We would not want to make it the default behavior.
PRs are welcome for this but it will require significant design and testing.