github4s icon indicating copy to clipboard operation
github4s copied to clipboard

Published 20 hours ago • verified publisher icon 47degrees

"Invalid tree info" - Fileless directories

Open blast-hardcheese opened this issue 5 years ago • 10 comments

Following along with sbt-microsite: Publish using GitHub4s, I've created a token with the repo scope, and have Enable SSO'd (which is required by my environment).

I was then able to use the Github API to perform some simple tests:

  • List commits for my repo
  • List entries for a given tree

Despite this, when attempting to use publishMicrosite, I get:

[error] Error committing filesGitHub returned an error: Failed invoking with status : 422 body :
 {"message":"Invalid tree info","documentation_url":"https://developer.github.com/v3/git/trees/#create-a-tree"}

        at sbtorgpolicies.github.syntax$GHResponseOps.execE(syntax.scala:53)
        at sbtorgpolicies.github.syntax$EitherTOps.execE(syntax.scala:43)
        at sbtorgpolicies.github.GitHubOps.commitDir(GitHubOps.scala:172)
        at sbtorgpolicies.github.GitHubOps.commitDir(GitHubOps.scala:148)
        at microsites.MicrositeAutoImportSettings.$anonfun$micrositeTasksSettings$13(MicrositeKeys.scala:305)
        at scala.Function1.$anonfun$compose$1(Function1.scala:44)
        at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
        at sbt.std.Transform$$anon$4.work(System.scala:67)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
        at sbt.Execute.work(Execute.scala:278)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: github4s.GithubResponses$UnsuccessfulHttpRequest: Failed invoking with status : 422 body :
 {"message":"Invalid tree info","documentation_url":"https://developer.github.com/v3/git/trees/#create-a-tree"}
        at github4s.HttpRequestBuilderExtensionJVM.toEntity(HttpRequestBuilderExtensionJVM.scala:84)
        at github4s.HttpRequestBuilderExtensionJVM.toEntity$(HttpRequestBuilderExtensionJVM.scala:74)
        at github4s.jvm.Implicits$.toEntity(Implicits.scala:21)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.$anonfun$runMap$3(HttpRequestBuilderExtensionJVM.scala:67)
        at scala.util.Try$.apply(Try.scala:209)
        at sbtorgpolicies.github.instances$$anon$2.capture(instances.scala:57)
        at sbtorgpolicies.github.instances$$anon$2.capture(instances.scala:56)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.runMap(HttpRequestBuilderExtensionJVM.scala:61)
        at github4s.HttpRequestBuilderExtensionJVM$$anon$1.run(HttpRequestBuilderExtensionJVM.scala:34)
        at github4s.HttpClient.post(HttpClient.scala:159)
        at github4s.api.GitData.createTree(GitData.scala:251)
        at github4s.free.interpreters.Interpreters$$anon$9.$anonfun$apply$7(Interpreters.scala:295)
        at cats.data.Kleisli.$anonfun$map$1(Kleisli.scala:19)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$2(Kleisli.scala:534)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:57)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:15)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$1(Kleisli.scala:534)
        at cats.data.Kleisli.$anonfun$map$1(Kleisli.scala:19)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$2(Kleisli.scala:534)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:57)
        at cats.instances.TryInstances$$anon$1.tailRecM(try.scala:15)
        at cats.data.KleisliFlatMap.$anonfun$tailRecM$1(Kleisli.scala:534)
        at github4s.Github$GithubIOSyntaxEither.exec(Github.scala:62)
        at sbtorgpolicies.github.syntax$GHResponseOps.execE(syntax.scala:51)
        ... 19 more

I found https://github.com/47deg/sbt-org-policies/issues/388#issuecomment-437045396, which was largely unhelpful other than to suggest that perhaps despite supplying the token, it doesn't work as advertised (possibly in my SSO environment).

Any assistance would be appreciated!

blast-hardcheese avatar Jul 13 '19 04:07 blast-hardcheese

Hi @blast-hardcheese! Is your microsite repo initialized? Also double check that your sbt values micrositeGithubOwner and micrositeGithubRepo are correct.

bilki avatar Jul 15 '19 09:07 bilki

@bilki I presume so. It would be easier to just link where I am at this point.

  • PR: https://github.com/twilio/guardrail/pull/347
  • build.sbt: https://github.com/twilio/guardrail/pull/347/files#diff-f96947b443ddd5823c201c553e5dad4cR11 -gh-pages: https://github.com/twilio/guardrail/tree/gh-pages

I was successfully able to publish to a non-SSO-protected repo with the API token, fwiw.

blast-hardcheese avatar Jul 16 '19 05:07 blast-hardcheese

Hey, @blast-hardcheese would you mind (if possible) trying to publish the microsite to another clean empty branch, just to be sure it's not the content of gh-pages that is provoking this bug?

bilki avatar Jul 17 '19 15:07 bilki

@bilki OK, so after digging quite a bit, I discovered that the root problem here is I'm getting NewTreeRequest(Some(...), Nil), as I have a tree containing only trees.

I've narrowed down what I suspect is the culprit: def getAllFiles: List[File] = Option(dirToCommit.listFiles()).toList.flatten.filter(_.isFile).

It seems as though we're doing top-down to create all the directories, but I suspect github really wants us to navigate depth first, creating the trees as the full hierarchy.

I'm wrapping up for the night, but this is where I got. If you've got some advice on how to proceed, (is flipping processing from top-down to bottom-up acceptable to you?) I'd appreciate it.

Thanks so far.

blast-hardcheese avatar Jul 19 '19 06:07 blast-hardcheese

Hey @blast-hardcheese, could you check if your site folder contains any empty dir (in the machine from where you are trying to publish)? I get the exact same error Invalid tree info when I try to commitDir a folder that contains empty dirs. As soon as I remove them, it works like a charm.

bilki avatar Jul 23 '19 14:07 bilki

@bilki I have a directory that is empty, save for other directories, which contain files. The problem, as I've seen it, is that we create trees only containing blobs, then go back and create the next deepest tree with those blobs, with the parent being another tree.

My proposal would be to do a depth-first approach, committing the deepest tree, then collecting that tree's hash as another sibling in the next highest up tree to generate.

What do you think?

blast-hardcheese avatar Jul 24 '19 16:07 blast-hardcheese

I am seeing the same error "Invalid tree info" when trying to use the API to add individual files to my repo. I don't have empty folders. Has a solution been found? Edit : I found other issue in my own code that was corrupting the data I was passing in. Ignore my question.

stanleydunne avatar Jul 26 '19 11:07 stanleydunne

Sorry about the delay @blast-hardcheese. That sounds like a possible solution, have you already tried it? I honestly don't know in what order github expects dir creations to happen. Anyway, you could open an issue at https://github.com/47deg/sbt-org-policies/issues, which is where the order on how commit dirs are updated is decided, and it's where your suggestion will apply.

bilki avatar Aug 08 '19 11:08 bilki

I've been using github4s through sbt-microsites too, and after messing a bit with this, I can confirm that the error will happen with a project structure like this. Source directory (src/main):

  • resources▼
    • microsite▼
      • data▼
        • menu.yml
  • scala▼
    • HelloWorld.scala
  • tut▼
    • docs▼
      • index.md
      • section-1-1.md
      • section-1-2.md
      • section-1.md
    • index.md

Then, processed through sbt-microsites, this is the actual content that will be committed with github4s:

  • css►
  • docs►
  • highlight►
  • img►
  • js►
  • microsite▼
    • data▼
      • menu.yml
  • _config.yml
  • index.html

I found that github4s will struggle to commit this structure (422), with the culprit being the fileless microsite directory. As a workaround I added a .gitkeep file near the data directory there and then github4s can proceed.

Somehow, a thing that indirectly solve this, is to use sbt-microsites in a different module, instead of the root one :man_shrugging: This solves it because using sbt-microsites that way, the microsite/data/menu.yml won't be generated (hence not committed), and the case is that it shouldn't (https://github.com/47deg/sbt-microsites/issues/335).

But, summarizing, and getting back to the bug, after some other tests, we can conclude that github4s will fail to commit a tree that include some kind of fileless directory like this (I don't think other considerations on the tree shape will matter really for this case):

  • static▼
    • subdir▼
      • data.yml

Being subdir the only child of the static directory.

calvellido avatar Sep 19 '19 10:09 calvellido

I have the same problem. While I had such a structure the publication was successful:

  • dir1
    • dir2
      • index.md
      • file1.md
      • file2.md
    • index.md
    • file1.md
    • file2.md
  • index.md
  • file1.md
  • file2.md

When I got such a structure that the publication began to fail with an error "Invalid tree info":

  • dir1
    • dir2
      • index.md
      • file1.md
      • file2.md
    • dir3
      • dir4
        • index.md
        • file1.md
        • file2.md
    • index.md
    • file1.md
    • file2.md
  • index.md
  • file1.md
  • file2.md

I got directory dir3 with no files in it - just another directory (dir4) The error "Invalid tree info" went away only after I added the file to the directory dir3:

  • dir1
    • dir2
      • index.md
      • file1.md
      • file2.md
    • dir3
      • dir4
        • index.md
        • file1.md
        • file2.md
      • index.md
    • index.md
    • file1.md
    • file2.md
  • index.md
  • file1.md
  • file2.md

artemkorsakov avatar Sep 17 '20 16:09 artemkorsakov