Bizarre "Running `Pkg.resolve` on `path/to/Project.toml` will change the content." errors

[chriselrod]

D:\a\Octavian.jl\Octavian.jl\Project.toml: Test Failed at C:\Users\runneradmin\.julia\packages\Aqua\DaPBP\src\project_toml_formatting.jl:7
  Expression: result ⊜ true
   Evaluated: ⟪result: 😭 FAILED: D:\a\Octavian.jl\Octavian.jl\Project.toml
    Running `Pkg.resolve` on `D:\a\Octavian.jl\Octavian.jl\Project.toml` will change the content.
    
    --- Original Project.toml
    +++ Pkg's output
    @@ -1,8 +1,19 @@
    -authors = ["Mason Protter", "Chris Elrod", "Dilum Aluthge", "contributors"]
     name = "Octavian"
     uuid = "6fd5a793-0b7e-452c-907f-f8bfe9c57db4"
    +authors = ["Mason Protter", "Chris Elrod", "Dilum Aluthge", "contributors"]
     version = "0.3.13"
     
    +[deps]
    +ArrayInterface = "4fba245c-0d91-5ea0-9b3e-6abc04ee57a9"
    +IfElse = "615f187c-cbe4-4ef1-ba3b-2fcf58d6d173"
    +LoopVectorization = "bdcacae8-1622-11e9-2a5c-532679323890"
    +ManualMemory = "d125e4d3-2237-4719-b19c-fa641b8a4667"
    +PolyesterWeave = "1d0040c9-8b98-4ee7-8388-3f51789ca0ad"
    +Requires = "ae029012-a4dd-5104-9daa-d747884805df"
    +Static = "aedffcd0-7271-4cad-89d0-dc628f76c6d3"
    +ThreadingUtilities = "8290d209-cae3-49c0-8002-c8c24d57dab5"
    +VectorizationBase = "3d5dd08c-fd9d-11e8-17fa-ed2836048c2f"
    +
     [compat]
     ArrayInterface = "3.1.14, 5.0.1"
     IfElse = "0.1"
    @@ -15,17 +26,6 @@
     VectorizationBase = "0.21.15"
     julia = "1.6"
     
    -[deps]
    -ArrayInterface = "4fba245c-0d91-5ea0-9b3e-6abc04ee57a9"
    -IfElse = "615f187c-cbe4-4ef1-ba3b-2fcf58d6d173"
    -LoopVectorization = "bdcacae8-1622-11e9-2a5c-532679323890"
    -ManualMemory = "d125e4d3-2237-4719-b19c-fa641b8a4667"
    -PolyesterWeave = "1d0040c9-8b98-4ee7-8388-3f51789ca0ad"
    -Requires = "ae029012-a4dd-5104-9daa-d747884805df"
    -Static = "aedffcd0-7271-4cad-89d0-dc628f76c6d3"
    -ThreadingUtilities = "8290d209-cae3-49c0-8002-c8c24d57dab5"
    -VectorizationBase = "3d5dd08c-fd9d-11e8-17fa-ed2836048c2f"
    -
     [extras]
     Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
     BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
    
⟫ ⊜ true
Stacktrace:
 [1] macro expansion
   @ C:\hostedtoolcache\windows\julia\nightly\x64\share\julia\stdlib\v1.9\Test\src\Test.jl:464 [inlined]
 [2] macro expansion
   @ C:\Users\runneradmin\.julia\packages\Aqua\DaPBP\src\project_toml_formatting.jl:7 [inlined]
 [3] macro expansion
   @ C:\hostedtoolcache\windows\julia\nightly\x64\share\julia\stdlib\v1.9\Test\src\Test.jl:1433 [inlined]
 [4] test_project_toml_formatting(packages::Module)
   @ Aqua C:\Users\runneradmin\.julia\packages\Aqua\DaPBP\src\project_toml_formatting.jl:5
Test Summary:                                 | Pass  Fail  Total   Time
Aqua.jl                                       |    6     1      7  25.6s
  Method ambiguity                            |              None   0.0s
  Unbound type parameters                     |    1            1   0.2s
  Undefined exports                           |    1            1   0.0s
  Compare Project.toml and test/Project.toml  |    1            1   0.0s
  Stale dependencies                          |    1            1  16.3s
  Compat bounds                               |    1            1   0.4s
  Project.toml formatting                     |          1      1   2.8s
    D:\a\Octavian.jl\Octavian.jl\Project.toml |          1      1   2.3s

log: https://github.com/JuliaLinearAlgebra/Octavian.jl/runs/5398185264?check_suite_focus=true

It claims the deps were below the compat. However, that is not the case https://github.com/JuliaLinearAlgebra/Octavian.jl/blob/fbe1c304cbc68301d15ab93c1170bea4b8b193ac/Project.toml I've suddenly started getting these failures in several packages. Maybe it's an issue in TOML.jl?

[tkf]

Hmm... Yes, that's strange. It's totally possible I mess up generating Pkg's output but it doesn't explain why the original file content is something different.

Just in case there are other people having the same issue, this part of the test can be disabled using Aqua.test_all(YourPackage; project_toml_formatting = false).

[bvdmitri]

Can confirm that our package also suffers from issue in CI. No issues if execute tests locally.

[fingolfin]

Is this still a problem? I just tried to reproduce it on your repository via CI there (see https://github.com/fingolfin/Octavian.jl/pull/1) but no success. But of course the error above is on Windows, so perhaps it only happens on non-CI runs and only Windows? Or did you also have it in CI and/or on Linux?

I don't think Aqua has the ability to "make up" that files in Project.toml are in a location they are not really. But I do notice that in the "wrong" input, all keys and sections are sorted alphabetically - just like a generic TOML formatter would put them, while Pkg.jl enforces a certain order (e.g. [deps] before [compat]). Could it be that there is / was a second script involved that is meant to "fix" the TOML formatting?

[chriselrod]

I haven't seen this particular problem in a while. If anyone else has, please feel free to reopen.