Pkg.jl Pkg's gc failing due to stray characters in manifest

The error message is

[ Info: We haven't cleaned this depot up for a bit, running Pkg.gc()...
┌ Error: GC failed
│   exception =
│    TOML Parser error:
│    ~/.julia/logs/manifest_usage.toml:1103:0 error: invalid bare key character: '\n'
│      3Z
│        ^
└ @ Pkg /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.7/Pkg/src/Pkg.jl:644

Dec 01 '21 16:12 BeastyBlacksmith

There is a suspiciously looking 3Z on this line in the file

[["~/.julia/dev/OCTOPOS/web/Manifest.toml"]]
time = 2021-11-10T11:07:35.083Z
3Z

But there are also more lines like this in the file. Failing on 1.7.0 and 1.6.2

Dec 01 '21 16:12 BeastyBlacksmith

The line with time = is a valid ISO-8601 datetime string with timezone information (that's the Z, here for Zulu or UTC time) - what shouldn't be there is imo the lonely 3Z. Does this occur at the end somewhere? It looks like a reused buffer with different lengths didn't get properly truncated when writing to it again :thinking:

Dec 01 '21 17:12 Seelengrab

Yeah, in the Yggdrasil buildmachine we have plenty of those lines. I suspect that may be due to parallel access to the file.

Dec 01 '21 17:12 giordano

Whats the best way to handle this if it happens to me? Just delete that file and let it rebuild?

Dec 03 '21 14:12 jonniedie

In Yggdrasil that file is close to 1 million lines, so for me it's easier to delete it than trying to fix it (something which I actually tried once, but gave up after half an hour when I realised I was at 10% of the file)

Dec 03 '21 14:12 giordano

Xref https://github.com/JuliaLang/Pkg.jl/issues/1664

Dec 13 '21 16:12 ericphanson

Apparently the same error. macOS ARM , Julia 1.8.3 & 1.8.4 Rebuilding Manifest doesn't help.

(MSCal) pkg> gc
ERROR: TOML Parser error:
/Users/elk/.julia/logs/manifest_usage.toml:94:0 error: invalid bare key character: '\n'
  747Z
      ^
Stacktrace:
  [1] parse
    @ ./toml_parser.jl:441 [inlined]
  [2] Base.CachedTOMLDict(p::Base.TOML.Parser, path::String)
    @ Base ./loading.jl:222
  [3] (::Base.var"#894#895"{String, Base.TOMLCache})()
    @ Base ./loading.jl:277
  [4] lock(f::Base.var"#894#895"{String, Base.TOMLCache}, l::ReentrantLock)
    @ Base ./lock.jl:185
  [5] parsed_toml
    @ ./loading.jl:274 [inlined]
  [6] parse_toml
    @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Types.jl:58 [inlined]
  [7] (::Pkg.API.var"#reduce_usage!#175")(f::Function, usage_filepath::String)
    @ Pkg.API /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:500
  [8] gc(ctx::Pkg.Types.Context; collect_delay::Dates.Day, verbose::Bool, force::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:511
  [9] gc (repeats 2 times)
    @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:469 [inlined]
 [10] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:406
 [11] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:387
 [12] do_cmd
    @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:377 [inlined]
 [13] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/REPLMode/REPLMode.jl:551
 [14] #invokelatest#2
    @ ./essentials.jl:729 [inlined]
 [15] invokelatest
    @ ./essentials.jl:726 [inlined]
 [16] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/REPL/src/LineEdit.jl:2510
 [17] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/REPL/src/REPL.jl:1248
 [18] (::REPL.var"#49#54"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:484

Dec 24 '22 19:12 Eben60

Weird that this remains after moving to pidfile locks.

Maybe the usage file writes need to be flushed before being closed and the pidfile is released?

I think I've seen this in cases where a single julia process is being used. It could be one of the artifact/build/test subprocesses clashing with the above.

Dec 24 '22 19:12 IanButterworth

I get a similar (related?) error when running Pkg.gc() under linux:

[ Info: We haven't cleaned this depot up for a bit, running Pkg.gc()...
┌ Error: GC failed
│   exception =
│    TOML Parser error:
│    /home/user/.julia/logs/manifest_usage.toml:5581:1 error: invalid bare key character: '\0'
│      [["/project/user/Manifest.toml"]]
│      ^                                                                                                                                                                                                                                                                                                                   
└ @ Pkg /cache/build/default-aws-shared0-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/Pkg/src/Pkg.jl:724

Jan 31 '23 21:01 pfarndt

Reading the error message helps: in the file /Users/elk/.julia/logs/manifest_usage.toml there was the apparently erroneous line 747Z

...
[["/Users/elk/Julia/Mendeleev.jl/Manifest.toml"]]
time = 2022-12-17T20:08:33.748Z
747Z
[["/Users/elk/Julia/MegaDiffEq.jl/Manifest.toml"]]
...

After removing it manually, gc works.

Feb 11 '23 21:02 Eben60

The point is that the Manifest.toml is machine generated and shouldn't have such an issue in the first place.

Feb 12 '23 07:02 Seelengrab

This still happens here with 1.8.5.

Mar 24 '23 10:03 oliviermilla

This is fixed in 1.9

I was confused previously. The fix was not backported to 1.8 and cannot be because it requires a stdlib change

Mar 24 '23 11:03 IanButterworth

I am still having this in 1.9.3:

ERROR: TOML Parser error:
/home/labs/tsodyks/yuvalw/.julia/logs/scratch_usage.toml:4377:2 error: invalid bare key character: '\"'
  l"]
   ^
Stacktrace:
  [1] parse
    @ ./toml_parser.jl:445 [inlined]
  [2] Base.CachedTOMLDict(p::Base.TOML.Parser, path::String)
    @ Base ./loading.jl:222
  [3] (::Base.var"#938#939"{String, Base.TOMLCache})()
    @ Base ./loading.jl:278
  [4] lock(f::Base.var"#938#939"{String, Base.TOMLCache}, l::ReentrantLock)
    @ Base ./lock.jl:229
  [5] parsed_toml
    @ ./loading.jl:275 [inlined]
  [6] parse_toml
    @ /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/Types.jl:60 [inlined]
  [7] (::Pkg.API.var"#reduce_usage!#178")(f::Function, usage_filepath::String)
    @ Pkg.API /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/API.jl:507
  [8] gc(ctx::Pkg.Types.Context; collect_delay::Dates.Day, verbose::Bool, force::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/API.jl:538
  [9] gc
    @ /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/API.jl:476 [inlined]
 [10] gc()
    @ Pkg.API /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/API.jl:476
 [11] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:409
 [12] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:390
 [13] do_cmd
    @ /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:380 [inlined]
 [14] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:557
 [15] #invokelatest#2
    @ ./essentials.jl:819 [inlined]
 [16] invokelatest
    @ ./essentials.jl:816 [inlined]
 [17] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2647
 [18] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1300
 [19] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:514

Nov 23 '23 13:11 yuvalwas

Is your system using multiple julia versions? Like is vscode set to before 1.9?

Nov 23 '23 13:11 IanButterworth

I don't think so, but it was on my institute's HPC, so I'm not sure. If it helps,

julia> versioninfo()
Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 80 × Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)
  Threads: 1 on 80 virtual cores
Environment:
  JULIA_DEPOT_PATH = :
  LD_LIBRARY_PATH = /apps/easybd/easybuild/software/Julia/1.9.3-linux-x86_64/lib:/usr/share/lsf/10.1/linux3.10-glibc2.17-x86_64/lib:/home/labs/testing/almoga/tmp/ncbi-magicblast-1.4.0-src/c++/local/ncbi-vdb-2.9.0-1/lib64::

Nov 23 '23 14:11 yuvalwas

I just saw this happen in a scenario I know only 1.9.4 was being used.

│    TOML Parser error:
│    /home/ian/.julia/logs/scratch_usage.toml:27346:3 error: invalid bare key character: '\"'
│      ml"]
│        ^

Nov 24 '23 02:11 IanButterworth

a colleague of mine ran into this issue on Julia 1.9.4, fwiw, with stacktrace:

Stacktrace:
  [1] parse
    @ ./toml_parser.jl:445 [inlined]
  [2] parsefile(f::String)
    @ TOML ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/TOML/src/TOML.jl:43
  [3] (::Pkg.Types.var"#51#54"{String, String, Dates.DateTime, String})()
    @ Pkg.Types ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Types.jl:513
  [4] mkpidlock(f::Pkg.Types.var"#51#54"{String, String, Dates.DateTime, String}, at::String, pid::Int32; kwopts::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:stale_age,), Tuple{Int64}}})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/FileWatching/src/pidfile.jl:82
  [5] mkpidlock
    @ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/FileWatching/src/pidfile.jl:79 [inlined]
  [6] #mkpidlock#6
    @ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/FileWatching/src/pidfile.jl:77 [inlined]
  [7] mkpidlock
    @ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/FileWatching/src/pidfile.jl:77 [inlined]
  [8] write_env_usage(source_file::String, usage_filepath::String)
    @ Pkg.Types ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Types.jl:511
  [9] Pkg.Types.EnvCache(env::Nothing)
    @ Pkg.Types ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Types.jl:366
 [10] EnvCache
    @ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Types.jl:345 [inlined]
 [11] add_snapshot_to_undo(env::Nothing)
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:2108
 [12] add_snapshot_to_undo
    @ ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:2104 [inlined]
 [13] activate(path::String; shared::Bool, temp::Bool, io::Base.PipeEndpoint)
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:1891
 [14] activate(path::String)
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:1850
 [15] top-level scope
    @ ~/.vscode-server/extensions/julialang.language-julia-1.76.2/scripts/languageserver/main.jl:8
in expression starting at /home/ubuntu/.vscode-server/extensions/julialang.language-julia-1.76.2/scripts/languageserver/main.jl:7

May 06 '24 15:05 ericphanson

encountered on Julia 1.10.4, the manifest_usage.toml file was corrupted.

Stacktrace:
  [1] parse
    @ ./toml_parser.jl:445 [inlined]
  [2] parsefile(f::String)
    @ TOML ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/TOML/src/TOML.jl:43
  [3] (::Pkg.Types.var"#51#54"{String, String, Dates.DateTime, String})()
    @ Pkg.Types ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Types.jl:541
  [4] mkpidlock(f::Pkg.Types.var"#51#54"{String, String, Dates.DateTime, String}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [5] mkpidlock
    @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:90 [inlined]
  [6] mkpidlock
    @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] write_env_usage(source_file::String, usage_filepath::String)
    @ Pkg.Types ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Types.jl:539
  [8] Pkg.Types.EnvCache(env::Nothing)
    @ Pkg.Types ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Types.jl:377
  [9] EnvCache
    @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Types.jl:356 [inlined]
 [10] Context
    @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Types.jl:406 [inlined]
 [11] gc()
    @ Pkg.API ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:479
 [12] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:409
 [13] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:390
 [14] do_cmd
    @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:380 [inlined]
 [15] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:557
 [16] #invokelatest#2
    @ ./essentials.jl:892 [inlined]
 [17] invokelatest
    @ ./essentials.jl:889 [inlined]
 [18] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/LineEdit.jl:2656
 [19] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:1312
 [20] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:386

(@v1.10) pkg>

Jun 16 '24 23:06 elsoroka

Pkg.jl
Pkg.jl copied to clipboard

Pkg's gc failing due to stray characters in manifest_usage.toml

Pkg.jl Pkg.jl copied to clipboard

Pkg's gc failing due to stray characters in manifest_usage.toml

Pkg.jl
Pkg.jl copied to clipboard