DerivationReadError in a .drv containing unicode chars

[nh2]

We encountered on ZuriHac:

nix-output-monitor error: DerivationReadError /nix/store/sy16h6b3k498dfpxqhq77l1q7nr6wgqb-source.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 195)

Contents of that file:

nom-issue-136.zip

[maralorn]

Ah, well. That’s then an issue in nix-derivation I think. Probably gonna drop that dependency …

[Libadoxon]

Is this planned to be fixed anytime soon? Would try to help but I suck at functional programming. When running the provided executable from nix-derivation pretty-derivation on a store path that throws a DerivationReadError in nix-output-monitor it works perfectly so maybe not a problem with nix-derivation.

[axman6]

I'm also running into this, usually with gettext which appears to be a dependency for a lot of things - I get dozens of lines of this exceptionwhen I build anything these days

nix-output-monitor error: DerivationReadError /nix/store/hszac7zz2wn5891b3x0viab3hvb9fv17-gcc-14-20241116.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/hszac7zz2wn5891b3x0viab3hvb9fv17-gcc-14-20241116.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/hszac7zz2wn5891b3x0viab3hvb9fv17-gcc-14-20241116.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/xcc17w1fkkkpqrbj9xlfiyndssp8sk99-db-5.3.28.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/qqhhxa7pppq70ibigljfyviis2glmd5a-source.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 195)
nix-output-monitor error: DerivationReadError /nix/store/0yyzjx8lf9m2nwhn4f9nzvq2s6rrily7-gfortran-14-20241116.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/hszac7zz2wn5891b3x0viab3hvb9fv17-gcc-14-20241116.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/hszac7zz2wn5891b3x0viab3hvb9fv17-gcc-14-20241116.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)
nix-output-monitor error: DerivationReadError /nix/store/v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv: hGetContents: invalid argument (cannot decode byte sequence starting from 226)

Offending derivation attached. v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv.zip

I've taken a look at nix-derivation and tested it with the file above, and it is at least able to parse it without errors (I thought there might be a bug in the parser when dealing with unicode symbols, bit attoparsec seems to handle that file with its use of Text.takeWhile).

~/Haskell/Haskell-Nix-Derivation-Library [main] $ cabal repl -b attoparsec                              
Resolving dependencies...
Build profile: -w ghc-9.10.1 -O1
In order, the following will be built (use -v for more details):
 - nix-derivation-1.1.3 (interactive) (lib) (first run)
Preprocessing library for nix-derivation-1.1.3...
GHCi, version 9.10.1: https://www.haskell.org/ghc/  :? for help
[1 of 4] Compiling Nix.Derivation.Types ( src/Nix/Derivation/Types.hs, interpreted )
[2 of 4] Compiling Nix.Derivation.Parser ( src/Nix/Derivation/Parser.hs, interpreted )
[3 of 4] Compiling Nix.Derivation.Builder ( src/Nix/Derivation/Builder.hs, interpreted )
[4 of 4] Compiling Nix.Derivation   ( src/Nix/Derivation.hs, interpreted )
Ok, four modules loaded.
ghci> import Data.Attoparsec.Text
ghci> import Nix.Derivation.Parser 
ghci> import Data.Text.IO as TIO
ghci> drv <- TIO.readFile "./v21bpc8cffd06bz080ayvs2xg78zizwr-gettext-0.22.5.drv" 
ghci> parse parseDerivation drv
Done "" (Derivation {outputs = fromList [("doc",DerivationOutput {path = "/nix/store/060h4lpb7ny25mkpjcnwijmca4njcf5l-gettext-0.22.5-doc", hashAlgo = "", hash = ""}),("info",DerivationOutput {path = "/nix/store/c937qmkc0qd9l1xin1is1g9rxb2xxwzh-gettext-0.22.5-info", hashAlgo = "", hash = ""}),("man",DerivationOutput {path = "/nix/store/pdb6kr6mw6qvwhw6ih9da51jjpnmz87w-gettext-0.22.5-man", hashAlgo = "", hash = ""}),("out",DerivationOutput {path = "/nix/store/m4454h58k2jyrpz9r96zby1aid2fv9qd-gettext-0.22.5", hashAlgo = "", hash = ""})], inputDrvs = fromList [("/nix/store/8qg7rhjjy2ilc2aln322y3hp09w4xhr1-stdenv-linux.drv",fromList ["out"]),("/nix/store/9fpkgd4qy3l5469macr0cspry10c8wh7-bash-5.2p37.drv",fromList ["dev","out"]),("/nix/store/d8y7z6y71ahy3jcgyqi7yiy6340ain6q-gettext-0.22.5.tar.gz.drv",fromList ["out"]),("/nix/store/sn4h2r3bn2qwgka79xw7j7ljzyj7vi6m-update-autotools-gnu-config-scripts-hook.drv",fromList ["out"])], inputSrcs = fromList ["/nix/store/ny42y6hs4p294rvnrwbmrpwzqghw2816-gettext-setup-hook.sh","/nix/store/p2fp6i7hjx9af1wbwr32k217wp2dxmiw-absolute-paths.diff","/nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh","/nix/store/v9034cqc4h5bm10z4vz3n1q2n55grv5y-role.bash","/nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh","/nix/store/yqwx9yln5i68nw61mmp9gz066yz3ri99-0001-msginit-Do-not-use-POT-Creation-Date.patch"], platform = "x86_64-linux", builder = "/nix/store/4fvc5fm8bszmkydng1ivrvr5cbvr1g60-bash-5.2p37/bin/bash", args = ["-e","/nix/store/vj1c3wf9c11a0qs6p3ymfvrnsdgsdcbq-source-stdenv.sh","/nix/store/shkw4qm9qcw5sc5n1k5jznc83ny02r39-default-builder.sh"], env = fromList [("LDFLAGS",""),("__structuredAttrs",""),("buildInputs","/nix/store/rrb1378zih3wygygg6j4h1nqz76ap940-bash-5.2p37-dev"),("builder","/nix/store/4fvc5fm8bszmkydng1ivrvr5cbvr1g60-bash-5.2p37/bin/bash"),("cmakeFlags",""),("configureFlags","--disable-csharp"),("depsBuildBuild",""),("depsBuildBuildPropagated",""),("depsBuildTarget",""),("depsBuildTargetPropagated",""),("depsHostHost",""),("depsHostHostPropagated",""),("depsTargetTarget",""),("depsTargetTargetPropagated",""),("doCheck",""),("doInstallCheck",""),("doc","/nix/store/060h4lpb7ny25mkpjcnwijmca4njcf5l-gettext-0.22.5-doc"),("enableParallelBuilding","1"),("enableParallelChecking",""),("enableParallelInstalling","1"),("gettextNeedsLdflags",""),("info","/nix/store/c937qmkc0qd9l1xin1is1g9rxb2xxwzh-gettext-0.22.5-info"),("man","/nix/store/pdb6kr6mw6qvwhw6ih9da51jjpnmz87w-gettext-0.22.5-man"),("mesonFlags",""),("name","gettext-0.22.5"),("nativeBuildInputs","/nix/store/hm8j4bp4hvjnz90vvmfzwsm5vwi1kcww-update-autotools-gnu-config-scripts-hook"),("out","/nix/store/m4454h58k2jyrpz9r96zby1aid2fv9qd-gettext-0.22.5"),("outputs","out man doc info"),("patches","/nix/store/p2fp6i7hjx9af1wbwr32k217wp2dxmiw-absolute-paths.diff /nix/store/yqwx9yln5i68nw61mmp9gz066yz3ri99-0001-msginit-Do-not-use-POT-Creation-Date.patch"),("pname","gettext"),("postPatch","# Older versions of gettext come with a copy of `extern-inline.m4` that is not compatible with clang 18.\n# When a project uses gettext + autoreconfPhase, autoreconfPhase will invoke `autopoint -f`, which will\n# replace whatever (probably compatible) version of `extern-inline.m4` with one that probalby won\8217t work\n# because `autopoint` will copy the autoconf macros from the project\8217s required version of gettext.\n# Fixing this requires replacing all the older copies of the problematic file with a new one.\n#\n# This is ugly, but it avoids requiring workarounds in every package using gettext and autoreconfPhase.\ndeclare -a oldFiles=($(tar tf gettext-tools/misc/archive.dir.tar | grep '^gettext-0\\.[19].*/extern-inline.m4'))\noldFilesDir=$(mktemp -d)\nfor oldFile in \"${oldFiles[@]}\"; do\n  mkdir -p \"$oldFilesDir/$(dirname \"$oldFile\")\"\n  cp gettext-tools/gnulib-m4/extern-inline.m4 \"$oldFilesDir/$oldFile\"\ndone\ntar uf gettext-tools/misc/archive.dir.tar -C \"$oldFilesDir\" \"${oldFiles[@]}\"\n\nsubstituteAllInPlace gettext-runtime/src/gettext.sh.in\nsubstituteInPlace gettext-tools/projects/KDE/trigger --replace \"/bin/pwd\" pwd\nsubstituteInPlace gettext-tools/projects/GNOME/trigger --replace \"/bin/pwd\" pwd\nsubstituteInPlace gettext-tools/src/project-id --replace \"/bin/pwd\" pwd\n"),("propagatedBuildInputs",""),("propagatedNativeBuildInputs",""),("setupHooks","/nix/store/v9034cqc4h5bm10z4vz3n1q2n55grv5y-role.bash /nix/store/ny42y6hs4p294rvnrwbmrpwzqghw2816-gettext-setup-hook.sh"),("src","/nix/store/whgzrixh2jwgz9bydmfsm3kx6a9szajl-gettext-0.22.5.tar.gz"),("stdenv","/nix/store/acjcsq0iim1lg14f8jvka05dhhz70c2c-stdenv-linux"),("strictDeps","1"),("system","x86_64-linux"),("version","0.22.5")]})
ghci> 

[axman6]

Looking at the error again, I should have noticed the hGetContents part of the error - it looks like hGetContents is failing to read from some handle - I wonder if the handle being used in this line has not been set to BinaryMode? I will explore this...