integrate more recent nix version

[qknight]

Using the awesome work from @tomberek aka Thomas Bereknyei we can adapt to a newer version of nix in the tour:

https://github.com/flox/nix/tree/wasm

To try it out:

  try {Module.ccall("main_nix_instantiate2","string",["string"],["let\n c = 2; in { inherit c; c = 2;}"]);} catch (e) { console.log(getExceptionMessage(e).toString()); };

try {Module.ccall("main_nix_instantiate2","string",["string"],["let c = 2; in { inherit c;}"]);} catch (e) { console.log(getExceptionMessage(e).toString()); };

try {Module.ccall("main_nix_instantiate2","string",["string"],["let\n c = 2; in { inherit c; f = builtins.readDir ./.;}"]);} catch (e) { console.log(getExceptionMessage(e).toString()); };

[qknight]

There is huge progress here since nix-instantiate 2.17 is now compiled and can evaluate.

It still lacks:

• integration in index.html since it has to be called differently now
• bundling of nixpkgs and assets has to be redesigned

See https://lastlog.de/blog/tour_of_nix.html

[qknight]

@tomberek I need nixpkgs in that build but to get there I have to adapt the src/nix-instantiate/local.mk section like this:

nix-instantiate.html_LDFLAGS = \
	  $(shell echo "$(NIX_LDFLAGS)" | cut -f 5- -d' ') \
	  -lcrypto \
	  -sNO_EXIT_RUNTIME=1 \
	  -sALLOW_MEMORY_GROWTH=1 \
	  -sEXPORTED_RUNTIME_METHODS=cwrap,ccall,UTF8ToString,ptrToString,allocateUTF8,UTF32ToString \
	  -sEXPORTED_FUNCTIONS=_processExpr,_main_nix_instantiate2 \
	  -sEXPORT_EXCEPTION_HANDLING_HELPERS \
	  --preload-file nixpkgs \
	  --preload-file test.nix \
	  --preload-file derivation.nix

And this makes our current attempt at generalizing the build much more problematic. To my knowledge there is no:

emstorage nix-instantiate.html --postload-file nixpkgs ...

You get it... Let's see how we can work around this.

[qknight]

@tomberek A solution could be to make nix-instantiate a library, which is stored in the derivation, then using another mkDerivation use this to build the final artifact.

  tourofnix = stdenv.mkDerivation {
              name = "tourofnix";
              nativeBuildInputs = [ nix ];

Which strongly suggests days of Type III Fun -> https://www.rei.com/blog/climb/fun-scale

[qknight]

In the flake.nix I've now added:

  nixpkgs2211 = stdenv.mkDerivation {
    name = "nixpkgs-nixos-22.11-small";
    src = fetchgit {
      url = "https://github.com/nixos/nixpkgs.git";
      rev = "380be19fbd2d9079f677978361792cb25e8a3635";
      sha256 = "sha256-Zffu01pONhs/pqH07cjlF10NnMDLok8ix5Uk4rhOnZQ=";
    };
    phases = [ "installPhase" ];
    installPhase = ''
      mkdir -p $out
      cp -R $src/* $out
      du -sh $out
    '';
  };

the nix build phase has now:

    buildPhase = ''
      ln -s ${nixpkgs2211} src/nix-instantiate/nixpkgs
      make src/libexpr/eval.o
      make -j6 src/nix-instantiate/nix-instantiate.html
    '';

and inside the src/nix-instantiate/local.mk i've tried various variants:

nix-instantiate.html_LDFLAGS = \
  $(shell echo "$(NIX_LDFLAGS)" | cut -f 5- -d' ') \
  -lcrypto \
  -sNO_EXIT_RUNTIME=1 \
  -sALLOW_MEMORY_GROWTH=1 \
  -sEXPORTED_RUNTIME_METHODS=cwrap,ccall,UTF8ToString,ptrToString,allocateUTF8,UTF32ToString \
  -sEXPORTED_FUNCTIONS=_processExpr,_main_nix_instantiate2 \
  -sEXPORT_EXCEPTION_HANDLING_HELPERS \
  --preload-file ./nixpkgs
  #--preload-file src/nix-instantiate/nixpkgs

not sure

emscripten-nix> Error: Embedding "src/nix-instantiate/nixpkgs/env-vars" which is below the current directory "/build/source". This is invalid since the current directory becomes the root that the generated code will see
emscripten-nix> em++: error: '/nix/store/fif8p12nknb7cw8ffbxbsbrhr2gml3v4-emscripten-3.1.17/share/emscripten/tools/file_packager src/nix-instantiate/nix-instantiate.data --from-emcc --export-name=Module --preload src/nix-instantiate/nixpkgs' failed (returned 1)
emscripten-nix> make: *** [mk/lib.mk:119: src/nix-instantiate/nix-instantiate.html] Error 1
error: builder for '/nix/store/qk66j3q3n5hp4r75b64xms757a3hvfww-emscripten-nix.drv' failed with exit code 2;
       last 10 log lines:
       > em++: warning: ignoring dynamic library libnixfetchers.so because not compiling to JS or HTML, remember to link it when compiling to JS or HTML at the end [-Wemcc]
       >   LD     src/nix-instantiate/nix-instantiate.html
       > em++: warning: ignoring unsupported linker flag: `-rpath` [-Wlinkflags]
       > em++: warning: ignoring unsupported linker flag: `-rpath` [-Wlinkflags]
       > em++: warning: ignoring unsupported linker flag: `-rpath` [-Wlinkflags]
       > em++: warning: ignoring unsupported linker flag: `-rpath` [-Wlinkflags]
       > em++: warning: ignoring unsupported linker flag: `-rpath` [-Wlinkflags]
       > Error: Embedding "src/nix-instantiate/nixpkgs/env-vars" which is below the current directory "/build/source". This is invalid since the current directory becomes the root that the generated code will see

it is troublesome that i have to wait soooo long for every try. should probably make nix-instantiate into a library and then use that from another project for faster turnaround times.

roundtrip time on my machine is:

real    3m41.723s
user    0m0.209s
sys     0m0.997s

don't know how to debug this tbh

[qknight]

Oh, symlinks don't work it seems:

    buildPhase = ''
      mkdir -p nixpkgs
      cp -R ${nixpkgs2211} ./nixpkgs/
      du -sh nixpkgs
      make src/libexpr/eval.o
      make -j6 src/nix-instantiate/nix-instantiate.html
    '';

But this in fact, works!

[qknight]

not there yet

try {Module.ccall("main_nix_instantiate2","string",["string"],["with import <nixpkgs> { }; with stdenv.lib; { a=(2+2); q=lib.isList [];}"]);} catch (e) { console.log(getExceptionMessage(e).toString()); };
exception of type nix::ThrownError: �[31;1merror:�[0m
       … <borked>

         �[34;1mat �[35;1m«none»:0�[0m:�[3m (source not available)�[0m

       … <borked>

         �[34;1mat �[35;1m«none»:0�[0m:�[3m (source not available)�[0m

       �[35;1m(stack trace truncated; use '--show-trace' to show the full trace)�[0m

       �[31;1merror:�[0m file '�[35;1mnixpkgs�[0m' was not found in the Nix search path (add it using $NIX_PATH or -I)

       �[34;1mat �[35;1m«none»:0�[0m:�[3m (source not available)�[0m

[qknight]

Nearly there, nixpkgs is now in the data and i can read it:

try {Module.ccall("main_nix_instantiate2","string",["string"],["{ a=builtins.readDir ./nixpkgs;}"]);} catch (e) { console.log(getExceptionMessage(e).toString()); };
'{"a":{"CONTRIBUTING.md":"regular","COPYING":"regular","README.md":"regular","default.nix":"regular","doc":"directory","flake.nix":"regular","lib":"directory","maintainers":"directory","nixos":"directory","pkgs":"directory"}}
' 

Only thing left to do is adding the parameters to nix-instantiate.cc in:

-I nixpkgs=nixpkgs/ --eval --strict --show-trace /test.nix

[qknight]

https://github.com/emscripten-core/emscripten/issues/20599