Generate/write nix derivations for all the haskell code

[akshaymankar]

Most of these nix files are generated using ./hack/bin/generate-nix-overrides.sh and ./hack/bin/generate-local-nix-overrides.sh.

The generate-nix-overrides.sh script reads pins.yaml (Yes, I invented another YAML thing :cry:) and generates nix derivations using cabal2nix and puts them in nix/haskell-overrides/. It also generates nix/haskell-overrides.nix which contains the overrides required to make these derivation part of any pkgs.haskell.packages.<ghc> package set.

The generate-local-nix-overrides.sh script finds all the cabal files in the repository and generates a nix derivation for each package using cabal2nix and puts each derivation next to the package itself in a default.nix file. It also generates nix/local-overrides.nix similar to the haskell-overrides.nix.

The build.nix at the root of the repo, combines these two overrides and also makes sure the packages which are not pinned by pins.yaml work. The reason for not pinning these packages is that we don't actually care if these are new and we should keep upgrading them until we can. This way we can keep the pinned things to a minimum.

TODO:

• [ ] Add details about how shellFor and docker images work.
• [ ] Make building without optimization configurable
• [ ] Ensure that HLS works

[smatting]

The build.nix at the root of the repo

This seems to be missing from the PR

[akshaymankar]

The build.nix at the root of the repo

This seems to be missing from the PR

Added it now.

[stephen-smith]

How much of this is generated vs. written? I naturally recoil at treating anything generated as source code and checking it in, though I certainly understand it's the path of least resistence in some cases.

[akshaymankar]

How much of this is generated vs. written? I naturally recoil at treating anything generated as source code and checking it in, though I certainly understand it's the path of least resistence in some cases.

All the new files in the PR except pins.yaml, build.nix and the 3 shell scripts in hack/ are generated. For all the default.nix files, we could use callCabal2nix function, but a file with list of those calls to the function for each package would either need to be generated or manually written. For all the pinned things, I think it is possible to write something with callCabal2nix and fetchGit, but it will be a little effort to do this. That said, I think this will also make things harder to debug, so I would still prefer just committing these files and have CI ensure that these files are not out of date.

[pcapriotti]

A few questions, sorry if they are dumb, but I still struggle with nix things.

• Why are you generating all these nix files? Can't cabal2nix or whatever just read the cabal file dynamically and create the corresponding derivation?
• Are we dropping stackage now? Is this necessary? Couldn't we be using the same pins as before and avoid the trouble of finding a consistent choice of versions? This might become an issue in the future.
• Relatedly, why do we need the hackage portion of pins.yaml? Can't we get versions from the existing yaml files (i.e. cabal.project.freeze).

[akshaymankar]

Why are you generating all these nix files? Can't cabal2nix or whatever just read the cabal file dynamically and create the corresponding derivation?

There exists a callCabal2nix but it only works for local paths. I think we can write something similar to it to avoid generating the files. We will still have to generate a file which calls this new function with the right parameters and creates the overrides.

Are we dropping stackage now? Is this necessary? Couldn't we be using the same pins as before and avoid the trouble of finding a consistent choice of versions? This might become an issue in the future.

The pkgs.haskell.packages.ghc8107 already follows some stackage pin, but it refers to a nightly snapshot. The overrides are similar to our overrides written in the stack.yaml.

Relatedly, why do we need the hackage portion of pins.yaml? Can't we get versions from the existing yaml files (i.e. cabal.project.freeze).

We want to not rely on stack.yaml so I chose to not read it. The cabal.project.freeze file is not a YAML file and I didn't feel like writing a parser and it contains more versions than we want to pin as we want to follow some stackage snapshot. So, for this approach to work we have to delete the cabal.project.freeze file along with the source-repository-package sections in the cabal.project file.

We could also try to get rid of the pins.yaml file if we can get a function similar to callCabal2nix. If we do that we can write all our pins in the build.nix file.

[smatting]

I rebased on latest develop and updated pins.yaml to track cabal.project in the rebased commit.

Currently build failures:

Configuring partial-isomorphisms-0.2.3.0...
CallStack (from HasCallStack):
  $, called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:1024:20 in Cabal-3.2.1.0:Distribution.Simple.Configure
  configureFinalizedPackage, called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:477:12 in Cabal-3.2.1.0:Distribution.Simple.Configure
  configure, called at libraries/Cabal/Cabal/Distribution/Simple.hs:625:20 in Cabal-3.2.1.0:Distribution.Simple
  confHook, called at libraries/Cabal/Cabal/Distribution/Simple/UserHooks.hs:65:5 in Cabal-3.2.1.0:Distribution.Simple.UserHooks
  configureAction, called at libraries/Cabal/Cabal/Distribution/Simple.hs:180:19 in Cabal-3.2.1.0:Distribution.Simple
  defaultMainHelper, called at libraries/Cabal/Cabal/Distribution/Simple.hs:116:27 in Cabal-3.2.1.0:Distribution.Simple
  defaultMain, called at Setup.hs:2:8 in main:Main
Setup: Encountered missing or private dependencies:
template-haskell >=2.17

[fisx]

i checkout this branch out on a fresh clone. direnv allow succeeded after 40 minutes. make c had this to say:

Configuring wire-message-proto-lens-0.1.0...
[ 9 of 16] Compiling Test.Schema.Util ( test/Test/Schema/Util.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/Util.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/Util.dyn_o )
[10 of 16] Compiling Test.Schema.UserSpec ( test/Test/Schema/UserSpec.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/UserSpec.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/UserSpec.dyn_o )
[11 of 16] Compiling Test.Schema.ResourceSpec ( test/Test/Schema/ResourceSpec.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/ResourceSpec.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/ResourceSpec.dyn_o )
[12 of 16] Compiling Test.Schema.PatchOpSpec ( test/Test/Schema/PatchOpSpec.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/PatchOpSpec.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/PatchOpSpec.dyn_o )
setup: filepath wildcard 'generic-message-proto/proto/otr.proto' does not
match any files.

[13 of 16] Compiling Test.Schema.MetaSchemaSpec ( test/Test/Schema/MetaSchemaSpec.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/MetaSchemaSpec.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/MetaSchemaSpec.dyn_o )
[14 of 16] Compiling Test.Schema.GroupSpec ( test/Test/Schema/GroupSpec.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/GroupSpec.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/GroupSpec.dyn_o )
[15 of 16] Compiling Test.Schema.AuthenticationSchemeSpec ( test/Test/Schema/AuthenticationSchemeSpec.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/AuthenticationSchemeSpec.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Test/Schema/AuthenticationSchemeSpec.dyn_o )
[16 of 16] Compiling Main             ( test/Spec.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Main.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec-tmp/Main.dyn_o )
Linking /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/hscim-0.3.6/t/spec/build/spec/spec ...
[26 of 26] Compiling Wire.Swagger     ( src/Wire/Swagger.hs, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/types-common-0.16.0/build/Wire/Swagger.o, /home/mf/tmp/wire-server/dist-newstyle/build/x86_64-linux/ghc-8.10.7/types-common-0.16.0/build/Wire/Swagger.dyn_o )
cabal: Failed to build wire-message-proto-lens-0.1.0 (which is required by
test:wire-api-tests from wire-api-0.1.0, test:wire-api-golden-tests from
wire-api-0.1.0 and others).

[smatting]

@fisx You need to fetch submodules in wire-server too, git submodule update --init --recursive

[fisx]

there was a disk space issue. with enough disk space, make full-clean c ci still fails in the same place.

[fisx]

git submodule update --init --recursive

that sounds helpful! is there a way to make this automatic? if not i'll add this trick to the PSA we already give i make c fails.

[smatting]

there was a disk space issue. with enough disk space, make full-clean c ci still fails in the same place.

What is the eror message? It should be different when you've pulled the submodules. For pulling submodules automatically I think https://stackoverflow.com/a/49427199

[fisx]

there was a disk space issue. with enough disk space, make full-clean c ci still fails in the same place.

What is the eror message? It should be different when you've pulled the submodules. For pulling submodules automatically I think https://stackoverflow.com/a/49427199

trying with submodules now, hopeful this will work.

[fisx]

git submodule update --init --recursive

that sounds helpful! is there a way to make this automatic? if not i'll add this trick to the PSA we already give i make c fails.

https://github.com/wireapp/wire-server/pull/2761/commits/c9acc5aa574c017973f130734c6334ba92a27300

[smatting]

direnv allow succeeded after 40 minutes

I'm wondering why it took so long. We don't build the devShell on ci I think, however all the wireServerPackages should be in the cache, right @akshaymankar ?

[fisx]

direnv allow succeeded after 40 minutes

I'm wondering why it took so long. We don't build the devShell on ci I think, however all the wireServerPackages should be in the cache, right @akshaymankar ?

I'm not sure and I lost the terminal and no logs, but I don't remember any building. Maybe the nix-cache server is just slow? Haven't done the numbers to decide how plausible that is. I'm on a good uplink, though.

[fisx]

another weird thing: every shell command output is not enriched with this direnv noise:

~/tmp/wire-server$ ls
cabal.project	      changelog.d   charts  dist-newstyle  hack      libs     Makefile	package-defaults.yaml  SECURITY.md  shell.nix  tools
cassandra-schema.cql  CHANGELOG.md  deploy  docs	   hie.yaml  LICENSE  nix	README.md	       services     snapshots
direnv: loading ~/tmp/wire-server/.envrc
direnv: loading ~/tmp/wire-server/.env/.env/dev-shell.sh
direnv: loading ~/tmp/wire-server/.env/profile-env/.env/profile-env/.profile
./.envrc:40: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
direnv: export +AR +AS +CC +CONFIG_SHELL +CXX +DETERMINISTIC_BUILD +HOST_PATH +IN_NIX_SHELL +LC_ALL +LD +LOCALE_ARCHIVE +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_BUILD_CORES +NIX_BUILD_TOP +NIX_CC +NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_CFLAGS_COMPILE +NIX_ENFORCE_NO_NATIVE +NIX_GHC +NIX_GHCPKG +NIX_GHC_DOCDIR +NIX_GHC_LIBDIR +NIX_HARDENING_ENABLE +NIX_INDENT_MAKE +NIX_LDFLAGS +NIX_PATH +NIX_PKG_CONFIG_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_STORE +NM +OBJCOPY +OBJDUMP +PKG_CONFIG +PKG_CONFIG_PATH +PYTHONHASHSEED +PYTHONNOUSERSITE +PYTHONPATH +RANLIB +READELF +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +XDG_DATA_DIRS +_PYTHON_HOST_PLATFORM +_PYTHON_SYSCONFIGDATA_NAME +buildInputs +builder +cmakeFlags +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +installPhase +mesonFlags +name +nativeBuildInputs +out +outputs +patches +phases +propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook +stdenv +strictDeps +system ~PATH ~WINDOWID

can this be turned off?

[fisx]

The build process was killed (i.e. SIGKILL). The
typical reason for this is that there is not enough memory available (e.g. the
OS killed a process using lots of memory).

trying again, with less system noise.

sigh

[akshaymankar]

I'm wondering why it took so long. We don't build the devShell on ci I think, however all the wireServerPackages should be in the cache, right @akshaymankar ?

@fisx do you have https://wire-server.cachix.org/ in your nix substituters?

another weird thing: every shell command output is not enriched with this direnv noise:

This happened to me once, I deleted .direnv in wire-server and it went away. I will look deeper into direnv to see if this step is unnecessary.

[fisx]

The build process was killed (i.e. SIGKILL). The
typical reason for this is that there is not enough memory available (e.g. the
OS killed a process using lots of memory).

trying again, with less system noise.

sigh

forgot to copy cabal.profile.local into the fresh clone. i'm not going to miss a single trap... :)

[akshaymankar]

forgot to copy cabal.profile.local into the fresh clone. i'm not going to miss a single trap... :)

Oh yeah, compiling with -O2 requires a lot of memory right now :(