Calling Haskell from C

[acarrico]

Suppose I create a haskell library intended to be called from C, such as in Calling Haskell from C. If I put install-includes in my cabal file, then the .h, .a, and .so files are available in the build, but how to use them in another nix derivation using stdenv.mkDerivation?

The includes (etc.) don't seem to end up in the system path found by the C compiler. They are inconveniently located in a cryptic paths like ${out}/ghc-8.8.4/x86_64-linux-ghc...........etc........../include.

Is there a way to locate these without searching? For example is there a way to tell cabal2nix to install or link them in ${out}/include, etc., or better yet integrate with stdenv, or to provide data for pkg-config?

[maralorn]

I am quite out of my depth here. And there might well be a correct solution to this problem.

But maybe you can just move the files in the postinstall phase? I don‘t think you can get cabal2nix to do that, but you can use pkgs.haskell.lib.overrideCabal to set the postInstall.

[sternenseemann]

If cabal has some kind of pkg-config support, that'd be ideal, otherwise we'll probably have to add a setup hook for this.

[acarrico]

Just to sketch out the "find" option:

[nix-shell:~/org/src/calling-haskell-from-c]$ find .
./hs-to-c/src/HsToC/Fib.hs
./hs-to-c/hs-to-c.cabal
./hs-to-c/default.nix
./hs-from-c
./hs-from-c/default.nix
./hs-from-c/Makefile
./hs-from-c/src
./hs-from-c/src/main.c

with ./hs-to-c/hs-to-c.cabal:

name: hs-to-c
author: Anthony Carrico
version: 0
license: PublicDomain
build-type: Simple
cabal-version: >=2.0

library
  hs-source-dirs: src
  ghc-options: -stubdir include
  include-dirs: include
  install-includes: HsToC/Fib_stub.h
  exposed-modules: HsToC.Fib
  build-depends: base
  default-language: Haskell2010

and ./hs-from-c/default.nix

{ stdenv, hs-to-c }:
stdenv.mkDerivation {
  pname = "hs-from-c";
  version = "0";
  src = ./.;
  buildInputs = [ hs-to-c ];
  hs-to-c = hs-to-c;
}

and ./hs-from-c/Makefile

fib: src/main.c
	${CC} -o $@ src/main.c -I ${shell find ${hs-to-c} -name include}

install: fib
	mkdir -p ${out}/bin && cp fib ${out}/bin

This actually doesn't build because it can't find HsFFI.h, but that is another issue.

  In file included from src/main.c:1:
  /nix/store/vc1zc693n4yrkgkc9g65ahw58psil9l8-hs-to-c-0/lib/ghc-8.8.4/x86_64-linux-ghc-8.8.4/hs-to-c-0-F08eZePK8mrFMbdPz3rxjP/include/HsToC/Fib_stub.h:1:10: fatal error: HsFFI.h: No such file or directory
      1 | #include "HsFFI.h"
        |          ^~~~~~~~~
  compilation terminated.

[acarrico]

I can get a little closer with this ./hs-from-c/default.nix:

{ stdenv, hs-to-c, haskellPackages }:
stdenv.mkDerivation {
  pname = "hs-from-c";
  version = "0";
  src = ./.;
  buildInputs = [ hs-to-c ];
  hs-to-c = hs-to-c;
  ghc = haskellPackages.ghc;
}

And ./hs-from-c/Makefile

fib: src/main.c
	${CC} -o $@ src/main.c -I ${shell find ${hs-to-c} -name include} -I ${shell find ${ghc} -name HsFFI.h | xargs dirname} -L ${shell find ${ghc} -name "libHSrts-ghc*.so" | xargs dirname} -L ${shell find ${hs-to-c} -name "libHShs-to-c*.so" | xargs dirname} -l HSrts -l HShs-to-c-0-F08eZePK8mrFMbdPz3rxjP-ghc8.8.4

install: fib
	mkdir -p ${out}/bin && cp fib ${out}/bin

I cheated with the HShs-to-c-0-F08eZePK8mrFMbdPz3rxjP-ghc8.8.4 name, and I am still missing libraries. This begs the question, what would actually need to be in a pkg-config? Obviously more than my .so and libHSrts-ghc.so (which was just a guess because I found hs_init in there with nm). Forgetting about cabal2nix, how do you even link with CC? The original example uses ghc to link, which defeats the purpose of providing a library for use from C, but I can try that just to see if it will work:

fib2: src/main.c
	${ghc}/bin/ghc --make -no-hs-main -optc-O src/main.c -o $@ -I${shell find ${hs-to-c} -name include} -L${shell find ${hs-to-c} -name "libHShs-to-c*.so" | xargs dirname} -l HShs-to-c-0-F08eZePK8mrFMbdPz3rxjP-ghc8.8.4

install: fib2
	mkdir -p ${out}/bin && cp fib2 ${out}/bin

This actually builds. It doesn't load properly though:

$ result/bin/fib2 
result/bin/fib2: symbol lookup error: /nix/store/vanx5iavd66hpwfkzp0rwn7ss2wn726q-ghc-8.8.4/lib/ghc-8.8.4/ghc-prim-0.5.3/libHSghc-prim-0.5.3-ghc8.8.4.so: undefined symbol: stg_setThreadAllocationCounterzh

[acarrico]

I've been able to link main.o written in C and compiled with $CC to a haskell Module Fib either

• compiled to fib.so with ghc, and linked with $CC

or

• compiled to fib.o with ghc, and linked with ghc

but not

• compiled to fib.o with ghc, and linked with $CC

I can get the last to build (using ghc -v3 or readelf -d libFib.so | grep NEEDED to figure out how to link), but unfortunately it segfaults at run time. Here are the two working Makefiles:

GHC=${ghc}/bin/ghc
ghc-version=${shell ${GHC} --numeric-version}
ghc-libdir=${shell ${GHC} --print-libdir}

NIX_CFLAGS_COMPILE=-L ${shell find ${ghc} -name 'libHSrts-ghc*.so' | xargs dirname} \
                   -l HSrts_thr-ghc${ghc-version} \
                   -I ${shell find ${ghc} -name HsFFI.h | xargs dirname}

# The -rpath here is just a temporary hack to avoid LD_LIBRARY_PATH=. while testing:
fib: main.o libFib.so
	$(CC) -o $@ main.o libFib.so  -Xlinker -rpath -Xlinker .

# Add ghc flag -v3 to observe CC calls.
# The NIX_CFLAGS_COMPILE above interfere with ghc's CC calls.
Fib.hi libFib.so Fib_stub.h: Fib.hs
	NIX_CFLAGS_COMPILE="" ${GHC} -O2 -dynamic -shared -fPIC -o libFib.so Fib.hs

main.o: main.c Fib_stub.h
	$(CC) -c main.c

and

GHC=${ghc}/bin/ghc

# Add ghc flag -v3 to observe CC calls.
fib: main.o Fib.o Fib.hi
	${GHC} --make -no-hs-main -optc-O main.o Fib -o $@

Fib.hi Fib.o Fib_stub.h: Fib.hs
	${GHC} -c -O Fib.hs

main.o: main.c Fib_stub.h
	$(CC) -I ${shell find ${ghc} -name HsFFI.h | xargs dirname} -c main.c

It would be nice if ghc had a way to export its magic to help you call the linker yourself (maybe it does), otherwise you seem to be stuck going through a ghc linked .so file, or linking with ghc. Anyway, these can probably be generated with cabal, and maybe provided with cabal2nix, and used as in the examples, but I haven't tried doing so with cabal or cabal2nix yet.

If that works, I guess the support needed from cabal2nix would mainly be the nix magic to setup the NIX_CFLAGS_COMPILE automatically, if it doesn't do that already (I haven't tried creating a .so with cabal2nix).

NOTES: Apparently there are two options for the library libHSrts and libHSrts_thr. Of course you would need to add the actual library, libFib.so in this case, and the include directory. Finally, the wiki mentions that pkg-config is needed when the headers go in a subdir, which seems likely given that haskell modules are arranged in subdirs.

[acarrico]

I've created a derivation that allows the C compiler and the linker to access ghc's includes and rts libraries:

{ stdenv, haskellPackages }:

let ghc = haskellPackages.ghc; in

stdenv.mkDerivation {
  pname = "ghc-from-c";
  version = "0";
  src = ./.;
  installPhase = ''
    mkdir -p $out
    ln -s ${ghc}/lib/ghc-${ghc.version}/include $out/include
    ln -s ${ghc}/lib/ghc-${ghc.version}/rts/ $out/lib
'';

  meta = {
    description = "Support for compiling and linking C code which calls Haskell Code";
    longDescription = ''
      When compiling and linking C code which calls Haskell code, it
      is necessary to access C includes from the ghc package
      (HsFFI.h). When linking, it may also be necessary to access
      ghc's rts libraries (although this can be avoided by arranging
      for C code linked by ghc to call hs_init and hs_exit). This
      derivation adds support. To access these resources simply
      include this in the C code's stdenv derivation.

      In particular, this derivation causes the cc-wrapper setup-hook to
      notice ghc's include directory, adding it to NIX_CFLAGS_COMPILE, and
      the bintools-wrapper setup-hook to notice ghc's rts library
      directory, adding it to NIX_LDFLAGS.

      See nixpkgs/pkgs/build-support/bintools-wrapper/setup-hook.sh and
      nixpkgs/pkgs/build-support/cc-wrapper/setup-hook.sh for details on
      the mechanism.
'';
    license = stdenv.lib.licenses.publicDomain;
    platforms = stdenv.lib.platforms.all;
  };
}

It is a concrete step in the right direction, but it doesn't solve the problem for the user's own libraries.

[acarrico]

For your own libraries, you can pull the same trick I did above in ghc-from-c. In the hs-to-c example, here is an export.nix file beside the cabal2nix generated default.nix to do the job:

{ stdenv, ghc, hs-to-c }:

stdenv.mkDerivation {
  pname = "hs-to-c-export";
  version = "0";
  src = ./.;
  installPhase = ''
    mkdir -p $out
    ln -s $(find ${hs-to-c} -name include) $out/include
    ln -s ${hs-to-c}/lib/ghc-${ghc.version} $out/lib
'';
}

Also, I've fixed the cabal file to create a foreign library, and included csrc/HsToC.c which wraps hs_init and hs_exit. This trick forces ghc to link its runtime system into the library, which avoids the linking problem later, although my ghc-from-c derviation above provides an alternative solution for linking the rts manually.

name: hs-to-c
author: Anthony Carrico
version: 0
license: PublicDomain
build-type: Simple
cabal-version: >=2.0

foreign-library HsToC
  type: native-shared
  lib-version-linux: 0.0.0
  hs-source-dirs: src
  c-sources: csrc/HsToC.c
  ghc-options: -stubdir include
  include-dirs: include
  install-includes: HsToC/Fib_stub.h csrc/HsToC.h
  exposed-modules: HsToC.Fib
  build-depends: base
  default-language: Haskell2010
  other-modules: HsToC.Fib

On the C side, the derivation looks like this:

{ stdenv, hs-to-c, hs-to-c-export, ghc-from-c }:

stdenv.mkDerivation {
  pname = "hs-from-c";
  version = "0";
  src = ./.;
  buildInputs = [ hs-to-c hs-to-c-export ghc-from-c ];
}

It is also possible to avoid creating hs-to-c-export and do it on the import side in your Makefile, or in the buildPhase, something like this:

  buildPhase = ''
    hs_to_c_include=$(find ${hs-to-c} -name include)
    export NIX_CFLAGS_COMPILE="-I$hs_to_c_include $NIX_CFLAGS_COMPILE"
    export NIX_LDFLAGS="-L${hs-to-c}/lib/ghc-${ghc.version} -rpath ${hs-to-c}/lib/ghc-${ghc.version} $NIX_LDFLAGS"
    make fib
'';

This last solution isn't very good for development, because those env vars aren't setup automatically when you enter nix-shell, as they would be in the prior solution.

[acarrico]

Ok. So my last two comments are a full solution to the original issue. To provide the solution to others, either

1. my ghc-from-c derivation should go in nixpkgs, and the export trick should be documented or
2. the links created by ghc-from-c should go in haskellPackages.ghc, and cabal2nix should somehow cause the links to be created for packages (at least those with foreign-library clauses).

And one final note, it seems impossible to avoid using find in the one spot above, I don't really know anything about cabal, but I guess that is because of its "nix-style builds", although I'm not sure why that is an issue for includes but not the library itself.

[MathiasSven]

@acarrico Do you happen to still have the source files for this example? I know some things have changed, but I am trying to replicate this, and I'm encountering some issues.

[acarrico]

@MathiasSven, I uploaded this for you. It seems like the default.nix there was probably the overlay I was using on this issue. I haven't tested this recently.