(Towards) stable C bindings for libutil, libexpr

[yorickvP]

This PR introduces a stable C API that can be used to interface with the Nix interpreter externally.

Motivation

In the past, we've wanted to interface Nix with other languages.

This had some problems:

• The C++ ABI is not compatible with many other languages
• The GC makes it hard to deal with references from other languages
• It was hard to manipulate C++ stdlib types such as std::string from other languages
• Ownership was handled by C++ RAII, which is hard to be compatible with
• The Nix C++ API exposed all of the Nix internals, causing a quickly-moving target that was hard to stay up-to-date with.

A Nix plugin ecosystem was proposed, but never got off the ground for these reasons.

Perl bindings are maintained upstream, and this was the intention for python bindings, as well.

To resolve these difficulties, a stable C API would be advantageous.

Related: https://github.com/NixOS/nix/issues/7271

State

These bindings have been used to develop python bindings, which we intend to move to nix-community after this PR is finished.

Context

This is the second approach to making python bindings, this time we're focusing on creating a stable Nix API, and binding to it via CFFI. This also make sure we don't have to duplicate the effort to interface with Nix for every language.

Implementation strategy

GC

Objects that are managed by the GC are ref-counted in this API. API consumers should call nix_gc_decref() when they are done with a Value. This makes sure the GC does not have to scan the API consumers heap.

Alternatives

• Wrap pointers to these objects
  • (+) Hard to use incorrectly
  • (+) Easy to understand
  • (-) Header to interoperate: would not be compatible, for example, the current C++ API.
  • (-) Overhead: any use of these would require an additional dereference
  • (-) Administration cost: creating multiple references on the user side would necessitate some bookkeeping.
• GC references that you can keep separately
  • (+) Minimal overhead
  • (+) Usage optional if Boehm can see into your heap
  • (-) Exposes implementation details (type of garbage collector)
  • (-) Weird separation between gc-ref and values
  • (-) Hard to understand and use correctly

Error handling

To handle C++ exceptions from a C API, a structure called nix_context has been created, which can be allocated by the API consumer and passed to functions. Error state (exception info, strings, error type) can be stored in this context. Many functions also return a nix_err code or NULL on errors.

Alternatives considered:

• More minimal C API

  • (+) Easier to maintain
  • (-) Lessens the benefit from this PR, users will still have to maintain their own wrappers.
  • (-) Would make it harder to do this:

    pkgs["hello"]["overrideAttrs"](lambda o: {
      "pname": str(o["pname"]) + "-test"
    })
    

• Stabilizing the C++ API

  • (+) We already have the C++ API
  • (-) The current API surface is unnecessarily big, it would be hard to document and sift through it all.
  • (-) This would make it harder to make changes to the codebase
  • (-) C++ is harder to interoperate from most other languages

• Creating a new, stable C++ API

  • (+) Doesn't use a language from the 70's
  • (-) As much work as C bindings
  • Something like pimpl could help with long-term compatibility
  • (-) Easier to accidentally expose implementation details such as dataclasses
  • (-) C++ is harder to interoperate from most other languages

• Keeping the status quo

  • (-) No stability guarantees are made, causing plugin authors and nix lib users to frequently have to update their code, and many projects to stop being maintained or resort to wrapping the CLI.
  • (-) Documentation isn't great, you have to read the source to understand what's going on.

How to read the diff

• Check out the Nixcon Talk
• Documentation has, for now, been rendered at https://pub.yori.cc/nix-c-api-docs/modules.html
• A C example is in the linked documentation, also check out the python examples (not part of this PR)

Checklist for me

• [x] Low-level documentation
• [ ] High-level documentation
• [ ] Tests
• [ ] libstore bindings

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

• [x] agreed on idea
• [x] agreed on implementation strategy
• [ ] tests, as appropriate
  • functional tests - tests/**.sh
  • unit tests - src/*/tests
  • integration tests - tests/nixos/*
• [ ] documentation in the manual
• [ ] documentation in the internal API docs
• [ ] code and comments are self-explanatory
• [ ] commit message explains why the change was made
• [ ] new feature or incompatible change: updated release notes

Priorities

Add :+1: to pull requests you find important.

Affiliation

This PR was sponsored by Antithesis

[alex-ameen]

You're my hero.

Please reach out if you'd like any help with similar work. libstore is on my wish list obviously.

[fricklerhandwerk]

Reviewed and discussed in Nix team meeting (maintainers will add their own detailed reviews on top):

• we're open to merge this as an unstable interface, documented such that application developers are encouraged to try it out
  • it could become stable if it sees a lot of use
• it would be good to have the PR for Python bindings so we can look at it
  • it should be in a separate derivation (as with the Perl bindings) so we don't grow the dependency closure
• remove thunks and GC (merge GCRef into Value) to make it simpler to use, demote externals (to discuss with @roberth what exactly that would entail)
• adapt the coding style to the rest of the project
  • use the LLVM formatter

Complete discussion

• @edolstra: very unenthusiastic about this
  • don't want to maintain a sizeable C library, it's a 1970s language
  • would not be a problem if it were maintained by someone else outside of the repo
  • @thufschmitt: there is no actual logic in C, only wrapper types
  • any change to the C++ API will need work on the C API
    • @thufschmitt: hopefully not, as the C API is supposed to be stable
  • @roberth: working with @yorickvp to better isolate the internals
  • if we were to have a stable API it would be C++
    • @fricklerhandwerk: the rationale is that other languages will have FFI infrastructure
    • @thufschmitt: could provide a C++ interface and add C in a library, but that would make the pipeline more complicated
• @fricklerhandwerk: can we agree that we want a stable API?
• @edolstra: practical example of limited maintainability: lazy tree branch changes the path type from string to a tuple.
  • @roberth: could add the path to the store
    • @thufschmitt: this would keep the original performance characteristics
• @edolstra: does this allow you to add primops?
  • if you can, it would make it extremely tied to interpreter internals
  • @thufschmitt: this is not exposed anywhere in stable interface
  • thinks like nix_set_thunk
• @fricklerhandwerk: some background:
  • @infinisil is writing Nixpkgs libraries we want to run lots of test cases on, such as for property tests
  • that requires reducing the overhead for evaluation, and originally @infinisil picked up @Mic92's Python bindings
  • those were broadly agreed upon, but deemed too invasive for now, so @yorickvp picked up on a C interface
  • @roberth: another use case is nixos-option which was updated recently
• @edolstra: this de facto requires the C++ API to be stable, as this is so closely tied to it
  • @roberth: we can stabilise it in stages, as C APIs tend to do
  • @ericson2314: we could start out of tree to demonstrate that APIs are not moving as fast as feared
  • @thufschmitt: if we want a stable C++ interface, we should design it such that it's a subset of what we use internally, and even then use that internally as much as possible
• @edolstra: we could have an interface that takes a string and evaluates to an opaque value that can be queried to be a string or int
• (initial review)
  • thunks should not be part of that interface
  • GCref exposes internals, probably should be integrated into Value
  • we should probably make it a bit simpler and safer overall, and possibly have an additional less-stable layer that allows for more fidelity
  • external should not be part of it, as that would expose a highly unstable interface even more
  • constructing and introspecting arbitrary Nix values is more involved than what we would want for a first version
  • not sure if we want explicit forcing, which relates to the question if we should expose thunks
    • you'd get strictness issues
    • as a consumer of the API you may not ever want to deal with thunks
    • we may want to have two interfaces, one which never returns thunks, it would be a lot easier to work with
      • Hercules CI does something similar, and it allows lightweight provenance tracking
  • having the API abstract enough would also allow wrapping it around the eval cache
  • would be good to see the Python bindings first, to clarify what's needed for C and why
    • alternatively, C bindings could live together with Python and be factored out when they are needed
• @ericson2314: general observations:
  • we haven't done a lot of the foundational work for these stable interface to be easy to do
  • we want as much as possible out of tree, so Nix can be used more as a library
  • there are different kinds of costs and different strategies to deal with them
  • we don't even know why people are not using C++ bindings
    • no stability guarantees?
    • no documentation?
    • scared of C++?
    • easier to use CLI?
  • we have to move forward somehow, and it seems we're opting for validation rather than testing in this case

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-07-17-nix-team-meeting-minutes-72/30574/1

[roberth]

• remove thunks

To clarify, we do want to keep the ability to "have" a thunk, as long as you don't try to observe the value; in other words keep it lazy, but force it in nix_get_type. NIX_TYPE_THUNK can then be removed. The goal is to make the interface simpler safer, while preserving laziness.

[infinisil]

we don't even know why people are not using C++ bindings

@Ericson2314 https://github.com/Mic92/pythonix used them, but iirc it stopped being maintained because Nix frequently changed the C++ bindings, requiring changes in pythonix too. See this commit from my initial Python bindings PR, that updates pythonix with newer C++ API changes. The original idea was to just upstream pythonix so that downstream wouldn't be burdened with this.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-cli-working-group/30517/14

[alex-ameen]

I use the C++ bindings all the time. They work great. Plugins are incredibly useful.

[lf-]

As the author of nix-doc, I have also experienced API breakage on my relatively minimal (a builtin that takes strings and lambdas) C++ Nix plugin on about 5 releases in 2 years. Anyone doing something more complex definitely would be hitting these both more severely and more frequently.

Nix's C++ API is definitely not currently very stable. One benefit of a C API that's not used as much internally in Nix is that it is more likely to be decoupled from the faster-changing Nix internals, and I'm in favour of this on this basis alone.

[puckipedia]

To clarify, we do want to keep the ability to "have" a thunk, as long as you don't try to observe the value [..] The goal is to make the interface simpler safer, while preserving laziness.

I'm unconvinced this actually makes the API much simpler; you still have to handle the same (or more!) amount of errors (what if nix_show_type tries to force a thunk that errors?), while losing an important introspection axis (translating thunks to a Promise object, which is how my use of the CFFI does it already).

In my uses, GCRef is also important for non-Value objects, like primops and external values; as they are how my bindings make native-code lambdas that can be called from Nix code and safely GC'd.

[andreabedini]

Just a drive by comment to mention that C++ API can be made stable using patterns like pimpl. Not to say a stable C API would not be appreciated, quite the opposite! 💜

[lf-]

Just a drive by comment to mention that C++ API can be made stable using patterns like pimpl. Not to say a stable C API would not be appreciated, quite the opposite! purple_heart

I don't think this is a good idea personally. The C++ API being unstable has enabled lots of the recent error messages work and other refactors and it feels like a mistake to try to stabilize the core APIs as they are rather than provide some second intentionally-stable wrapper APIs for other clients. (meta-thought: I don't know how many people writing Nix extensions actually want to be writing C++ if it's avoidable. I don't, and the two extensions I maintain are basically C ABI shims to call Rust where all the actual work is done; I would be happy to stop maintaining this-PR-but-worse)

[andreabedini]

I don't think this is a good idea personally. The C++ API being unstable has enabled lots of the recent error messages work and other refactors and it feels like a mistake to try to stabilize the core APIs as they are rather than provide some second intentionally-stable wrapper APIs for other clients

I didn't mean the internal API has to be stable (you are right, how could the code base evolve then!?). Instead I meant that one can offer a new external stable C++ API, which wraps the internal code just like the C wrapper would do. In other words: "C vs C++" and "stable vs unstable" are orthogonal issues.

[roberth]

Title updated to reflect that we won't promise stability immediately after merge.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-unit-a-nix-unit-testing-runner-compatible-with-runtests/30765/2

[DieracDelta]

the two extensions I maintain are basically C ABI shims to call Rust where all the actual work is done

@lf- what extensions are these? Are these using the nix plugin infra?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-07-27-documentation-team-meeting-notes-67/30998/1

[yorickvP]

Thank you all for the feedback!

I've made several updates:

• Considered several alternatives in the PR description
• Incorporated some review comments: Exprs are now gone
• Improved documentation a bit: https://pub.yori.cc/nix-c-api-docs/
• Moved from the separate GC references to an internal reference counter (using std::unordered_map on object pointers).

For a motivating example for this API, check out the WIP python bindings: https://github.com/tweag/nix/tree/nix-c-bindings-python/python

Changing everything to be forcing could be possible, but would require passing State* pointers around, which I'm not entirely happy with.

[Ericson2314]

Apologies if I forgot to comment on this before, but I feel the C code ought to be in separate libraries. We don't want other parts of Nix using this stuff --- C++ should only use the C++ and not C wrapper --- and separate libraries enforces that.

[Ericson2314]

Thanks very much @yorickvP. I think I would slightly prefer it to flatten all the "subprojects" to one layer within src, like src/libexpr-tests, src/libexprc, but that is a departure from what is currently there, whereas what you did matches how the tests are handled. So makes totally sense you didn't do that yet.

[edolstra]

@yorickvP Can you reformat the C/C++ code to follow the Nix coding style? See https://github.com/NixOS/nix/blob/29d6097b906ce36581549d8d7f001c54a56a9a0a/.clang-format. The same applies to file names, e.g. nix_api_expr.cc should be nix-api-expr.cc.

[fricklerhandwerk]

Reviewed in discussed in Nix team meeting 2023-08-25:

• @ericson2314: The layout I would like to tweak, testing packages as we do today with test/ is very fragile because we get nested -I flags and the precedence between that and sibling file (same dir) #include "..." lookup starts to matter. I would like to fix that up soon both to set a clear pattern for this, and to make the person working on the Meson port less miserable fighting the same issues.

• @fricklerhandwerk: proposal:

  • remove externals, can still add them later if deemed necessary
    • can use custom primops as stand-in
  • declare unstable with the goal to make it stable
  • merge when tests are added and documentation is fixed up

• @tomberek: may look like "yet another experimental interface"

  • @ericson2314: At least it is not greenfield code new features, but just exposing what we've got in a more flexible way. Ultimately, making it easier to make downstream programs that use Nix more easily should reduce the preassure to get features in upstream Nix.

• @edolstra: not opposed, but one consideration is that it adds maintenance burden

  • @fricklerhandwerk: also in terms of documentation. but it adds a lot of value to consumers
  • @roberth: don't expect significant issues from this; the underlying mechanisms are very unlikely to change

• idea approved

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-08-25-nix-team-meeting-minutes-82/32283/1

[zmitchell]

It's unclear to me how you would construct a Value and then have nix print it for you.

I see a couple of functions that take a nix_printer, but I don't see any functions that construct a nix_printer. From browsing the source it seems like nix_printer is a wrapper around something like stdout, but I'm not exactly sure. I also browsed the Python bindings, but all I see there are the standard __repr__, etc methods for telling Python how to print the value.

[yorickvP]

It's unclear to me how you would construct a Value and then have nix print it for you.

I see a couple of functions that take a nix_printer, but I don't see any functions that construct a nix_printer. From browsing the source it seems like nix_printer is a wrapper around something like stdout, but I'm not exactly sure. I also browsed the Python bindings, but all I see there are the standard __repr__, etc methods for telling Python how to print the value.

Currently, the API does not have a way to print arbitrary values. (The functions you found are part of the soon-to-be-removed externals support, which do the other way around (get called from nix to print an external value).)

However, you can work around this by calling builtins.toJSON or similar on a Nix value.

[yorickvP]

Progress update:

• Formatted
• Applied @fricklerhandwerk's documentation suggestions
• Changed PrimOps to contain an std::function (thoughts on this would be appreciated) and used this to supply some user data and a wrapper function that handles errors and decouples us from the internal PrimOp implementation.

Left to do:

• [ ] Remove Externals
• [ ] Tests
• [ ] Make the documentation visible

[zmitchell]

The Rust bindings I discussed in my talk are here if you'd like to take a look: https://github.com/zmitchell/nix-ffi-rs/tree/master/nix-bindgen

[yorickvP]

Another thought: when building a plugin, authors should probably not link against libnix*-c, since they'll link to the build-time libs and not the runtime libs. Should Nix load it for them when loading plugins?

[lf-]

Another thought: when building a plugin, authors should probably not link against libnix*-c, since they'll link to the build-time libs and not the runtime libs. Should Nix load it for them when loading plugins?

I think you have to link against it regardless but nix should definitely load its own one before loading plugins to make sure it's the right one.

Related: recently I patched nix-doc (which uses the old plugin api) to check that nixVersion at runtime is the same as the version of nix i have from pkg-config before actually invoking any Nix abi code. This required overriding nixpkgs hardening and setting weird rust options to make sure DT_BIND_NOW is not set on my object and was generally a pain.

I think it would make things slightly easier for such checks if they could use a constant in a header but also, imo, they shouldn't be in plugins in the first place and probably plugin dlopen failure shouldn't be fatal by default. Maybe the ideal case is that plugins could include a symbol of their expected abi version and nix itself would check it is compatible when loading them.

[farcaller]

Thanks for your great work on this! I've started work on go bindings: https://github.com/farcaller/gonix, and it was mostly seamless apart from a few string passing issues.

[farcaller]

I'm looking into the ExternalValue and, either I miss something obvious, or it will have issues with the GCd host languages.

Consider this scenario (based on Go):

1. I create some go value randomExtrernal = ...
2. I call ev = nix_create_external_value(..., randomExtrernal)
3. I create a new value and call nix_set_external(..., val, ev)
4. I store val in my complimentary go structure that handles the values, goval.

Now, val refcounts ev, and goval refcounts val. No one refcounts the original randomExtrernal though, and the Go GC is free to collect it. It is relatively simply fixable, though, because we can special-case it. If the set value is an external value, then we can have goval hold a reference at both val and randomExtrernal. This way, as long as goval is valid, it properly chains down.

Here's a problem. What if the value referenced by val is then stored in e.g. a list nix side and then returned in a listval? As we wrap listval in golistval, the latter has no visibility whatsoever on listval referencing val. More so, at that point, you cannot back-reference goval.

Now here's some GC specifics: it's impossible to force GC to ignore the value, i.e. the value must have a valid reference in the go heap to stay alive. I stumbled on this issue in a very simple code, though where I create string Values and add them into a list.

One option to keep goval and all they reference afloat is to e.g. have a map from int (being the val pointer address) to the goval. And this works, because now you have a globally referenced goval wrappers, they reference external values, everyone's happy. The problem, of course, is that now no values will be ever garbage-collected, because val will always have a strong reference from goval and goval always have a strong reference from the go heap.

Is my thought experiment in here correct?