Add nix-shell

[bromanko]

I prefer using nix to manage my development environments and dependencies. This PR adds a nix-shell config with development dependencies, and a direnv configuration to automatically source it when entering the directory.

I am curious if this is a helpful addition. If so, perhaps someone else can test it and we can add some reference to it in the documentation for local development setup.

[dxmh]

Hey @bromanko, total nix newbie here. I'm in the process of building a new machine right now and so with a fresh Linux install I figured it would be a good opportunity to test this PR and gain some nix experience (I usually work with ZMK inside of a docker container).

I'm running Fedora 35, and I have a functional nixpkgs setup. I cloned the ZMK repo, applied your PR and a couple others, entered the nix-shell and followed the ZMK "Basic Setup" documentation to set things up. I don't have the exact commands to hand, but if I remember correctly it was essentially this:

[nix-shell:~/code/zmk]$ west init -l app/
[nix-shell:~/code/zmk]$ west update
[nix-shell:~/code/zmk]$ west zephyr-export

At this point whenever I tried to build the firmware it would fail:

Click to expand build failure output…

[nix-shell:~/code/zmk/app]$ west build --pristine --board nice_nano -- -DSHIELD=a_dux_left -DZMK_CONFIG=/home/dom/code/zmk-config/config
-- west build: making build dir /home/dom/code/zmk/app/build pristine
-- west build: generating a build system
-- ZMK Config directory: /home/dom/code/zmk-config/config
-- Board: nice_nano, /home/dom/code/zmk/app/boards/arm/nice_nano, a_dux_left, a_dux
-- ZMK Config Kconfig: /home/dom/code/zmk-config/config/a_dux.conf
-- Using keymap file: /home/dom/code/zmk-config/config/a_dux.keymap
-- Using keymap file: /home/dom/code/zmk-config/config/a_dux.keymap
Including boilerplate (Zephyr base): /home/dom/code/zmk/zephyr/cmake/app/boilerplate.cmake
CMake Deprecation Warning at /home/dom/code/zmk/zephyr/cmake/app/boilerplate.cmake:37 (cmake_policy):
  The OLD behavior for policy CMP0079 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.
Call Stack (most recent call first):
  /home/dom/code/zmk/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
  /home/dom/code/zmk/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:35 (include_boilerplate)
  CMakeLists.txt:17 (find_package)


-- Application: /home/dom/code/zmk/app
-- Zephyr version: 2.4.0 (/home/dom/code/zmk/zephyr)
-- Found Python3: /nix/store/5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6/bin/python3.9 (found suitable exact version "3.9.6") found components: Interpreter 
-- Found west (found suitable version "0.12.0", minimum required is "0.7.1")
-- git describe failed: fatal: No tags can describe 'd7d3e0984a3734af52cfa143748927467f8c11aa'.
Try --always, or create some tags.;
   BUILD_VERSION is left undefined
-- Board: nice_nano
-- Cache files will be written to: /home/dom/.cache/zephyr
-- Found dtc: /nix/store/0yf1y6yfjhph22jl5v6nwyxsqvyvzw5l-dtc-1.6.1/bin/dtc (found suitable version "1.6.1", minimum required is "1.4.6")
-- Found toolchain: gnuarmemb (/nix/store/6s19wmv7w528kxwfkdir8d5adf6jqwiz-gcc-arm-embedded-10.3.1)
-- Found BOARD.dts: /home/dom/code/zmk/app/boards/arm/nice_nano/nice_nano.dts
-- Found devicetree overlay: /home/dom/code/zmk/app/boards/shields/a_dux/a_dux_left.overlay
-- Found devicetree overlay: /home/dom/code/zmk-config/config/a_dux.keymap
nice_nano.dts.pre.tmp:15.39-20.5: Warning (interrupt_provider): /soc/interrupt-controller@e000e100: Missing #address-cells in interrupt provider
  also defined at nice_nano.dts.pre.tmp:337.7-339.3
-- Generated zephyr.dts: /home/dom/code/zmk/app/build/zephyr/zephyr.dts
-- Generated devicetree_unfixed.h: /home/dom/code/zmk/app/build/zephyr/include/generated/devicetree_unfixed.h

warning: PM_DEVICE (defined at /home/dom/code/zmk/app/Kconfig:342) defined without a type

warning: <choice LVGL_THEME_DEFAULT_FONT_NORMAL> (defined at /home/dom/code/zmk/app/src/display/Kconfig:59) defined with type unknown

warning: <choice LVGL_THEME_DEFAULT_FONT_NORMAL> (defined at /home/dom/code/zmk/app/src/display/Kconfig:59) defined without a prompt

warning: the default selection LVGL_THEME_DEFAULT_FONT_NORMAL_MONTSERRAT_16 (defined at lib/gui/lvgl/Kconfig.themes:351) of <choice LVGL_THEME_DEFAULT_FONT_NORMAL> (defined at /home/dom/code/zmk/app/src/display/Kconfig:59) is not contained in the choice

warning: <choice LVGL_THEME_DEFAULT_FONT_SMALL> (defined at /home/dom/code/zmk/app/src/display/Kconfig:67) defined with type unknown

warning: <choice LVGL_THEME_DEFAULT_FONT_SMALL> (defined at /home/dom/code/zmk/app/src/display/Kconfig:67) defined without a prompt

warning: the default selection LVGL_THEME_DEFAULT_FONT_SMALL_MONTSERRAT_12 (defined at lib/gui/lvgl/Kconfig.themes:234) of <choice LVGL_THEME_DEFAULT_FONT_SMALL> (defined at /home/dom/code/zmk/app/src/display/Kconfig:67) is not contained in the choice

warning: the default selection PM_POLICY_APP (undefined) of <choice SYS_PM_POLICY> (defined at /home/dom/code/zmk/app/Kconfig:338, subsys/power/policy/Kconfig:3) is not contained in the choice

warning: <choice CBPRINTF_IMPLEMENTATION> (defined at /home/dom/code/zmk/app/Kconfig:494) defined with type unknown

warning: <choice CBPRINTF_IMPLEMENTATION> (defined at /home/dom/code/zmk/app/Kconfig:494) defined without a prompt

warning: the default selection CBPRINTF_NANO (undefined) of <choice CBPRINTF_IMPLEMENTATION> (defined at /home/dom/code/zmk/app/Kconfig:494) is not contained in the choice
Parsing /home/dom/code/zmk/app/Kconfig
Loaded configuration '/home/dom/code/zmk/app/boards/arm/nice_nano/nice_nano_defconfig'
Merged configuration '/home/dom/code/zmk-config/config/a_dux.conf'

error: Aborting due to Kconfig warnings

CMake Error at /home/dom/code/zmk/zephyr/cmake/kconfig.cmake:239 (message):
  command failed with return code: 1
Call Stack (most recent call first):
  /home/dom/code/zmk/zephyr/cmake/app/boilerplate.cmake:591 (include)
  /home/dom/code/zmk/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
  /home/dom/code/zmk/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:35 (include_boilerplate)
  CMakeLists.txt:17 (find_package)


-- Configuring incomplete, errors occurred!
FATAL ERROR: command exited with status 1: /nix/store/mnq9fvg9yqan4g1pgis01l0mqfkihpqs-cmake-3.21.2/bin/cmake -DWEST_PYTHON=/nix/store/1xsv6ykl54q7107vsyqvd098s740qvkn-python3-3.8.12/bin/python3.8 -B/home/dom/code/zmk/app/build -S/home/dom/code/zmk/app -GNinja -DBOARD=nice_nano -DSHIELD=a_dux_left -DZMK_CONFIG=/home/dom/code/zmk-config/config

I was aware I'd skipped the following step from the documentation:

pip3 install --user -r zephyr/scripts/requirements-base.txt

As somebody new to nix, I'm not yet familiar with the nix-shell configuration. But I did notice that not all of the requirements from that file were in your shell.nix, so I just tried adding them like this:

diff --git a/shell.nix b/shell.nix
index a445734..a1784ff 100644
--- a/shell.nix
+++ b/shell.nix
@@ -14,6 +14,14 @@ in mkShell {
     gcc-arm-embedded
     python38Packages.west
     python38Packages.pyelftools
+    python38Packages.pyyaml
+    python38Packages.pykwalify
+    python38Packages.canopen
+    python38Packages.packaging
+    python38Packages.progress
+    python38Packages.psutil
+    python38Packages.anytree
+    python38Packages.intelhex
   ];
 
   shellHook = ''

With this change in place, I am able to successfully build my firmware using the same west build ... command above.

I'm not sure if this is the appropriate way to resolve this (or even if it's the intended process to follow), but wanted to share with you and get your feedback. Thanks!

[bromanko]

Thanks for testing this out. I'm not that expert with Python, so maybe the two of us can get this to a better place. It looks like I had manually installed some of those dependencies in my nix shell and that's why it worked for me. In doing more research, perhaps this should be updated to use something like mach-nix to source the dependencies from requirements.txt. What do you think?

[dxmh]

Nice @bromanko, I was hoping there would be a solution like that.

I've submitted a PR to this branch in your repo:

• https://github.com/bromanko/zmk/pull/1

[MangoIV]

Hi! As a nixos-user I would like to ask some questions:

• Does it make sense to merge this PR with legacy nix being deprecated soon? Especially with no documentation on which versions you used for your nixpkgs. Nix after all is about reproducible builds and if this shell is not pure it might not do the same thing in all cases, which it should.
• Do you think that this PR is complete? For my feeling when doing a PR for adding nix support, it should also include a build recipe for the whole firmware. From my knowledge for everyone who uses nix/ nixos this shell is rather trivial.
• When testing this be sure to have a pure environment. As you stated yourself, you had some python dependencies available that aren't declared in the shell.nix. This is against the principle of nix which tries to make builds independent of your systems state.

[aumuell]

Hi! As a direnv-user - but not nixos-user - I would like to propose a change to this PR: instead of adding a .envrc, please add it as e.g. .envrc-nixos and add an entry for .envrc to .gitignore.

This way, everyone who wants to use nixos, can use the supplied .envrc-nixos by simply symlinking it to .envrc. And other users would retain the freedom to add a .envrc of their own choosing in their working copy. If the PR is merged as is, then a direnv-enabled shell will try to enable nixos, even when the user would like to use .envrc for something else.

Background .envrc is processed by direnv to automatically set additional environment variables when the shell enters the directory containing .envrc or one of its subdirectories.

[bromanko]

I've followed @aumell's suggestion and moved the .envrc file and added it to .gitignore.

[bromanko]

@MangoIV

• Does it make sense to merge this PR with legacy nix being deprecated soon? Especially with no documentation on which versions you used for your nixpkgs. Nix after all is about reproducible builds and if this shell is not pure it might not do the same thing in all cases, which it should.

I've added a flake configuration. I didn't realize that nix 2.4 was released.

• Do you think that this PR is complete? For my feeling when doing a PR for adding nix support, it should also include a build recipe for the whole firmware. From my knowledge for everyone who uses nix/ nixos this shell is rather trivial.

I was hoping to get something basic in place as a starting point. It's basic but it does simplify installing the dependencies and configuring the environment for development. As someone new to ZMK this was a pain point for me. Can you further clarify what you mean by "build the whole firmware"?

• When testing this be sure to have a pure environment. As you stated yourself, you had some python dependencies available that aren't declared in the shell.nix. This is against the principle of nix which tries to make builds independent of your systems state.

I agree with this point. @dxmh have been discussing it in https://github.com/bromanko/zmk/pull/1. I'm curious about how you'd handle the Python dependencies. There is no lockfile for requirements-base.txt pulled in from Zephyr.

[MangoIV]

I've added a flake configuration. I didn't realize that nix 2.4 was released.

apparently it got reverted on the current stable release (21.11), however, I think that it's still best practice or at least a really good idea to add a flake as it doesn't change anything about the usage of the legacy part, aka the shell.nix. + that it was released on stable once means that at least syntactically not much will change.

I was hoping to get something basic in place as a starting point. It's basic but it does simplify installing the dependencies and configuring the environment for development. As someone new to ZMK this was a pain point for me. Can you further clarify what you mean by "build the whole firmware"?

When further thinking about it, I think that adding a basic one is a good idea for now, with building the whole thing I mean being able to do something like nix build .#build-target to build a hex

I agree with this point. @dxmh have been discussing it in bromanko#1. I'm curious about how you'd handle the Python dependencies. There is no lockfile for requirements-base.txt pulled in from Zephyr.

I think if it builds with one specific version and you pull in the python packages from nix, you won't need any further lockfiles besides the flake.lock, this will also pin the python packages versions.

[dxmh]

I wanted to share in this discussion a link to @moergo-sc’s fork of ZMK:

https://github.com/moergo-sc/zmk (permalink to codebase at time of writing)

I haven’t had the chance to get too familiar with the code yet, but it seems @chrisandreae has made good use of Nix for ZMK development and firmware builds.

[moergo-sc]

Hey @dxmh, @chrisandreae did all the work for nix support, which is working very well for us.

Also see https://github.com/moergo-sc/zmk/blob/main/README-NIX.md for a very rough guide how to use the nix-support. It is more meant as a reminder for me.

[petejohanson]

As a non-nix user, I'm not much help in adding value, or pushing this forward. @dxmh is this something you want to own/steer/review?

[dxmh]

I agree with @MangoIV in that ultimately it would be great to be able run nix build .#build-target to build a hex (e.g. nix build .#bt60 to build the BT60 firmware). If I remember correctly MoErgo is doing something like this in the links above.

In the meantime, I think the functionality in this PR does provide a valuable first step – basic nix support providing a local ZMK development environment.

I'd like to test and confirm that the flake in this PR works with the latest ZMK, and I'd like to make sure that the lock file works as expected (e.g. the flake creates reproducible builds despite pulling in external deps from Zephyr's requirements-base.txt file, as per @MangoIV's last sentence above).

It's been a year since this PR was written, a lot has probably changed in this time. Any new feedback or suggestions from nix users would be welcomed. (Are you still using this setup to run ZMK @bromanko?)

[dxmh]

I had an opportunity to look at this PR again today. Unfortunately I wasn't able to get it working due to gcc-arm-embedded being unsupported on my aarch64-darwin machine, but that's probably a separate issue at this stage…

I mainly wanted to check if the development environment is reproducible despite pulling in external dependencies. Well, there's no problem with the dependencies from Zephyr's requirements-base.txt file as these are all managed with nix in this PR (via mach-nix).

However, once inside the dev environment we still need to run west init and west update (as per ZMK docs). These west commands fetch or update modules which are then not managed/pinned in the nix flake – this means the environment is not 100% reproducible and so this is still an issue:

• https://github.com/zmkfirmware/zmk/issues/1391

Does anyone have any ideas about how these west commands / zephyr modules could be handled? (@MangoIV? @bromanko?)

I checked the MoErgo ZMK nix setup – it's a bit too advance for my current nix experience, but I think there's some clever stuff going on to pin the Zephyr modules fetched by west (see manifest.json).

Alternatively, Pete pointed out in #1391 that pinning a specific hash in west.yml will always build against that revision. When used with the current flake, maybe this could be a simple "good enough" solution for reproducibility for now?

[kdb424]

Based on the current readme for Nix, West is not needed, therefor doesn't need handled. It could cause issues being upstream here as it would mean that users aren't running "official" builds, but they are at least documenting the Nix in it's own readme at this point. I'm no Nix master, but I'll give it a look as I'm also on Nix Darwin and want to be able to do local building within Nix.

[adisbladis]

I've taken another stab at a nix development shell in https://github.com/zmkfirmware/zmk/pull/1974.