LemmyNet
Add a Nix flake option for natively working with Lemmy's codebase
Requirements
- [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
- [X] Did you check to see if this issue already exists?
- [X] Is this only a feature request? Do not put multiple feature requests in one issue.
- [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.
- [X] Do you agree to follow the rules in our Code of Conduct?
Is your proposal related to a problem?
I want to add another dev environment and build option that leverages the power of Nix to make building and working on the codebase much more foolproof.
I cloned the lemmy and lemmy-ui repos and attempted to follow the instructions to get it working on my main machine. However, since I am using Nix and didn't want to have to use ansible or Docker, I was out of luck.
I forked Lemmy to use the code in my own project. Since I intend for my fork to roll with this repo for at least a long time and I am a Nix/NixOS Linux user, I have been attempting to build Lemmy using a flake but I ran into many issues.
Describe the solution you'd like.
Here's what I have so far in my flake:
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
rust-overlay = {
url = "github:oxalica/rust-overlay";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs = { self, nixpkgs, rust-overlay, flake-utils, ... }:
flake-utils.lib.eachDefaultSystem (system:
let
overlays = [
rust-overlay.overlays.default
(final: prev: {
rustToolchain = prev.rust-bin.fromRustupToolchainFile ./rust-toolchain.toml;
})
];
pkgs = import nixpkgs {
inherit system overlays;
};
lib = pkgs.lib;
cargoToml = builtins.fromTOML (builtins.readFile ./Cargo.toml);
version = cargoToml.workspace.package.version;
# versionPatch = pkgs.writeText "version-patch.patch" ''
# diff --git a/crates/utils/src/version.rs b/crates/utils/src/version.rs
# new file mode 100644
# index 0000000..e69de29
# --- /dev/null
# +++ b/crates/utils/src/version.rs
# @@ -0,0 +1 @@
# +pub const VERSION: &str = "${version}";
# '';
# A *very* necessary OpenSSL wrapper
openssl_wr = pkgs.symlinkJoin {
name = "openssl-dev-out";
paths = with pkgs; [ openssl.dev openssl.out ];
};
in
with pkgs;
{
packages.default = pkgs.rustPlatform.buildRustPackage {
inherit (cargoToml.package) name;
pname = "lemmy_server";
src = ./.;
cargoLock.lockFile = ./Cargo.lock;
# preConfigure = ''
# echo 'pub const VERSION: &str = "${version}";' > crates/utils/src/version.rs
# '';
buildInputs = [ postgresql pkg-config openssl.dev openssl libiconv protobuf ];
nativeBuildInputs = [ pkg-config openssl.dev openssl rustfmt protobuf ];
# Explicitly setting OpenSSL lib and include directories
RUST_SRC_PATH = "${pkgs.rust.packages.stable.rustPlatform.rustLibSrc}";
# Set OpenSSL environment variables
OPENSSL_LIB_DIR = "${openssl_wr}/lib";
OPENSSL_INCLUDE_DIR = "${openssl_wr}/include";
OPENSSL_DIR = openssl_wr; # No .out needed because it's a symlinkJoin
PROTOC = "${pkgs.protobuf}/bin/protoc";
PROTOC_INCLUDE = "${pkgs.protobuf}/include";
PKG_CONFIG_PATH = "${openssl_wr}/lib/pkgconfig:${pkgs.pkg-config}/lib/pkgconfig";
};
# Devshell (to be broken out to separate shell.nix later)
devShells.default = mkShell {
name = "lemmy-shell";
buildInputs = [ openssl openssl.dev postgresql libiconv protobuf pkg-config ];
nativeBuildInputs = [ rustToolchain pkg-config rustfmt protobuf ];
packages = [ rustToolchain cargo-deny cargo-edit cargo-watch rust-analyzer ];
# Dev Environment variables
RUST_SRC_PATH = "${pkgs.rust.packages.stable.rustPlatform.rustLibSrc}";
RUST_BACKTRACE = "1";
# Dev Environment variables
OPENSSL_LIB_DIR = "${openssl_wr}/lib";
OPENSSL_INCLUDE_DIR = "${openssl_wr}/include";
OPENSSL_DIR = openssl_wr;
PROTOC = "${pkgs.protobuf}/bin/protoc";
PROTOC_INCLUDE = "${pkgs.protobuf}/include";
PKG_CONFIG_PATH = "${openssl_wr}/lib/pkgconfig:${pkgs.pkg-config}/lib/pkgconfig";
# Environment variable to connect to the database
LEMMY_DATABASE_URL = "postgres://lemmy:password@localhost:5432/lemmy";
shellHook = ''
echo "LEMMY_DATABASE_URL is set to $LEMMY_DATABASE_URL";
# Any other setup steps can be added here
nix eval --raw .
echo
echo "evaluated successfully"
'';
};
}
);
# unsing caches where applicable
nixConfig = {
extra-experimental-features = ["nix-command flakes" "ca-derivations"];
allow-import-from-derivation = "true";
# This sets the flake to use nix cache.
# Nix should ask for permission before using it,
# but remove it here if you do not want it to.
extra-substituters = [
"https://cache.nixos.org"
"https://cache.iog.io"
];
extra-trusted-public-keys = [
"cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
"hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ="
];
};
}
Describe alternatives you've considered.
I considered:
- starting from scratch with an entirely new codebase
- working in a VM dev environment using Ubuntu
- ansible
- Docker
- dream2nix
- cargo2nix -etc
Additional context
here is the most recent log of the build failure:
Compiling lemmy_utils v0.19.3 (/build/4hblyr8vifjl9j2dkb9q8f545vpa3hyb-source/crates/utils)
Compiling chumsky v0.9.3
Compiling crossbeam-channel v0.5.11
Compiling crossbeam-epoch v0.9.18
Compiling http-signature-normalization v0.7.0
Compiling enum_delegate_lib v0.2.0
Compiling email-encoding v0.2.0
Compiling derivative v2.2.0
Compiling enum-map-derive v0.17.0
Compiling readonly v0.2.12
Compiling linkify v0.10.0
Compiling unicode-width v0.1.11
Compiling entities v1.0.1
Compiling downcast-rs v1.2.0
Compiling urlencoding v2.1.3
Compiling encoding_index_tests v0.1.4
Compiling retain_mut v0.1.9
Compiling email_address v0.2.4
Compiling argparse v0.2.2
Compiling triomphe v0.1.11
Compiling tagptr v0.2.0
Compiling fastrand v2.0.1
Compiling quoted_printable v0.5.0
Compiling lettre v0.11.3
Compiling deadpool v0.9.5
Compiling html2text v0.6.0
Compiling markdown-it v0.6.0
Compiling enum-map v2.7.3
Compiling enum_delegate v0.2.0
Compiling http-signature-normalization-reqwest v0.10.0
error: failed to run custom build command for `lemmy_utils v0.19.3 (/build/4hblyr8vifjl9j2dkb9q8f545vpa3hyb-source/crates/utils)`
Caused by:
process didn't exit successfully: `/build/4hblyr8vifjl9j2dkb9q8f545vpa3hyb-source/target/release/build/lemmy_utils-4a67c36804b984e8/build-script-build` (exit status: 1)
--- stderr
Error: FileRead { file: "translations/email/en.json", source: Os { code: 2, kind: NotFound, message: "No such file or directory" } }
warning: build failed, waiting for other jobs to finish...
this issue seems related to the lemmy-ui submodule structure of this project and I get this error because I need to figure out how to get Nix to build the whole project within this context. If I can get this to work, it will make collaboration with others a lot more foolproof than even Docker, IMO.
edit: I don't mean to bikeshed. I just want to help you pull in as many devs as possible and your code is going to be invaluable in my fork. Since I kind of require Nix end-to-end in all of my personal projects, if I am to fork this project, I figure need to learn how to get it working completely declaratively without touching any of your code before I start work on a fork of it.
In the end, for lemmy, I hope this perhaps cleans up the process that devs have to go through to get the dev env setup to exact specs while also rolling along with the various updates automatically.
You need to do the initial clone with git clone --recursive to include the translations submodule. Or in an existing repo run git submodule init && git submodule update.
We also have docs for native installation in case you didnt see: https://join-lemmy.org/docs/administration/from_scratch.html https://join-lemmy.org/docs/contributors/02-local-development.html
You need to do the initial clone with
git clone --recursiveto include the translations submodule. Or in an existing repo rungit submodule init && git submodule update.We also have docs for native installation in case you didnt see:
https://join-lemmy.org/docs/administration/from_scratch.html
https://join-lemmy.org/docs/contributors/02-local-development.html
I did that. It's not you. It's me.
In my current estimation, It didn't work due to the deterministic nature of the script that adds the translation upon ''sumbmodule init'' and Nix not seeing those files in its sandbox. I have to figure out a way to get that whole submodule init translations script into the flake where I can instruct Nix to use it. Eelco Dolstra himself may be done by now but I've only been using Nix for 8 months.
I'm posting this request to announce my intentions and what stage I'm on in implementing flakified Lemmy which will allow me to contribute to the project while also working on my own fork of this project.
I have some work to do to figure out how to get flakes working with this submodule style. I just wanted to give you a heads up that I'm working on this feature so we can have less, "it builds on my machine".
Keep up the great work, comrade. :)
''' nix build '.?submodules=1' '''
might be all I'm missing. (Unable to test right now.)
Resolved would mean you got the Lemmy project working end to end with Nix. Is that the case? I've been out of the loop lately (and my fork lays stagnant). I reached an impasse in my flake-centric build then had to return to my main project as I do all of this as a hobbyist who prefers Haskell.
We dont have any plans to add another official development method, native and docker are enough. But if you get nixos development working, feel free to link the instructions from our documentation.
No problem. I'll share if or when I do end up getting this Nix build option up and running in sync with your codebase.