NixOS Packaging

[IceDBorn]

I've managed to package the native connector for NixOS, but the extension does not recognize it as installed. I think that it has something to do with how NixOS handles installed apps. NixOS does not place applications in /usr/bin, instead each application has it's own path in the nix store. @filips123 I'm asking for help for making the package recognizable by the extension.

[IceDBorn]

https://github.com/NixOS/nixpkgs/issues/47340

[IceDBorn]

I managed to fix the issue by copying firefoxpwa.json to ~/.mozilla/native-messaging-hosts/ and specifying the absolute path of firefoxpwa-connector.

[IceDBorn]

I got this when trying to launch a web app: Failed to patch the runtime: Path "/usr/share/firefoxpwa/userchrome/runtime" does not exist or you don't have access!

[filips123]

Path /usr/share/firefoxpwa/userchrome/ contains files from native/userchrome/ that are required to modify Firefox so it behaves like a web app. I'm not sure how NixOS handles this, but you will need to include that directory inside NixOS package.

In case modifying global /usr/share is not possible on NixOS, you can also include those files in some other directory and set FFPWA_SYSDATA build- or run-time environment variable to the correct path. You will probably also have to set FFPWA_EXECUTABLES to a dir that contains firefoxpwa and firefoxpwa-connector executables to make launching web apps work. You can read more details about FFPWA environment variables here, and checking how Homebrew formula handles paths with environment variables might also be useful.

For native messaging manifest, if there is no automatic way of installing it to the required location, displaying instructions how to create a symlink to the manifest after installing the package (as it's done for Homebrew) is enough.

[spikespaz]

I made some progress on this.

# <https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=firefox-pwa>
{
  stdenv,
  rustPlatform,
  fetchFromGitHub,
  openssl,
  pkg-config,
  maintainers,
}: let
  version = "2.1.2";
  source = fetchFromGitHub {
    owner = "filips123";
    repo = "PWAsForFirefox";
    rev = "v${version}";
    sha256 = "sha256-zJSrZOLHyvvu+HoHrPkDDISuY9GqpKtwGn/7jKzg5pI=";
  };
in
  rustPlatform.buildRustPackage {
    pname = "firefox-pwa";
    inherit version;

    src = "${source}/native";
    cargoSha256 = "sha256-zLl7WvGzN/ltc7hT5cAsp3ByrlThQmRXrGM5rKbntdY=";

    doCheck = false;

    nativeBuildInputs = [pkg-config];
    buildInputs = [openssl.dev openssl];

    preConfigure = ''
      # replace the version number in the manifest
      sed -i 's;version = "0.0.0";version = "${version}";' Cargo.toml
      # replace the version in the lockfile, otherwise Nix complains
      sed -zi 's;name = "firefoxpwa"\nversion = "0.0.0";name = "firefoxpwa"\nversion = "2.1.2";' Cargo.lock
      # replace the version number in the profile template files
      sed -i $'s;DISTRIBUTION_VERSION = \'0.0.0\';DISTRIBUTION_VERSION = \'${version}\';' userchrome/profile/chrome/pwa/chrome.jsm
    '';

    installPhase = let
      target = "target/${stdenv.targetPlatform.config}/release";
    in ''
      runHook preInstall

      # Executables
      install -Dm755 ${target}/firefoxpwa $out/bin/firefoxpwa
      install -Dm755 ${target}/firefoxpwa-connector $out/lib/firefoxpwa/firefoxpwa-connector

      # Manifest
      install -Dm644 manifests/linux.json $out/lib/mozilla/native-messaging-hosts/firefoxpwa.json
      sed -i "s;/usr/libexec/firefoxpwa-connector;$out/lib/firefoxpwa/firefoxpwa-connector;" $out/lib/mozilla/native-messaging-hosts/firefoxpwa.json

      # Completions
      install -Dm755 ${target}/completions/firefoxpwa.bash $out/share/bash-completion/completions/firefoxpwa
      install -Dm755 ${target}/completions/firefoxpwa.fish $out/share/fish/vendor_completions.d/firefoxpwa.fish
      install -Dm755 ${target}/completions/_firefoxpwa $out/share/zsh/vendor-completions/_firefoxpwa

      # Documentation
      install -Dm644 ${source}/README.md $out/share/doc/firefoxpwa/README.md
      install -Dm644 README.md $out/share/doc/firefoxpwa/README-NATIVE.md
      install -Dm644 ${source}/extension/README.md $out/share/doc/firefoxpwa/README-EXTENSION.md
      install -Dm644 packages/deb/copyright $out/share/doc/firefoxpwa/copyright

      # UserChrome
      mkdir -p $out/share/firefoxpwa/userchrome/
      cp -r userchrome/* $out/share/firefoxpwa/userchrome/

      runHook postInstall
    '';
  }

{self, ...}: {
  config,
  lib,
  pkgs,
  ...
}: let
  inherit (lib) types;
  cfg = config.programs.firefox.pwa;
in {
  options = {
    programs.firefox.pwa = {
      enable = lib.mkEnableOption (lib.mdDoc "enable");
      package = lib.mkOption {
        type = types.package;
        default = pkgs.firefox-pwa;
        description = lib.mdDoc '''';
        example = lib.literalExpression '''';
      };
      executables = lib.mkOption {
        type = types.path;
        readOnly = true;
        default = "${cfg.package}/bin";
        description = lib.mdDoc '''';
        example = lib.literalExpression '''';
      };
      sysData = lib.mkOption {
        type = types.path;
        readOnly = true;
        default = "${cfg.package}/share";
        description = lib.mdDoc '''';
        example = lib.literalExpression '''';
      };
      userData = lib.mkOption {
        type = types.path;
        default = "${config.xdg.dataHome}/firefoxpwa";
        description = lib.mdDoc '''';
        example = lib.literalExpression '''';
      };
    };
  };
  config = lib.mkIf cfg.enable {
    home.sessionVariables = {
      FFPWA_EXECUTABLES = cfg.executables;
      FFPWA_SYSDATA = cfg.sysData;
      FFPWA_USERDATA = cfg.userData;
    };
    home.file.".mozilla/native-messaging-hosts/firefoxpwa.json".source = "${cfg.package}/lib/mozilla/native-messaging-hosts/firefoxpwa.json";
    home.packages = [
      (cfg.package.overrideAttrs (_: {
        postInstall = ''
          ln -s $out/lib/firefoxpwa/firefoxpwa-connector $out/bin;
        '';
      }))
    ];
  };
}

Now, the extension recognizes the binaries and the native messaging description. I can create profiles and add PWAs from the extension popup.

I face an issue where the extension assumes the path of /usr/bin, when I start for example Google Messages, I see:

Failed to patch the runtime: Path "/usr/share/firefoxpwa/userchrome/runtime" does not exist or you don't have access!

When I launch from the .desktop file using Rofi I see:

Failed to execute: '/usr/bin/firefoxpwa site launch 01GJF401WT97CK1Q5NFTFNQSTS --protocol' Error: 'Failed to execute child process "/usr/bin/firefoxpwa" (No such file or directory)

[spikespaz]

Showing recent changes:

Home Manager module:

    programs.firefox.package = cfg.firefoxPackage.overrideAttrs (old: {
      nativeBuildInputs = old.nativeBuildInputs ++ [pkgs.makeWrapper];
      postFixup = ''
        wrapProgram ${lib.getExe cfg.firefoxPackage} \
          --set FFPWA_EXECUTABLES '${cfg.executables}' \
          --set FFPWA_SYSDATA '${cfg.sysData}' \
          --set FFPWA_USERDATA '${cfg.userData}'
      '';
    });

Debugging:

$ firefoxpwa runtime install
07:34:24 [WARN] This will download the unmodified Mozilla Firefox and locally modify it
07:34:24 [WARN] Firefox is licensed under the Mozilla Public License 2.0
07:34:24 [WARN] Firefox is a trademark of the Mozilla Foundation in the U.S. and other countries
07:34:24 [WARN] This project is not affiliated with the Mozilla Foundation in any way
07:34:24 [WARN] By using this project you also agree to the Firefox Privacy Notice: https://www.mozilla.org/privacy/firefox/
07:34:24 [WARN] Check the Firefox website for more details: https://www.mozilla.org/firefox/
07:34:24 [INFO] Downloading the runtime archive
07:54:28 [INFO] Extracting the runtime archive
07:54:37 [INFO] Copying the runtime
07:54:37 [INFO] Runtime installed!

$ firefoxpwa site launch 01GJF97WY5Q6P74R5J2JRZARFV
07:54:55 [INFO] Patching the runtime
07:54:55 [INFO] Runtime patched!
07:54:55 [INFO] Patching the profile
07:54:55 [INFO] Profile patched!
07:54:55 [INFO] Launching the web app
07:54:55 [ERROR] No such file or directory (os error 2)

What exactly is not found?

$ echo $FFPWA_EXECUTABLES && ls $FFPWA_EXECUTABLES
/nix/store/zf1mqcdpcb4d2ga54wjpv8mygzxfw03p-firefox-pwa-2.1.2/bin
firefoxpwa firefoxpwa-connector
$ echo $FFPWA_SYSDATA && ls $FFPWA_SYSDATA
/nix/store/zf1mqcdpcb4d2ga54wjpv8mygzxfw03p-firefox-pwa-2.1.2/share/firefoxpwa
userchrome
$ echo $FFPWA_USERDATA && ls $FFPWA_USERDATA
/home/jacob/.local/share/firefoxpwa
config.json firefoxpwa.log profiles runtime

Also, in previous messages the code shown does not include firefoxpwa-connector in the bin or in $FFPWA_EXECUTABLES as should be. This is fixed.

[filips123]

It looks like the site launch failed when trying to launch firefox (somewhere after here, which calls site.launch and then runtime.run). The only way I see how this could happen is if the firefox executable from the runtime was not found. However, I don't know how the program went so far as launching it, as it should fail earlier with "Runtime not installed".

[pasqui23]

Here is my attempt: https://github.com/pasqui23/nixpkgs/blob/firefox-pwa/pkgs/tools/networking/firefox-pwa/default.nix

However if I try to open a PWA I get the error:

Failed to patch the runtime: Path "/home/me/.local/state/firefox-pwa/userchrome/runtime" does not exist or you don't have access!

[spikespaz]

@pasqui23 It's a bad idea to use ~ in package derivations, I don't even think Nixpkgs would accept it. Could be wrong.

Why did you use postInstall rather than installPhase? In this case, the behavior is 100% custom and therefore should override the default from buildRustPackage, unless I'm mistaken and we still need that default phase.

After line 47, where wrapArgs is defined; are you sure you don't want to use ${wrapArgs[@]} in usages?

[pasqui23]

On 10/02/23 02:16, Jacob Birkett @.***> wrote:

@pasqui23 https://github.com/pasqui23 It's a bad idea to use |~| in package derivations, I don't even think Nixpkgs would accept it. Could be wrong. I was trying to fix the error Failed to patch the runtime: Path "/usr/share/userchrome/runtime" does not exist or you don't have access! and found that FFPWA_SYSDATA is read at build time, not runtime. I thought that if it pointed to inside the home directory then it would have higher chance to have write permissions to the path but I was wrong. This is absolutely not ready for pr and in fact I wanted to ask the author for suggestions. Why did you use |postInstall| rather than |installPhase|? In this case, the behavior is 100% custom and therefore should /override/ the default from |buildRustPlatform|, unless I'm mistaken and we still need that default phase.

After line 47, where |wrapArgs| is defined; are you sure you don't want to use |${wrapArgs[@]}| in usages?

I'll take those in consideration after I managed to made it work.

[filips123]

However if I try to open a PWA I get the error:

This error is probably caused because UserChrome stuff hasn't been copied to the correct location. I think the problem is that you have set FFPWA_SYSDATA to ~/.local/state/firefox-pwa, while you copy UserChrome to $out/share/firefoxpwa/userchrome.

PWAsForFirefox expects userchrome to be located inside a directory set in FFPWA_SYSDATA, so you should probably set it to $out/share/firefoxpwa (but I'm not familiar with Nix and don't know what that $out is). Also, although I don't know how Nix handles "global" directories for packages, I think that you shouldn't use ~ for FFPWA_SYSDATA, as that directory is meant for static global data that are not specific for each user. It also does not need write permission (only FFPWA_USERDATA needs it).

and found that FFPWA_SYSDATA is read at build time, not runtime.

It should be read both at build and runtime.

[IceDBorn]

@filips123 $out is the folder created for the package on /nix/store, it should only contain files that you won't modify. /nix/store is read-only too.

[spikespaz]

@filips123

It should be read both at build and runtime.

That is incredibly useful to know. Will try exposing the var at build time and getting back to people @ here.

[spikespaz]

@filips123 How I make use of this information depends on: Does compiling the binary hard-code the value of FFPWA_SYSDATA, or does it simply read file contents at the path provided at FFPWA_SYSDATA?

[spikespaz]

Looks like it is either-or. https://github.com/filips123/PWAsForFirefox/blob/014b1a4c5b21e14789a007f85c56b6b0c4956613/native/src/directories.rs#L63

[spikespaz]

What does "on windows" mean? Is it firefoxpwa-connector only required on Windows? Or because you expect the binary to be in libexec on Linux? https://github.com/filips123/PWAsForFirefox/blob/014b1a4c5b21e14789a007f85c56b6b0c4956613/native/src/directories.rs#L36

[filips123]

What does "on windows" mean? Is it firefoxpwa-connector only required on Windows? Or because you expect the binary to be in libexec on Linux?

It's because firefoxpwa-connector is in libexec on Linux. I will probably improve that comment at some point so this is more clear.

Does compiling the binary hard-code the value of FFPWA_SYSDATA, or does it simply read file contents at the path provided at FFPWA_SYSDATA?

It just sets the location of the system data directory to the value of FFPWA_SYSDATA, and reads that directory when needed at runtime. (Also, even if you set FFPWA_SYSDATA at buld time, it can still be overwritten at run time, but I think that shouldn't really matter.)

Basically, directories are set in this way:

1. If there is no environment variable: Use default value (/usr/share/firefoxpwa/ on normal Linux).
2. If there is build-time environment variable: Set directory to that instead.
3. If there is run-time environment variable: Set directory to that instead.
4. When needed at run-time: Access directory contents, etc.

$out is the folder created for the package on /nix/store, it should only contain files that you won't modify. /nix/store is read-only too.

Then I think FFPWA_SYSDATA should point to somewhere in that directory, as it only contains read-only data.

---

Here is a rough overview of how I think the package should be set up to work:

• $XDG_DATA_HOME/firefoxpwa/:

  This is FFPWA_USERDATA. This should be stored per user and must have write permission. Unless Nix requires some specific directory for user data, I don't think much work is needed here as PWAsForFirefox will already automatically detect XDG data directory.

• $NIX_PACKAGE_ROOT/bin/:

  This is FFPWA_EXECUTABLES and should be stored globally. It needs to contain firefoxpwa binary, and should probably be added to PATH (or what Nix uses for binaries), so users can easily execute it. Additionally, this needs to be some unversioned path (i.e., it must not contain any version or other stuff that changes between package updates in the directory name), as otherwise, web apps would break on every update (like they did on Homebrew when I forgot to use unversioned paths #55).

• $NIX_PACKAGE_ROOT/libexec/:

  This is where firefoxpwa-connector needs to be stored. It does not need to be added to PATH, but outside programs (Firefox) still need to be able to launch it. ~~It should also be in an unversioned path, if possible (but I think here it's not necessary, as long as the manifest gets updated every time).~~ Edit: It can be versioned, but then the manifest needs to be updated every time the path changes to point to the correct one.

• $NIX_PACKAGE_ROOT/share/:

  This is FFPWA_SYSDATA, should be stored globally and only needs read permission. It needs to contain userchrome directory. ~~It should also be in an unversioned path, if possible.~~ Edit: It can be versioned without any additional requirements.

• /usr/lib/mozilla/native-messaging-hosts/firefoxpwa.json and /usr/lib64/mozilla/native-messaging-hosts/firefoxpwa.json:

  Should be in the root Linux filesystem, without any Nix stuff, so Firefox can detect them properly. Or, if Nix does not allow such global files, the manifest should be stored per user in ~/.mozilla/native-messaging-hosts/firefoxpwa.json. The path in the manifest should point to the correct firefoxpwa-connector location.

• Shell completions:

  I don't know how Nix handles this, but shell completions should be stored in appropriate locations so shells can automatically load them.

[IceDBorn]

@filips123 I don't think that you can have unversioned paths for the executables, because everytime nix builds a package it stores it with a hash in the path. Would links work? Because it is possible to do something like environment.etc."firefoxpwa".source = "${(pkgs.callPackage self-built/firefoxpwa.nix {})}/";, which links the package's auto-generated path to /etc/firefoxpwa/

[filips123]

Links should probably work. The reason why it needs unversioned paths or something similar is that generated .desktop entries for installed web apps simply call {exe} site launch {id} when launched. On classic Linux (and Windows), this is not a problem, because the path will always remain the same (/usr/bin/firefoxpwa), but on Nix (and Homebrew), it can cause the entries to point to invalid executable path from the previous version. Homebrew has opt_bin which is not versioned and probably uses links, so I think links should also work here.

firefoxpwa-connector can versioned if that is easier, but the package needs to make sure that the manifest always points to the correct version.

Also, after checking some thing a bit, share/FFPWA_SYSDATA can also be versioned. I think that shouldn't cause any problems, because it is only used internally to copy some files from it.

[pasqui23]

• $NIX_PACKAGE_ROOT/share/: This is FFPWA_SYSDATA, should be stored globally and only needs read permission. It needs to contain userchrome directory. It should also be in an unversioned path, if possible.

Again I've already tried to do FFPWA_SYSDATA="$out/share" at build time but the error remain the same, only referencing the nix store path instead. So what is firefoxpwa really trying to do? Why does it says "Path does not exist or you don't have access!"? When I copy the path I'm given it effectively does not exist, even when i tried to point FFPWA_SYSDATA to inside the nix store.

[spikespaz]

Not worried about any of that, already got all of the paths set up how you've described. I only asked for clarification because of the wording of one of @filips123's comments and quickly resolved by looking at directories.rs. I thought the documentation there was clear enough, it was foolish of me to ask here first.

@filips123 Would it be possible for you to publish your two (or three?) Git-vendored crates on crates.io? They are quite difficult to get working with Nix because something something hashing something something.

[filips123]

@pasqui23 I think the problem is that I've said $NIX_PACKAGE_ROOT/share/ instead of $NIX_PACKAGE_ROOT/share/firefoxpwa/. It should be FFPWA_SYSDATA="$out/share/firefoxpwa". This probably makes more sense as there is also other stuff in share (like completions) that is not really sysdata.

@spikespaz Well, data-url and mime are just forks of already existing crates, but with some changes that are not yet merged to the upstream (and unfortunately probably won't be anytime soon as they are not actively developed anymore). web_app_manifest is my own crate that I would like to publish eventually, but it depends on my fork of mime and I think you can't publish crates that depend on git crates to crates.io. I could probably publish them under different names than original, but I don't know what crates.io policies/recommendations are about publishing more or less temporary forks of existing crates.

[spikespaz]

@filips123 Mind making them submodules? I totally understand if you don't want to, I get that submodules aren't always the easiest for you, and you've gotta keep them up to date before the lock file. Probably really annoying.

[filips123]

Yeah, submodules can be quite annoying... Is there some other way of creating Nix packages with crates from Git? I don't know how useful that is, but you can still obtain commit hashes for Git dependencies in Cargo.lock.

[spikespaz]

@filips123 Nix is dumb strict about hashes for everything. It isn't satisfied by commit hashes. I've argued about it before. I'll look deeper tomorrow, I don't even want you to deal with submodules. I'll find some sort of hack, or maybe extract one of the release .deb packages. Won't be able to get this into nixpkgs that way, but at least it's something. I have a feeling your project is in high demand among NixOS users.

[filips123]

Thanks! I think it would still be nice if the package can get into nixpkgs, so if you cannot find any other way that is accepted, I can still publish data-url and mime under different names, and then web_app_manifest. It seems that although it's generally not recommended to publish forks, it's still fine to publish them with the username as a prefix and appropriate description, if that's really needed.

[pasqui23]

The reason why it needs unversioned paths or something similar is that generated .desktop entries for installed web apps simply call {exe} site launch {id} when launched.

Note that the {exe} can and should be a bare command, not an absolute path.For example nixpkgs' firefox.desktop has just Exec=firefox %U.

[filips123]

Does Nix then automatically handle adding binaries to the PATH and can they be accessed even from outside their packages, or does it have some handling specific for Firefox? If firefoxpwa can be automatically added to the PATH (and Nix will handle paths when they change on updates), we probably don't even need unversioned path for FFPWA_EXECUTABLES. The only problem is that I don't know what how firefoxpwa will behave if you set an empty FFPWA_EXECUTABLES, but I guess I can fix it if needed.

Edit: I think it actually works if you set FFPWA_EXECUTABLES to an empty string. Then, the desktop entry will simply contain Exec=firefoxpwa site launch ID --protocol %u. But still needs to be tested with Nix though...

[IceDBorn]

@filips123 I think that it does handle that automatically if you're using a package builder. In this particular case buildRustPackage.

[spikespaz]

Does Nix then automatically handle adding binaries to the PATH

If the package has bin in $out, it can be added to to one of the options environment.systemPackages or home.packages. The package's bin directory contents will all be symlinked into /run/current-system/sw/bin or $XDG_USER_HOME/.nix-profile/bin, corresponding to a system or user "installation".

~/.nix-profile/bin is in PATH.

All packages reside in /nix/store, and are named like ^(?P<derivation_hash>[a-z\d]{32})-(?P<name_version>[^INVALID_CHARS]+)(?P<suffix>\.\w+)?$.

Packages can contain more than just bin. Directories such as lib, share, etc, etc. all have corresponding locations that will end up containing links to files in the package's output path.

If firefoxpwa can be automatically added to the PATH

Nix users, if they know what they're doing (why they want to use Nix in the first place) like to keep their PATH clear of clutter. Usually programs like FirefoxPWA will be wrapped further by Nix code, to keep interactions between it and other programs in relative isolation. For example, I've written (half of) a module that will generate the .desktop files needed to launch the PWAs with your program. All paths will be hard-coded at what we call "generation build-time", and perhaps have a separate instance of Firefox (and profiles). These generated .desktop files (declaratively described in the Nix language) will be put into a package's $out/share/applications directory, which then can be added to home.packages.

Optionally, if the user does not desire a declarative approach to creating PWAs (why are they using Nix then, the fools) I will provide traditional integration between the browser extension and native executable (because it is useful for testing, and also the originally intended functionality).

Fundamentally, your Nix configurations function as a monolithic declaration of the "generation switch" script that is "compiled" by the Nix interpreter. Nix will 1. build the packages, put them into the store and 2. compile and run a big-ass shell script that creates and updates symlinks across the filesystem.

Think of it as one of those dotfiles managers that people use to make links in the outer world to the innards of a "dotfiles" git repository, but on steroids and everything is hashed. In theory, the entire system is mathematically "pure" (in practice, the definition is a little loose).