Nix flake is broken

[Jaculabilis]

I was excited to see the flake configs in this repo, but it seems like it isn't working.

$ nix flake show github:obsidian-html/obsidian-html
github:obsidian-html/obsidian-html/edc24eba8e43a28e989c433ba03620a5e74cf0dd
├───apps
│   ├───aarch64-darwin
│   │   └───default: app
│   ├───aarch64-linux
│   │   └───default: app
│   ├───i686-linux
│   │   └───default: app
│   ├───x86_64-darwin
│   │   └───default: app
│   └───x86_64-linux
│       └───default: app
├───checks
│   ├───aarch64-darwin
error: cannot build '/nix/store/8mwh2yk5rz3xmh3zh8m17arpk4n703k2-setup.cfg.drv' during evaluation because the option 'allow-import-from-derivation' is disabled
(use '--show-trace' to show detailed location information)

Attempting to use nix run or nix build gives the same unusual versioning error:

$ nix build .#default
error: builder for '/nix/store/gpygjmazgs473c1r17ckq458rpcd4nh2-obsidianhtml-3.3.1.drv' failed with exit code 1;
       last 10 log lines:
       > The conflict is caused by:
       >     The user requested obsidianhtml 3.3.0 (from /build/j6x5zdvp8nkyija2wvzs18a58r15i5d0-source/dist/obsidianhtml-3.3.0-py3-none-any.whl)
       >     The user requested obsidianhtml 3.3.1 (from /build/j6x5zdvp8nkyija2wvzs18a58r15i5d0-source/dist/obsidianhtml-3.3.1-py3-none-any.whl)
       >
       > To fix this you could try to:
       > 1. loosen the range of package versions you've specified
       > 2. remove package versions to allow pip attempt to solve the dependency conflict
       >
       > ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
       >
       For full logs, run 'nix log /nix/store/gpygjmazgs473c1r17ckq458rpcd4nh2-obsidianhtml-3.3.1.drv'.

nix develop seems to work fine, though.

[dwrolvink]

I'm not very knowledgeable on Nix flakes, perhaps @Programmerino can help you?

[Jaculabilis]

Based on some local testing, deleting the 3.3.0 files under dist solves the version issue. It looks like the build process ends up trying to install those, which results in the package's version (3.3.1) conflicting with the dist version (3.3.0).

When I delete those locally, I get an error from your regression test that runs in the build step:

$ nix run .#default -- -h
warning: Git tree '/tmp/obsidian-html' is dirty
error: builder for '/nix/store/iqp5q02w0yb6vi43xi9ca5hgcghcqk9f-obsidianhtml-3.3.1.drv' failed with exit code 1;
       last 10 log lines:
       > AssertionError: 1 != 0 : Issues found with filtering
       > - File excl exists but it should not.
       >
       >
       > ----------------------------------------------------------------------
       > Ran 3 tests in 2.005s
       >
       > FAILED (failures=1)
       > DEFERRED: closed webserver
       > /nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6/lib/python3.10/subprocess.py:1070: ResourceWarning: subprocess 351 is still running
       For full logs, run 'nix log /nix/store/iqp5q02w0yb6vi43xi9ca5hgcghcqk9f-obsidianhtml-3.3.1.drv'.

And when I comment out that test, everything else builds fine.

[dwrolvink]

Hmm those dist files don't need to be there. Let me make a PR that might fix this

[dwrolvink]

Okay so I'm not sure why, but the copy_vault_to_tempdir feature is not working in the nix install part, and instead of failing, it just keeps chugging along?

The issue is that both the copy_vault_to_tempdir feature and the filetree component do exclusions, and apparently the latter had a bug in its exclusion system causing it to not exclude the excl folder.

I fixed that, removed the dist folder, and now after running nix run '.#default' -- version I get 3.3.1, which I think means it works again.

[Jaculabilis]

If you create a branch off master with your changes, I can test it on my end and double check that it works.

[dwrolvink]

Not sure why you need a separate branch, but I created nix_test just now. I just ran that command in the git folder, if you'd do a git clone of the master branch you'd have the same files.

But good luck, let me know how it goes

[Jaculabilis]

A separate branch isn't necessary if you're already pushing the changes to master, I was thinking of the case where the changes would be tested before merging to the trunk.

$ nix run github:obsidian-html/obsidian-html#default -- version
3.3.1

$ nix run github:obsidian-html/obsidian-html/nix_test#default -- version
3.3.1

Looks like your fixes worked! Thanks for the prompt attention, I'm looking forward to trying this out.

[Programmerino]

Hey! Are there any lasting questions about the Nix build or does everything look like it's fixed? For the allow-import-from-derivation error in the first comment, the build requires IFD which as far as I can tell is enabled by default, but you might want to try editing your system nix.conf file and set allow-import-from-derivation to true. If this is a recurring problem, it can be included in the build as well.

Finally, if there are any changes I could make to make it easier to maintain, let me know.

[dwrolvink]

For me an open question is how to setup a regression test in github actions for this. When I ran it the first time on my local machine it started with downloading a gig of data to build all the requirements from the ground up. I don't really want to have that step run every time there is a PR.

I could just write a local test of course, but then I'll have to remember to run it each time. I guess I could build a container image with all the stuff cached in it, but not sure how that would work and if there are better ways.

Know anything out of the top of your head?

[Jaculabilis]

Nix builds are deterministic, so if you run locally once, it should cache anything it doesn't need to rebuild if you run it a second time. I'm not familiar with GitHub Actions, so I don't know how what tech exists for bridging it to Nix builds. I suspect using GitHub-hosted Action runners would mean re-downloading everything on each run, especially if there was containerization that prevented subsequent builds from sharing a Nix store.

This particular failure involved the regression test failing, so I think I just found the repo at a bad time between patches.

[Jaculabilis]

@Programmerino Is outputs.mkProject a well-known output used by some external integration? I don't see it used anywhere in the project, and it's not one of the known flake outputs listed in the flakes wiki page. Are compile and run supposed to be targets for nix build or nix check?

the build requires IFD which as far as I can tell is enabled by default

I'm not sure that's true. I still get that error for the current HEAD:

$ nix flake show github:obsidian-html/obsidian-html?rev=28d6fb09d9505d9f4f3469ec206abd0f30e0fa8b
github:obsidian-html/obsidian-html/28d6fb09d9505d9f4f3469ec206abd0f30e0fa8b
├───apps
│   ├───aarch64-darwin
│   │   └───default: app
│   ├───aarch64-linux
│   │   └───default: app
│   ├───i686-linux
│   │   └───default: app
│   ├───x86_64-darwin
│   │   └───default: app
│   └───x86_64-linux
│       └───default: app
├───checks
│   ├───aarch64-darwin
error: cannot build '/nix/store/3aykyyzv7crik07b5zp22l4c0iippjqi-setup.cfg.drv' during evaluation because the option 'allow-import-from-derivation' is disabled

Adding --allow-import-from-derivation instead produces this error:

error: a 'aarch64-darwin' with features {} is required to build '/nix/store/3aykyyzv7crik07b5zp22l4c0iippjqi-setup.cfg.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

This seems to be a known issue with using IFD in flakes: https://github.com/NixOS/nix/pull/5253

[Programmerino]

compile and run are both derivations that are created if you pass valid configuration details to mkProject as a separate Nix derivation. It is there so that you can generate HTML output with a Nix build, which is helpful if you want to use this project in a CI/CD configuration with something like Fleek. I would be happy to create a example GitHub repo to show this off.

As for IFD, that pull request you linked is only for certain Nix operations, like the nix flake show you are using. If you use it to compile or run tests, you shouldn't have any problems with it. The error about aarch64-darwin is a result of certain Nix commands trying to evaluate all derivations, even ones unsupported on the current system. If you want to run tests, they are done by default when compiling anyways, so I would avoid using nix flake check or related commands.

[Programmerino]

For me an open question is how to setup a regression test in github actions for this. When I ran it the first time on my local machine it started with downloading a gig of data to build all the requirements from the ground up. I don't really want to have that step run every time there is a PR.

I could just write a local test of course, but then I'll have to remember to run it each time. I guess I could build a container image with all the stuff cached in it, but not sure how that would work and if there are better ways.

Know anything out of the top of your head?

I'm sorry for missing this earlier! It seems like this is the most efficient solution right now and should cache all of the downloads you were encountering. Instead of ./#devShell.x86_64-linux, you can probably just use ./

[langfingaz]

Hi there! It seems, as if the nix flake is currently broken again. I ran nix --extra-experimental-features flakes --extra-experimental-features nix-command run .#default -- version and got this output:

--------------------- IndexFromTags -----------------------------
...
[ INFO  ] module.controller.run_module() :: copy_vault_to_tempdirectory.run()
Traceback (most recent call last):
...
File "/nix/store/sz0j8k8ljh7y8qgyfxgqb3ws11bcy4gs-python3-3.10.6/lib/python3.10/pathlib.py", line 1175, in mkdir                                                                                                            self._accessor.mkdir(self, mode)                                                                         FileNotFoundError: [Errno 2] No such file or directory: '/homeless-shelter/.config/obsidianhtml/tmpdir/input'
...
self.parent.mkdir(parents=True, exist_ok=True)
...
PermissionError: [Errno 13] Permission denied: '/homeless-shelter'

IndexFromTags:   > (self check) config dict should have been fetched                                         .             
                                                                                   IndexFromTags:   > index.html should exist in the expected path                                              F

Sorry for the bad formatting, I tried to copy this from Termux on my Android mobile 😂

[dwrolvink]

Some contributor made the nix flake so I don't know much about that, but the error here is suggests that it tries to create the folder /homeless-shelter/.config/obsidianhtml/tmpdir/input, but does not have permissions to write there.

Normally, it should try to create this folder at $HOME/.config/obsidianhtml/tmpdir/input, is /homeless-shelter your home dir?

Kind of a bummer, but this path was never made configurable. You might try disabling copy-vault-to-tempdir, though the compiled config file should also be written to /homeless-shelter/.config/obsidianhtml, so it might go wrong there then anyways.

ps. Pretty sure that this is not a nix flake related issue, unless you can attest that is works the "normal" way but not with the nix flake. If it's not a nix flake related issue, please make a new issue for your problem.