danth
treewide: new base16 generator
Because the Haskell generator does not respect the base16 style guide and creates very monochrome themes in many cases, I would like to suggest providing an alternative. This is a continuation of the conversation started in PR #892 and was inspired by the fact that stylix is now adding other generators.
I am currently working on a alternative at https://github.com/Mikilio/base16-generator. There I have a notebook outlining the process, criteria and reasoning that led to this algorithm.
I hope to receive feedback on it, and I am willing to make big changes if warranted.
Great idea. I'm also looking for a base16 generator option in another language with a smaller download size when rebuilding the configuration. I'm tired of downloading 300MB+ just to change the wallpaper. Is it possible to rewrite the generator in a smaller, faster language like rust or golang?
I plan to write the algorithm, that currently exists in the form of a Jupyter notebook, in Rust. I expect it to run in under 1sec on a normal laptop, but I have no idea about the size of the binary. (should be less than 300MB, though)
Realistically, given my motivation and availability, it should be done by Christmas.
do base16 color scheme generators not already exist?
-- https://github.com/nix-community/stylix/pull/892#issuecomment-3575942423
What about using one of the following existing tools:
nix run nixpkgs#flavours -- generate --stdout dark $IMAGE
nix run nixpkgs#gowall -- extract --colors 16 $IMAGE
I prefer the interpolation results from flavours.
I do not know how these tools extract their colors because I just read enough documentation to figure out what command to run. This also means that I did not vet their source code.
Although this is totally irrelevant, flavours is made by Misterio77.
I have actually encountered both solutions in my preliminary research.
Flavors is actually good, yes. But I know that my algorithm produces color schemes that follow the spec more closely, but also differs too much from the approach of flavors for me to just contribute there to meet the extra requirements I place on the scheme. For those, who don't care about these requirements, I'd love to have flavors in stylix. Personally, I wanted something of at least the same quality as popular themes out there, but instead customized. (Catppuccin, Nord, Dracula, Gruvbox, etc ...) (FYI: my requirements are very mathematical, based on color theory. I also put in some effort to ensure good schemes for colorblind people. Additionally, I make sure that red is a reddish color and green is a greenish color, etc. to ensure that error highlighting and status lights stay recognizable across themes.)
Gowall was just bad, though.
My conclusion: I'm already halfway there and about to produce something with better results than any of these (at least for my requirements), so I'm probably just going to finish it.
Flavors is actually good, yes. But I know that my algorithm produces color schemes that follow the spec more closely, but also differs too much from the approach of flavors for me to just contribute there to meet the extra requirements I place on the scheme.
If the original base16 specification results in poor colors, we can just make our specification. Is it better to follow the original base16 specification than to follow flavours's behavior?
For those, who don't care about these requirements, I'd love to have flavors in stylix. Personally, I wanted something of at least the same quality as popular themes out there, but instead customized. (Catppuccin, Nord, Dracula, Gruvbox, etc ...)
Would be awesome if you managed to implement SOTA customizable color extraction.
Side note: Maybe surprisingly, before https://github.com/nix-community/stylix/pull/2014, I always used the default stylix.image mentioned in the Stylix documentation and used its generated color scheme. Since the new image resolution results in very vibrant colors after https://github.com/nix-community/stylix/pull/2014, I manually reverted stylix.base16Scheme to the previously generated color scheme. Since a couple days ago, I am using a polished version of the previously generated color scheme by interpolating between base00--base07 and base08--base0F in the Oklab color space. I manually picked the four color edges so they roughly match the original vibe. So far I am really liking it, and unlike before, everything is actually legible now. Since none of the existing base16 schemes match my current scheme requirements, I was basically forced to make my own.
(FYI: my requirements are very mathematical, based on color theory. I also put in some effort to ensure good schemes for colorblind people. Additionally, I make sure that red is a reddish color and green is a greenish color, etc. to ensure that error highlighting and status lights stay recognizable across themes.)
Yes, when I previously skimmed through your notebook, I could tell color analysis and extraction was very methodical and intentional.
Gowall was just bad, though.
Agreed. Maybe their color extraction was just an afterthought without much interest, resulting in a very basic implementation.
My conclusion: I'm already halfway there and about to produce something with better results than any of these (at least for my requirements), so I'm probably just going to finish it.
By now, you have my interest :)
Since a couple days ago, I am using a polished version of the previously generated color scheme by interpolating between base00--base07 and base08--base0F in the Oklab color space. I manually picked the four color edges so they roughly match the original vibe. So far I am really liking it, and unlike before, everything is actually legible now. Since none of the existing base16 schemes match my current scheme requirements, I was basically forced to make my own.
Actually, my approach is not so different from what you did there manually. -> Some ML to generate a scheme that represents the vibe of the image -> from there, use some math on oklab color space to make an ergonomic theme
I plan to write the algorithm, that currently exists in the form of a Jupyter notebook, in Rust.
Great to have this in Rust. It definitely helps with build sizes when you're just trying to change a wallpaper. The Haskell implementations are fine, but in places with slow or expensive internet, pulling down a 300MB compiler just for a wallpaper change is a waste of time and data #1862 .
I plan to write the algorithm, that currently exists in the form of a Jupyter notebook, in Rust.
Great to have this in Rust. It definitely helps with build sizes when you're just trying to change a wallpaper. The Haskell implementations are fine, but in places with slow or expensive internet, pulling down a 300MB compiler just for a wallpaper change is a waste of time and data #1862 .
Although I like Rust primarily because its type model matches my programming style, compiling Rust binaries from source might require significantly more CPU and download bandwidth than the current Haskell implementation. On NixOS, compiling from source essentially happens when binaries are not packaged and cached by Nixpkgs. If source compilation is the biggest problem, the simplest workaround is to upstream the package to Nixpkgs. To avoid fragmenting the Stylix code base, this has not been done, although this has been briefly discussed in the past:
With the palette generator as a function, we could potentially move it to its own repository - it's only part of Stylix because it's been here since the very beginning.
IMHO, replacing the current mono repo architecture does not add much benefit, especially, when the only use case for those micro repositories is Stylix. Managing these separately does not seem worth it for now.
-- https://github.com/nix-community/stylix/pull/477#issuecomment-2254170449
This does not mean I am against a Rust implementation. I suppose any non-immutable language, unlike Nix or Haskell, should be sufficiently performant, including GC languages, like Python or Go.
This does not mean I am against a Rust implementation. I suppose any non-immutable language, unlike Nix or Haskell, should be sufficiently performant, including GC languages, like Python or Go.
I have no issues with the Haskell implementation itself; it's strictly the download footprint that's challenging for me. Using Rust or Go sounds like a solid plan. I can only imagine how snappy wallpaper switches would be with a lightweight tool. for comparison, matugen comes in at under 10MB (2.44MB in the latest release).
simplest workaround is to upstream the package to Nixpkgs
I totally agree on the Nixpkgs point; getting it cached is key to avoiding those massive compile times.
should be sufficiently performant, including GC languages, like Python or Go.
I'd really love to see this resolved, and I'm more than happy to help contribute to the Rust/Go implementation/porting if needed.
Although I like Rust primarily because its type model matches my programming style, compiling Rust binaries from source might require significantly more CPU and download bandwidth than the current Haskell implementation. On NixOS, compiling from source essentially happens when binaries are not packaged and cached by Nixpkgs. If source compilation is the biggest problem, the simplest workaround is to upstream the package to Nixpkgs. To avoid fragmenting the Stylix code base, this has not been done, although this has been briefly discussed in the past:
Regarding this point: stylix is part of the nix-community org since a while ago. We can and should use their cache.
Although I like Rust primarily because its type model matches my programming style, compiling Rust binaries from source might require significantly more CPU and download bandwidth than the current Haskell implementation. On NixOS, compiling from source essentially happens when binaries are not packaged and cached by Nixpkgs. If source compilation is the biggest problem, the simplest workaround is to upstream the package to Nixpkgs. To avoid fragmenting the Stylix code base, this has not been done, although this has been briefly discussed in the past:
Regarding this point: stylix is part of the nix-community org since a while ago. We can and should use their cache.
The compiled binary is already cached on https://nix-community.cachix.org, and installing it requires only downloading 1 MB:
$ curl https://nix-community.cachix.org/1nb3vnwcjfgdwd38rdmqd0i9xs6qwbim.narinfo
StorePath: /nix/store/1nb3vnwcjfgdwd38rdmqd0i9xs6qwbim-palette-generator
URL: nar/e87fe43966b29d35522f398d0bf6dfff9cbe7d283812eaac8e4dab9fd6b562b6.nar.zst
Compression: zstd
FileHash: sha256:e87fe43966b29d35522f398d0bf6dfff9cbe7d283812eaac8e4dab9fd6b562b6
FileSize: 1115833
NarHash: sha256:1fl83xc8nvc3kf9sv8g19zqn5wqqjfz4afnvz9mprsxya034h1gs
NarSize: 5382456
References: 1nqqjacc6dnj61jlpgz5hk9zdjbfidbr-elfutils-0.194 54jkwsavi3fdciqfyjmbilq0jhvv4jga-gmp-with-cxx-6.3.0 b9p0zpa93hwvh4d0r1rmgc2500yx2ldn-libffi-3.5.2 gdni20c8009xdz8gms6yn1r2hfhmk1jk-numactl-2.0.18 l7xwm1f6f3zj2x8jwdbi8gdyfbx07sh7-zlib-1.3.1 xx7cm72qy2c0643cm1ipngd87aqwkcdp-glibc-2.40-66
Deriver: my3ljxzsjc8kbxk607lp5plzlqnmh7yf-palette-generator.drv
Sig: nix-community.cachix.org-1:H0n89aeN+buppRPlLxm7dKT+4WFLNwIcnM8sST4fCn124LSstC2GJ1BaZ5Jpr7gEoaUVR5d4jCytUjHLnfbmBA==
Most people just do not enable this cache. In that regard, the caching problem has already been solved for a while.
Although I like Rust primarily because its type model matches my programming style, compiling Rust binaries from source might require significantly more CPU and download bandwidth than the current Haskell implementation. On NixOS, compiling from source essentially happens when binaries are not packaged and cached by Nixpkgs. If source compilation is the biggest problem, the simplest workaround is to upstream the package to Nixpkgs. To avoid fragmenting the Stylix code base, this has not been done, although this has been briefly discussed in the past:
Regarding this point: stylix is part of the nix-community org since a while ago. We can and should use their cache.
Users shouldn't use untrusted caches (such as nix-community) for sensitive use cases (e.g. a nixos configuration). See https://garnix.io/blog/stop-trusting-nix-caches for more information.
I read that, however:
I am a nix-community member and I can't write to the stylix cache. It seems to me like only a few selected members actually have write-access. Each to their own, but I wouldn't call the nix-community cache untrusted. If someone has secrets worth protecting from this kind of attack vector, to the point they'd like to build their own binaries, I hope they possess the financial means to operate a build machine and host their own cache.
For whatever measures we can do to mitigate that:
- upload binaries to nixpkgs (if it's considered more trusted)
- run daily checks for cache poisoning
Either way, the color generator rebuilds in like less than 10 seconds and will probably not be updated that often, because it's a specialized tool and not a configurable tool kit. Dependencies are slim. We are talking less than 2MB.
I read that, however:
I am a nix-community member and I can't write to the stylix cache. It seems to me like only a few selected members actually have write-access.
This is not true. Anyone with write access to a nix-community repo has the ability to read the cachix secret key via CI (even on unmerged prs), which by a very conservative estimate is over 250 people[^1]
[^1]: calculated by subtracting home-manager maintainers from nix-community members.
@0xda157 I can't see the settings of the cache, but from what you are saying: Does that means then that nix-community does not use the API tokens for fine-grained access rights?
Also: this is going off-topic, but I would like to know where this kind of discussion is taking place. (And I hope it is taking place)
I read that, however:
I am a nix-community member and I can't write to the stylix cache. It seems to me like only a few selected members actually have write-access.
This is not true. Anyone with write access to a nix-community repo has the ability to read the cachix secret key via CI (even on unmerged prs), which by a very conservative estimate is over 250 people[^1]
[^1]: calculated by subtracting home-manager maintainers from nix-community members.
I always assumed repositories under the nix-community GitHub organization inherit its secrets without them being readable or writable in individual repositories, with the only exception being intentional leaking of the environment variable in CI logs, which again is only visible to authorized people. Would be weird if GitHub did not operate like this, although this would not really surprise me...
Although https://nix-community.cachix.org may be convenient, it is good that it is not enabled by default on NixOS, as it increases the attack surface to less protected entities. For this reason, our documentation should not recommend this cache, although many projects indeed recommend using their binary cache. It might be safe to assume that anyone who actually knows (not just copy-pastes provided code) about binary caches is aware of its security implications, meaning those benefiting from this feature do not need us to remind them about it. As an aside, https://github.com/nix-community/stylix/pull/1796 assumes reasonable Nix knowledge.
this is going off-topic
Indeed. Whether Nix packages are cached should not impact our decision. I will mark related messages as off-topic for now.