Hardware attributes, global package storage, and binary dependencies

[happysalada]

Hey, first of all thank you so much for this, the state of package management in python is so horribly broken and needs this!

A bit of context, im one of the numerous maintainers of nixpkgs, which is a repo to build packages for nixos. Its a linux distribution that makes configuring linux system "easy" .

Python packages are notoriously hard because there is no reproducibility. Poetry is one step in the right direction, but many python project refuse to use it because it doesnt have "hardware" attributes (amd or nvidia gpu for example). Did you consider those ?

Pnpm in the js world is popular mostly because it provides a central local storage to deduplicate packages installed on a system. Its not a vital feature, but its a very nice to have . (the package directory is then populated with symlinks)

None of the package managers have anything to say about binary dependencies. This one is admittedly a hard one. That is the whole reason of existence of the nix package manager. Having something other than the readme that details binary dependencies would be amazing though.

Just some thoughts, hope those help, thanks again !

[cnpryer]

Hi thanks for this!

... it because it doesnt have "hardware" attributes (amd or nvidia gpu for example). Did you consider those ?

I have not! Sounds interesting.

... it provides a central local storage to deduplicate packages installed on a system. ...

I have not used pnpm but would you say this is along the lines of what https://github.com/konstin/poc-monotrail demonstrates?

None of the package managers have anything to say about binary dependencies. ...

I want to make sure I understand this, so could you provide me with an example?

[happysalada]

| have not used pnpm but would you say this is along the lines of what https://github.com/konstin/poc-monotrail demonstrates? yes on the surface poc-monotrail sounds like it's doing exactly that. It's a feature where the details are hard to get right though. For npm, the folder structure inside the packages directory matter, I don't know too much about python though.

For an example of the second point, look at https://github.com/Unstructured-IO/unstructured in the readme, they mention that you need libreoffice libmagick ... Those are binary dependencies that need to be on your system but the requirements aren't in the code. This is what nix does for you. you can specify binary dependency requirements and those are actually fetched and enforced. This last feature is really hard though, so feel free to skip.