conda-forge.github.io
conda-forge.github.io copied to clipboard
conda-forge
Package naming policies for Julia packages
Your question:
Question
How should Julia packages be named within conda-forge?
Introduction
The julia-feedstock is now current with upstream Julia at version 1.8.0. Support is currently best on Linux. Mac OS X support is possible on x86_64 and arm64 but needs further work and testing. Windows support is likely not possible within conda-forge without a comprehensive mingw-w64 ecosystem as opposed to a Microsoft Visual C.
We current have two feedstocks employing a dual language Python - Julia strategy within conda-forge:
- https://github.com/ngam/pysr-feedstock
- https://github.com/conda-forge/xbitinfo-feedstock
These two packages primarily are Python interfaces for the following Julia packages:
Currently the installation of the Julia packages needs to be initiated by the user and is handled completely outside of the conda or mamba package management. The Julia feedstock configures the Julia depot within the conda prefix rather than the default ~/.julia depot location.
It would best if these Julia packages could be directly installed by conda rather than having the user initiate a manual step. I have developed a mechanism using the Julia depot stack that allows conda to directly install Julia packages into the user's Julia load path if they use a preconfigured Julia project environment. Thus we are now prepared to propose conda-forge feedstocks for individual Julia packages and now need to decide upon a naming scheme.
Julia package naming guidelines
Julia packages typically include capitalization according to camel case and a "file extension" with a preceding period. For example:
Julia package naming guidelines are located at the following URL: https://pkgdocs.julialang.org/v1/creating-packages/#Package-naming-guidelines-1
Potential naming schemes within conda-forge
My first preference would be to preserve the original package names. Julia packages can be uniquely identified by the ".jl" extension from those in other languages.
From #18, many seem to prefer no capitalization. If we do want to conform to a language prefix with all lowercase, my preference would be "julia-" since this would be unambiguous:
- julia-pycall
- julia-conda
- julia-hdf5
- julia-hdf5_jll
- julia-clustermanagers
- julia-symbolicregression
- julia-bitinformation
I could also see an argument for using "jl-" based on brevity and similarity to the original name:
- jl-pycall
- jl-conda
- jl-hdf5
- jl-hdf5_jll
- jl-clustermanagers
- jl-symbolicregression
- jl-bitinformation
cc: @ngam, @isuruf @ocefpaf
References
- https://github.com/conda-forge/julia-feedstock/issues/14
- https://github.com/conda-forge/pysr-feedstock/pull/43#issuecomment-1233129264
If we do want to conform to a language prefix with all lowercase, my preference would be "julia-" since this would unambiguous.
This would be my vote too.
Question though: Do we really need the _jll ones at all like julia-hdf5_jll? Or do we simply need these wrappers only? If the latter, we could probably tug those under julia-hdf5 for the case of julia-hdf5_jll.
To move the ball forward, we should target a live case to convince the community of this approach. Let's try to get these two packages completely set up and functional. Then we can proceed with a CFEP for the community to consider.
Here's yet another alternative proposal:
- py_call-jl
- conda-jl
- hdf5-jl
- hdf5_jll-jl
- cluster_managers-jl
- symbolic_regression-jl
- bit_information-jl
The -jl suffix does preserve the original name a bit more. I'm not so sure if converting CamelCase to snake_case makes sense though. Is there any precedent for snake_case in conda-forge?
The -jl suffix does preserve the original name a bit more. I'm not so sure if converting CamelCase to snake_case makes sense though. Is there any precedent for snake_case in conda-forge?
Typically, package names are converted to all lower case using the jinja template {{ name|lower }}, so there are no CamelCase package names. Some packages use snake_case in order to match the python module name, but most packages are kebab-case.
@mkitti a good general precedent for this is R-packages (not sure if you're aware of this...), https://github.com/orgs/conda-forge/repositories?q=r-&type=all&language=&sort=
My understanding from meeting with conda-forge core is that . is allowed so we could proceed with lower case names as follows:
- pycall.jl
- conda.jl
- hdf5.jl
- hdf5_jll.jl
- clustermanagers.jl
- symbolicregression.jl
- bitinformation.jl
I think the Julia community would be happier with that - which is a good thing (!) for making them open to an alternative package management system to Pkg
I don't think keeping the dots is a good idea. If the julia community actually cares about the dots beyond the silly appearance, then Pkg would not work without the dots. But it does:
(@v1.8) pkg> add SymbolicRegression
Updating registry at `~/.julia/registries/General.toml`
Resolving package versions...
Installed DynamicPolynomials ───────────── v0.4.5
Please think about this from the point of view of users using conda-forge already. What we really should care about is the integrity and ease of this ecosystem, not appeasing some esoteric styling choices by others. Julia fans are never going to believe in or use anything outside of julia, so it is a moot point keeping the styling to lure them.
Would would feel more natural to someone using conda?
conda install julia-pycallconda install pycall.jl
Yes, one could use the dots if you really want them to, but we simply don't use them as much for a reason. It is just not common and it is awkward (the dot being significantly smaller than a dash and so on).
My understanding has always been that .jl was just to distinguish julia packages on github, but nothing to do with the actual packages per se.
My understanding has always been that
.jlwas just to distinguish julia packages on github, but nothing to do with the actual packages per se.
Good point, I take back my comment. julia-* is still my vote then.
I am personally very open to any naming scheme tbh (despite my seemingly strongly worded comment above, sorry 😅).
I am just pointing out the facts (as I understand them) so that more experienced julia users (like @MilesCranmer, @mkitti, and @cjdoris) can make an informed call in the context of conda-forge. At the end of the day, the votes that matter most are those of core and experienced julia users. (That is, my vote shouldn't count here!)
In the Julia package manager, you can actually type add SymbolicRegression.jl and it will work. In that context, it's just superfluous though.
(jl_IyuX1c) pkg> add SymbolicRegression.jl
Resolving package versions...
No Changes to `/tmp/jl_IyuX1c/Project.toml`
No Changes to `/tmp/jl_IyuX1c/Manifest.toml`
The other thing I'm noting is that the conda does not seem to care about capitalization. I could spell it NumPy and conda will install the numpy package.
$ conda create -n test -c conda-forge NumPy
This means I could do
$ conda create -n test -c conda-forge SymbolicRegression.jl
and conda will install the package symbolicregression.jl.
@mkitti, we can also do outputs to alias packages, e.g. see grayskull/greyskull
https://github.com/conda-forge/grayskull-feedstock/blob/6c06869fb6207fa440398030ae0c96ccdbf636ab/recipe/meta.yaml#L59-L64
So if you want we can make jl-*, julia-*, and *.jl simultaneously. Which one do you feel most productive?
My first preference is *.jl. My second preference is julia-*. jl-* does not seem that useful to me.
Both for the naming of the feedstocks and outputs on anaconda.org? (Feedstock names don't have to match output names exactly btw)
So, julia-symbolicregression-feedstock or symbolicregression.jl-feedstock? We can easily output both symbolicregression.jl and julia-symbolicregression once inside a feedstock and even symbolicregression
Is there a way to differentiate if a package has been explicitly installed in a conda environment rather than installed because it is a dependency of another package that was requested?
What is the mechanism to resolve a feedstock name from a package name?
They are unrelated. You can have xyz-feedstock as the repo name, then have "name: abc" in meta.yaml (so it will be xyz-feedstock on github, but abc on anaconda.org and conda install abc). There is no special mechanism. Look at conda-forge/pytorch-cpu-feedstock and conda-forge/grayskull-feedstock for example.
Is there a way to differentiate if a package has been explicitly installed in a conda environment rather than installed because it is a dependency of another package that was requested?
There should be, but it could be a bit tortuous. One way is to get the history of the environment, --from-history, to generate a env.yml file for the packages in a given environment. Someone with more experience probably know of a neater way. There are also repoquery whoneeds and repoquery depends which may help here. What sort of situation are envisioning here?
I'm fairly ambivalent, so long as we're consistent and cannot clash with anything that already exists. I mildly prefer julia-pycall (as it's really clear) then pycall.jl (=PyCall.jl since package names are case insensitive).
I think ngam is suggesting we have both julia-pycall and pycall.jl with one of those being an alias.
conda install -c conda-forge PyCall.jl
One reason I'm interested in calling the package PyCall.jl is search engine optimization. If you searched for "PyCall.jl", julia-pycall might not come up in the results. One might not realize there is a conda-forge package for it.
SEO is a strong argument, though you'd hope that if we put "PyCall.jl" in the description of the Conda package (a description policy would be good too) then that would suffice.
I will try to reengage with this once Julia 1.9 is released, which is imminent. There are some interesting possibilities there for conda-forge such as distributing precompiled binary shared libraries.
cc: @cjdoris
Hi @majidaldo , do you have a specific application in mind?
No. But as a polyglot, I just don't want to deal with each programming language's packaging and have to deal with an integration mess as a result.
It should be in the interest of the julia team to work /with/ conda while still keeping their specific way of doing things as has been done with python and R.
multilanguage is the future (already here?) and conda is the only way.
It should be in the interest of the julia team to work /with/ conda while still keeping their specific way of doing things as has been done with python and R.
The "julia team" here is basically me with some help from @ngam. I do not expect the core Julia developers to responsible for building Julia here nor do I expect CPython developers to be building CPython here either.
We currently have one prototype package online.
https://anaconda.org/conda-forge/versionparsing.jl
A couple others were staged but stalled.
Parsers.jl: https://github.com/conda-forge/staged-recipes/pull/20655 MacroTools.jl: https://github.com/conda-forge/staged-recipes/pull/20654
We also need to resolve what to do about pyjuliacall
https://github.com/conda-forge/staged-recipes/pull/20379
If you are interested in volunteering, let me know. There are a lot of issues to resolve.