cubed-dev
build: use hatchling
~This is just a suggestion, happy to discuss, but thinking to the long time based on some things we learned in dask in high energy physics. Specifically that our graphs can get very large and a compiled taskgraph api may be something we desire, likewise being able to easily include extensions may help the project in the future. I know this is a premature optimization, but it also doesn't negatively affect anything.~
~For now, running cmake in scikit-build-core is turned off since this repository has no compiled code. If the project goes with rust, maturin is an easy change from here.~
I have removed the setup.py and setup.cfg, so all configuration is in pyproject.toml and changed the build system to hatchling.
I have also moved cubed's implementation into a src/ directory so there's no possibility for clobbering packages.
Tests may fail for a bit here where I find all the rough edges from the change.
Curious to hear your thoughts!
Thanks for the PR @lgray - a few comments below.
Specifically that our graphs can get very large and a compiled taskgraph api may be something we desire
Cubed should be able to help with this since each node in the graph is an array not a chunk.
For now, running cmake in scikit-build-core is turned off since this repository has no compiled code.
I would rather make this change when it is needed - i.e. alongside the PR that introduces the compiled code.
If the project goes with rust, maturin is an easy change from here.
I can see some parts of the Cubed stack migrating to Rust in the future. For storage we already have obstore support (#715), and zarrs-python (#731) is on the TODO list. Both of these are simply using Rust-based Python libraries though so don't need any build changes, but we could have Rust code in Cubed for the runtime, primitive (blockwise, rechunk) and core (planning) layers. I think it's natural to make the necessary build changes at the point we introduce the compiled code.
I have removed the setup.py and setup.cfg, so all configuration is in pyproject.toml.
This is a good change - could you break this off as a separate PR and we'll get it merged.
OK - I'll move everything to hatchling in the meantime. Do you have any opinion on src/ vs the package in the root of the repository?
OK - I'll move everything to hatchling in the meantime. Do you have any opinion on
src/vs the package in the root of the repository?
I don't have a strong opinion on that. @TomNicholas do you have any thoughts?
Gotcha - I only did it here because not doing so has gotten me into trouble with consistent testing a few times in the past.
OK - kept the pyproject.toml-only changes, used a python-only / more minimal build system. All the code is now beneath src/
Thanks! I'll see what I can do to get it all going. It seems to not be picking up the mypy config correctly either.
Thanks. Don't worry about array API, lithops and Zarr v2 - they are known failures on CI at least (#752, #753).
Hey - sorry for dropping off here. I did start a vacation last week, given this is fairly low priority I'll tie it up on my return.
Thank you very much for the info about known failing tests so I don't pull my hair out. :-)
No problem @lgray! Moving the src dir may require a bit of coordination with other PRs so let me know when you're ready.
I've also been fixing some of the flaky test failures.
There are commits from the primary author as of two days ago. Probably external issues or something: https://github.com/pypa/hatch/commits/master/