soda-core
soda-core copied to clipboard
sodadata
Adoption barrier related to `soda-pandas-dask` dependencies
Not strictly a bug, but significant adoption barrier.
Summary: soda-core-pandas-dask and soda-pandas-dask cannot be installed on a system with a recent (last 3yrs) version of Python and (last 1yr) Rust compiler.
Details
soda-core-pandas-dask and soda-pandas-dask pin dask-sql to <2023.10.0:
https://github.com/sodadata/soda-core/blob/d7786fb47199a53cd7c80ee789fb0c7e1a3901bc/soda/dask/setup.py#L10
In practice this resolves to 2023.8.0 on a fresh install, which only has prebuilt binaries available for Python <= 3.10 (https://pypi.org/project/dask-sql/2023.8.0/#files). This means that on any Python installation 3.11+ (i.e. last 3 yrs) you need to compile from the rust sources.
By itself already a bit of a barrier, but compounding the issue is that the dask-sql 2023.8.0 sources don't compile out of the box with Rust versions >= 1.80.0 (released July 2024) due to a breaking change introduced in that version (specifically: It cannot compile a transitive dependency, the time rust library, due to breaking changes related to type inference in 1.80.0, see here). Therefore users, if they bother to go down the rabbit hole, need to figure out the Rust compilation error and downgrade their rust compiler to, e.g., 1.79.0
Due to rust's EOL policy, all compatible versions of the rust compiler are already deemed end-of-life. Therefore as far as I can tell there is no way to use this Soda library on Python 3.11+ without installing EOL software.
It is worth pointing out as well (as you're probably already aware) that dask-sql itself is currently unmaintained (https://github.com/dask-contrib/dask-sql/issues/1344), and has not seen any development activity in close to a year.
Related
https://github.com/sodadata/soda-core/issues/2085
