narwhals-dev
REFACTOR: Unite Your Requirements!
With the arrival of a lightning-fast package manager, mighty UV, the requirements.txt is like yesterday's news. Instead of navigating through a jungle of dependencies, like a mismatched sock drawer, everything can now unite in one glorious spot—just like Italy in 1861!
This marvelous change even makes installing dependencies blazingly fast (except for cudf—polars is better anyways). Now it's as easy as pie with just one command: uv sync --all-extras. No more typing three commands when you can do it in one! 1 > 3 — gives you more time to enjoy that delicious Italian caffè.
What type of PR is this? (check all applicable)
- [x] 💾 Refactor
- [ ] ✨ Feature
- [ ] 🐛 Bug Fix
- [ ] 🔧 Optimization
- [ ] 📝 Documentation
- [ ] ✅ Test
- [ ] 🐳 Other
Checklist
- [ ] Code follows style guide (ruff)
- [ ] Tests added
- [ ] Documented the changes
If you have comments or can explain your changes, please do so below.
P.S. I've updated pyproject.toml and banished those pesky .txt files. More changes are coming; just testing the waters before diving in!
Buongiorno Francesco (@FBruzzesi),
Embracing change can indeed be challenging, and that's perfectly normal.
- If someone's workflow is not on
uv, then how can he/she/they pick up dev-dependencies?
It's understandable that this might seem like a hurdle. However, I see it as an exciting chance to introduce the efficiency of uv. For those who prefer sticking with pip (those poor souls), I've ensured that the pyproject.toml is structured to allow all optional dependencies to be installed effortlessly with the command:
pip install .[docs,modin,pandas,polars,pyarrow,dask,dev]
uv syncis too strict for my (but I guess many) development settings, and to avoid that it doesn't uninstall the rest of the libraries in the env, I need to remember to runuv sync --inexactinstead.
I understand that might be inconvenient, but this rigidity also promotes consistency in workflows. uv sync --inexact is a helpful option to adapt it to your specific needs.
- For a few setups based on Docker (e.g., devcontainers and Codespaces), having a file allows for easier copying and installation of dependencies before even copying the Python package, which leads to better caching (at least in my experience).
From what I understand, Docker caching should remain intact whether using pyproject.toml or requirements.txt. If the current setup is causing issues, I'm keen to explore alternatives that work better for you. Here’s an example to illustrate:
FROM python:latest
COPY ./pyproject.toml /pyproject.toml
RUN pip install .
RUN echo installing dev && pip install .[dev]
RUN echo installing polars && pip install .[polars]
Please let me know if there's anything specific you find challenging, and I’d be happy to discuss further.
Embracing change can indeed be challenging, and that's perfectly normal.
Definitely, and we would like to let people use whatever workflow works for them (not only pip) as well. Thanks for adjusting the pyproject section.
From what I understand, Docker caching should remain intact whether using
pyproject.tomlorrequirements.txt. If the current setup is causing issues, I'm keen to explore alternatives that work better for you. Here’s an example to illustrate:
FROM python:latest COPY ./pyproject.toml /pyproject.toml RUN pip install . RUN echo installing dev && pip install .[dev] RUN echo installing polars && pip install .[polars]Please let me know if there's anything specific you find challenging, and I’d be happy to discuss further.
The above snippet would run from RUN pip install . each time something changes in the codebase.
What I usually do is something along the lines:
...
COPY ./requirements.txt /requirements.txt
RUN uv pip install -r requirements.txt # this will be cached until something changes in requirements.txt
COPY <rest of codebase> <rest of codebase>
RUN uv pip install .
...
There should be a way to link requirements.txt dynamically into pyproject, so personally that's how I would do it but happy to discuss
thanks - you'll need to update the instructions for this to work (e.g. ci, contributing)
fancy making a PR first, so that github automatically runs ci for you, and then coming back to this?
Ciao @FBruzzesi, Sorry for late reply, what you think about this setup?
FROM python:3.12-slim-bookworm
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
COPY ./pyproject.toml pyproject.toml
RUN uv venv
RUN uv pip compile pyproject.toml --extra dev > requirements.txt
RUN uv pip install -r requirements.txt
RUN echo hello
RUN uv pip install -r pyproject.toml --extra docs
COPY . .
RUN uv sync --all-extras
When trying on the local machine it felt like there was a caching of
RUN uv pip compile pyproject.toml --extra dev > requirements.txt
RUN uv pip install -r requirements.txt