Installation from git is broken

[jbbarth]

Problem:

pip uninstall -y simpleflow; pip install git+https://github.com/botify-labs/simpleflow.git@62c4fd24e87c754d253438d78fd85b2e733133f2

results in:

% ls .venv/lib/python3.11/site-packages/simpleflow
__init__.py  __pycache__     activity.py  canvas.py   constants.py  exceptions.py  executor.py  futures.py  log.py              marker.py     registry.py  storage.py  workflow.py
__main__.py  _decorators.py  base.py      command.py  download.py   execute.py     format.py    history.py  logging_context.py  metrology.py  signal.py    task.py

We're missing all the subdirectories.

Versions in case it matters:

% python --version
Python 3.11.5
% pip --version
pip 23.1.2 from /Users/jean-baptiste.barth/Library/Caches/pypoetry/virtualenvs/xxxx-4VfeYLr8-py3.11/lib/python3.11/site-packages/pip (python 3.11)

I'm digging

[ewjoachim]

See https://github.com/botify-labs/simpleflow/blob/main/pyproject.toml#L86C2-L88 and https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html . We're missing tool.setuptools.packages.find and the find directive

[jbbarth]

Thanks @ewjoachim I just saw exactly that!

For posterity:

• ❌ I confirmed it was the same with an old commit (pip uninstall -y simpleflow; pip install git+https://github.com/botify-labs/simpleflow.git@1e755cea0693f55dcc1013a3235507c88df79e39 => same result)
• ❌ the behaviour is the same locally: pip uninstall -y simpleflow; pip install --verbose ../../botify-labs/simpleflow => makes it easy to test
• ✅ changing "tool.setuptools.packages" from a static value to find = {} works

I'll check everything twice and submit a PR this morning

[ybastide]

Oopsie—I've been bitten by this a few times but didn't investigate it; thank you both!

[jbbarth]

The issue is that as soon as I include all sub directories correctly, I get "build", "examples", "venv" and "tests" installed into my site-packages. The "exclude" directive doesn't seem to work, at least not as documented. I'm digging more. Send help if you don't hear about me in 2 days.

[jbbarth]

@ybastide @ewjoachim I randomly stumbled upon Hatch via Stackoverflow, and it works nicely out of the box. I'm tempted to dig into that direction, and in particular check if it has any impact on "script/release". Any opinion on the topic?

[ybastide]

Looks like it would simplify script/release bigly 🙂 Joachim has a lot of experience in such subjects, I'll defer to both of you.

[ewjoachim]

Not a huge fan of hatch compared to poetry (both are equivalent tools) but it's true that hatch, being born later, uses the standard PEP-defined format for pyproject.toml while poetry uses its own format.

Honnestly, I'm ok with both. Let's go with hatch, this way I'll learn.

[jbbarth]

Alternatively we can keep setuptools if we find a config that works (== that doesn't leak "examples" in my site-packages 😆)

About poetry: I'm using it daily and I absolutely hate it. It failed us so many times (not in the context of a lib though). I'd be very happy to discuss this with you over a beer one day!

[ybastide]

👎 on keeping setuptools, we all want to learn 🙂

[jbbarth]

@ybastide @ewjoachim I tested hatch a bit more and submitted a PR here => #435. Lmk what you think, no hurry!