Build tiktoken (ie. Rust package) as part of workspace

[foges]

I'm trying to add the tiktoken source code as a package in my uv workspace. It's available through pip, but I need to make some local modifications. The core of tiktoken is written in Rust and has a setup.py compile step. Things I've tried:

• Adding it to my uv workspace and importing it directly
  • This doesn't build the package so fails to find from tiktoken import _tiktoken
• Building it using either uv pip install . (in the tiktoken directory), uv run python setup.py install
  • I then just get the error that there's a circular import due to the from tiktoken import _tiktoken in the tiktoken module.
• uv build && uv pip install .
  • Building works and it adds it to .venv/lib, but as soon as I run anything in other workspace directories, it overrides the .venv/lib package with a workspace reference.
• Leaving it out of the workspace and just building with uv run python setup.py install
  • It now doesn't get built to .venv/lib.

My directory structure is

pyproject.toml        <-- `members = ["packages/*"]` and `package = false` in here. 
packages/
  mylib/
    mylib/            <-- Trying to import tiktoken from here
    pyproject.toml
  tiktoken/           <-- Copied from the tiktoken github
    tiktoken/
    pyproject.toml

One thing to note is that tiktoken uses a different build backend "setuptools.build_meta", instead of "hatchling.build", which seems to cause some issues in terms of package discoverability. I'm not sure it's easy to get around that due to "setuptools-rust>=1.5.2".

At this point I'm out of ideas. Any suggestions are greatly appreciated :)