migrate to pyproject.toml

[ddobrinskiy]

In this PR I will attempt to migrate nbdev to pyproject.toml, and learn something new along the way

Why do this at all?

I am using nbdev as my primary dev framework, and here are some pain points that I hope to address:

• the setup.py format is declared legacy by the PSF (link)
• reduce the learning curve for devs trying nbdev for the first time:
  • the config will be located in pyproject.toml, which they are already familiar with
  • (??)some of nbdev's custom commands can be hidden behind pdm hooks, if we choose to go that way
• dependency management - need to manually keep settings.ini up-to-date
  • (??) can be automated via pdm
• venv management - also quite manual right now
  • (??)can be automated via pdm
• need to remember nbdev entry points, and to run them at appropriate times (I usually setup elaborate Makefiles)
  • pdm can take care of that via "hooks", e.g.:
    • always run nbdev_export before building
    • always run nbdev_test and nbdev_readme before publishing
    • etc

(??) - marks points where I am yet undecided

Preliminary plan

• [ ] migrate nbdev itself
• [ ] migrate nbdev_new
  • [ ] update https://github.com/fastai/nbdev-template
• [ ] create a script to migrate existing projects from settings.ini and setup.py to pyproject.toml
  • this will allow us to publish a breaking change, and not worry about backwards compatibility
• [ ] migrate nbdev_pypi from twine to pdm/hatch?
• [ ] update ci, especially cache-dependency-path https://github.com/fastai/workflows/blob/master/nbdev-ci/action.yml#L29

Some open questions:

• which build backend to use?
  1. pdm-pep517
  2. setuptools
  3. flit-core
  4. hatchling
  5. pdm-backend
• should we add a toml-management tool, such as pdm or hatch?
  • do the benefits justify the extra dependency? need to dig deeper
    • counterargument: we already depend on twine for nbdev_pypi, and pdm can also replace twine
  • at first glance, these almost have feature-parity? need to dig deeper

	flit	pdm	hatch	manual (no tool)
dependency management	❌	✅ (+) takes care of deps via pdm add PACKAGE_NAME (-) one extra tool for user to learn	✅ (+) can be hidden from the user and invoked behind the scenes (-) deps are added/removed manually in pyproject.toml	❌ (+) no new external dependencies (-) deps are added/removed manually in pyproject.toml
venv management	❌	✅	✅	❌
build	tbd	(+) supports multiple build backends	(?) backend support	tbd
publication (replaces twine)	✅	✅	✅	❌
pep-582 support	?	✅	?	?

• ℹ️ Looking at the table above, I am gravitating towards using pdm by default, since it look like the only tool that

  • allows the user to change the build backend
  • supports pep-582 (but this is likely to change in the future)

• should nbdev_test always run in a virtualenv?

  • currently nbdev uses the user's active env by default, and then creates a fresh env from settings.ini's dependencies during CICD
  • this may lead to unexpected CICD fails
  • by enforcing that nbdev_test runs in a venv, we reduce the distance b/w CI-CD and DEV environment, which is always good

[ddobrinskiy]

@jph00 @hamelsmu hi guys, I'm just starting to go down the pyproject.toml rabbit hole, would you have any comments to the plan and questions outlined above?

Would be interested to hear your take

[ddobrinskiy]

Also, I feel that this would be better as multiple PRs.

But since this would be a breaking change, we would have to roll it out all at once with support for:

• new pkg init
• migrating existing pkg
• addition of pdm?

What if we go this way:

• create a new branch (B) in this repo
• create multiple PRs into (B)
• merge (B) into (main)
• publish new release

[LinuxIsCool]

Hi @ddobrinskiy. I am interested in interoperability between nbdev and poetry. Do you think this is possible with what you are proposing? Thanks!

[aviadr1]

is this PR also hoping to achive poetry<-->nbdev integration? in the sense that users would be able to use poetry to add dependencies to their projects, without needing to tinker with the settings.ini file?

[jph00]

Sorry it's been so long - but I've investigated this on and off over the last year, and I keep running into things that the toml format doesn't support. So I'm going to close this, unless more stuff is added to pyproject.toml.

[ddobrinskiy]

Agreed, that's why I ran into a wall here as well