Refactor lazy imports

[ofek]

Currently, to keep the CLI responsive we import most things where they are used rather than at the top of modules. The scientific Python community has coalesced on a design and a reference implementation that we should adopt:

• https://scientific-python.org/specs/spec-0001/
• https://github.com/scientific-python/lazy_loader

This will be used ubiquitously throughout not just Hatch but also Hatchling.

Special care must be taken when adding dependencies to Hatchling and I think this is fine for the following reasons:

1. The aforementioned package has no dependencies itself
2. That package builds with setuptools so re-distributors will not encounter cyclical dependencies
3. We will be getting rid of the dependency pluggy soon so it will just be a trade

[hoechenberger]

We have adopted this lazy loading approach in MNE-Python and I have to say I'm not a fan. It feels extremely hacky and basically abuses .pyi files to achieve something that's not part of the Python standard library. My personal suggestion is, if you can live with nested imports, use these instead.

The scientific Python community has coalesced on a design

I'd also say this is not quite correct. A few people wrote a proposal, and a handful of packages have adopted it. That's about it.

There are very good reasons why the PEP for lazy loading was rejected not so long ago.

[ofek]

Oh very interesting, thank you!

[hoechenberger]

Don't get me wrong, I'm not telling you what and what not to do :) I just wanted to share my personal feelings and experiences here.

[hugovk]

Upstream at CPython, there have been recent PRs to defer some imports in stdlib modules to improve import time: https://github.com/python/cpython/issues/109653

If you see some other candidates, feel free to comment on the issue. No guarantees though, some maintainers are cautious about these kind of changes.

[ofek]

I was unaware of such an effort, that's awesome! That will help CLIs everywhere in perpetuity.

edit: also apps running in function-based models like AWS Lambda

[djcopley]

Forgive me if this is duplicative information, but python importlib has lazy loading utilities available: https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader

[ofek]

Yes but you have to use that programmatically rather than importing as usual or something that resembles imports.