ENH: support installing yt_idefix as `pip install yt[idefix]`; plug it in docs

[neutrinoceros]

This external frontend has been stable for a couple years, but is hardly discoverable, so I'd like to make it more visible in yt docs. It also seems that pretty nice to me that, combining https://github.com/yt-project/yt/pull/4272 and https://github.com/yt-project/yt/pull/4285, we're actually able to make pip install 'yt[idefix]' seamlessly provide as much functionality as pip install 'yt[ramses]' at a very low maintenance cost. However, there's no precedent for doing this so this is just a proposal, please do raise any objection !

[chrishavlin]

I'm 100% on board with adding external frontends to the docs! in this case I think it makes sense to add where you did, but it could also be nice to have a separate "externally packaged frontends" section if there are others we could add (I don't have a sense for how many external frontends exist, also, totally possible this exists in the docs already??).

But I'm on the fence about adding the install option. I'm not sure it helps with discoverability much and I'm slightly worried about the possibility of an external frontend becoming unmaintained. e.g., if a future yt breaks an orphaned external frontend that was added to yt's install options, I feel like it's a bit more annoying than just a list of external plugins in yt's docs going stale. So a wider discussion of some criteria for both addition to and possible future removal from the install options might be warranted? I may be over thinking this though :)

[neutrinoceros]

I don't think you're overthinking it, it's a reasonable objection. I'm obviously biased so I'd probably better leave it up others to define the criteria to allow this (and again, maybe we shouldn't).

[neutrinoceros]

it could also be nice to have a separate "externally packaged frontends" section if there are others we could add

That's a great idea but I think we'd need to make the rounds just to have an idea what's out there first, and I'd rather have this discussion be a separate issue/PR if that's alright.

By the way, don't you have an extension frontend somewhere ? 😄

But I'm on the fence about adding the install option. I'm not sure it helps with discoverability much

that's true, it doesn't help discoverability. I realise this is a rather weak argument but the appeal to me is that if yt_idefix was ever integrated as a first party frontend (why not ?), this would make the transition absolutely transparent to users.

Actually, the main reason why I'm not proposing yt_idefix for adoption in the core repo is that I do want to keep maintaining it and avoid being tied to the release pace of the whole library. If I changed my mind and didn't want to maintain it anymore, my hope is that it could find a home in the main library.

[chummels]

I am a bit confused. I thought all of the code frontends were automatically included in the yt source install. When I download yt, I sort of expect that the code is present to be able to load in any arbitrary code frontend with a ds = yt.load() command. But maybe I'm misunderstanding something here?

[neutrinoceros]

The idefix frontend is literally a separate package, which yt knows nothing about, so it's never included in any install unless explicitly requested (which is intended). At runtime however, yt.load knows how to import such extension frontends so users don't have to: having yt_idefix installed makes yt behave as if the frontend wasn't special.

Any clearer ?

[jzuhone]

I suspect the main thing is that I don't think @chummels knows about the external frontend capability: https://yt-project.org/doc/developing/extensions.html#frontends-as-extensions

[chummels]

It's true. I was unaware of this functionality. I guess I just don't understand why idefix's frontend wouldn't just be included in yt by default, as opposed to being a separate package like many other frontends? Is it a large package?

[neutrinoceros]

Initially the frontend was moving fast because it was built onto experimental yt functionality (stretched grids), so I needed to be able to iterate and release it much more quickly than the core package. Having a separate package meant that I didn't need to go through the review process for every little change and could also build on stable versions instead of having to compile from source (which I don't like pushing on my users). It stayed separate because I didn't feel the need to unload the maintenance burden on core package maintainers (which in my humble opinion shouldn't be the default for frontends to do). Anyway, that's my reasoning.

[neutrinoceros]

A side goal of this PR is to raise awareness that doing this is an option we support, so I guess this works already ! :-)

[chummels]

OK, this all makes sense, and I can understand why you did it this way. When we were writing Trident, it definitely changed the development process by requiring more oversight in changes made on the yt side. It's good to have this as a more modular approach moving forward for packages interfacing with yt. Thanks for the info!

[chrishavlin]

By the way, don't you have an extension frontend somewhere ? (@neutrinoceros)

hah, yes. yt_aspect. and it's at a point where i'd include it in a list in the docs somewhere, but wouldn't add it as an install option yet. and I agree we can move the discussion of adding a list in the docs off this PR.

@chummels one additional perk to building a frontend as an external extension that @neutrinoceros didn't mention is when a frontend needs extra dependencies -- you can just add dependencies as you need them without worrying about yt's optional dependency framework. The main downside I've encountered while developing and maintaining an external frontend is not having yt's answer testing framework available.

And in terms of this PR, it seems so far no one has any objections to adding idefix as an install option here?

[chrishavlin]

I'd say this has been around long enough with no real objections, so +1 from me for getting it in!