Unclear how to evaluate a marker for a set of extras

[pfmoore]

This is possibly similar to #448, but I think it's a different use case. I'm not completely clear on what "operations" #448 is meant to cover, so apologies if I missed that this is already part of that discussion.

I have two requirements:

a_req = Requirement("a[b,c]")
a_dep = Requirement("foo; extra == 'b'")

I want to check whether a_dep should apply, given that I am looking at it in the context of a_req. So, I have a set of extras, and I want to evaluate a_dep.marker to see if any apply. In the given case, I expect to get True.

The simplest way I can see is any(a_dep.marker.evaluate(environment={"extra": e}) for e in a_req.extras).

Is that the best way of doing this? Would it be useful to have an additional argument to Marker.evaluate() that specified a set of extras to check - then I could do a_dep.marker.evaluate(extras=a_req.extras) which seems more natural.

[pradyunsg]

Well, that's definitely the best way to do it, as of today. Here's the relevant part in pip:

https://github.com/pypa/pip/blob/1df8934a97ac77f064e2b858b554d3fb5ce43fae/src/pip/_internal/req/req_install.py#L253

[McSinyx]

Does PEP 508 forbid e.g. foo; extra == 'a' and extra == 'b'? I suppose no one is actually using this due to the way Marker.evaluate is implemented, but I couldn't find where it is explicitly forbidden.

[uranusjr]

It does not, but it is also very unclear what the expression means. PEP 508 is very vague about extra in markers, and both the name extra and the operator == are technically wrong (I think I said this somewhere on Discourse a while ago).

[pfmoore]

I agree, one of the reasons I found it hard to work out how to do this is that extras are conceptually a set, but packaging expects you to supply just one at a time, and that links back to PEP 508 using == which is only even remotely meaningful if an extra is a single value.

I doubt there's much hope of redesigning the semantics without a very extended debate, though, so we're probably stuck with what we have. I don't know if packaging could offer an API that smoothed over the weirdness at all (the idea of an extras argument to evaluate that I suggested above, for example). That's really the only reason for leaving this issue open, in case someone wants to do something with the API. Anything more fundamental would need a PEP and a wider discussion.

[uranusjr]

I’ve been planning a PEP in my head to “fix” extras once and for all. There’s no draft, but I’m thinking about doing three things:

• Definitely define what is a valid extra name and how it’s normalised in the resolver (by standardising whatever pip is doing right now)
• Introduce a new marker variable extras that’s a set of normalised extra names. The only valid operators are in and not in (which is actually implied since a marker_var can only be either a string literal or variable name, but it’s better to be explicit).
• Remove extra and bump metadata version to 2.0.

BTW since this is going to bump the major version, I’m also thinking about doing two more things not strictly relavant here (but also related to extras):

• A new metadata field Requires-Extra that specifies “default extras” to install when a package is requested with extra specification
• Introduce a new syntax package[] to explicitly request installing a package without any extras.

Nothing concrete at this time though. Also anyone please feel free to steal any of the above idea if you have time to write a PEP 😄

[brettcannon]

https://peps.python.org/pep-0685/ takes care of the normalization problem (once it's accepted).

[uranusjr]

I proposed the rest of the comment above to discuss at the Packaging Summit next week.

[pradyunsg]

A new metadata field Requires-Extra that specifies “default extras” to install when a package is requested with extra specification

nit: Call it Default-Extra (multiple use) please. :)