Qiskit
Allow PassManagers and transpile to support Pubs as inputs and outputs
What should we add?
With primitive V2 inputs changing from raw circuits to parametric SamplerPubs and EstimatorPubs, and also requiring ISA circuits for their run methods it becomes more difficult for a user to prepare optimal inputs, especially in the case of the Estimator, where they need to do tedious manual remapping arrays of observables.
The natural thing to do would be to allow transpile and PassManagers to take Pubs as inputs and outputs, so that they apply transpilation to the circuit in the pub, apply the layout mapping to observables in the case of an EstimatorPub, and return an ISA Pub for running on a primitive.
In the future this would also be useful for implementing more performant transpilation passes that can work with the parameteric circuits + BindingsArray's in a pubs and do parameter re-writing to remap the input parameters to new parameters if the parametric structure of a circuit is optimized.
Just for visibility: this is a big feature request and we're talking about prioritisation / team availability and the design-space way forwards internally first. There's a lot of overlap with request #11643.
For what it's worth, my initial feeling is that this is a better way to go than #11643, but expanding the transpiler IR is not a decision to be taken lightly. That said, we can fulfil the base-case request of "automatically update the observables in transpile" without affecting the IR - that's something that can be done purely in the input/output conversion in PassManager, but there's still design discussion needed about how to do that in a safely extensible way, given that there's a high possiblity that primitives implementers are going to want to change their ObservablesArray in some form in the near future, whereas transpile will be consuming the base interface definition, which isn't really set up for extensibility.
Changing up the IR in a more fundamental manner (such as to enable the sorts of optimisation passes talked about here and in #11643) is possibly more like a Qiskit 2.0 thing, since we're already planning for major DAGCircuit changes to come in then (hopefully).
Perhaps having a separate transpile function, and a specific PassManager subclass for this could be a method to do so without changing the base PassManager/transpiler IR or waiting for Qiskit 2.0, though eventually I think you probably do want to have something in place for those sorts of optimization passes in 2.0.
In regards to #11643 think there are a lot of subtle issues with directly adding expectation values as an instruction to a circuit. For one an EstimatorPub we also natively support ND-Arrays of observables for each input circuit, so now you would either need to make a new circuit for each index of the array, or have some ND-Array observable instruction.
Also if these are allowed in circuits, then your are moving the semantics of an estimator program into the circuit itself, which could be sent to run on a Sampler or backend which maybe involve assuming how they are decomposed into rotations and measure instructions which is not independent of post-processing in the absence of direct backend support for these as a measurement return type. One other subtle question is what would it mean to have two expectation value instructions in a circuit, are they marginal or correlated on a single shot basis. If we don't want to deal with all these situations, then the safest thing seems to be to work with the EstimatorPub format directly which provides a well defined program to be executed on an Estimator primitive.
Perhaps having a separate transpile function [and pass manager, etc]
This is potentially possible yeah, and one way we could go about things. I think the decision here will be more around what kind of timescales we expect these optimisation passes to become necessary and when we expect them to be written; if we're not going to have them in place before Qiskit 2.0 or they can wait - there are workarounds - it's possibly not worth the user pain of fragmenting the transpile design, which is already tricky for users to understand.
Also if these are allowed in circuits, then your are moving the semantics of an estimator program into the circuit itself
I 100% agree with this (and the rest of the paragraph) - I don't think expectation values should be on the circuit, and imo it breaks the circuit-model abstraction - before the changes to the IBM Runtime, our path forwards was going to be Ali sketching out a design draft so we could talk more about requirements and alternate approaches, rather than necessarily going that route verbatim. I think what you're proposing here is much a cleaner path to the capabilities that are needed, and are much more in line with the abstraction levels.