Speed up `get_pending_observation_features` by caching (on the experiment) which trials have received observations.

[mgrange1998]

Summary: "Speed up get_pending_observation_features by caching (on the experiment) which trials have received observations."

"We would cache points that are non-pending, as forever non-pending, since we're not going to "lose" them from the data so they don't need to be checked again once they are there"

Change

This diff adds simple caching for pending metric names in the add_arm_to_pending_features method.

• For trials with status is_deployed, non-pending metrics (i.e metrics with values in the output of trial.lookup_data) are cached in experiment.not_pending_cache
• When add_arm_to_pending_features is next called, if a trial has status is_deployed, and all of the experiment's metrics are present in the not_pending_cache, evaluation of that trial is skipped
• Additionally, for trials which are not is_deployed, "lookup_data" is skipped entirely, as its outputs are not used.

Differential Revision: D54218989