Add metadata capture

[skrawcz]

You want to +1 or -1 a response for later evaluation. We need to expose a way to do that.

idea

# annotated current returned state 
app.annotate({"value": 1, }) # implicit app_id, sequence_id

# annotate some thing afterwards
Tracker().annotate(app_id, sequence_id, {"value": 1})
Tracker().annotate(app_id, sequence_id, {"value": 1, "target": SOME_STATE_KEY})

You'd have to provide more spcificis on what the metadata is yourself I think. e.g. what it targets in the state.

This should work for arbitrary metadata annotation.

We'd then want some export functionality.

[elijahbenizzy]

Some thoughts out loud:

More can go through the app -- having it on the tracker is weird...

app.annotate({"value": 1})
app.annotate({"value": 1, }, sequence_id=10)
app.annotate({"value": 1, }, sequence_id=-10) # 10 back from current

Does the target make sense? Or does that just tie to the value?

[elijahbenizzy]

Types of metadata we want:

1. Artifacts -- these are associated with actions, and optionally traces. These are immutable, and only logged during execution. Examples:
   • An embedding of something
   • The cost of an OpenAI call
   • A metric associated with an epoch for evaluating loss
   • A set of metrics that we might want to query for
   • A dataset (training data, whatnot)
2. Annotations -- these are added after the fact to allow for post-hoc analysis. E.G.
   • A user marking a response as good/bad
   • A user saying they like a model/like the results
   • Marking a dataset as useful/interesting in some way