Wire oso_agent discord bot to run evals

[ravenac95]

What is it?

There are probably multiple ways for us to do this but this is my current thought for how to make this work. The discord bot doesn't currently execute anything against the agent code, it simply responds to !run_eval so this issue is to actually map the agent so we can easily trigger agent evals.

Implementation thoughts:

• We take advantage of the UX in phoenix to be able to "run experiments" against data in uploaded datasets in phoenix.
  • Experiments are run using arize phoenix's phoenix.experiments.run_experiment. See: https://docs.arize.com/phoenix/datasets-and-experiments/how-to-experiments/run-experiments
• When triggering the bot, we should just use click to do the string parsing for the command. Eventually we could make it just use an llm to chat with but that seems too heavy weight for this very specific thing now?
  • Whatever the command is you can invoke with an arbitrary list[str] by calling Command#main. See: https://click.palletsprojects.com/en/stable/api/#commands
    • For example, in our oso_agent.cli package there's a cli function that is a Command object. We would call this like cli.main([...list of strings to use for command parsing...])

[linear[bot]]

OSO-753 Wire oso_agent discord bot to run evals

[ravenac95]

Oh just noticed this: https://discordpy.readthedocs.io/en/stable/ext/commands/index.html

Definitely Might be preferable to what i suggested above.

[ryscheng]

Fixed the plumbing here https://github.com/opensource-observer/oso/pull/3902

Uses the Discord bot commands