dspy
dspy copied to clipboard
Add Weights & Biases logging to BootstrapFewShot
wandb_enabled
to BootstrapFewShot
Here is a minimal first step to adding Weights & Biases optimization logging to DSPy.
This can be used for monitoring runs to BootstrapFewShot
in order to see the metric_val
returned for each bootstrapped example. To motivate the use case, you may have a rating on a scale of 1 to 5 for answers and you only want to use examples that achieve a 5 in your prompt.
There are definitely more directions to take with DSPy <> Weights & Biases logging, hopefully this is a nice first step. I've found this to be pretty interesting particularly when you are testing multiple teacher models for bootstrapping.
Docs
Note, this contains a quick note in 6-optimizers.md
. I think it would be a premature refactoring, but I think it's likely we see a lot of the "callback" style logging of PyTorch / Keras carry over.
Hi @CShorten , thanks for the PR! left a few comments that will also address the failing test here by fixing the import with a try-except block. There is also a merge conflict to fix. Should be good to merge after that!