dspy icon indicating copy to clipboard operation
dspy copied to clipboard

Add Weights & Biases logging to BootstrapFewShot

Open CShorten opened this issue 10 months ago • 1 comments

wandb_enabled to BootstrapFewShot

Here is a minimal first step to adding Weights & Biases optimization logging to DSPy.

This can be used for monitoring runs to BootstrapFewShot in order to see the metric_val returned for each bootstrapped example. To motivate the use case, you may have a rating on a scale of 1 to 5 for answers and you only want to use examples that achieve a 5 in your prompt.

There are definitely more directions to take with DSPy <> Weights & Biases logging, hopefully this is a nice first step. I've found this to be pretty interesting particularly when you are testing multiple teacher models for bootstrapping.

Docs

Note, this contains a quick note in 6-optimizers.md. I think it would be a premature refactoring, but I think it's likely we see a lot of the "callback" style logging of PyTorch / Keras carry over.

CShorten avatar Apr 17 '24 03:04 CShorten

Hi @CShorten , thanks for the PR! left a few comments that will also address the failing test here by fixing the import with a try-except block. There is also a merge conflict to fix. Should be good to merge after that!

arnavsinghvi11 avatar May 06 '24 01:05 arnavsinghvi11