dspy
dspy copied to clipboard
optimize_signature for typed predictors needs more documentation
The documentation doesn't specify what to do with the result of optimize_signature
The code specifies that it should return a dspy.Program:
https://github.com/stanfordnlp/dspy/blob/303c66991899a7fc5123ee5642f4ff4d7f76560f/dspy/teleprompt/signature_opt_typed.py#L149C1-L149C19
However this is not the case and it returns instead a list of (scores, signatures) It is unclear what one should do with the signatures. One possibility is to manually install the signature as follows:
result = optimize_signature(student_program=predictor, ....)
from copy import deepcopy
predictor2 = deepcopy(predictor)
best = result.scores.index(max(result.scores))
predictor2.signature = result.signatures[best]["self"]
This is not good UX (and possibly brittle, eg the key "self" used to be "base" in previous versions) but seems to work locally. It has one major caveat though: the signature won't survive saving the model as it is not returned by predictor2.dump_state()
, which makes me feel it's not the intended way.
Hi @alcinos , thanks for raising this! feel free to push a PR updating the typing inconsistencies and adding documentation. The OptimizerResult
is akin to the return values in dspy.Evaluate
, but this can be modified to be included with a flag.
Tagging @thomasahle for more context on TypedOptimizers and use cases for the returned signatures.