Run programs with multiple beams

[jvhoffbauer]

trafficstars

Describe the solution you'd like

I want to run programs with multiple beams and choose one or many beams with the highest score according to my LM. .

[slundberg]

Beam search is not currently supported, but I don't think there is any reason why it couldn't be. Happy to review any PRs. (or course this would only be for open models, since if an endpoint does not support beam search there is no way to add it)

[jvhoffbauer]

I'm excited about this! I think it makes sense to start in the transformer model class. Can you potentially give me one or two starting points to get up to speed quickly?

[slundberg]

Sure. Right now we call the generate command. https://github.com/microsoft/guidance/blob/059ca7d64139dcc0f423ae0c0336b42eef3bcab5/guidance/llms/_transformers.py#L32 With a set of args that defines some custom logit processors and stopping conditions. https://github.com/microsoft/guidance/blob/059ca7d64139dcc0f423ae0c0336b42eef3bcab5/guidance/llms/_transformers.py#L363 or https://github.com/microsoft/guidance/blob/059ca7d64139dcc0f423ae0c0336b42eef3bcab5/guidance/llms/_transformers.py#LL369C13-L369C13

We should be able to just pass the appropriate beam search options to the transformers generate call. However there might be some tricky interactions with our monkey-patch to support guidance acceleration: https://github.com/microsoft/guidance/blob/059ca7d64139dcc0f423ae0c0336b42eef3bcab5/guidance/llms/_transformers.py#L165-L213

I would just start though by trying to add the params and pass them right to transformers and then see if it works :)

[SimJeg]

@marcotcr @slundberg up on this issue, especially for substrin :) Following this issue I think it would make sense to use beam search for substrings, but it would imply some more refactoring as currently substring relies on select so you would need to return logprobs and re-create the tree (not sure you can use huggingface generate here)