outlines icon indicating copy to clipboard operation
outlines copied to clipboard

Published 20 hours ago • verified publisher icon dottxt-ai

Add Anthropic integration

Open rlouf opened this issue 2 years ago • 7 comments

rlouf avatar Nov 16 '23 15:11 rlouf

Correct me if I'm wrong - for what I know, the anthropic/claude API does not have an API for outputting token probabilities yet

jakaline-dev avatar Nov 19 '23 12:11 jakaline-dev

That's correct, but we try to add other providers so people can compare their output to guided generation in outlines. I often need to do this.

rlouf avatar Nov 19 '23 17:11 rlouf

Anthropic is now available via AWS Bedrock: https://aws.amazon.com/bedrock/

I think it would be great to add support for Bedrock alltogether!

sztanko avatar Dec 15 '23 20:12 sztanko

why was this idea of support for Bedrock dropped?

shabie avatar Feb 18 '24 22:02 shabie

The team has very limited resources

rlouf avatar Feb 19 '24 09:02 rlouf

What if I volunteered to do this?

shabie avatar Feb 19 '24 10:02 shabie

Correct me if I'm wrong - for what I know, the anthropic/claude API does not have an API for outputting token probabilities yet

Yes, with Bedrock, what is returned from the API request varies from one model to the next. For instance, the Anthropic models response does not contain token probabilities in the response. On the other hand, the Cohere Command models response will contain an array of per token likelihoods if you specify the return_likelihoods input parameter. I am not familiar enough with Outlines to know if that is enough.

jeremyallenjacobson avatar May 01 '24 21:05 jeremyallenjacobson