langflow-ai
Add model ids corresponding to inference profiles supported by AWS Bedrock
AWS has something called "Inference Profiles" for performing cross-region inference for calling Bedrock APIs for specific models from specific regions. This is typically used by using particular model ids, which is added with this commit.
Read more here:
- https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html
- https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html
I noticed this when setting up Langflow and trying to use the models in Bedrock. It is a simple constants change for now. But maybe is it possible to allow a user to type-in a model id, if in the dropdown the option user tries to choose does not exist? Because API providers would keep adding new models every other week and we would need to keep playing catch up :)
CodSpeed Performance Report
Merging #5696 will degrade performances by 15.76%
Comparing taghash:main (4f728e6) with main (d84ea5c)
Summary
❌ 1 regressions
✅ 18 untouched benchmarks
:warning: Please fix the performance issues or acknowledge them on CodSpeed.
Benchmarks breakdown
| Benchmark | BASE |
HEAD |
Change | |
|---|---|---|---|---|
| ❌ | test_cancel_nonexistent_build |
8.3 ms | 9.8 ms | -15.76% |
![codspeed-hq[bot] avatar](https://avatars.githubusercontent.com/in/257293?v=4 "Published by a pub.dev verified publisher"){kind=small}
@ogabrielluiz If you see in the linked docs, according to region + model, the model id should be changed. There doesn't seem to be a consistent logic there. It's just a table with values that need to be mapped. Hence I tried to keep it simple, since I'm blocked on this :sweat_smile:
Can we merge this?
@ogabrielluiz could you specify exactly what changes you are looking for?
@ogabrielluiz bumping this up in case it got lost in the inbox :sweat_smile:
Happy to make changes in the MR, but I'm not able to understand what you are looking for. I feel like current approach is cleaner.
If you could have a feature that dynamically loads the model list, it would be great. Also, wouldn’t the region input be sufficient for this change instead of adding the region to the model IDs?
@shrikrishnaholla cc. @ogabrielluiz
@edwinjosechittilappilly @ogabrielluiz Request you to go through the relevant AWS documentation (which is also included in comment in code): https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html
There is no straightforward logic that can tell you which model is supported in which region. Not every model is supported in every region. Not every model needs inference profiles. Hence I included them in the model ids list. It would be better if we even turned it from a dropdown to a text input because AWS keeps updating this list often and from the rate of activity I see I don't feel confident we will be able to keep up. If we keep this a dropdown and the model I have access to, I'm not able to use because it can't be selected, then user will give up trying to use.
Just my 2 cents. If you want me to make any changes, I'm happy to do so, but please help me with a more specific advice on the changes needed.
Also, if we just pass region + model id (without the inference profile), which I tried, then AWS Bedrock returns an error. That is what led me to make this change.
@edwinjosechittilappilly @ogabrielluiz do you advise that I close this PR? I don't really see it going anywhere. I neither get any guidance, nor approvals. I understand you may be busy, maybe it's not a priority area in the project. I don't know how much longer I'd need to track this PR.
@shrikrishnaholla this change would be welcomed as currently it prevents working with the latest Anthropic models via Bedrock (their most popular according to AWS). The issue is that the models in the dropdown list do not contain the inference IDs and changing the region does not fix the problem. It was mentioned already and I fully agree that pasting in the model ID or ARN is the ultimate fix, especially as new models come out almost weekly. This would allow full flexibility. I'm happy to contribute to PRs and such.
@ogabrielluiz @edwinjosechittilappilly could you please advise on the changes needed? Any pointers on a right direction would be helpful. I had mentioned the challenges earlier. Would love to hear any alternative ideas you might have to handling cross-region inference profiles better :pray:
any update about this ? im also meet this issue when testing agent with bedrock provider
I've resorted to forking AWS / Gemini components and replacing dropdowns with text boxes so that I can input any model id I want, and experience has gotten so much better!
The field of LLMs is progressing too fast and models get outdated by newer models too soon. The older paradigms of restricting to "tested" options with dropdowns are not going to be scale well, unless those are kept up to date live with changes in the providers' offerings. That can be time and human bandwidth consuming, so best to just turn model selections to be free flow input box. We could keep a dropdown to make onboarding easier, but also allow the dropdowns to accept user inputs as well (like freesolo)
just my two cents @ogabrielluiz @edwinjosechittilappilly