[BUG] You're NOT Crazy the Published API Doesn't Support Inference models and Returns false success

[Peter-Lankton]

🚨 Please Note 🚨

To ensure efficient investigation of the issue, please fill out the fields below with as much detail as possible. Reports that do not follow this template may be closed without notification. We appreciate your cooperation.

🐞Describe the bug

The published API code uses 'ConverseStream' operation which is not supported by Anthropic Models

🔄 To Reproduce

Steps to reproduce the behavior:

1. Create a bot
2. Publish it
3. POST to the '/conversation' endpoint with any anthropic model, my screenshots use claude-v3.5-sonnet-v2
4. The response will lie and say 'success' here's your ConversationId and MessageID
5. Go to SQS queue and see that there's no messages waiting and there's none in flight
6. Go to DynamoDB and see that nothing was created

📷 Screenshots

1. ERROR with Anthropic Model

1.a POST Message with Model

1.b Error Logs on SQS Handler Lambda

2. Success with non anthropic model

2.a POST Message with Amazon Nova model

2.b SUCCESS Logs on SQS Handler Lambda

🔎 Logs for Chat Issues

n.a.

🔎 Logs for Bot Creation/Update Issues

n.a.

📝 Additional context

This was a tough bug to hunt down with the response saying it was successful on endpoint, but failing during the execution of the SQS Lambda which is a container image, so can't look at code in a straightforward manner.

[Peter-Lankton]

Not sure on how to update the SQSLambda code but if you point me in the correct direction I don't mind giving a try :-).

[statefb]

Thanks for reporting this issue. Your detailed debugging helped identify this problem.

The initial 200 response is expected (API is asynchronous), but you're right that the backend is failing silently if the model is not activated.

possible solution:

• Update schema to return proper error messages when error happens

Do you think update the implementation with keeping backward compatibility in terms of schema?

[Peter-Lankton]

I think it is fair to update the schema to let users know the lambda is failing. For the SQS Handler code is it possible to know which file(s) in the project are related to making the container image that it runs as so that I can try to debug it? From online the solution is here: AWS Invoke Error e.g. in the code it should reference us.anthropic.claude-3-5-haiku-20241022-v1:0 (I tried passing that to the API Endpoint and didn't have any luck) instead of claude-3-5-haiku.

[statefb]

#862 merged. Please check if the problem resolved. Thank you

[github-actions[bot]]

This issue has been labeled as "stale" due to no response by the reporter within 1 month (and 14 days after last commented by someone). And it will be closed automatically 14 days later if not responded.

[github-actions[bot]]

This issue has been closed due to no response within 14 days after labeled as "stale", 14 days after last reopened, and 14 days after last commented.