aws-amplify
OpenSearch 2.3 R2022114-P1 (Latest) errors in Importing Lambda
How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
v18.9.0
Amplify CLI Version
10.6.2
What operating system are you using?
Mac
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
Yes - upgraded the version of OpenSearch to the latests in the console.
Version 2.3 Service software version: R20221114-P1
Describe the bug
New results were not coming back from my search models, this is only happening in the lower dev environment with the latest version of OpenSearch installed.
I checked the logs of the lambda - amplify-XXXX-de-OpenSearchStreamingLambd-XXXXXXXXX - and I can see there are some errors in the log that weren't there before:
[ERROR] 2023-01-31T20:20:11.253Z 377fa735-553e-457d-91bb-12516ee75de9 Traceback (most recent call last): File "/var/task/python_streaming_function.py", line 260, in lambda_handler return _lambda_handler(event, context) File "/var/task/python_streaming_function.py", line 254, in _lambda_handler post_to_opensearch(opensearch_payload) # Post to OpenSearch with exponential backoff File "/var/task/python_streaming_function.py", line 101, in post_to_opensearch opensearch_ret_str = post_data_to_opensearch( File "/var/task/python_streaming_function.py", line 76, in post_data_to_opensearch raise Searchable_Exception(res.status_code, res._content) python_streaming_function.Searchable_Exception: Searchable_Exception: status_code=400, payload=b' { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Action/metadata line [1] contains an unknown parameter [_type]" } ], "type": "illegal_argument_exception", "reason": "Action/metadata line [1] contains an unknown parameter [_type]" }, "status": 400 } '
Expected behavior
New items in the DDB table should be indexed by the opensearch service.
Reproduction steps
Deploy a searchable model, upgrade to the latest version of open search from the AWS console.
Project Identifier
No response
Log output
# Put your logs below this line
Additional information
No response
Before submitting, please confirm:
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
- [X] I have removed any sensitive information from my code snippets and submission.
According to https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-troubleshoot-cloudwatch-logs/
In OpenSearch Service versions 2.0 and later, the _type parameter is removed from API endpoints. To resolve this error, you must also remove the parameter from your Lambda function's code.
Which makes sense in that I am getting the error:
{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Action/metadata line [1] contains an unknown parameter [_type]" } ], "type": "illegal_argument_exception", "reason": "Action/metadata line [1] contains an unknown parameter [_type]" }, "status": 400 } '
Hey @jeff-accountably, Thank you for bringing this to our attention. currently work around is to rollback to previous OpenSearch version. we are working on supporting the OpenSearch 2.0 version on amplify. once it is done, amplify will automatically update the versions on client apps, no action required on your side.
Thanks, is there any documentation on rolling back to previous opensearch versions? I can't seem to find any, I'd rather not have to blow away my dev stage just to roll back.
I am getting this error in the lambda that insert/update/delete from dynamo db to opensearch:
1df028edb045 Traceback (most recent call last): File "/var/task/python_streaming_function.py", line 260, in lambda_handler return _lambda_handler(event, context) File "/var/task/python_streaming_function.py", line 254, in _lambda_handler post_to_opensearch(opensearch_payload) # Post to OpenSearch with exponential backoff File "/var/task/python_streaming_function.py", line 101, in post_to_opensearch opensearch_ret_str = post_data_to_opensearch( File "/var/task/python_streaming_function.py", line 76, in post_data_to_opensearch raise Searchable_Exception(res.status_code, res._content)python_streaming_function.Searchable_Exception: Searchable_Exception: status_code=400, payload=b'{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Action/metadata line [1] contains an unknown parameter [_type]" } ], "type": "illegal_argument_exception", "reason": "Action/metadata line [1] contains an unknown parameter [_type]" }, "status": 400 }'
In the aws documentation i found it: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html.
Can't downgrade after upgradeIn-place upgrades are irreversible, but if you contact AWS Support, they can help you restore the automatic, pre-upgrade snapshot on a new domain. For example, if you upgrade a domain from Elasticsearch 5.6 to 6.4, AWS Support can help you restore the pre-upgrade snapshot on a new Elasticsearch 5.6 domain. If you took a manual snapshot of the original domain, you can perform that step yourself.
That information says that the aws support can help you restore the automatic, pre-upgrade snapshot on a new domain. But how do I tell amplify to take that new domain that was restored?
Thanks Frank
I was having the same issue @jeff-accountably. The issue was solved using the solution provided by AWS Support. Please look into the last solution here: https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-troubleshoot-cloudwatch-logs
If you have a python script created by Amplify CLI, please let me know I can guide you on how to solve it.
@AnilMaktala do you have a ticket to watch/follow for adding OpenSearch 2.0 support to Amplify?
I was able to fix this by changing the Lambda function code and removing the _type parameter in 3 places. Does anyone know if the next amplify push might overwrite my band-aid change?
Hi @jeff-accountably and @reikje, We have discovered a workaround that allows us to revert to the previous version of OpenSearch until Amplify supports the latest version. Please see the steps below for further details.
Schema:
type Student @model @searchable{
name: String
dateOfBirth: AWSDate
email: AWSEmail
examsCompleted: Int
}
Remove the 'https://github.com/searchable' directive from the schema as shown below. type Student @model { name: String dateOfBirth: AWSDate email: AWSEmail examsCompleted: Int } Run "amplify push". This will delete the current search domain from the AWS console.
Add the 'https://github.com/searchable' directive back to the schema as shown below.
type Student @model @searchable{ name: String dateOfBirth: AWSDate email: AWSEmail examsCompleted: Int } Run 'Amplify push' again. This will create a new OpenSearch domain in the AWS console that is supported by Amplify.
Finally, please follow the steps outlined here to backfill the indexes from the DynamoDB table.
We recommend trying these steps in a lower environment before attempting them in a production environment. If you have any questions or concerns, please do not hesitate to let us know.
According to https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-troubleshoot-cloudwatch-logs/
In OpenSearch Service versions 2.0 and later, the _type parameter is removed from API endpoints. To resolve this error, you must also remove the parameter from your Lambda function's code.
Which makes sense in that I am getting the error:
{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Action/metadata line [1] contains an unknown parameter [_type]" } ], "type": "illegal_argument_exception", "reason": "Action/metadata line [1] contains an unknown parameter [_type]" }, "status": 400 } '
Removing _type from the code worked for me. Thanks
We are working on supporting the OpenSearch 2.0 version on amplify. once it is done, amplify will automatically update the versions on client apps, no action required on your side.
Hi @AnilMaktala Where can we track progress/release of the 2.x support?
We want to expose the Dashboards, but we don't want to release Kibana, and then swap it over to OSD for our users.
Thanks.
Hi @jeff-accountably and @reikje, We have discovered a workaround that allows us to revert to the previous version of OpenSearch until Amplify supports the latest version. Please see the steps below for further details.
Schema:
type Student @model @searchable{ name: String dateOfBirth: AWSDate email: AWSEmail examsCompleted: Int }Remove the 'https://github.com/searchable' directive from the schema as shown below. type Student @model { name: String dateOfBirth: AWSDate email: AWSEmail examsCompleted: Int } Run "amplify push". This will delete the current search domain from the AWS console.
Add the 'https://github.com/searchable' directive back to the schema as shown below.
type Student @model @searchable{ name: String dateOfBirth: AWSDate email: AWSEmail examsCompleted: Int } Run 'Amplify push' again. This will create a new OpenSearch domain in the AWS console that is supported by Amplify.
Finally, please follow the steps outlined here to backfill the indexes from the DynamoDB table.
We recommend trying these steps in a lower environment before attempting them in a production environment. If you have any questions or concerns, please do not hesitate to let us know.
"Discovered a workaround" - wtf of cause we all know that deleting search, add it again and filling everything would work. What about people that have data for multiple indexes / types and don't want to spend hours backfilling everything again?
Its 2025 now (More than 2 year later), any update on supporting OpenSearch 2.0, if it seems that you only need to delete "_type"? Or at least tell us what needs to be migrated, then we could do it ourself.
