aws-amplify
OpenSearch: @searchable - wildcard, regexp matching returns 0 results
Before opening, please confirm:
- [X] I have searched for duplicate or closed issues and discussions.
- [X] I have read the guide for submitting bug reports.
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
JavaScript Framework
React, React Native
Amplify APIs
GraphQL API
Amplify Categories
api
Environment information
# Put output below this line
System:
OS: macOS 12.1
CPU: (8) x64 Apple M1 Pro
Memory: 145.91 MB / 16.00 GB
Shell: 5.8 - /bin/zsh
Binaries:
Node: 12.22.10 - ~/.nvm/versions/node/v12.22.10/bin/node
Yarn: 1.22.17 - ~/.nvm/versions/node/v12.22.10/bin/yarn
npm: 6.14.16 - ~/.nvm/versions/node/v12.22.10/bin/npm
Watchman: 2022.03.21.00 - /opt/homebrew/bin/watchman
Browsers:
Brave Browser: 103.1.40.107
Chrome: 103.0.5060.114
Safari: 15.2
npmPackages:
@alfonmga/react-lottie-light-ts: ^0.0.1 => 0.0.1
@aws-amplify/pushnotification: ^4.3.8 => 4.3.8
@babel/core: ^7.8.6 => 7.14.8 (7.17.5, 7.9.0, 7.16.0)
@cometchat-pro/chat: ^3.0.4 => 3.0.6
@cometchat-pro/react-native-chat: 3.0.0 => 3.0.0
@expo-google-fonts/inter: ^0.2.0 => 0.2.0
@expo/match-media: ^0.3.0 => 0.3.0
@expo/webpack-config: ^0.16.6 => 0.16.16
@react-native-async-storage/async-storage: ^1.12.1 => 1.15.9
@react-native-clipboard/clipboard: ^1.9.0 => 1.9.0
@react-native-community/cameraroll: ^4.1.2 => 4.1.2
@react-native-community/checkbox: ^0.5.1 => 0.5.8
@react-native-community/masked-view: 0.1.10 => 0.1.10
@react-native-community/netinfo: 6.0.2 => 6.0.2
@react-native-community/push-notification-ios: ^1.8.0 => 1.8.0 (1.0.3)
@react-native-community/slider: 3.0.3 => 3.0.3
@react-native-picker/picker: ^2.1.0 => 2.2.1
@react-navigation/bottom-tabs: ^6.0.5 => 6.0.5
@react-navigation/drawer: ^6.3.1 => 6.3.1
@react-navigation/native: ^6.0.2 => 6.0.2
@react-navigation/stack: ^6.0.7 => 6.0.7
@sentry/react-native: ^3.2.4 => 3.2.5
@storybook/addon-actions: ^6.0.21 => 6.3.6
@storybook/addon-knobs: ^6.0.21 => 6.3.0
@storybook/addon-ondevice-actions: ^5.3.21 => 5.3.23
@storybook/addon-ondevice-knobs: ^5.3.21 => 5.3.25
@storybook/react-native: ^5.3.21 => 5.3.25
@stripe/react-stripe-js: ^1.7.0 => 1.7.0
@stripe/stripe-js: ^1.23.0 => 1.23.0
@svgr/webpack: ^5.4.0 => 5.5.0
@types/howler: ^2.2.4 => 2.2.4
@types/lodash: ^4.14.161 => 4.14.172
@types/react: ~16.9.35 => 16.9.56 (17.0.15)
@types/react-native: ~0.63.2 => 0.63.53 (0.65.1)
@types/uuid: ^8.3.0 => 8.3.1
HelloWorld: 0.0.1
amazon-cognito-identity-js: ^4.4.0 => 4.6.3 (5.2.3)
aws-amplify: ^4.3.11 => 4.3.11
aws-amplify-react-native: ^6.0.2 => 6.0.2
aws-sdk: ^2.1056.0 => 2.1057.0
babel-jest: ~25.2.6 => 25.2.6 (25.5.1)
babel-loader: ^8.2.3 => 8.2.3 (8.1.0)
babel-plugin-react-native-web: ^0.17.5 => 0.17.5 (0.13.18)
buffer: ^6.0.3 => 6.0.3 (4.9.2, 5.7.1)
dayjs: ^1.9.4 => 1.10.6
dotenv-webpack: ^7.1.0 => 7.1.0
emoji-mart-native: ^0.6.2-beta => 0.6.2-beta
expo: ^39.0.0 => 39.0.5
expo-application: ~2.3.0 => 2.3.0
expo-av: ^9.2.3 => 9.2.3
expo-document-picker: ^9.2.3 => 9.2.4
expo-font: ^9.2.1 => 9.2.1 (8.3.0)
expo-image-picker: ^10.2.2 => 10.2.2
expo-linking: 1.0.7 => 1.0.7
expo-permissions: ^12.1.1 => 12.1.1 (9.3.0)
expo-splash-screen: ~0.6.2 => 0.6.2
expo-status-bar: ~1.0.2 => 1.0.4
expo-updates: ~0.3.2 => 0.3.5
faker: 5.5.3 => 5.5.3
file-loader: ^6.2.0 => 6.2.0 (6.0.0)
hermes-inspector-msggen: 1.0.0
howler: ^2.2.3 => 2.2.3
install: ^0.13.0 => 0.13.0
jest: ~25.2.6 => 25.2.7
lodash: ^4.17.20 => 4.17.21
lottie-react-native: ~2.6.1 => 2.6.1
numeral: ^2.0.6 => 2.0.6
patch-package: ^6.4.7 => 6.4.7
plaid: ^10.1.0 => 10.1.0
postinstall-postinstall: ^2.1.0 => 2.1.0
prettier: 2.5.1 => 2.5.1
query-string: ^7.0.1 => 7.0.1
react: 16.13.1 => 16.13.1 (16.14.0)
react-animated: 0.1.0
react-dom: 16.13.1 => 16.13.1
react-image-file-resizer: ^0.4.7 => 0.4.7
react-is: ^16.13.1 => 16.13.1
react-lottie: ^1.2.3 => 1.2.3
react-native: ~0.63.4 => 0.63.4
react-native-android-keyboard-adjust: ^1.2.0 => 1.2.0
react-native-autolink: ^4.0.0 => 4.0.0
react-native-blob-courier: 3.0.2 => 3.0.2
react-native-config: ^1.4.5 => 1.4.5
react-native-device-info: ^8.3.0 => 8.3.1
react-native-elements: ^3.4.2 => 3.4.2
react-native-fs: ^2.18.0 => 2.18.0
react-native-gesture-handler: ^1.10.3 => 1.10.3
react-native-get-random-values: ^1.5.0 => 1.7.0
react-native-hyperlink: ^0.0.19 => 0.0.19
react-native-image-resizer: ^1.4.5 => 1.4.5
react-native-keep-awake: ^4.0.0 => 4.0.0
react-native-keyboard-aware-scroll-view: https://github.com/APSL/react-native-keyboard-aware-scroll-view#fb40649a9c84aa23d6a55355e9e0e432795967a6 => 0.9.2
react-native-mov-to-mp4: ^0.2.2 => 0.2.2
react-native-plaid-link-sdk: ^7.2.1 => 7.2.1
react-native-portalize: ^1.0.7 => 1.0.7
react-native-reanimated: 1.13.2 => 1.13.2
react-native-safe-area-context: ^3.3.2 => 3.3.2 (3.1.4)
react-native-screens: ^3.7.2 => 3.7.2
react-native-sound: ^0.11.1 => 0.11.1
react-native-svg: 12.1.0 => 12.1.0
react-native-svg-transformer: ^0.14.3 => 0.14.3
react-native-swipe-list-view: ^3.2.9 => 3.2.9
react-native-unimodules: ^0.14.6 => 0.14.6
react-native-vector-icons: ^8.1.0 => 8.1.0
react-native-web: ~0.17.5 => 0.17.5
react-native-webview: ^11.13.0 => 11.13.0
react-plaid-link: 3.2.2 => 3.2.2
react-responsive: ^9.0.0-beta.6 => 9.0.0-beta.6
react-test-renderer: ~16.13.1 => 16.13.1
read-excel-file: ^5.3.5 => 5.3.5
read-excel-file/commonjs: undefined ()
read-excel-file/node: 1.0.0
read-excel-file/schema: 1.0.0
read-excel-file/web-worker: 1.0.0
reanimated-bottom-sheet: ^1.0.0-alpha.22 => 1.0.0-alpha.22
rn-fetch-blob: ^0.12.0 => 0.12.0
stripe: ^8.135.0 => 8.209.0
tipsi-stripe: ^8.0.2 => 8.0.2
typescript: ^4.3.2 => 4.5.5
url-loader: ^4.1.1 => 4.1.1
uuid: ^8.3.2 => 8.3.2 (3.4.0, 3.3.2, 7.0.2, 7.0.3)
webpack: ^4.43.0 => 4.46.0 (4.43.0)
npmGlobalPackages:
@aws-amplify/cli: 5.1.0
expo-cli: 5.3.0
npm: 6.14.16
yarn: 1.22.17
Describe the bug
Running this code gives me zero results (Index contains data).
I am running app on iOS & Android simulator. I get the same response even when I am running react-native in web environment.
I am using "ElasticSearchInstanceType": "t2.small.elasticsearch".
I would expect it returned something back.
- EDIT The same happens for regexp with payload "Lo" or "\Lo\" or "*Lo*".
- EDIT2: (Searchable) Data were imported after creating table with @searchable annotation using NodeJS, Dynamo Document Client and its PutOperation.
const variables: SearchSchoolsQueryVariables = {
filter: {
schoolName: {
wildcard: "*Lo*"
},
},
};
const response = (await API.graphql(
graphqlOperation(searchSchools, variables)
)) as GraphQLResult<SearchSchoolsQuery>;
schema.graphql:
type School
@model(subscriptions: null)
@auth(
rules: [
{ allow: private, operations: [read] }
{ allow: public, operations: [read] }
]
)
@key(
name: "schoolsByName"
fields: ["schoolName"]
queryField: "allSchoolsByName"
)
@searchable
{
id: ID!
schoolName: String!
createdAt: AWSDateTime!
updatedAt: AWSDateTime!
}
Expected behavior
It would return non-empty response.
Reproduction steps
Run the search & add dummy data.
Code Snippet
// Put your code below this line.
Log output
// Put your logs below this line
aws-exports.js
No response
Manual configuration
No response
Additional configuration
No response
Mobile Device
No response
Mobile Operating System
Web, iOS, Android
Mobile Browser
No response
Mobile Browser Version
No response
Additional information and screenshots
No response
Hi @majirosstefan 👋 thank you for raising this issue. Are you experiencing the same result when attempting the query in the AppSync console?
If it helps, we are using OpenSearch / ElasticSearch engine with version 6.2:
@majirosstefan thank you for the info, we will be trying to reproduce this issue and get some feedback for you from the Amplify data team
@majirosstefan would you happen to know if you might've created the records you're querying for before or after you added @searchable to the model? If you added them before, you might have to backfill the data using a script:
https://docs.amplify.aws/cli/graphql/troubleshooting/#backfill-opensearch-index-from-dynamodb-table
@chrisbonifacio Although we were using searchable for another model before, this time we only needed to add/create another new model that is also searchable.
So I created the model that you could see above, and then imported data to DynamoDB // manually using nodeJS script that uses putOperation and Dynamo's Document Client. Not 100% sure, but Ddb uses streaming or something like that to propagate changes to Opensearch, no ?
Also, if the data simply was not present in OpenSearch so backfilling would be needed, how could it be possible that search with other conditions is working but it's only ignoring regexp and wildcard ?
So I created the model that you could see above, and then imported data to DynamoDB // manually using nodeJS script that uses putOperation and Dynamo's Document Client. Not 100% sure, but Ddb uses streaming or something like that to propagate changes to Opensearch, no ?
Yes, any records created after opensearch has been added to the model will be streamed from DDB to OpenSearch and should be available via AppSync query.
Also, if the data simply was not present in OpenSearch so backfilling would be needed, how could it be possible that search with other conditions is working but it's only ignoring regexp and wildcard ?
Can you share the filtered queries that are working and which are not? If the records don't exist in OpenSearch but are still exist in the DDB table, they could still be queried by standard filters.
I noticed you are on version 5.1.0 of the Amplify CLI. Can you upgrade to the latest version, redeploy your schema and see if that makes a difference? Our attempts to reproduce the issue have been using version 9.1.0, we're just forcing the use of graphql transformer V1 by changing some settings in the cli.json file so we can deploy the schema you provided.
Hi @majirosstefan I attempted to reproduce this issue however, I am able to get the expected response:
Seeing the use of @key we assume you are using the v1 transformer.
CLI.json
"features": {
"graphqltransformer": {
...
"useexperimentalpipelinedtransformer": false,
...
"transformerversion": 1,
...
},
...
}
schema.graphl (copied from above)
type School
@model(subscriptions: null)
@auth(
rules: [
{ allow: private, operations: [read] }
{ allow: public, operations: [read] }
]
)
@key(
name: "schoolsByName"
fields: ["schoolName"]
queryField: "allSchoolsByName"
)
@searchable
{
id: ID!
schoolName: String!
createdAt: AWSDateTime!
updatedAt: AWSDateTime!
}
AppSync response
- Yes, we are using transformer v1.
- I tried to find CLI.json in the project, but seems there is none (I have added all version of all dependencies when creating this ticket )
- Only matchPhrasePrefix, and match are working:
@majirosstefan Thanks for the confirmation.
Out of curiosity, have you tried running the queries in the OpenSearch generated Kibana URL to see if there's a response? Hoping to isolate the root of your issue.
If you haven't done this before here are some tips:
- The Kibana URL is located under the OpenSearch domain in the General information section
- Create index pattern for your type (should be
schoolbased on schema), then in the Discover tab run your queries. Ex.school:*Lo* - In case you face anonymous user errors: https://aws.amazon.com/premiumsupport/knowledge-center/anonymous-not-authorized-opensearch/
will do, but in next few days, not today
Btw, should I take a look also on logs somewhere there ? Or just see if it returns anything (or an error maybe)
@majirosstefan yes. You may need to enable logging on the AppSync API if you haven't, then check the associated CloudWatch logs after running a new Query.
Also it can be helpful to check/share the AppSync Resolver mapping template for the searchSchools query as well.
Btw, seems like "match" filter was removed from the docs:
https://docs.amplify.aws/cli-legacy/graphql-transformer/searchable/#usage
EDIT (no, it's still there // long hours in the work, sorry)
@majirosstefan just want to follow up with you and see if you're still running into this issue. Also, have you tried upgrading your Amplify CLI version to the latest and see if that helps after redeploying your schema?
Also, when you can, can you please confirm that you are not able to query even newly created records that should fulfill those regex and wildcard filter expressions?
yeah, still running into this. Did not have time to upgrade CLI or look into OpenSearch yet.
"Also, when you can, can you please confirm that you are not able to query even newly created records that should fulfill those regex and wildcard filter expressions?" == yes
@majirosstefan No worries, let us know when you get a chance to test if upgrading makes a difference!
same issue here I just created my search with amplify looks like it provisions Elasticsearch 6.2 by default and only match and matchPhrasePrefix are working even though I didnt see them in the docs wildcard and regex dont seem to work
I will transfer this issue to Amplify CLI repo, they will handle this better than me
Hi @majirosstefan, Have you had an opportunity to upgrade to the latest version and confirm?
@internetammo, Are you still encountering the issue? If so, Would you be able to provide us with screenshots of the request and response, please?
Hi @majirosstefan, I am checking to see if you are still experiencing this issue? If not, please let us know we will close this bug.
Hello Anil, I am sorry, but currently I have no bandwidth to upgrade the app, or to provide more information.
I do not know if this can be closed - based on my experience bugs in the software will not be resolved on their own (but if it's your policy, feel free to close this issue, even if it was not resolved or triaged properly).
I got the same issue while upgrading some OpenSearch clusters in different environments. When the version is "OpenSearch 2.5 (latest)", wildcard does NOT work. When the version is "Elasticsearch 7.10", wildcard does work.
Any suggestion on how to revert clusters upgraded to the nonworking version is appreciated.
Adding more details
I'm using
- graphql transformer v2
- Amplify CLI 10.4.0
- Amplify for Swift 2.8.0
Hi @dral3x, Kindly follow the instructions provided here to revert back to the previous version of Elasticsearch.
Just small update from me:
I created new project (independent of the one that has issues above) with amplify-cli (11.0.3), and "aws-amplify": "^5.3.8" and it seems to work. Elasticsearch version is 7.10 ( In the AWS Console).