FlowiseAI
[BUG] Error after upgrading to Flowise v2.0.5 with Milvus failing
Description
I recently upgraded Flowise to version 2.0.5. After the upgrade, Milvus started to fail using the same credentials that worked in the previous Flowise version 1.8.4. I am now getting the following error message:
Error: Error searching data: {"status":{"extra_info":{},"error_code":"UnexpectedError","reason":"fail to search on QueryNode 63: queryHook execution failed: service unavailable: search params in wrong format
character 'o' looking for beginning of value","code":2,"retriable"
,"detail":"failed to search: failed to search/query delegator 63 for channel in01-0fb909e951340bb-rootcoord-dml_7_447376391097125769v0: fail to search on QueryNode 63: queryHook execution failed: service unavailable: search params in wrong format
character 'o' looking for beginning of value"},"results":[]}
Sample flow that I am testing:
I'm having the same problem. Curious if this also affects other people who use Milvus and Flowise 2.x.x?
I got the same issue with flowise 2.0.7. Here is the debug info from console.
''' 10:chain:RetrievalChainWithNoHistory > 12:retriever:VectorStoreRetriever] [1.41s] Retriever run errored with error: "Error searching data: {"status":{"extra_info":{},"error_code":"UnexpectedError","reason":"fail to search on QueryNode 2: worker(2) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: [o","code":65535,"retriable":false,"detail":"failed to search: failed to search/query delegator 2 for channel by-dev-rootcoord-dml_5_452377787821047827v0: fail to search on QueryNode 2: worker(2) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: [o"},"results":[]}
Error: Error searching data: {"status":{"extra_info":{},"error_code":"UnexpectedError","reason":"fail to search on QueryNode 2: worker(2) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: [o","code":65535,"retriable":false,"detail":"failed to search: failed to search/query delegator 2 for channel by-dev-rootcoord-dml_5_452377787821047827v0: fail to search on QueryNode 2: worker(2) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: [o"},"results":[]} at similaritySearchVectorWithScore (/usr/local/lib/node_modules/flowise/node_modules/flowise-components/dist/nodes/vectorstores/Milvus/Milvus.js:247:15) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async vectorStore.similaritySearchVectorWithScore (/usr/local/lib/node_modules/flowise/node_modules/flowise-components/dist/nodes/vectorstores/Milvus/Milvus.js:186:20) at async Milvus.similaritySearch (/usr/local/lib/node_modules/flowise/node_modules/@langchain/community/node_modules/@langchain/core/dist/vectorstores.cjs:108:25) at async VectorStoreRetriever.getRelevantDocuments (/usr/local/lib/node_modules/flowise/node_modules/@langchain/community/node_modules/@langchain/core/dist/retrievers/index.cjs:72:29) at async VectorStoreRetriever._streamIterator (/usr/local/lib/node_modules/flowise/node_modules/@langchain/community/node_modules/@langchain/core/dist/runnables/base.cjs:165:9) at async VectorStoreRetriever.transform (/usr/local/lib/node_modules/flowise/node_modules/@langchain/community/node_modules/@langchain/core/dist/runnables/base.cjs:402:9) at async RunnableSequence._streamIterator (/usr/local/lib/node_modules/flowise/node_modules/@langchain/core/dist/runnables/base.cjs:1208:30)" '''
+1
- flowise:
flowiseai/flowise:2.1.1 - milvus:
milvusdb/milvus:v2.4.11-gpu
It looks like the search_params which passing to Milvus is not correct json format.
[search_params="[{\"key\":\"anns_field\",\"value\":\"langchain_vector\"},{\"key\":\"topk\",\"value\":\"4\"},{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"params\",\"value\":\"[object Object]\"}]"]
It contains a \"value\":\"[object Object]\" in it, which caused the error**
parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: '[o'"]
Milvus said in console:
[2024/09/26 08:14:48.156 +00:00] [WARN] [proxy/impl.go:3015] ["Search failed to WaitToFinish"] [traceID=151bc25f777de00a3e9538a26ad8167d] [role=proxy] [db=default] [collection=fo] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=8204] [OutputFields="[loc,langchain_text]"] [search_params="[{\"key\":\"anns_field\",\"value\":\"langchain_vector\"},{\"key\":\"topk\",\"value\":\"4\"},{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"params\",\"value\":\"[object Object]\"}]"] [ConsistencyLevel=Bounded] [useDefaultConsistency=false] [nq=1] [error="failed to search: failed to search/query delegator 7 for channel by-dev-rootcoord-dml_0_452808391727775224v0: fail to search on QueryNode 7: worker(7) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: '[o'"] [errorVerbose="failed to search: failed to search/query delegator 7 for channel by-dev-rootcoord-dml_0_452808391727775224v0: fail to search on QueryNode 7: worker(7) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: '[o'\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).Execute\n | \t/workspace/source/internal/proxy/task_search.go:552\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n | \t/workspace/source/internal/proxy/task_scheduler.go:474\n | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n | \t/workspace/source/internal/proxy/task_scheduler.go:553\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/root/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\nWraps: (2) failed to search\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:186\n | [...repeated from below...]\nWraps: (4) failed to search/query delegator 7 for channel by-dev-rootcoord-dml_0_452808391727775224v0\nWraps: (5) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/proxy.(*searchTask).searchShard\n | \t/workspace/source/internal/proxy/task_search.go:739\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry.func1\n | \t/workspace/source/internal/proxy/lb_policy.go:178\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).ExecuteWithRetry\n | \t/workspace/source/internal/proxy/lb_policy.go:147\n | github.com/milvus-io/milvus/internal/proxy.(*LBPolicyImpl).Execute.func2\n | \t/workspace/source/internal/proxy/lb_policy.go:215\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (6) fail to search on QueryNode 7\nWraps: (7) worker(7) query failed: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: '[o'\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) merr.milvusError"]
Flowise 2.1.1 crashes completely for me, when trying to upsert a vector store:
2024-10-07 16:52:16 flowise-1 | Error: 4 DEADLINE_EXCEEDED: Deadline exceeded after 15.000s,name resolution: 0.004s,metadata filters: 0.001s,Waiting for LB pick
2024-10-07 16:52:16 flowise-1 | at callErrorFromStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
2024-10-07 16:52:16 flowise-1 | at Object.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/client.js:193:76)
2024-10-07 16:52:16 flowise-1 | at /usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/call-interface.js:78:35
2024-10-07 16:52:16 flowise-1 | at Object.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Grpc.js:146:25)
2024-10-07 16:52:16 flowise-1 | at InterceptingListenerImpl.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/call-interface.js:73:23)
2024-10-07 16:52:16 flowise-1 | at Object.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
2024-10-07 16:52:16 flowise-1 | at Object.onReceiveStatus (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
2024-10-07 16:52:16 flowise-1 | at /usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
2024-10-07 16:52:16 flowise-1 | at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
2024-10-07 16:52:16 flowise-1 | for call at
2024-10-07 16:52:16 flowise-1 | at ServiceClientImpl.makeUnaryRequest (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/client.js:161:32)
2024-10-07 16:52:16 flowise-1 | at ServiceClientImpl.<anonymous> (/usr/local/lib/node_modules/flowise/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
2024-10-07 16:52:16 flowise-1 | at /usr/local/lib/node_modules/flowise/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:32:31
2024-10-07 16:52:16 flowise-1 | at new Promise (<anonymous>)
2024-10-07 16:52:16 flowise-1 | at /usr/local/lib/node_modules/flowise/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:29:16
2024-10-07 16:52:16 flowise-1 | at Generator.next (<anonymous>)
2024-10-07 16:52:16 flowise-1 | at fulfilled (/usr/local/lib/node_modules/flowise/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:5:58)
I have submitted a PR that fix the symptoms maybe not the cause of the problem but it works #3336
I am still getting a similar issue with Milvus on Flowise 2.1.5
