k-NN
k-NN copied to clipboard
Adds profiler for knn query
Description
Adds exact and ann timings and counts with knn query profile API
You can find sample response here: concurrent_nofilter.json concurrent.json profile_filter.json faiss_profile.json
CIs will fail since it require a change in Opensearch-core
Pending:
- Opensearch PR https://github.com/opensearch-project/OpenSearch/pull/17146
- unit tests
Related Issues
Resolves https://github.com/opensearch-project/k-NN/issues/2286
Partial response
"profile": {
"shards": [
{
"id": "[6lfotEMuRUqgq0kM6BfLSQ][target_index_faiss][0]",
"inbound_network_time_in_millis": 0,
"outbound_network_time_in_millis": 0,
"searches": [
{
"query": [
{
"type": "NativeEngineKnnVectorQuery",
"description": "NativeEngineKnnVectorQuery[]...KNNQuery[]",
"time_in_nanos": 27173585,
"breakdown": {
"set_min_competitive_score_count": 0,
"exact_knn_search": 0,
"match_count": 0,
"shallow_advance_count": 0,
"next_doc": 3542,
"score_count": 3,
"compute_max_score_count": 0,
"advance": 0,
"advance_count": 0,
"score": 2459,
"shallow_advance": 0,
"create_weight_count": 1,
"build_scorer": 614792,
"ann_search": 0,
"set_min_competitive_score": 0,
"match": 0,
"next_doc_count": 4,
"compute_max_score": 0,
"build_scorer_count": 2,
"create_weight": 26552792,
"ann_search_count": 0,
"exact_knn_search_count": 0
},
"children": [
{
"type": "KNNQuery",
"description": "",
"time_in_nanos": 2821209,
"breakdown": {
"set_min_competitive_score_count": 0,
"exact_knn_search": 0,
"match_count": 0,
"shallow_advance_count": 0,
"next_doc": 0,
"score_count": 0,
"compute_max_score_count": 0,
"advance": 0,
"advance_count": 0,
"score": 0,
"shallow_advance": 0,
"create_weight_count": 0,
"build_scorer": 0,
"ann_search": 2821209,
"set_min_competitive_score": 0,
"match": 0,
"next_doc_count": 0,
"compute_max_score": 0,
"build_scorer_count": 0,
"create_weight": 0,
"ann_search_count": 1,
"exact_knn_search_count": 0
}
}
]
}
],
"rewrite_time": 29292,
"collector": [
{
"name": "SimpleTopScoreDocCollector",
"reason": "search_top_hits",
"time_in_nanos": 381916
}
]
}
],
"aggregations": []
}
Check List
- [x] New functionality includes testing.
- [x] Commits are signed per the DCO using
--signoff. - [ ] Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.
@shatejas Can you post the sample response as text in comment if it is small so that we don't have to download?
@VijayanB there are multiple of them and aren't small so uploaded them. I can add partial if it helps