Indexing knn vector fields is rejected as circuit breaker triggered

[zordark]

Good day, we have an error during a search by k-NN vector and can't identify a problem. We have an index with a nested object which contains k-NN vector. After some time search by k-NN has stopped working. We observe an error in logs during search:

MapperParsingException[failed to parse field [images.vector] of type [knn_vector] in document with id '541969203748146'. Preview of field's value: '[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]']; nested: IllegalStateException[Indexing knn vector fields is rejected as circuit breaker triggered. Check PATH for detailed state]; at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:263) at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) at org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:542) at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:417) at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:502) at org.elasticsearch.index.mapper.DocumentParser.parseNonDynamicArray(DocumentParser.java:592) at org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:544) at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:417) at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:227) at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:849) at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:826) at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:798) at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:285) at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:175) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:220) at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:126) at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:85) at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:234) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:752) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at PATH(ThreadPoolExecutor.java:1128) at PATH(ThreadPoolExecutor.java:628) at PATH(Thread.java:829)Caused by: java.lang.IllegalStateException: Indexing knn vector fields is rejected as circuit breaker triggered. Check PATH for detailed state__AMAZON_INTERNAL__ at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:244) ... 30 more

we are using opendistro:

{
  "name" : "5ecfb90b51677dd6fdf7a06e9c7f4f17",
  "cluster_name" : "539592047266:xxxxxxxxxxxxxxx",
  "cluster_uuid" : "uXzPtoFsRrCuQHlxFT3qPQ",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2021-09-20T07:12:51.966483Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[jmazanec15]

Hi @zordark , we block writes when the circuit breaker is triggered. In order to resume indexing, you will need to clear out the cache.

[zordark]

Hi @zordark , we block writes when the circuit breaker is triggered. In order to resume indexing, you will need to clear out the cache.

@jmazanec15 thx for the quick response. Did you mean index cache which can be cleared with:

POST /my-index-000001/_cache/clear

or k-NN has his own?