Cache.clear() requires index [DATACOUCH-606]

[spring-projects-issues]

achoice opened DATACOUCH-606 and commented

 

Cache.clear() requires an primary index. Not obvious for new usres.

See https://github.com/spring-projects/spring-data-couchbase/blob/master/src/main/java/org/springframework/data/couchbase/cache/DefaultCouchbaseCacheWriter.java#L106

Exception on a new bucket below, as seen on server Couchbase 6.6.

Problem solved after manually creating index.

Perhaps initialization logic should check if an index exist?

com.couchbase.client.core.error.PlanningFailureException: The server failed planning the querycom.couchbase.client.core.error.PlanningFailureException: The server failed planning the query {"completed":true,"coreId":"0x4f359f4900000001","errors":[{"code":4000,"message":"No index available on keyspace default that matches your query. Use CREATE INDEX or CREATE PRIMARY INDEX to create an index, or check that your expected index is online."}],"idempotent":false,"lastDispatchedFrom":"127.0.0.1:24849","lastDispatchedTo":"localhost:8093","requestId":12,"requestType":"QueryRequest","retried":3,"retryReasons":["ENDPOINT_TEMPORARILY_NOT_AVAILABLE"],"service":{"operationId":"03399eff-1e81-4c19-b2d4-860534813a4d","statement":"DELETE FROM `default` where meta().id LIKE $pattern","type":"query"},"timeoutMs":75000,"timings":{"dispatchMicros":23165,"totalMicros":176313}} at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51) at com.couchbase.client.java.Cluster.query(Cluster.java:393) at org.springframework.data.couchbase.cache.DefaultCouchbaseCacheWriter.clear(DefaultCouchbaseCacheWriter.java:106) at org.springframework.data.couchbase.cache.CouchbaseCache.clear(CouchbaseCache.java:146)

 UPDATE:

Sample code I ended up using (or similar)

if(couchbaseClientFactory
        .getCluster()
        .queryIndexes()
        .getAllIndexes(couchbaseClientFactory.getBucket().name())
        .stream().filter(queryIndex -> queryIndex.primary())
        .findFirst()
        .isEmpty())
{
    LOG.info("Creating primary index, needed for .clear() and .invalidate()");
    couchbaseClientFactory
            .getCluster()
            .queryIndexes()
            .createPrimaryIndex(
                    couchbaseClientFactory.getBucket().name(), CreatePrimaryQueryIndexOptions.createPrimaryQueryIndexOptions().ignoreIfExists(true));
}

 

UPDATE 2:

Also noted that this confusing/unexpected behavior can happen. I believe it is due to main index was not update immediately and  hence the clear() query did not find the put item. If my theory is correct I think DOC should mention this.

@Test // this test FAILS (local empty (i.e. fast) Couchbase installation)
public void clearFail() {
 final Cache cache = cacheManager.getCache(CacheConfig.CACHE_NAME);
 cache.put("KEY","VALUE");
 cache.clear();
 assertFalse(cache.get("KEY") != null); // fails, clear failed to delete
}

@Test // this WORKS
public void clearWithDelayOk() throws InterruptedException {
  final Cache cache = cacheManager.getCache(CacheConfig.CACHE_NAME);
  cache.put("KEY","VALUE");
  Thread.sleep(50); // give main index time to update
  cache.clear();
  assertFalse(cache.get("KEY") != null);
}

 

---

Affects: 4.0.3 (Neumann SR3)

[maciej-ilnicki-allegro]

+1

[mikereiche]

I agree that the documentation could use some work.

We have steered away from doing any type of work during initialization that could have been done ahead of time because the applications might need to start up very quickly. Even if there was a perfect way to implement this, I'm not sure we'd do it just for that reason.

Maintaining a primary key incurs considerable overhead. During cache initialization, it is not possible to know if clear() or invalidate() will be used on the cache. If a primary key was created during initialization and never used it would incur unnecessary overhead maintaining the primary key and also creating it. We could introduce an option to indicate that the cache will use clear/invalidate - and therefore initialization should check/create a primary key. The catch is that the current behavior is to not check/create, so that would have to remain as-is, and the check/create would only be done if the option is set - in which case the consumer would need to already know about additional requirements for clear/invalidate.

When clear()/invalidate() is called with no primary, the message from the server is helpful.

"No index available on keyspace default that matches your query. Use CREATE INDEX or CREATE PRIMARY INDEX to create an index, or check that your expected index is online."

Also noted that this confusing/unexpected behavior can happen. I believe it is due to main index was not update immediately and hence the clear() query did not find the put item.

Yes. This is a common situation in tests which insert a document and immediately query on the inserted document. The delete query could be executed with REQUEST_PLUS (based on an option), but again there is a performance penalty. There is also the possibility of documents not yet being deleted by clear() when the subsequent cache.get("KEY") executes - but I think only in a multi-node cluster.

[mikereiche]

fixed.

Note that to use cache.clear() or cache.invalidate(), the bucket must have a primary key.