Transaction.supportsFiltering check disappeared from InMemoryDataStoreTransaction.loadObject

[krlc]

Transaction.supportsFiltering check disappeared from InMemoryDataStoreTransaction.loadObject after upgrade, and now when filterExpression is not null, InMemoryDataStoreTransaction never calls an overridden DataStoreTransaction.loadObject and instead relies on the original DataStoreTransaction.loadObject which performs an in-memory filtering on all available objects in a datastore, even when a per-object filtering capability is available. Is this an expected behavior?

Expected Behavior

A way to make InMemoryDataStoreTransaction call a custom datastore loadObject even when EntityProjection.filterExpression is not null.

Current Behavior

InMemoryDataStoreTransaction ignores a custom datastore loadObject because of filterExpression and instead calls DataStoreTransaction.loadObject which performs filtering on all available objects in-memory.

Possible Solution

Return the tx.supportsFiltering == FeatureSupport.FULL check back (commit). Or is there another work-around?

Steps to Reproduce (for bugs)

1. Create a model, a NoopDataStore and NoopDataStoreTransaction with loadObject, loadObjects
2. Define a filterExpression for the model (in our case it's a security check)
3. Do a request
4. Start debugging and find out that only the loadObjects is called, but loadObject is ignored
5. Freak out about the JVM memory usage (not tested, but assumed)

Context

We have a datastore derived from NoopDataStore with both overridden loadObject and loadObjects. Our loadObject can efficiently get a single object using Serializable id (and handle filterExpression). During runtime, however, our overridden DataStoreTransaction seems be wrapped in InMemoryDataStoreTransaction which never calls our loadObject because EntityProjection.filterExpression is not null. We would like for InMemoryDataStoreTransaction to call our loadObject instead of relying on DataStoreTransaction.loadObject which calls our loadObjects and then performs in-memory filtering of all available objects in our datastore. Is something like this possible?

Your Environment

• Elide version used: 6.1.1
• Environment name and version (Java 1.8.0_152): Java 14.0.2

[aklish]

Transaction.supportsFiltering logic moved into DataStoreIterable:

https://github.com/yahoo/elide/blob/master/elide-core/src/main/java/com/yahoo/elide/core/datastore/DataStoreIterable.java#L45-L47

Transaction.loadObjects can return a custom implementation of this interface which can tell Elide whether or not the wrapped Iterable should be filtered in memory or be left alone. This was changed because the original interface lacked sufficient context for all data stores to tell Elide whether or not filtering, pagination, or sorting should be done in-memory on the server.

The second issue you raise is whether InMemoryDataStoreTransaction could always call loadObject in the wrappedTransaction rather than delegate to DataStoreTransaction.loadObject. I generally agree something like this would make sense, but it might change the behavior and break other implementations. We might need to do an Elide 7 release to make that kind of change.