Gaffer
Gaffer copied to clipboard
ProxyStore should support Lazy Iterable streaming
Describe the new feature you'd like I want ProxyStore to handle large volumes of elements better.
Why do you want this feature? Currently ProxyStore when receiving Lazy Iterable over HTTP it will try to resolve the entire Lazy Iterable received from remote store, before returning it.
Describe alternatives you've considered Proxy Store to support compression. #2807 #2863
Additional context Most problematic with GetAllElements followed by Limit 1 when not correctly wrapped in an operation chain. Possible solution is to get ProxyStore to use JsonSerialisation streaming.