Gaffer icon indicating copy to clipboard operation
Gaffer copied to clipboard

ProxyStore should support Lazy Iterable streaming

Open GCHQDev404 opened this issue 2 years ago • 0 comments

Describe the new feature you'd like I want ProxyStore to handle large volumes of elements better.

Why do you want this feature? Currently ProxyStore when receiving Lazy Iterable over HTTP it will try to resolve the entire Lazy Iterable received from remote store, before returning it.

Describe alternatives you've considered Proxy Store to support compression. #2807 #2863

Additional context Most problematic with GetAllElements followed by Limit 1 when not correctly wrapped in an operation chain. Possible solution is to get ProxyStore to use JsonSerialisation streaming.

GCHQDev404 avatar Nov 08 '22 16:11 GCHQDev404