Gaffer icon indicating copy to clipboard operation
Gaffer copied to clipboard

Published 20 hours ago • verified publisher icon gchq

ProxyStore support compression

Open GCHQDev404 opened this issue 2 years ago • 2 comments

Describe the new feature you'd like I want ProxyStore to handle large volumes of elements better.

Why do you want this feature? If proxy store was able to support compression over HTTP then Federation of data would be sped up.

Describe alternatives you've considered Proxy Store to support streaming.

See: lower priority #2806

GCHQDev404 avatar Nov 08 '22 16:11 GCHQDev404

@GCHQDev404 "If proxy store was able to support compression over HTTP then Federation of data would be sped up."

It is yet to be established how performant FederatedStore will be. Hoping to test this after release of alpha 4. Do you have any concerns about likely performance? If so, perhaps this issue should be addressed sooner?

n3101 avatar Nov 15 '22 09:11 n3101

One of our customers had some IO bottle necking. The usage of the old FederatedOperationChain being replaced with FederatedOperation improved their bottle neck. But this should still be improved.

GCHQDev404 avatar Jan 11 '23 13:01 GCHQDev404