pmaw
pmaw copied to clipboard
parquet for caching responses
Use parquet
instead of pickle
for caching responses.
Need to assess if this is a reasonable improvement:
- Benchmark
parquet
cache speed for storing and retrieval againstpickle
- Compare resulting cache sizes
This could be expanded by increasing the generality of the caching mechanism to identify the type of file used in the cache and load accordingly.
Saving responses to cache should be performed with a default type (pickle/parquet/feather/jay) - whichever is fastest, with optional config to select a caching function.