pmaw icon indicating copy to clipboard operation
pmaw copied to clipboard

parquet for caching responses

Open mattpodolak opened this issue 3 years ago • 1 comments

Use parquet instead of pickle for caching responses.

Need to assess if this is a reasonable improvement:

  • Benchmark parquet cache speed for storing and retrieval against pickle
  • Compare resulting cache sizes

mattpodolak avatar Feb 17 '21 21:02 mattpodolak

This could be expanded by increasing the generality of the caching mechanism to identify the type of file used in the cache and load accordingly.

Saving responses to cache should be performed with a default type (pickle/parquet/feather/jay) - whichever is fastest, with optional config to select a caching function.

mattpodolak avatar May 27 '21 10:05 mattpodolak