julep icon indicating copy to clipboard operation
julep copied to clipboard

Add a xxhash based cache

Open creatorrr opened this issue 1 year ago • 0 comments
trafficstars

Use xxhash for hashing, create a cozo table for saving these. lookup before every call to openai

xxhash(json.dumps({ chatml, parameters })) => cache lookup => openai

the hash is the key, save the response as the value

creatorrr avatar May 07 '24 12:05 creatorrr