v6d icon indicating copy to clipboard operation
v6d copied to clipboard

The `import/export_kv_cache_buffer` show bad performance

Open dashanji opened this issue 1 year ago • 2 comments

Describe your problem

During the integration with llama.cpp, we find the import_kv_cache_buffer and export_kv_cache_buffer show bad performance as follows.

image

The Import time includes query vineyard + import_kv_cache_buffer. The Export time includes update vineyard + export_kv_cache_buffer.

dashanji avatar Mar 12 '24 13:03 dashanji

/cc @sighingnow, this issus/pr has had no activity for a long time, please help to review the status and assign people to work on it.

github-actions[bot] avatar Apr 12 '24 00:04 github-actions[bot]