sqlite-vec icon indicating copy to clipboard operation
sqlite-vec copied to clipboard

[question] Do I need to recreate the vec table after changing distance_metric

Open arkohut opened this issue 1 year ago • 4 comments

The default distance_metric is l2_distance which is not the one using for embedding. I need to change it to distance_metric=cosine.

How to migrate this change and minimize the impact? Can I save the existing vector data?

arkohut avatar Dec 11 '24 07:12 arkohut

I'm in the same boat. I think we/you could just create a new table with the other distance metric, do a insert into new_table select * from old_table, and it's good? I'm going to try this.

markuswustenberg avatar Dec 11 '24 13:12 markuswustenberg

I think that may work, as long as the old table isn't dropped and the new one renamed. I'm getting an SQL logic error in that case. Maybe renaming on virtual tables isn't supported?

markuswustenberg avatar Dec 11 '24 13:12 markuswustenberg

https://github.com/asg017/sqlite-vec/issues/125

And there is also other issues I encountered: https://github.com/asg017/sqlite-vec/issues/154

arkohut avatar Dec 23 '24 03:12 arkohut

Relatedly, how do I change the distance metric to cosine using python? Do I need to recompile the .so somehow?

tkswanson avatar Jan 11 '25 00:01 tkswanson