Add a duplicate prevention mechanism

[dgarnitz]

Right now most vector DBs support uploading duplicate vectors as long as the ID is unique. It might be beneficial for vectorflow to 1) prevent the uploading of duplicate vectors 2) deduplicate an index/collection/class object store

But is it possible to enforce these things in a performant way

[dgarnitz]

The default deduplication field should be the source filename. We do not need to go down to the chunk level. Whoever builds this should take into account how to check this for each vector DB. If it's not possible to search qdrant/weaviate/pinecone by metadata field or ID, then we may need to use a SQL table in postgres. I don't want to add Redis just for this though