qdrant-client icon indicating copy to clipboard operation
qdrant-client copied to clipboard

How to pytest for collection config with QdrantLocal?

Open meichenseer opened this issue 2 years ago • 1 comments

Hi, we want to test in CI/CD, that a collection does have expected configuration values.

from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance,

client = QdrantClient(':memory:')  # same with client = QdrantClient(path="qdrant_data")
client.create_collection(
  collection_name="my_collection", 
  on_disk_payload=False, 
  vectors_config=VectorParams(size=100, distance=Distance.COSINE)
)
collection = client.get_collection("my_collection")

assert collection.config.params.on_disk_payload == False  # Fails
assert collection.config.params.on_disk_payload == None   # Succeeds

I want the first assertion to succeed. Is this somehow possible? for QdrantClient(path="qdrant_data") there is a meta.json written to the qdrant_data folder. Couldn't there end up also the passed configuration?

Our use case: We create a collection that is loaded into memory. When we create a new collection, we want to keep the old collection and label it as backup collection. The backup collection should be moved to disk. Now we want to test with pytest that the config changes and e. g. on_disk_payload is set to True after it is labeled as backup collection.

meichenseer avatar Sep 18 '23 13:09 meichenseer

cp from discord Local mode ignores all additional parameters like hnsw config, optimizers config, etc. however it saves those parameters which were provided in vectors_config e.g. input:

cl.create_collection('col', vectors_config=models.VectorParams(size=10, distance=models.Distance.COSINE, hnsw_config=models.HnswConfigDiff(m=32, ef_construct=160)))

output:

CollectionInfo(status=<CollectionStatus.GREEN: 'green'>, optimizer_status=<OptimizersStatusOneOf.OK: 'ok'>, vectors_count=0, indexed_vectors_count=0, points_count=0, segments_count=1, config=CollectionConfig(params=CollectionParams(vectors=VectorParams(size=10, distance=<Distance.COSINE: 'Cosine'>, hnsw_config=HnswConfigDiff(m=32, ef_construct=160, full_scan_threshold=None, max_indexing_threads=None, on_disk=None, payload_m=None), quantization_config=None, on_disk=None), shard_number=None, replication_factor=None, write_consistency_factor=None, on_disk_payload=None), hnsw_config=HnswConfig(m=16, ef_construct=100, full_scan_threshold=10000, max_indexing_threads=0, on_disk=None, payload_m=None), optimizer_config=OptimizersConfig(deleted_threshold=0.2, vacuum_min_vector_number=1000, default_segment_number=0, max_segment_size=None, memmap_threshold=None, indexing_threshold=20000, flush_interval_sec=5, max_optimization_threads=1), wal_config=WalConfig(wal_capacity_mb=32, wal_segments_ahead=0), quantization_config=None), payload_schema={})

joein avatar Sep 18 '23 13:09 joein