Qdrant import / collection not working

[tkreuder]

Details

curl -L -X POST 'http://localhost:6333/collections/my_imported_collection/points/search' -H 'Content-Type: application/json' \                                                                                                                                                                                      
                                                                                        --data-raw '{
                                                                                   "vector": [0.69,0.69,0.59,0.74,0.18,0.44,0.91,0.76,0.35,0.8,0.31,0.61,0.62,0.49,0.85,0.41,0.26,0.43,0.09,0.34,0.41,0.07,0.16,0.75,0.24,0.87,0.89,0.29,0.43,0.78,0.55,0.78,0.97,0.28,0.68,0.44,0.01,0.41,0.63,0.64,0.41,0.69,0.36,0.92,0.29,0.52,0.37,0.49,0.83,0.53,0.24,0.04,0.78,0.7,0.04,0.48,0.81,
0.33,0.39,0.24,0.26,0.68,0.26,0.02,0.69,0.38,0.76,0.67,0.65,0.83,0.28,0.98,0.59,0.49,0.67,0.42,0.4,0.11,0.1,0.94,0.89,0.45,0.73,0.87,0.76,0.7,0.06,0.45,0.95,0.27,0.3,0.84,0.11,0.06,0.6,0.94,0.56,0.68,0.99,0.33,0.22,0.71,0.49,0.6,0.84,0.63,0.71,0.1,0.96,0.59,0.9,0.92,0.5,0.03,0.65,0.39,0.96,0.72,0.87,0.02,0.38,0.61,0.91,0.34,0.95,0.47,0.82,0.81,0.47,0.31,0.72,0.27,0.88,0.47,0
.51,0.04,0.5,0.99,0.09,0.77,0.13,0.5,0.78,0.33,0.26,0.28,0.1,0.27,0.27,0.92,0.15,0.49,0.88,0.55,0.4,0.2,0.61,0.6,0.88,0.29,0.48,0.1,0.41,0.21,0.41,0.27,0.73,0.25,0.52,0.89,0.94,0.28,0.16,0.17,0.03,0.34,0.85,0.52,0.83,0.38,0.17,0.57,0.28,0.29,0.94,0.89,0.46,0.81,0.93,0.88,0.07,0.62,0.6,0.75,0.63,0.63,0.37,0.2,0.41,0.15,0.34,0.39,0.96,0.63,0.02,0.53,0.23,0.2,0.11,0.09,0.4,0.42
,0.64,0.68,0.78,0.13,0.45,0.47,0.32,0.95,0.66,0.41,0.02,0.22,0.16,0.1,0.51,0.31,0.3,0.41,0.85,0.36,0.85,0.27,0.26,0.96,0.73,0.62,0.99,0.96,0.47,0.3,0.78,0.53,0.46,0.24,0.39,0.58,0.64,0.4,0.68,0.02,0.86,0.04,0.38,0.11,0.69,0.9,0.4,0.88,0.14,0.96,0.65,0.74,0.32,0.59,0.83,0.22,0.91,0.55,0.7,0.87,0.23,0.19,0.94,0.98,0.16,0.86,0.76,0.18,0.43,0.18,0.69,0.07,0.31,0.52,0.93,0.91,0.9
7,0.32,0.15,0.98,0.23,0.36,0.48,0.18,0.56,0.77,0.21,0.87,0.65,0.1,0.3,0.52,0.57,0.9,0.65,0.62,0.94,0.96,0.33,0.24,0.89,0.02,0.75,0.77,0.1,0.75,0.84,0.49,0.15,0.19,0.37,0.12,0.2,0.56,0.99,0.44,0.74,0.08,0.52,0.36,0.07,0.07,0.78,0.8,0.39,0.79,0.58,0.16,0,0.46,0.38,0.05,0.26,0.18,0.27,0.21,0.57,0.07,0.86,0.54,0.31,0.25,0.14,0.56,0.14,0.98,0.06,0.14,0.76,0.6,0.93,0.58,0.9,0.18,0
.46,0.33,0.27,0.34,0.89,0.27,0.69,0.89,0.41,0.05,0.07,0.28,0.28,0.17,0.88,0.62,0.81,0.92,0.05,0.13,0.11,0.82,0.23,0.96,0.88,0.07,0.71,0.12,0.38,0.11,0.1,0.94,0.63,0.38,0.25,0.54,0.85,0.93,0.65,0.33,0.52,0.6,0.99,0.24,0.47,0.09,0.94,0.65,0.44,0.52,0.35,0.24,0.66,0.59,0.59,0.68,0.37,0.3,0.22,0.28,0.25,0.4,0.6,0.98,0.94,0.88,0.33,0.94,0.59,0.2,0.48,0.96,0.52,0.56,0.13,0.1,0.05,
0.14,0.97,0.14,0.35,0.67,0.36,0.22,0.58,0.29,0.85,0.07,0.18,0.77,0.5,0.13,0.51,0.11,0.92,0.53,0.34,0.85,0.63,0.7,0.07,0.31,0.12,0.64,0.47,0.56,0.17,0.54,0.68,0.95,0.2,0.3,0.12,0.42,0.83,0.42,0.23,0.34,0.7,0.66,0.77,0.15,0.97,0.22,0.26,0.6,0.99,0.67,0.1,0.82,0.03,0.72,0.8,0.88,0.59,0.71,0.77,0.65,0.88,0.59,0.6,0.24,0.22,0.08,0.17,0.69,0.34,0.63,0.24,0.94,0.17,0.12,0.08,0.12,0
.18,0.44,0.16,0.25,0.91,0.41,0.72,0.66,0.63,0.86,0.94,0.1,0.91,0.52,0.17,0.13,0.23,0.97,0.29,0.12,1,0.95,0.7,0.31,0.09,0.68,0.18,0.82,0.88,0.11,0.43,0.75,0.51,0.77,0.41,0.87,0.3,0.39,0.99,0.27,0.44,0.42,0.84,0.86,0.77,0.28,0.26,0.62,0.99,0.32,0.41,0.45,0.95,0.85,0.13,0.28,0.18,0.1,0.54,0.37,0.86,0.86,0.79,0.36,0.64,0.98,0.79,0.07,1,0.87,0.4,0.04,0.24,0.78,0.91,0.41,0.98,0.63
,0.73,0.49,0.35,0.55,0.87,0.24,0.26,0.14,0.67,0.1,0.97,0.09,0.44,0.04,0.87,0.24,0.4,0.49,0.32,0.5,0.44,0.91,0.57,0.67,0.39,0.22,0.81,0.28,0.14,0.73,0.93,0.27,0.63,0.67,0.06,0.89,0.8,0.82,0.09,0.81,0.32,0.16,0.19,0.71,0.03,0.84,0.19,0.89,0.83,0.52,0.61,0.79,0.2,0.54,0.64,0.13,0.46,0.24,0.15,0.84,0.31,0.43,0.52,0.4,0.49,0.32,0.63,0.05,0.69,0.67,0.48,0.94,0.89,0.02,0.28,0.4,0.3
4,0.34,0.03,0.02,0.65,0.85,0.78,0.72,0.47,0.67,0.32,0.03,0.11,0.27,0.8,0.59,0.58,0.83,0.92,0.87,0.95,0.61,0.06,0.72,0.13,0.63,0.47,0.59,0.1,0.95,0.75,0.37,0.63,0.99,0.58,0.1,0.37,0.8,0.36,0.05,0.92,0.76,0.41,0.99,0.03,0.06,0.09,0.28,0.86,0.99,0.69,0.52,0.62,0.79,0.33,0.12,0.7,0.58,0.56,0.02,0.46,0.77,0.88,0.09,0.37,0.51,0.87,0.45,0.5,0.45,0.42,0.43,0.64,0.66,0.13,0.94,0.35,0
.35,0.62,0.61,0.64,0.8,0.18,0.59,0.77,0.02,0.54,0.94,0.56,0.26,0.97,0.55,0.71,0.08,0.89,0.82,0.26,0.24,0.18,0.27,0.77,0.79,0.18,0.03,0.68,0.56,0.63,0.08,0.15,0.76,0.91,0.41,0.94,0.44,0.78,0.95,0.52,0.68,0.45,0.16,0.29,0.08,0.86,0.26,0.65,0.21,0.39,0.77,0.45,0.19,0.81,0.48,0.9,0.55,0.42,0.03,0.37,0.72,0.99,0.4,0.01,0.86,0.96,0.69,0.67,0.12,0.85,0.79,0.95,0.19,0.46,0.57,0.76,0
.82,0.63,0.51,0.86,0.29,0.05,0.24,0.42,0.99,0.13,0.55,0.73,0.63,0.24,0.94,0.67,0.34,0.79,0.47,0.89,0.98,0.98,0.99,0.32,0.93,0.42,0.95,0.74,0.64,0.33,0.98,0.64,0.35,0.91,0.4,0.9,0.97,0.22,0.82,0.65,0.47,0.61,0.94,0.24,0.42,0.64,0.24,0.63,0.17,0.39,0.42,0.31,0.84,0.34,0.41,0.78,0.13,0.02,0.63,0.87,0.58,0.37,0.62,0.31,0.9,0.02,0.47,0.12,0.39,0.84,0.96,0.55,0.24,0.21,0,0.96,0.56
,0.31,0.85,0.28,0.61,0.35,0.76,0.93,0.37,0.73,0.07,0.08,0.36,0.21,0.77,0.75,0.53,0.45,0.21,0.79,0.57,0.67,0.74,0.39,0.97,0.46,0.41,0.24,0.85,0,0.51,0.19,0.4,0.41,0.48,0.64,0.86,0.75,0.79,0.65,0.4,0.59,0.69,0.44,0.77,0.85,0.95,0.06,0.94,0.61,0.2,0.19,0.21,0.32,0.38,0.99,0.58,0.38,0.16,0.9,0.2,0.18,0.16,0.78,0.41,0.22,0.57,0.88,0.78,0.56,0.91,0.12,0.14,0.74,0.97,0.68,0.32,0.8,
0.56,0.93,0.15,0.4,0.93,0.33,0.28,0.67,0.5,0.93,0.89,0.54,0.79,0.87,0.74,0.25,0.66,0.42,0.03,0.24,0.06,0.36,0.54,0.41,0.05,0.05,0.39,0.24,0.15,0.89,0.67,0.84,0.47,0.3,0.9,0.79,0.58,0.1,0.95,0.49,0.78,0.75,0.93,0.42,0.53,0.17,0.12,0.01,0.34,0.65,0.46,0.32,0.04,0.15,0.03,0.6,0.29,0.07,0.87,0.29,0.6,0.6,0.15,0.14,0.29,0.52,0.34,0.58,0.44,0.85,0.23,0.08,0.26,0.21,0.02,0.4,0.89,0
.8,0.66,0.65,0.28,0.56,0.14,0.36,0.61,0.22,0.85,0.32,0.83,0.44,0.44,0.72,0.85,0.01,0.23,0.51,0.65,0.22,0.58,0.99,0.31,0.34,0.85,0.45,0.78,0.49,0.94,0.36,0.95,0.15,0.86,0.96,0.01,0.72,0.08,0.92,0.22,0.57,0.39,0.31,0.47,1,0.34,0.29,0.82,0.31,0.44,0.07,0.5,0.25,0.25,0.77,0.03,0.66,0.96,0.46,0.52,0.1,0.52,0.92,0.61,0.07,0.24,0.81,0.73,0.81,0,0.12,0.92,0.9,0.37,0.49,0.4,0.98,0.58
,0.35,0.59,0.11,0.05,0.17,0.61,0.85,0.07,0.33,0.5,0.97,0.87,0.22,0.63,1,0.1,0.28,0.15,0.49,0.5,0.25,0.68,0.8,0.24,0,0.05,0.59,0.54,0.88,0.49,0.17,0.36,0.03,0.46,0.03,0.31,0.46,0.39,0.55,0.05,0.92,0.42,0.37,0.87,0.49,0.52,0.21,0.2,0.86,0.49,0.75,0.97,0.65,0.57,0.74,0.72,0.77,0.51,0.04,0.02,0.08,0.52,0.5,0.66,0.39,0.85,0.63,0.11,0.37,0.51,0.85,0.03,0.19,0.65,0.65,0.33,0.28,0.7
1,0.68,0.88,0.31,0.99,0.52,0.08,0.83,0.59,0.22,0.31,0.86,0.44,0.59,0.4,0.61,0.33,0.3,0.23,0.83,0.02,0.8,0.98,0.11,0.35,0.48,0.45,0.58,0.23,0.18,0.23,0.36,0.16,0.35,0.88,0.25,0.39,0.9,0.03,0.81,0.69,0.2,0.36,0.61,0.85,0.17,0.43,0.5,0.57,0.95,0.94,0.67,0.27,0.42,0.76,0.63,0.41,0.46,0.99,0.37,0.56,0.21,0.32,0.98,0.7,0.7,0.32,0.93,0.34,0.84,0.34,0.94,0.18,0.31,0.07,0.59,0.98,0.9
1,0.3,0.06,0.17,0.73,0.72,0.93,0.5,0.04,0.22,0.54,0.06,0.21,0.09,0.77,0.67,0.94,0.39,0.66,0.93,0.03,0.11,0.92,0.47,0.81,0.38,0.03,0.06,0.84,0.66,0.01,0.82,0.68,0.86,0.97,0.15,0.99,0.29,0.56,0.41,0.4,0.11,0.65,0.34,0.47,0.1,0.22,0.56,0.32,0.74,0.18,0.15,0.42,0.69,0.18,0.17,0.07,0.98,0,0.86,0.92,0.56,0.53,0.12,0.01,0.58,0.72,0.28,0.59,0.17,0.95,0.21,0.29,0.78,0.45,0.19,0.26,0.
8,0.29,0.33,0.48,0.53,0.73,0.44,0.95,0.9,0.49,0.32,0,0.37,0.72,0.29,0.3,0.09,0.91,0.25,0.51,0.23,0.27,0.86,0.73,0.4,0.63,0.7,0.5,0.03,0.46,0.66,0.13,0.28,0.22,0.77,0.24,0.19,0.59,0.32,0.12,0.28,0.83,0.45,0.96,0.14,0.45,0.93,0.28,0.46,0.97,0.4,0.94,0.57,0.87,0.57,0.22,0.35,0.9,0.34,0.41,0.7,0.35,0.38,0.04,0.27,0.25,0.69,0.02,0.91,0.35,0.76,0.62,0.46,0.49,0.46,0.45,0.9,0.1,0.3
2,0.09,0.91,0.13,0.87,0.83,0.06,0.84,0.1,0.97,0.11,0.31,0.18,0.02,0.76,0.51,0.5,0.17,0.61,0.12,0.25,0.51,0.65,0.01,0.93,0.59,0.27,0.35,0.22,0.43,0.02,0.7,0.55,0.9,0.37,0.92,0.41,0.32,0.21,0.57,0.49,0.64,0.54,0.85,0.98,0.87,0.14,0.43,0.15,0.04,0.71,0.01,0.43,0.1,0.72,0.32,0.96,0.34,0.83,0.72,0.96,0.82,0.07,0.95,0,0.51,0.15,0.43,0.8,0.57,0.11,0.27,0.14,0.56,0.01,0.03,0.04,0.99
,0.92,0.49,0.39,0.64,0.13,0.82,0.66,0.1,0.94,0.47,0.61,0.3,0.3], "top": 3 }'
{"status":{"error":"Wrong input: Vector params for  are not specified in config"},"time":0.001418866}

the config.json of the collection looks strange to me:

{
    "params": {
        "vectors": {
            "vector": {
                "size": 1536,
                "distance": "Cosine"
            }
        },
        "shard_number": 1,
        "replication_factor": 1,
        "write_consistency_factor": 1,
        "on_disk_payload": true
    },
    "hnsw_config": {
        "m": 16,
        "ef_construct": 100,
        "full_scan_threshold": 10000,
        "max_indexing_threads": 0,
        "on_disk": false
    },
    "optimizer_config": {
        "deleted_threshold": 0.2,
        "vacuum_min_vector_number": 1000,
        "default_segment_number": 0,
        "max_segment_size": null,
        "memmap_threshold": null,
        "indexing_threshold": 20000,
        "flush_interval_sec": 5,
        "max_optimization_threads": null
    },
    "wal_config": {
        "wal_capacity_mb": 32,
        "wal_segments_ahead": 0
    },
    "quantization_config": null
}

Probably the reason for this issue is: "vectors": { "vector": { ... instead of

{
    "params": {
        "vectors": {
                "size": 1536,
                "distance": "Cosine"
        },
...}

Branch

No response

Checklist

• [X] Modify src/vdf_io/import_vdf/qdrant_import.py ✓ https://github.com/AI-Northstar-Tech/vector-io/commit/01405c0d8695fcdffbab2a783bfa29199c2868c5 Edit

[sweep-ai-deprecated[bot]]

🚀 Here's the PR! #94

See Sweep's progress at the progress dashboard!

💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID: 4725fe3daf)

---

[!TIP] I can email you next time I complete a pull request if you set up your email here!

---

Actions (click)

• [ ] ↻ Restart Sweep

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

https://github.com/AI-Northstar-Tech/vector-io/blob/ad971da78fbc1e01f4977f5d6c2d8c7eaf0149ae/src/vdf_io/import_vdf/qdrant_import.py#L1-L469

---

Step 2: ⌨️ Coding

• [X] Modify src/vdf_io/import_vdf/qdrant_import.py ✓ https://github.com/AI-Northstar-Tech/vector-io/commit/01405c0d8695fcdffbab2a783bfa29199c2868c5 Edit

Modify src/vdf_io/import_vdf/qdrant_import.py with contents: Update the `vectors_config` dictionary in the `upsert_data` method to correctly structure the vector configuration.

<original_code> vectors_config = { vector_column_name: VectorParams( size=dims, distance=distance, on_disk=on_disk, ) for vector_column_name in vector_column_names } </original_code>

<new_code> vectors_config = { vector_column_name: VectorParams( size=dims, distance=distance, ) for vector_column_name in vector_column_names } </new_code>

Remove the nested "vector" key and specify the vector configuration directly under the "vectors" key, with the vector column name as the key and the VectorParams object as the value. Also remove the on_disk parameter as it is not part of the VectorParams configuration.

---

Step 3: 🔁 Code Review

I have finished reviewing the code for completeness. I did not find errors for sweep/qdrant_import_collection_not_working.

---

🎉 Latest improvements to Sweep:

• New dashboard launched for real-time tracking of Sweep issues, covering all stages from search to coding.
• Integration of OpenAI's latest Assistant API for more efficient and reliable code planning and editing, improving speed by 3x.
• Use the GitHub issues extension for creating Sweep issues directly from your editor.

💡 To recreate the pull request edit the issue title or description. ^{Something wrong? Let us know.}

This is an automated message generated by Sweep AI.

[greptile-apps[bot]]

The issue seems to stem from the collection configuration format used during the import process. Specifically, the vectors_config setup in the upsert_data method of qdrant_import.py expects a dictionary with keys corresponding to vector column names and their configurations. However, your collection's config.json indicates a mismatch in expected structure, particularly under the params -> vectors section. To resolve this, ensure the collection configuration passed to self.client.create_collection within upsert_data matches Qdrant's expected format. This involves adjusting the vectors_config dictionary construction to align with your collection's actual vector dimension and distance metric, ensuring it accurately reflects the structure shown in your issue description.

References

/src/vdf_io/import_vdf/qdrant_import.py

Ask Greptile

[dhruv-anand-aintech]

The

"params": {
        "vectors": {
            "vector": {
                "size": 1536,
                "distance": "Cosine"
            }
        },
		...

is matching the format for named vectors.

I want to better understand the sequence of your operations. Did you first import a vdf dataset into your qdrant instance, and then try to do a search via the REST API?