Redis doesn't return results when using query endpoint

[zainabalthafeeri1]

I've added documents to the Redis search using the upsert API, and the documents were added successfully. Still, when I want to use the query API, the results returned are always empty, I have tried different files with different types (PDF, DOCX), but nothing returned in the response.

[CyberAmyntas]

When you uploaded your document with Upsert, What did the debug look like? Here is an example

mimetype: application/pdf
file.file: <tempfile.SpooledTemporaryFile object at 0x12f164f60>
file:  <starlette.datastructures.UploadFile object at 0x12f1640f0>
extracted_text:
INFO:     127.0.0.1:55035 - "POST /upsert-file HTTP/1.1" 200 OK

When you query, What does the response look like? Here is an example of one that worked, and two that didn't.

Worked INFO: 127.0.0.1:54955 - "POST /query HTTP/1.1" 200 OK

Error

INFO:     127.0.0.1:55152 - "POST /query HTTP/1.1" 422 Unprocessable Entity
Error: 1 validation error for DocumentChunkWithScore
metadata -> source
  value is not a valid enumeration member; permitted: 'email', 'file', 'chat' (type=type_error.enum; enum_values=[<Source.email: 'email'>, <Source.file: 'file'>, <Source.chat: 'chat'>])
INFO:     127.0.0.1:55154 - "POST /query HTTP/1.1" 200 OK

[zainabalthafeeri1]

• The response for uploading the documents was:

  mimetype: application/pdf
  file.file: <tempfile.SpooledTemporaryFile object at 0x7ff6eae56560>
  file:  <starlette.datastructures.UploadFile object at 0x7ff6eae566e0>
  extracted_text:
  INFO:     127.0.0.1:56796 - "POST /upsert-file HTTP/1.1" 200 OK
  

• The query returned 200 but still the response is empty

[tylerhutcherson]

@zainabalthafeeri1 You can validate that the documents are indeed indexed in Redis? (use redis-cli to inspect). Any other details you can provide?

[goldsag]

I'm also getting the error: Error: 1 validation error for DocumentChunkWithScore metadata -> source value is not a valid enumeration member; permitted: 'email', 'file', 'chat' (type=type_error.enum; enum_values=[<Source.email: 'email'>, <Source.file: 'file'>, <Source.chat: 'chat'>])

Uploaded documents via the /upsert endpoint. the source fields of all my documents is the string "file". any idea?

---UPDATE--- I restarted the app (shut down and reran locally), and suddenly it worked! :)

[zainabalthafeeri1]

@zainabalthafeeri1 You can validate that the documents are indeed indexed in Redis? (use redis-cli to inspect). Any other details you can provide?

Yes, the documents were indexed in Redis with its respective embedding

[jongio]

the sample docs have "source_id", remove that and delete all documents with the delete endpoint

[ccqpein]

I just encountered the same issue where Redis doesn't return any results when I provide the "document_id" in the filter:

• /upsert endpoints call is successful
• Results are returned with this command: curl -H "Authorization: Bearer {token}" -H "Content-Type: application/json" -X POST -d '{"queries": [{"query": "query with documents id"}]}' http://0.0.0.0:8000/query
• An empty response is returned with this command: curl -H "Authorization: Bearer {token}" -H "Content-Type: application/json" -X POST -d '{"queries": [{"query": "query with documents id", "filter":{"document_id": "290d130f-4517-4ff7-a88b-f648df1c44da"}}]}' http://0.0.0.0:8000/query. The 'document_id' is the same as the one in the response from the successful and /upsert call.
• When switching to Milvus, the above commands return results.

I an running the redis and milvus using the docker-compose in chatgpt-retrieval-plugin/examples/docker.

[tylerhutcherson]

@ccqpein I think I have a fix for your issue.

[ccqpein]

@ccqpein I think I have a fix for your issue.

Hi @tylerhutcherson , I just pull the latest main branch and run the test. Unfortunately, this time redis doesn't return the result whether I give it document_id or not. This is the step and the commands I use:

# in chatgpt-retrieval-plugin/examples/docker/redis
docker-compose up -d

# back to chatgpt-retrieval-plugin
# DATASTORE="redis"
# BEARER_TOKEN and API token are in env also
poetry run start

# in other shell
# upsert the read me of chatgpt-retrieval-plugin
curl \
      -H "Authorization: Bearer {token}" \
      -X POST \
      -F '[email protected];type=text/markdown' \
      http://0.0.0.0:8000/upsert-file

# return 200 and the document id

curl -H "Authorization: Bearer {token}"\
      -H "Content-Type: application/json"\
      -X POST \
      -d '{"queries": [{"query": "start up"}]}' http://0.0.0.0:8000/query

# return empty results.
# and in terminal that poetry is running, it give a log "Failed to query, error: 'MilvusDataStore' object has no attribute 'col'"

Everything is fine when I use milvus. Thanks!

[tylerhutcherson]

@ccqpein I think I have a fix for your issue.

Hi @tylerhutcherson , I just pull the latest main branch and run the test. Unfortunately, this time redis doesn't return the result whether I give it document_id or not. This is the step and the commands I use:

# in chatgpt-retrieval-plugin/examples/docker/redis
docker-compose up -d

# back to chatgpt-retrieval-plugin
# DATASTORE="redis"
# BEARER_TOKEN and API token are in env also
poetry run start

# in other shell
# upsert the read me of chatgpt-retrieval-plugin
curl \
      -H "Authorization: Bearer {token}" \
      -X POST \
      -F '[email protected];type=text/markdown' \
      http://0.0.0.0:8000/upsert-file

# return 200 and the document id

curl -H "Authorization: Bearer {token}"\
      -H "Content-Type: application/json"\
      -X POST \
      -d '{"queries": [{"query": "start up"}]}' http://0.0.0.0:8000/query

# return empty results.
# and in terminal that poetry is running, it give a log "Failed to query, error: 'MilvusDataStore' object has no attribute 'col'"

Everything is fine when I use milvus. Thanks!

This particular issue looks like something different. upsert-file is not working properly for Redis. Opening a new issue

[tylerhutcherson]

my PR is open which should address this and other things!