ChatQnA readme issues

[tbykowsk]

trafficstars

Hi, I followed ChatQnA Application readme and encountered some problems. I started with code from the master branch.

1. In the step Start the Backend Service, any request to the streaming endpoint returns an error:

INFO:     Application startup complete.
	INFO:     Uvicorn running on http://0.0.0.0:8000/ (Press CTRL+C to quit)
	INFO:     <IP>:50773 - "OPTIONS /v1/rag/chat_stream HTTP/1.1" 200 OK
	[rag - chat_stream] POST request: /v1/rag/chat_stream, params:{'query': 'What is the total revenue of Nike in 2023?"', 'knowledge_base_id': 'default'}
	[rag - chat_stream] history: []
	[rag - reload retriever] reload with index: rag-redis
	INFO:     <IP>:50773 - "POST /v1/rag/chat_stream HTTP/1.1" 200 OK
	score_threshold is deprecated. Use distance_threshold instead.score_threshold should only be used in similarity_search_with_relevance_scores.score_threshold will be removed in a future release.
	Metadata key source not found in metadata. Setting to None.
	Metadata fields defined for this instance: ['source', 'start_index']
	[...]
	Metadata key start_index not found in metadata. Setting to None.
	Metadata fields defined for this instance: ['source', 'start_index']
	ERROR:    Exception in ASGI application
	Traceback (most recent call last):
	  File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 265, in __call__
	    await wrap(partial(self.listen_for_disconnect, receive))
	  File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 261, in wrap
	    await func()
	  File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
	    message = await receive()
	              ^^^^^^^^^^^^^^^
	  File "/home/user/.local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 538, in receive
	    await self.message_event.wait()
	  File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
	    await fut
	asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f0a23abf1d0
	 
	During handling of the above exception, another exception occurred:
	 
	  + Exception Group Traceback (most recent call last):
	  |   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
	  |     result = await app(  # type: ignore[func-returns-value]
	  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  |   File "/home/user/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
	  |     return await self.app(scope, receive, send)
	  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  |   File "/home/user/.local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
	  |     await super().__call__(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
	  |     await self.middleware_stack(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
	  |     raise exc
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
	  |     await self.app(scope, receive, _send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
	  |     await self.simple_response(scope, receive, send, request_headers=headers)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 148, in simple_response
	  |     await self.app(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
	  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
	  |     raise exc
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
	  |     await app(scope, receive, sender)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
	  |     await self.middleware_stack(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
	  |     await route.handle(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
	  |     await self.app(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
	  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
	  |     raise exc
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
	  |     await app(scope, receive, sender)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
	  |     await response(scope, receive, send)
	  |   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 258, in __call__
	  |     async with anyio.create_task_group() as task_group:
	  |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 678, in __aexit__
	  |     raise BaseExceptionGroup(
	  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
	  +-+---------------- 1 ----------------
	    | Traceback (most recent call last):
	    |   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 261, in wrap
	    |     await func()
	    |   File "/home/user/.local/lib/python3.11/site-packages/starlette/responses.py", line 252, in stream_response
	    |     chunk = chunk.encode(self.charset)
	    |             ^^^^^^^^^^^^
	    | AttributeError: 'NoneType' object has no attribute 'encode'
	    +------------------------------------

This error causes that the Frontend Service is unresponsive, because it connects to /v1/rag/chat_stream endpoint which is broken.

When I reverted app/server.py to this commit the streaming endpoint started to work. It would be useful to add information to the instruction about a commit/release it was validated with.

2. The Frontend Service allows to add new data sources Please upload your local file or paste a remote file link, and Chat will respond based on the content of the uploaded file. In the log new data sources are ingested correctly:

[rag - create] POST request: /v1/rag/create, filename:architecture-instruction-set-extensions-programming-reference.pdf
[rag - create kb folder] upload path: upload_dir/kb_0f19a9e4/upload_dir, persist path: upload_dir/kb_0f19a9e4/persist_dir
[rag - create] file saved to local path: upload_dir/kb_0f19a9e4/upload_dir/2024-04-17-16:54:11-architecture-instruction-set-extensions-programming-reference.pdf
[rag - create] starting to create local db...
[rag - create retriever] create with index: rag-rediskb_0f19a9e4
[nltk_data] Downloading package punkt to /home/user/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /home/user/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
`index_schema` does not match generated metadata schema.
If you meant to manually override the schema, please ignore this message.
index_schema: {'text': [{'name': 'content'}, {'name': 'source'}], 'numeric': [{'name': 'start_index'}], 'vector': [{'name': 'content_vector', 'algorithm': 'HNSW', 'datatype': 'FLOAT32', 'dims': 768, 'distance_metric': 'COSINE'}]}
generated_schema: {'text': [{'name': 'source'}], 'numeric': [{'name': 'start_index'}], 'tag': []}

[rag - create] kb created successfully
INFO:     <IP>:51563 - "POST /v1/rag/create HTTP/1.1" 200 OK

[rag - upload_link] POST request: /v1/rag/upload_link, link list:['https://en.wikipedia.org/wiki/American_white_pelican']
[rag - create kb folder] upload path: upload_dir/kb_2ce41686/upload_dir, persist path: upload_dir/kb_2ce41686/persist_dir
[rag - upload_link] starting to create local db...
start fetch %s... https://en.wikipedia.org/wiki/American_white_pelican
`index_schema` does not match generated metadata schema.
If you meant to manually override the schema, please ignore this message.
index_schema: {'text': [{'name': 'content'}, {'name': 'source'}], 'numeric': [{'name': 'start_index'}], 'vector': [{'name': 'content_vector', 'algorithm': 'HNSW', 'datatype': 'FLOAT32', 'dims': 768, 'distance_metric': 'COSINE'}]}
generated_schema: {'text': [{'name': 'source'}, {'name': 'identify_id'}], 'numeric': [], 'tag': []}


[rag - upload_link] kb created successfully
INFO:     <IP>:20643 - "POST /v1/rag/upload_link HTTP/1.1" 200 OK

However, new data sources do not seem to be instantly included during response generation. A restart of app/server.py is required for new information to be available, even though in the backend an index changes with each uploaded document, and is reloaded with each response, example of log: [rag - reload retriever] reload with index: rag-rediskb_147637c0. To confirm this problem, I have used a working app/server.py from the aforementioned commit, the rest of code was from the master branch.

Please look into those issues. Thanks!

[lvliang-intel]

Hi @tbykowsk,

Thanks for raising this issue. For the first issue, I fixed it by PR https://github.com/opea-project/GenAIExamples/pull/88. I'm currently attempting to reproduce the second issue. I'll keep you updated on my progress.

[lvliang-intel]

Hi @tbykowsk,

Currently, our knowledge base creation process lacks an appending strategy. Whenever a file is uploaded, a new knowledge base ID is generated. Similarly, when a html link is pasted, it also generates a new knowledge base ID. As a result, the frontend utilizes the most recent knowledge base ID as the default for chatting. Consequently, if a question pertains to the uploaded file, the system searches only within the knowledge base associated with the uploaded link.

To enhance this process, we plan to refine the code and implement a knowledge appending strategy. This strategy will involve appending new knowledge to existing knowledge bases rather than generating new IDs each time.

[tbykowsk]

Hi @lvliang-intel, thank you for such a quick response!

I can confirm that the first issue is resolved.

Thank you for the details about the knowledge base implementation. Maybe information how it functions could be added to Readme, so it is clear what to expect.

[tbykowsk]

@lvliang-intel, are you able to say when do you plan to implement a knowledge appending strategy?

[apahleva]

@lvliang-intel, I am following the steps in the README, and after I successfully build the Docker container and set up my Hugging Face token, I encounter an issue that I don't know how to resolve. Can you guide me?

curl -v --noproxy '*' 127.0.0.1:8080/generate -X POST -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":32}}' -H 'Content-Type: application/json' Note: Unnecessary use of -X or --request, POST is already inferred.

• Trying 127.0.0.1:8080...
• TCP_NODELAY set
• Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)

POST /generate HTTP/1.1 Host: 127.0.0.1:8080 User-Agent: curl/7.68.0 Accept: / Content-Type: application/json Content-Length: 70

• upload completely sent off: 70 out of 70 bytes
• Recv failure: Connection reset by peer
• Closing connection 0 curl: (56) Recv failure: Connection reset by peer

[preethivenkatesh]

@tbykowsk Please let us know if this issue is fixed and can be closed

[letonghan]

@tbykowsk Since the dataprep microservice supports the knowledge base append strategy now, we will close this issue. Please let us know if there is other issues.