Relevancy for Chroma retriever results for non relevant questions

[My3VM]

While it is known that ultimately it is the responsibility of the prompt to control the responses to QA task, the ConversationalRetrievalChain running with ConversationSummaryBufferMemory occasionally responds with strange replies to non relevant questions. In the prompt it is mentioned like "Please do not refer to document sources while responding to off-topic questions."

During the middle of a conversation, asking non relevant questions like 'hello', 'how are you' repeats one of the previous responses back as fresh response. While I am using ConversationSummaryBufferMemory to serve as memory to the bot, am not sure if I have to consider the accuracy of the retriever for handling such scenarios? As it is already known, vector store with Chroma db doesn't allow the search relevance threshold based on similarity score. Printing the accuracy of docs_and_scores from

Printing similarity_search revealed that the similarity always ranges between 3.1 to 4.1 for both relevant /irrelevant responses. Configuring retreiver with (search_type="similarity", search_kwargs={"k":2}) also doesnt help the situation much.