[BUG] Conversational Retrieval QA Chain behaves different on flowise and embed website

[emrahtoy]

Describe the bug The chain behaves and replies differently on both sides.

To Reproduce Create Conversational Retrieval QA Chain chat flow based on the template or created yourself. I used a text file document with an in-memory vector store.

1. Limit your prompt within the border of the document or use the default prompt which works same way. I also added my own prompt.
2. Test your chat flow on Flowise editor chat panel.
3. Say "Merhaba" or "Hello" ( merhaba means hello in turkish language )
4. Do the same thing on your website with an embedded Flowise chat panel.
5. Also forgets the language I limited. ( you can use the document and prompt I added below )

Expected behavior Both should give the same answers regarding to the document given Flowise editor chat panel works as expected and relies on variables i set on chat model. The embedded website panel does not care about the content of the document and gives answers unrelated to the prompt and doesn't care about variables I set on the chat model. Also forgets about prompted language even though I strictly limit it.

Screenshots

Document ( 1500 chunk default variables on character text splitter ) YTU Startup House.txt

Prompt I want you to act as an officer working at Yıldız Technical University. Your name is Ahmet. Stay within the boundaries of the document. Reply, answer, write, and talk in Turkish only. When something irrelevant is asked or said reply "Size Yıldız Teknik Üniversitesi Startup House hakkında yardımcı olmak için buradayım." shortly in Turkish and then redirect people to the boundaries of the document politely. Be helpful and do not give long answers. Never break character.

[emrahtoy]

Logs are giving kind of a hint about behavioral difference.

embed.log flowsiePanel.log

[emrahtoy]

@HenryHengZJ how can I debug components?

[emrahtoy]

Logs show that the Embed version is jumping the "[1:chain:ConversationalRetrievalQAChain > 2:chain:StuffDocumentsChain]" step.

To me, it should have got both the text ( in base64 format ) and the chain data from chat flow data requested from the database.

[HenryHengZJ]

@HenryHengZJ how can I debug components?

you can put console log in the component file

I tried it there it seems to behave same for me: Flowise Chat:

Embedded Chat:

[emrahtoy]

So strange, I tried again and have had different results. I will clean everything and try again.

[emrahtoy]

@HenryHengZJ it is still happening. Plus son that there are new bugs... embed gets 401 unauthorized error at first but then receives stream ( probably over websocket ) and API requests causes another error ( when you have a brand new flow and trigger it over API at the first time )

I still see the difference I mentioned before in the logs. I am trying really hard to use flowise with our products but keep facing with the new bugs. I would like to contribute more but I am seriously in need of help to learn how to debug this project end to end ( by debug I don't mean console.log.. I mean debug points where I can see call stack, etc.. )

[HenryHengZJ]

@emrahtoy do you have some projects in mind that have good debug points where we can take reference from?

when error happens, logs like this should be able to tell you where the error is from:

[emrahtoy]

Any kind of log only gives information about the error and maybe call-stack at limited deep. On the other hand, debugging with a debug point gives you the state of the app and information on the variables inside the current context.

For example, you have no idea what was the json string giving the error above. If you could set a debug point on line 43 of the PromptTemplate.js file then you could see the JSON string because the application flow would stop at that point right before the error happens. Most probably you are not going to able to do that because of the structure of the mono repo and how modules connected to each other in development mode.

You are using distribution packages of flowise-ui and flowise-components in the server module, this is where the "yarn link" comes in handy as it will make it to use the real dist folder inside the workspace. Yet still not enough. At this point, source maps may help as stated here but I couldn't spare enough time to test.

[mibtim]

Henry and emrahtoy, hope all is well. guys just checking into see if this issue embed issue is resolved? for it doesn't still work. thanks

[emrahtoy]

@mibtim i believe it is not resolved. I figure out how to debug chains and I will check again today

[mibtim]

thank you heaps mate, please keep me updated thanks heaps

[HenryHengZJ]

@emrahtoy @mibtim see update here - https://github.com/FlowiseAI/Flowise/issues/473

[mibtim]

@HenryHengZJ , mate thank heaps and i will have a look. also wanted give you a massive shout out for the amazing work you guys do and for the awesome product. tc

[chungyau97]

Hi @emrahtoy @mibtim, once you guys verified that it's working on your side.

kindly help us close the issue. Much appreciated.

[mibtim]

@HenryHengZJ @chungyau97 thank you heaps i think i have to wait for the next release as im using docker/npm. good to hear this is now resolved and thank you heaps for the awesome work you all do.

[emrahtoy]

@chungyau97 i will do a check tonight and share the result.

[emrahtoy]

@HenryHengZJ @chungyau97 It seems like solved. Both Flowise and embed bubble chat are giving the same answers and the logs are identical. Thank you for your efforts, I gladly close this issue 👍