David

> this is a pretty big change to the logic right? > > > > i wrote this loader, and IIRC I did this because body by itself contained a...

Not to mention the AI is pretty good at determining relevant information. In conclusion it's better to have irrelevant content scraped then no content at all aslong as you can...

> Would it be better to use the tags if they exist and if not then fall back to a different strategy of using the body tag ? Yes.

Yes, i'll try and get something up and working asap.

🔎 Searching I'm searching for relevant snippets in your repository. Code Snippets Found https://github.com/Haste171/langchain-chatbot/blob/5f913d074827946f318181c11e17229846441352/.gitignore#L0-L15 https://github.com/Haste171/langchain-chatbot/blob/5f913d074827946f318181c11e17229846441352/.gitignore#L15-L30 https://github.com/Haste171/langchain-chatbot/blob/5f913d074827946f318181c11e17229846441352/.gitignore#L30-L45 https://github.com/Haste171/langchain-chatbot/blob/5f913d074827946f318181c11e17229846441352/.gitignore#L45-L60 https://github.com/Haste171/langchain-chatbot/blob/5f913d074827946f318181c11e17229846441352/.gitignore#L60-L75

Added to the To-Do list

{"tag": "COMMENT"} I suggest adding compatibility with offline models to the project.

{"tag": "CONSOLIDATION", "workflow": [{"step": "Step 1", "name": "Read task requirements", "input": "Task requirements", "output": "Understanding of task requirements", "async_score": 10, "assigned_agent": "User", "dependencies": []}, {"step": "Step 2", "name": "Analyze current...

{"tag": "COMMENT"} I suggest adding a new step to the workflow to address the compatibility with offline models.

{"tag": "CONSOLIDATION", "workflow": [{"step": "Step 1", "name": "Read task requirements", "input": "Task requirements", "output": "Understanding of task requirements", "async_score": 10, "assigned_agent": "User", "dependencies": []}, {"step": "Step 2", "name": "Analyze current...