Research with local documents

[ElishaKay]

Fixes an error when running on mac:

docker-compose up --build

The error (both on this branch & master branch) is:

Failed to fetch http://deb.debian.org/debian-security/pool/updates/main/f/firefox-esr/firefox-esr_115.3.1esr-1%7edeb11u1_arm64.deb 404 Not Found [IP: 199.232.82.132 80] #0 2.622 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[assafelovic]

Thanks @ElishaKay ! Did you see that there are many conflicts?

[arsaboo]

Don't we also need to modify the docker-compose to provide the mount path?

[ElishaKay]

@arsaboo better to include the documents within the repository itself. Reason being: if we mount local folders into a docker container, it can lead to permission issues & other complexity

[olafgeibig]

@ElishaKay @arsaboo I think the mounting a docker volume is a good idea. This is being used by many dockerized applications. Maybe it could be optional to keep the default simple. I guess very often production deployments wouldn't like to have their docs in git. Thanks for this work. Can't wait that it's merged.

[ElishaKay]

2-minute internet and documents reports work after merging latest master. 5-minute reports need work :)

[ElishaKay]

@assafelovic,morning sir.

• All 6 potential combos work: - 2-min, 5-min & resource-reports for both internet & document sources.
• Don't believe it should have affected the multi-agent feature since I believe that folder was added after @proy9714's original branch verged off master.

And best of all.... zero conflicts with master 🎉 Would be thrilled if we could put other features on hold so as not to create new conflicts 😅

@proy9714 feel free to leave feedback here as well - solid stuff! Let's do great things gentlemen

[assafelovic]

Thank you @proy9714 and @ElishaKay for this super super demanded feature! I'll stop all PRs until we get this through (except for hot fixes)! I will dive into it on Wed and we should release it next Monday

[assafelovic]

@ElishaKay can you update the documentation here: https://github.com/assafelovic/gpt-researcher/blob/master/docs/docs/gpt-researcher/tailored-research.md

[ElishaKay]

@ElishaKay can you update the documentation here: https://github.com/assafelovic/gpt-researcher/blob/master/docs/docs/gpt-researcher/tailored-research.md

@assafelovic, Done. Documentation added both for the pip package and running on localhost:8000 - with links and teaser on Readme.md

[ElishaKay]

Open issues for consideration: A) we have 6 report_types and I only tested the 3 report_types available in the UI. @proy9714 do you think it will support all 6 potential report_types available from the pip package? If not, let's clarify that limit in the Docs.

B) @assafelovic, if the multi_agent feature is leveraging the same GPTResearcher Class, we should re-test the multi-agent feature on this branch? @assafelovic probably best to pass that to you since you're best acquainted with it

[proy9714]

Open issues for consideration: A) we have 6 report_types and I only tested the 3 report_types available in the UI. @proy9714 do you think it will support all 6 potential report_types available from the pip package? If not, let's clarify that limit in the Docs.

B) @assafelovic, if the multi_agent feature is leveraging the same GPTResearcher Class, we should re-test the multi-agent feature on this branch? @assafelovic probably best to pass that to you since you're best acquainted with it

Hey! Yeah I guess it should work for all the report types, but I have not tested extensively. I mean if I am not wrong the other basic report types only differ in the prompts?!

Also I should mention that a new issue was logged where the PydanticOutputParser from the current version of langchain fails to work with gpt-4o. So detailed report generation fails, as the subtopics are failed to be parsed from the output of the LLM. I have suggested using the gpt-4-turbo model for the time being.

Here are the issues:

1. Issue no. #442
2. Langchain Issue : https://github.com/langchain-ai/langchain/discussions/21635

[ElishaKay]

proy9714

@proy9714 , I invited you now to collaborate on my fork of the repo - feel free to do whatever you gotta do on this branch. This is your baby & I trust you bro!

Also, if someone reading this has time budget to test the multi_agent feature on this branch, feel free

[ElishaKay]

Looks like the multi_agent cli feature broke 🙈 Will need to have a closer look this weekend

Error: TypeError: GPTResearcher.__init__() got an unexpected keyword argument 'verbose'

[assafelovic]

@ElishaKay did you merge master into this branch? I've added a param to GPTResearcher called verbose that disabled console prints if false (for production purposes).

[assafelovic]

@ElishaKay check out this commit: https://github.com/assafelovic/gpt-researcher/commit/a091fd5e65ba6de3a7fda0ba6e9e22a2b73c2c5c