morphic
morphic copied to clipboard
miurla
Enable deep searching of specific URLs
When a specific URL is included in the query, retrieve the data from that URL instead of searching.
Firecrawl is pretty slow when I tested it Is this fine? I'm sure the user would understand as long as we provide a spinner
Ahh nevermind, the crawl api which scrapes multiple pages is slow, just scraping one page is decent
It's a good try! I got the following advice on our Discord. I'm going to try this.
You don't need firecrawl for this, IMHO, can just use Playwright/fetch and then use turndown to convert html to markdown
Firecrawl uses playwright + turndown 😆 We can also use this one https://jina.ai/reader
I'll give it a shot, already got the researcher able to refer to the previous sources it pulled, just that the types are a bit tricky. Just need to integrate those API's
I'll work on this one first and I'll see if I can get back to video section next week, it's been busy at work
Thank you. I was stuck with the chat history feature. I'll be able to start working on a different task soon.
Hey @miurla, fixed the types related to enabling follow up conversations and also found a way to enable function calling with groq so now I'm moving forward with adding FireCrawl, planning to push a PR by Sunday!
@albertdbio Wow, that's great. Is the function calling an implementation that uses groq's sdk to replace the researcher's model? Looking forward to the PR.
@albertdbio I've merged significant changes, so it's best to develop from main.
@miurla not using the groq sdk, it turns out the vercel ai sdk needed to be upgraded to a minor version. That minor version set's content as optional in the OpenAI zod schema. Groq doesn't return content when doing function calls so it would throw an invalid response error. Going to test it out with the inquire agent to verify that it's working. Sounds good, I'll use main!
Got caught up with work over the weekend. Will continue chipping away at it this week. First I'll rebase on to main.