Scrapegraph-ai
Scrapegraph-ai copied to clipboard
ScrapeGraphAI
Python scraper based on AI
**Is your feature request related to a problem? Please describe.** Lots of LLMs support only 32k tokens. And many webpage content has tokens more than 32k. When I send the...
following lines should be moved to a unit testing file or be wrapped with main guards: https://github.com/VinciGit00/Scrapegraph-ai/blob/62fdbcca52012c34d714232b1f6987c83e841c62/scrapegraphai/utils/parse_state_keys.py#L106 I haven't checked the rest, but there might be other similar instances throughout...
**Describe the bug** When running the following code: ``` from langchain_openai import AzureChatOpenAI, AzureOpenAIEmbeddings import os from scrapegraphai.graphs import SmartScraperGraph, SearchGraph, SpeechGraph, SmartScraperMultiGraph os.environ["AZURE_OPENAI_ENDPOINT"] = "" os.environ["AZURE_OPENAI_API_KEY"] = "" llm_model_instance...
OpenAI models uses the message parameter for the prompt. ScrapegraphAI also use this parameter to link to the prompt argument on scrapper invocation. However, sometimes when using openAI models, we...
Love this library! Would you be open to adding an integration to Indexfiy(https://getindexify.ai) as a destination for the scraped data? Developers would be able to build complex pipelines on the...
I am in a situation where by i want to scrape a page that is behind auth, so i have to login first, how would i implement the use of...
**Is your feature request related to a problem? Please describe.** Is it possible to get all the dynamically loaded content from a url? E.g. getting all the reviews from IMDB...
it’s not apparent how to request a series of prompts on the knowledge graph. Like if I wanted to ask a series of questions. I don’t want to make multiple...
