Scrapegraph-ai
Scrapegraph-ai copied to clipboard
Adding message parameter support for OpenAI models
OpenAI models uses the message parameter for the prompt. ScrapegraphAI also use this parameter to link to the prompt argument on scrapper invocation. However, sometimes when using openAI models, we need to make multiple prompts to better guide the response (like in this article and this documentation).
Is it possible to replace the standart scrapGraphAI prompt when providing message argument in the graph_config ?
Example :
import os
from scrapegraphai.graphs import SmartScraperGraph
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
system_prompt = "Provide output in valid JSON."
user_prompt = "List me all the news article with a brief description for each one."
graph_config = {
"llm": {
"api_key": OPENAI_API_KEY,
"model": "gpt-3.5-turbo",
"response_format": {"type": "json_object"},
"seed": 0,
"temperature": 0,
"messages": [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
],
},
"verbose": True,
}
smart_scraper_graph = SmartScraperGraph(
prompt=user_prompt,
# also accepts a string with the already downloaded HTML code
source="https://perinim.github.io/projects",
config=graph_config,
)
# TypeError: openai.resources.chat.completions.Completions.create() got multiple values for keyword argument 'messages'```