Scrapegraph-ai icon indicating copy to clipboard operation
Scrapegraph-ai copied to clipboard

Adding message parameter support for OpenAI models

Open Hkllopp opened this issue 8 months ago • 5 comments

OpenAI models uses the message parameter for the prompt. ScrapegraphAI also use this parameter to link to the prompt argument on scrapper invocation. However, sometimes when using openAI models, we need to make multiple prompts to better guide the response (like in this article and this documentation).

Is it possible to replace the standart scrapGraphAI prompt when providing message argument in the graph_config ?

Example :

import os
from scrapegraphai.graphs import SmartScraperGraph

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

system_prompt = "Provide output in valid JSON."
user_prompt = "List me all the news article with a brief description for each one."

graph_config = {
    "llm": {
        "api_key": OPENAI_API_KEY,
        "model": "gpt-3.5-turbo",
        "response_format": {"type": "json_object"},
        "seed": 0,
        "temperature": 0,
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
    },
    "verbose": True,
}

smart_scraper_graph = SmartScraperGraph(
    prompt=user_prompt,
    # also accepts a string with the already downloaded HTML code
    source="https://perinim.github.io/projects",
    config=graph_config,
)

# TypeError: openai.resources.chat.completions.Completions.create() got multiple values for keyword argument 'messages'```

Hkllopp avatar Jun 14 '24 10:06 Hkllopp