firecrawl plugin request error with system prompt

[22Goose]

Self Checks

• [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [x] I have searched for existing issues search for existing issues, including closed ones.
• [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [x] Please do not modify this template :) and fill in all the required fields.

Dify version

Version 1.0.0 - offical website

Cloud or Self Hosted

Cloud

Steps to reproduce

I was using firecrawl plugin with single scrape to get my content. when i use the system prompt or prompt, i got this error

{
  "text": "",
  "files": [],
  "json": [
    {
      "details": [
        {
          "code": "custom",
          "message": "When 'extract' or 'json' format is specified, corresponding options must be provided, and vice versa",
          "path": []
        }
      ],
      "error": "Bad Request",
      "success": false
    }
  ]
}

when i get away with my prompt, it will be seccess. i have tested on several website, all turn into the same

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[dosubot[bot]]

Hello, @22Goose! I'm here to assist you with any bugs, questions, or contributions you might need help with. Let's work together to resolve the issue you're facing!

The error message you're encountering indicates that when using the 'extract' or 'json' format with the Firecrawl plugin, specific configuration options must be provided. This means that if you specify either of these formats, you need to ensure that all required options for that format are correctly set up. Unfortunately, the specific configuration options required for these formats are not detailed in the available documentation or knowledge base.

To resolve this issue, you should:

1. Double-check the configuration settings for the 'extract' or 'json' format to ensure all necessary options are included.
2. Verify that the system prompt or any other prompts used are correctly formatted and compatible with the Firecrawl plugin's requirements.
3. Consider reviewing any available documentation or examples related to the Firecrawl plugin to ensure compliance with its expected input and output formats.

If the problem persists, you might need to explore further documentation or community discussions for more detailed guidance on configuring these formats.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[crazywoola]

I think you did not turn on the switch in the model's settings?

[22Goose]

I think you did not turn on the switch in the model's settings?

what do you mean? firecrawl settings? or other LLM models'?

it wents wrong when i only run the firecrawl node, so basically it does nothing wrong with the LLM model's setting

[22Goose]

I think you did not turn on the switch in the model's settings?

i have try with the http request with the firecrawl cURL function, it wents well when use json format. i was concerned that the firecrawl node didn't real set the 'format' into 'json', so when node request the api, firecrawl response this part ‘When 'extract' or 'json' format is specified, corresponding options must be provided, and vice versa’

{
  "text": "",
  "files": [],
  "json": [
    {
      "details": [
        {
          "code": "custom",
          "message": "When 'extract' or 'json' format is specified, corresponding options must be provided, and vice versa",
          "path": []
        }
      ],
      "error": "Bad Request",
      "success": false
    }
  ]
}

besides， here is the cURL seccuessful test case with the http request node, baybe you can try out where goes wrong

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer <YOUR_API_KEY>' \
    -d '{
      "url": "https://docs.firecrawl.dev/",
      "formats": ["json"],
      "jsonOptions": {
        "prompt": "Extract the company mission from the page."
      }
    }'

[zingrigger]

I think you did not turn on the switch in the model's settings?

i have try with the http request with the firecrawl cURL function, it wents well when use json format. i was concerned that the firecrawl node didn't real set the 'format' into 'json', so when node request the api, firecrawl response this part ‘When 'extract' or 'json' format is specified, corresponding options must be provided, and vice versa’

{
  "text": "",
  "files": [],
  "json": [
    {
      "details": [
        {
          "code": "custom",
          "message": "When 'extract' or 'json' format is specified, corresponding options must be provided, and vice versa",
          "path": []
        }
      ],
      "error": "Bad Request",
      "success": false
    }
  ]
}

besides， here is the cURL seccuessful test case with the http request node, baybe you can try out where goes wrong

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer <YOUR_API_KEY>' \
    -d '{
      "url": "https://docs.firecrawl.dev/",
      "formats": ["json"],
      "jsonOptions": {
        "prompt": "Extract the company mission from the page."
      }
    }'

I also encountered the same issue. While practicing the workflow from this article https://mp.weixin.qq.com/s/5ulE6cqhyHDNhhDT08hOXA in Dify 1.0, I imported the workflow configuration https://www.ginonotes.com/dify/wenrun.yml, which uses Firecrawl.

When running the 【single page scraping】 node alone, it gave the same error message. Later, I found that after clearing the value in 【systemPrompt】, it was able to correctly fetch what I wanted from Firecrawl.

[rosenjiang]

I think you did not turn on the switch in the model's settings?

i have try with the http request with the firecrawl cURL function, it wents well when use json format. i was concerned that the firecrawl node didn't real set the 'format' into 'json', so when node request the api, firecrawl response this part ‘When 'extract' or 'json' format is specified, corresponding options must be provided, and vice versa’

{
  "text": "",
  "files": [],
  "json": [
    {
      "details": [
        {
          "code": "custom",
          "message": "When 'extract' or 'json' format is specified, corresponding options must be provided, and vice versa",
          "path": []
        }
      ],
      "error": "Bad Request",
      "success": false
    }
  ]
}

besides， here is the cURL seccuessful test case with the http request node, baybe you can try out where goes wrong

curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer <YOUR_API_KEY>' \
    -d '{
      "url": "https://docs.firecrawl.dev/",
      "formats": ["json"],
      "jsonOptions": {
        "prompt": "Extract the company mission from the page."
      }
    }'

I also encountered the same issue. While practicing the workflow from this article https://mp.weixin.qq.com/s/5ulE6cqhyHDNhhDT08hOXA in Dify 1.0, I imported the workflow configuration https://www.ginonotes.com/dify/wenrun.yml, which uses Firecrawl.

When running the 【single page scraping】 node alone, it gave the same error message. Later, I found that after clearing the value in 【systemPrompt】, it was able to correctly fetch what I wanted from Firecrawl.

that's work fine. thx