firecrawl
How to set the api key when running locally?
Problem Description How to access the API key when launching the Firecrawl locally.
@artificialzjy when running locally, you should set USE_DB_AUTHENTICATION=false, and you can use any string value for the API key. Take a look at the self host guide and running locally guide.
Thank you! @rafaelsideguide Follow the self_host_guide and running_locally_guide， i set any API Key in the .env and run pnpm install. Then i fill Dify with API Key, but invoke a problem: Failed to authorize. Status code: 401. Error: Unauthorized: Invalid token.
The setting of .env is as followed:
===== Required ENVS ======
NUM_WORKERS_PER_QUEUE=8 PORT=3002 HOST=0.0.0.0 REDIS_URL=redis://localhost:6379
To turn on DB authentication, you need to set up supabase.
USE_DB_AUTHENTICATION=false
===== Optional ENVS ======
Supabase Setup (used to support DB authentication, advanced logging, etc.)
SUPABASE_ANON_TOKEN= SUPABASE_URL= SUPABASE_SERVICE_TOKEN=
Other Optionals
TEST_API_KEY=${API key:fc-9***(randomly generated)}$** #use if you've set up authentication and want to test with a real API key SCRAPING_BEE_API_KEY= #Set if you'd like to use scraping Be to handle JS blocking OPENAI_API_KEY= # add for LLM dependednt features (image alt generation, etc.) BULL_AUTH_KEY= # LOGTAIL_KEY= # Use if you're configuring basic logging with logtail PLAYWRIGHT_MICROSERVICE_URL= # set if you'd like to run a playwright fallback LLAMAPARSE_API_KEY= #Set if you have a llamaparse key you'd like to use to parse pdfs
The problem is as followed:
I am facing the same problem. Below is the error log from the docker-api-1 of Dify. This issue started occurring after updating both Firecrawl and Dify to the latest versions (Dify 0.7.1).
I am not sure whether this is caused by Dify or Firecrawl...
2024-08-21 15:38:05 api-1 | 2024-08-21 06:38:05,517.517 ERROR [Dummy-36] [app.py:838] - Exception on /console/api/api-key-auth/data-source/binding [POST]
2024-08-21 15:38:05 api-1 | Traceback (most recent call last):
2024-08-21 15:38:05 api-1 | File "/app/api/controllers/console/auth/data_source_bearer_auth.py", line 51, in post
2024-08-21 15:38:05 api-1 | ApiKeyAuthService.create_provider_auth(current_user.current_tenant_id, args)
2024-08-21 15:38:05 api-1 | File "/app/api/services/auth/api_key_auth_service.py", line 21, in create_provider_auth
2024-08-21 15:38:05 api-1 | auth_result = ApiKeyAuthFactory(args['provider'], args['credentials']).validate_credentials()
2024-08-21 15:38:05 api-1 | File "/app/api/services/auth/api_key_auth_factory.py", line 14, in validate_credentials
2024-08-21 15:38:05 api-1 | return self.auth.validate_credentials()
2024-08-21 15:38:05 api-1 | File "/app/api/services/auth/firecrawl.py", line 37, in validate_credentials
2024-08-21 15:38:05 api-1 | self._handle_error(response)
2024-08-21 15:38:05 api-1 | File "/app/api/services/auth/firecrawl.py", line 55, in _handle_error
2024-08-21 15:38:05 api-1 | raise Exception(f'Failed to authorize. Status code: {response.status_code}. Error: {error_message}')
2024-08-21 15:38:05 api-1 | Exception: Failed to authorize. Status code: 401. Error: Unauthorized: Invalid token
2024-08-21 15:38:05 api-1 |
2024-08-21 15:38:05 api-1 | During handling of the above exception, another exception occurred:
2024-08-21 15:38:05 api-1 |
2024-08-21 15:38:05 api-1 | Traceback (most recent call last):
2024-08-21 15:38:05 api-1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
2024-08-21 15:38:05 api-1 | rv = self.dispatch_request()
2024-08-21 15:38:05 api-1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
2024-08-21 15:38:05 api-1 | return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
2024-08-21 15:38:05 api-1 | File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
2024-08-21 15:38:05 api-1 | resp = resource(*args, **kwargs)
2024-08-21 15:38:05 api-1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/views.py", line 110, in view
2024-08-21 15:38:05 api-1 | return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return]
2024-08-21 15:38:05 api-1 | File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
2024-08-21 15:38:05 api-1 | resp = meth(*args, **kwargs)
2024-08-21 15:38:05 api-1 | File "/app/api/controllers/console/setup.py", line 75, in decorated
2024-08-21 15:38:05 api-1 | return view(*args, **kwargs)
2024-08-21 15:38:05 api-1 | File "/app/api/libs/login.py", line 93, in decorated_view
2024-08-21 15:38:05 api-1 | return current_app.ensure_sync(func)(*args, **kwargs)
2024-08-21 15:38:05 api-1 | File "/app/api/controllers/console/wraps.py", line 22, in decorated
2024-08-21 15:38:05 api-1 | return view(*args, **kwargs)
2024-08-21 15:38:05 api-1 | File "/app/api/controllers/console/auth/data_source_bearer_auth.py", line 53, in post
2024-08-21 15:38:05 api-1 | raise ApiKeyAuthFailedError(str(e))
2024-08-21 15:38:05 api-1 | controllers.console.auth.error.ApiKeyAuthFailedError: 500 Internal Server Error: Failed to authorize. Status code: 401. Error: Unauthorized: Invalid token
2024-08-21 15:38:05 nginx-1 | 172.21.0.1 - - [21/Aug/2024:06:38:05 +0000] "POST /console/api/api-key-auth/data-source/binding HTTP/1.1" 500 127 "http://localhost/datasets/create" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "-"
@artificialzjy it seems like you're sending the requests to api.firecrawl.dev instead of your local environment. Could you try changing the base URL on Dify and see if that works? Firecrawl usually runs on http://localhost:3002.
Don't you set USE_DB_AUTHENTICATION=true and prompt the Supabase client is not configured when using LLM Extract
Can't make /extract work locally. it seems docker is not recognizing my OPENAI base URL.
I've set USE_DB_AUTHENTICATION to false and set TEST_API_KEY, but whether it's an api endpoint or accessed through a locally built dify, the set key is invalid, the api endpoint doesn't need the Authorization header, and dify can be accessed with any key configured!
I've set USE_DB_AUTHENTICATION to false and set TEST_API_KEY, but whether it's an api endpoint or accessed through a locally built dify, the set key is invalid, the api endpoint doesn't need the Authorization header, and dify can be accessed with any key configured!
It can be confirmed that the same thing happens in my deployed Firecrawl.
@artificialzjy awesome! Closing this issue.太棒了！关闭此问题。
we have some problems, just liks :I've set USE_DB_AUTHENTICATION to false and set TEST_API_KEY, but whether it's an api endpoint or accessed through a locally built dify, the set key is invalid, the api endpoint doesn't need the Authorization header, and dify can be accessed with any key configured! how to fix it?
i use docker image i set USE_DB_AUTHENTICATION: false however i get 500 every time i try this code: import requests
url = "http://localhost:3002/v1/scrape" payload = { "url": "https://firecrawl.dev" }
response = requests.post(url, json=payload) print("Status Code:", response.status_code) print("Response:", response)
======================= or 401 when i use this code: from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="")
Scrape a website:
scrape_result = app.scrape_url( url='firecrawl.dev', formats=['markdown', 'html'] )
@artificialzjy when running locally, you should set
USE_DB_AUTHENTICATION=false, and you can use any string value for the API key. Take a look at the self host guide and running locally guide.
@artificialzjy does it mean when we self host, we cannot use USE_DB_AUTHENTICATION?
because I'm wondering how to set an API key if we want to use db authentication here.
what should we when we want to use authentication in self hosted? I'm already configured supabase though
