firecrawl icon indicating copy to clipboard operation
firecrawl copied to clipboard

[Self-Host] /v1/crawl/ endpoint error when no supabase

Open daniel5gh opened this issue 3 weeks ago • 4 comments

Describe the Issue I have a docker compose setup as described in SELF_HOST.md. I have not configured supabase or authentication.

When I hit the crawl status endpoint I see the following error:

error [:]: Error occurred in request! (/v1/crawl/6ebe383b-742b-4098-9c41-a1213cdce1df) -- ID 35396e67f7b54070855f1eec0ac31994 -- {"message":"Supabase client is not configured.","name":"Error","stack":"Error: Supabase client is not configured.\n    at Proxy.<anonymous> (/app/dist/src/services/supabase.js:41:23)\n    at crawlStatusController (/app/dist/src/controllers/v1/crawl-status.js:154:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)"} {}

To Reproduce Steps to reproduce the issue:

  1. Configure the environment or settings without SUPABASE_URL (or I am using an empty value)
  2. Start the stack with docker compose up
  3. POST to http://localhost:3002/v1/crawl with a valid job
  4. GET status endpoint with ID from previous step: http://localhost:3002/v1/crawl/6ebe383b-742b-4098-9c41-a1213cdce1df
  5. Observe the error on the api service container: see description above

Expected Behavior I expect a status 200 and a crawl result.

Environment (please complete the following information):

  • OS: docker on WSL
  • Firecrawl Version: 5894076
  • Node.js Version: node:23-slim
  • Docker Version (not applicable): 27.4.0
  • Database Type and Version: n/a

Logs

error [:]: Error occurred in request! (/v1/crawl/6ebe383b-742b-4098-9c41-a1213cdce1df) -- ID 35396e67f7b54070855f1eec0ac31994 -- {"message":"Supabase client is not configured.","name":"Error","stack":"Error: Supabase client is not configured.\n    at Proxy.<anonymous> (/app/dist/src/services/supabase.js:41:23)\n    at crawlStatusController (/app/dist/src/controllers/v1/crawl-status.js:154:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)"} {}

Additional Context I have solved the issue by using the following check, going off what is done in other endpoints (crawlCancelController for example) this seems the right thing to do when no supabase is configured:

  const useDbAuthentication = process.env.USE_DB_AUTHENTICATION === "true";

  if (useDbAuthentication && totalCount === 0) {
    const x = await supabase_service
      .from('firecrawl_jobs')
      .select('*', { count: 'exact', head: true })
      .eq("crawl_id", req.params.jobId)
      .eq("success", true)
    
    totalCount = x.count ?? 0;
  }

at https://github.com/mendableai/firecrawl/blob/5894076fda71ef320dcb0e7dcee34ce589059522/apps/api/src/controllers/v1/crawl-status.ts#L247-L255

daniel5gh avatar Feb 03 '25 23:02 daniel5gh