[BUG] curl to https://clubhubuk.co.uk/club-search/... breaks with a 500 error

[calebpeffer]

Here is the curl request and response I'm getting that is giving the 500 error:

Curl -v --request POST
--url https://api.firecrawl.dev/v1/crawl
--header 'Authorization: Bearer <Removed>'
--header 'Content-Type: application/json'
--data '{ "url": "https://clubhubuk.co.uk/club-search/?search_location_static=IG8+0ST&search_location=IG8%25200ST&advanced_lat=51.6123148&advanced_lng=0.0218892&search_age=3%2C4%2C5%2C6%2C7&category=&submit=Search", "maxDepth": 2, "ignoreSitemap": true, "limit": 10, "scrapeOptions": { "formats": [ "markdown", "links", ], "onlyMainContent": true, } }'

Response:

Note: Unnecessary use of -X or --request, POST is already inferred.

• Host api.firecrawl.dev:443 was resolved.
• IPv6:<Removed>
• IPv4:
• Trying [2a09:8280:1::30:df95:0]:443...
• Connected to api.firecrawl.dev (2a09:8280:1::30:df95:0) port 443
• ALPN: curl offers h2,http/1.1
• (304) (OUT), TLS handshake, Client hello (1):
• CAfile: /etc/ssl/cert.pem
• CApath: none
• (304) (IN), TLS handshake, Server hello (2):
• (304) (IN), TLS handshake, Unknown (8):
• (304) (IN), TLS handshake, Certificate (11):
• (304) (IN), TLS handshake, CERT verify (15):
• (304) (IN), TLS handshake, Finished (20):
• (304) (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / / [blank] / UNDEF
• ALPN: server accepted h2
• Server certificate:
• subject: CN=api.firecrawl.dev
• start date: Aug 5 23:25:47 2024 GMT
• expire date: Nov 3 23:25:46 2024 GMT
• subjectAltName: host "api.firecrawl.dev" matched cert's "api.firecrawl.dev"
• issuer: C=US; O=Let's Encrypt; CN=E5
• SSL certificate verify ok.
• using HTTP/2
• [HTTP/2] [1] OPENED stream for https://api.firecrawl.dev/v1/crawl
• [HTTP/2] [1] [:method: POST]
• [HTTP/2] [1] [:scheme: https]
• [HTTP/2] [1] [:authority: api.firecrawl.dev]
• [HTTP/2] [1] [:path: /v1/crawl]
• [HTTP/2] [1] [user-agent: curl/8.7.1]
• [HTTP/2] [1] [accept: /]
• [HTTP/2] [1] [authorization: <Removed>
• [HTTP/2] [1] [content-type: application/json]
• [HTTP/2] [1] [content-length: 343]

POST /v1/crawl HTTP/2 Host: api.firecrawl.dev User-Agent: curl/8.7.1 Accept: / Authorization: Bearer Content-Type: application/json Content-Length: 343

• upload completely sent off: 343 bytes < HTTP/2 500 < x-powered-by: Express < content-type: application/json; charset=utf-8 < content-length: 160 < etag: W/"a0-pFvQdbGCiEVH5UwNq+HPrtga9Tg" < date: Tue, 03 Sep 2024 01:34:02 GMT < server: Fly/81d92dece (2024-08-28) < via: 2 fly.io < fly-request-id: 01J6TR8X48C9BTKD1DYPZFGR1V-sjc <
• Connection #0 to host api.firecrawl.dev left intact {"success":false,"error":"An unexpected error occurred. Please contact [email protected] for help. Your exception ID is d1f26be7-e81e-48fc-9738-6b624ce0f189"}