intellectual Cloudflare blocking instance

Is there an existing issue for this?

[X] I have searched the existing issues

Are you using the latest version of Intellectual?

[X] I am using the latest version of Intellectual.

Describe the bug

Using the docker image

Internal Error Content type error

If this continues to occur, please report it.

Anything else?

I am using Cloudflare.

Jul 01 '23 21:07 Jeyso215

Is this reproducible on the official instance? If so, what URL is it occurring at?

Jul 01 '23 21:07 Insprill

Is this reproducible on the official instance? If so, what URL is it occurring at?

no, https://sing.whateveritworks.us

Jul 01 '23 21:07 Jeyso215

@Insprill

Jul 02 '23 01:07 Jeyso215

Please do not repeatedly ping me. I work on this project in my free time. I'll get back to you when I can. With that said, is there anything in the logs?

Jul 02 '23 05:07 Insprill

21:28:50 [INFO] Starting Intellectual v0.5.0, listening on 0.0.0.0:8080!
21:28:50 [INFO] starting 6 workers
21:28:50 [INFO] Actix runtime found; starting in Actix runtime
21:29:03 [ERROR] Content type error
21:29:08 [ERROR] Content type error
21:31:43 [ERROR] Content type error
21:31:49 [ERROR] Content type error
21:33:36 [ERROR] Content type error
21:34:07 [ERROR] Content type error
21:59:01 [ERROR] Content type error
22:19:27 [ERROR] Content type error
22:40:03 [ERROR] Content type error
01:10:39 [ERROR] Content type error
05:04:15 [ERROR] Content type error
05:04:36 [ERROR] Content type error
05:32:14 [ERROR] Content type error

Jul 02 '23 13:07 Jeyso215

I'm not sure why this would be happening, as I can't reproduce it locally. If you try to fetch https://genius.com/api/songs/2875511 from the same machine hosting Intellectual, what does Genius respond with?

Jul 07 '23 21:07 Insprill

https://genius.com/api/songs/2875511

21:28:50 [INFO] Starting Intellectual v0.5.0, listening on 0.0.0.0:8080!
21:28:50 [INFO] starting 6 workers
21:28:50 [INFO] Actix runtime found; starting in Actix runtime
21:29:03 [ERROR] Content type error
21:29:08 [ERROR] Content type error
21:31:43 [ERROR] Content type error
21:31:49 [ERROR] Content type error
21:33:36 [ERROR] Content type error
21:34:07 [ERROR] Content type error
21:59:01 [ERROR] Content type error
22:19:27 [ERROR] Content type error
22:40:03 [ERROR] Content type error
01:10:39 [ERROR] Content type error
05:04:15 [ERROR] Content type error
05:04:36 [ERROR] Content type error
05:32:14 [ERROR] Content type error
20:00:18 [ERROR] Content type error
20:00:22 [ERROR] Content type error
14:47:09 [INFO] SIGTERM received; starting graceful shutdown
14:47:09 [INFO] accept thread stopped
14:47:09 [INFO] shutting down idle worker
14:47:09 [INFO] shutting down idle worker
14:47:09 [INFO] shutting down idle worker
14:47:09 [INFO] shutting down idle worker
14:47:09 [INFO] shutting down idle worker
14:47:09 [INFO] shutting down idle worker
14:47:50 [INFO] Starting Intellectual v0.5.0, listening on 0.0.0.0:8080!
14:47:50 [INFO] starting 6 workers
14:47:50 [INFO] Actix runtime found; starting in Actix runtime
20:58:09 [ERROR] Content type error
21:05:49 [ERROR] Content type error
02:46:32 [ERROR] Content type error
02:47:04 [ERROR] Content type error

Jul 08 '23 02:07 Jeyso215

Sorry, not in Intellectual. On the machine you're hosting Intellectual on, run the following command.

curl -s -I https://genius.com/api/songs/2875511

And send the output.

Jul 08 '23 03:07 Insprill

Sorry, not in Intellectual. On the machine you're hosting Intellectual on, run the following command.
curl -s -I https://genius.com/api/songs/2875511
And send the output.

curl -s -I https://genius.com/api/songs/2875511

HTTP/2 403 
date: Sat, 08 Jul 2023 04:28:37 GMT
content-type: text/html; charset=UTF-8
cf-chl-bypass: 1
cross-origin-embedder-policy: require-corp
cross-origin-opener-policy: same-origin
cross-origin-resource-policy: same-origin
permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
referrer-policy: same-origin
x-frame-options: SAMEORIGIN
cf-mitigated: challenge
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
server: cloudflare
cf-ray: 7e3594db5ef6bbf2-FRA

Jul 08 '23 04:07 Jeyso215

Ah, it looks like your instance is being challenged by Cloudflare. I'll look into workarounds.

Jul 08 '23 04:07 Insprill

Ah, it looks like your instance is being challenged by Cloudflare. I'll look into workarounds.

Ah, I see my dumbass lol… genius is using cloudflare.com which means dumb and your genius privacy frontend might be not able to work. If they are using bot protection + their own custom WAF rules.

EDIT: But i also a method in mind, You can make the code serverless code and automatic deploy in Cloudflare workers which will bypass genius Cloudflare which they probably are using Cloudflare ipv4/ipv6 IP addresses

Jul 08 '23 05:07 Jeyso215

Please don't repeatedly ping me. As I've said, I work on this in my free time, and this isn't a trivial issue to solve. I'll need to do some research on how to reliably get around Cloudflare, then implement it. If anyone has experience with similar issues, insight or PRs are welcome!

Jul 29 '23 22:07 Insprill

Please don't repeatedly ping me. As I've said, I work on this in my free time, and this isn't a trivial issue to solve. I'll need to do some research on how to reliably get around Cloudflare, then implement it. If anyone has experience with similar issues, insight or PRs are welcome!

Workaround for Genius:

Proxy server to bypass Cloudflare protection https://github.com/FlareSolverr/FlareSolverr

Sep 03 '23 00:09 Jeyso215