Reuters returns 401

[acidus99]

Reuters currently returns a 401. Looks like there are detecting that a non-human, non-browser is access them. Even when mimicking the headers of modern Safari I still get blocked by datadome. Oddly though, curl does not get blocked...

What .NET is sending (+ a host header):

=== Request Headers ===
GET https://www.reuters.com/world/ HTTP/2.0
User-Agent: Mozilla/5.0, (Macintosh; Intel Mac OS X 14_0), AppleWebKit/605.1.15, (KHTML, like Gecko), Version/17.0, Safari/605.1.15
Accept: text/html, application/xhtml+xml, application/xml; q=0.9, */*; q=0.8
Accept-Language: en-US, en; q=0.9
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1

=== Response Headers ===
HTTP/1.1 401 Unauthorized
Connection: keep-alive
Server: CloudFront
Date: Thu, 02 Oct 2025 22:48:13 GMT
x-datadome: protected
accept-ch: Sec-CH-UA,Sec-CH-UA-Mobile,Sec-CH-UA-Platform,Sec-CH-UA-Arch,Sec-CH-UA-Full-Version-List,Sec-CH-UA-Model,Sec-CH-Device-Memory
charset: utf-8
Cache-Control: no-store, must-revalidate, no-cache, max-age=0, private
Pragma: no-cache
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: x-dd-b, x-set-cookie
Access-Control-Allow-Origin: *
x-datadome-cid: AHrlqAAAAAMA2r_kpJHz-j4Aoo5oHg==
x-dd-b: 3
Set-Cookie: datadome=~MFqW9BfPT7zFeWXSCGTZCRH0RdciZNCF3JpdNhUMuIQjsnLW2RANwmNUHG0f2nTCW4vPTgUggn62uWDnjYzaFO8Pcmn58vzm4R5k6sEqFGEE7~syMP8qOjCfC1Hj26b; Max-Age=31536000; Domain=.reuters.com; Path=/; Secure; SameSite=Lax
X-Cache: LambdaGeneratedResponse from cloudfront
Via: 1.1 e41d0925024fb5d00109c45acc3a4290.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: ATL59-P13
X-Amz-Cf-Id: MmwfcGCtDs7pGnTlFxDC-VTKXZTSNtHFy4MkE8PKmOp9rX8gR5lTLg==
Content-Security-Policy: frame-ancestors 'self'; report-uri https://reuters.report-uri.com/r/t/csp/enforce; report-to report-uri
Report-To: {"endpoints":[{"url":"https://reuters.report-uri.com/a/t/g"}],"group":"report-uri","include_subdomains":true,"max_age":31536000}
Content-Type: text/html; charset=utf-8
Content-Length: 774

What Curl is sending/getting:

> HEAD /world/ HTTP/2
> Host: www.reuters.com
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> Accept-Language: en-US,en;q=0.9
> Accept-Encoding: gzip, deflate, br
> Connection: keep-alive
> Upgrade-Insecure-Requests: 1
> Sec-Fetch-Dest: document
> Sec-Fetch-Mode: navigate
> Sec-Fetch-Site: none

curl --head  "https://www.reuters.com/world/"   -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15"   -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"   -H "Accept-Language: en-US,en;q=0.9"   -H "Accept-Encoding: gzip, deflate, br"   -H "Connection: keep-alive"   -H "Upgrade-Insecure-Requests: 1"   -H "Sec-Fetch-Dest: document"   -H "Sec-Fetch-Mode: navigate"   -H "Sec-Fetch-Site: none" 
HTTP/2 200 
content-type: text/html; charset=utf-8
content-length: 61385
server: openresty
x-arc-pb-request-id: e50b80fe-c51a-4047-95f3-8a7278bb670c
content-encoding: gzip
etag: W/"5742f-1HbraiWmax6xY6ZY6I/7DjEy3rw"
x-arc-pb-mx-id: 00000000
last-modified: Sat, 04 Oct 2025 01:43:53 GMT
mpulse_cdn_cache: HIT
mpulse_origin_time: 0
cache-control: private, max-age=60
expires: Sat, 04 Oct 2025 01:53:01 GMT
date: Sat, 04 Oct 2025 01:52:01 GMT
server-timing: ak_p; desc="1759542721260_389506223_938116562_5268_16699_6_0_-";dur=1
set-cookie: reuters-geo={"country":"-", "region":"-"}; path=/; secure
content-security-policy: frame-ancestors 'self'; report-uri https://reuters.report-uri.com/r/t/csp/enforce; report-to report-uri
strict-transport-security: max-age=31536000
x-arc-ttl: 900
x-arc-request-id: 0.af643717.1759542721.37ea85d2
vary: Accept-Encoding
x-cache: Miss from cloudfront
via: 1.1 6aaf059c8fb1a1e31354f1b3cdcd9c90.cloudfront.net (CloudFront)
x-amz-cf-pop: ATL59-P13
x-amz-cf-id: 7RljuceU8PCtSdg96NM4bhju7jbFHL8Wdgr5IRPi0kkbel2QqAoRFw==

Even forcing .NET to use H2, I'm still getting detected/blocked by datadome. I'm running from the same IP, and using Proxyman, I see the headers are identical (though not the right order). Perhaps this is TLS fingerprinting?

[acidus99]

BTW, it's not an IP filtering/blocking thing. This blocking happening from my local machine. curl works from both my local machine, and from the server that runs gemi.dev also works.

It could be header order, or fingerprinting the client's TLS handshake.