urly.fi (cloudflare) always returning 404

[hb9eue]

Hi, very usefull module.

At the moment I come acress a lot of spam using Cloudflare's shortener urly.fi

Esample from a spam mail: https://urly.fi/1cAF

When accessed manually thought openssl I get a result:

$ openssl s_client -connect urly.fi:443 -servername urly.fi [...] GET /1cAF HTTP/1.0 Host: urly.fi

HTTP/1.1 301 MOVED PERMANENTLY Date: Fri, 17 May 2019 13:40:58 GMT Content-Type: text/html; charset=utf-8 Connection: close Set-Cookie: __cfduid=d2d87edbcdde7c2e41c88b24941f433d01558100458; expires=Sat, 16-May-20 13:40:58 GMT; path=/; domain=.urly.fi; HttpOnly Location: https://besthotvalentines.com/qjqumernhzkmksy?t=goam Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" Server: cloudflare CF-RAY: 4d8601770ad0cc36-ZRH

But when I look at the debug output of SpamAssassin firing this module:

May 17 15:54:17.940 [11506] dbg: DecodeShortURLs: URL is not redirect: https://urly.fi/1cAF = 403 Forbidden

So I wonder where this 403 forbidden comes from. Does CloudFlare recognize the 'LWP' User agent and distrusts it? Is it a https problem?

-Benoît-

[MASHtm]

I had the same effect with tiny.cc. I think they block some user-agents and "libwww" is on this list. I added $self->{ua}->agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0'); to the LWP::UserAgent initialization and then it works.