WebToEpub icon indicating copy to clipboard operation
WebToEpub copied to clipboard

Mtlarchive is having some 403 issues again.

Open Franzeeee opened this issue 1 year ago • 8 comments

The mtlarchive.com is having some 403 errors again. Even if i already did the captcha and already logged in, it's still giving me the 403 error.

Franzeeee avatar Aug 24 '24 05:08 Franzeeee

Looks like we've hit an issue with sending the cf cookies again; not entirely sure why this is hitting specifically on mtlarchive. I'll try to look into this in the evening if no one picks it up. The default request cookie in the domain appears to have cf_clearance after a new line - might be the reason. image

cf_clearance is completely missing.

Kiradien avatar Aug 24 '24 10:08 Kiradien

@Franzeeee Firefox works, Chrome is broken.

gamebeaker avatar Aug 24 '24 10:08 gamebeaker

Looks to me like it is going to be a bigger problem. I just tested https://www.scribblehub.com/ and it also stopped working. At the moment i am not sure what the problem is. It looks like something changed with the cookies: let name1 = "cf_clearance"; cookies = await chrome.cookies.getAll({name: name1}); returns all "old" wesites with cloudflare cookie. Example i get a cookie for discord -> visit discord -> execute js again now the cookie for discord isn't returned anymore. It looks to me like cloudflare changed some flags for the cookie cf_clearance: old: image new: image edit: I just visited an old issue: #1306 and in the comment i uploaded a screenshot where the flags are the same as in the "new" screenshot:

I think the problem is, that the cloudflare cookie is not sent with the request. If you open Inspect -> Network click the failed request -> cookies there is this message: Screenshot 2024-05-21 230645 Link from message: https://developers.google.com/privacy-sandbox/3pcd/chips?utm_source=devtools For fanfiction.net the missing cookies name is cf_clearance: grafik WebToEpub in firefox sends this cookie: grafik edit: @Kiradien i missed your comment sry.

Originally posted by @gamebeaker in https://github.com/dteviot/WebToEpub/issues/1306#issuecomment-2123450414

gamebeaker avatar Aug 24 '24 12:08 gamebeaker

It looks like something changed with the cookies:

It looks like the problem is in Chrome rather than the cookies themselves; I've been digging through the setting method and the following request returns an empty array. let cookies = await chrome.cookies.getAll({partitionKey: {topLevelSite: topLevelSite}});

It shouldn't be a permission issue, as the old standby does still return cookies... however it does not include the ones we need. await chrome.cookies.getAll({url: 'https://mtlarchive.com'})

Edit: After fiddling around with the cookie itself, the parameter Partition Key Site seems to be the source of issues. The moment this value is set, all requests seem to ignore the cookies very existence. Doing a basic getAll with no properties shows how it appears/disappears depending on how it's set. image

This is either a permissions thing, in which I can't find the setting required, or Google has completely cut out access to these items.

No idea where to go from here -_-

Kiradien avatar Aug 24 '24 13:08 Kiradien

I have opened an issue with chromium: https://issues.chromium.org/issues/361862746

gamebeaker avatar Aug 24 '24 18:08 gamebeaker

@Kiradien @dteviot would be cool if you guys could vote "I am impacted" in the issue tracker: https://issues.chromium.org/issues/361862746

gamebeaker avatar Aug 24 '24 23:08 gamebeaker

@Franzeeee Test versions for Firefox and Chrome have been uploaded to https://github.com/dteviot/WebToEpub/releases/tag/developer-build. Pick the one suitable for you, follow the "How to install from Source (for people who are not developers)" instructions at https://github.com/dteviot/WebToEpub/tree/ExperimentalTabMode#user-content-how-to-install-from-source-for-people-who-are-not-developers and let me know how it goes. Tested with:

  • https://mtlarchive.com/novel/traveling-through-thousands-of-realms-starting-from-obtaining-the-ten-marvel-rings

gamebeaker avatar Aug 25 '24 12:08 gamebeaker

Chrome code fix got merged https://issues.chromium.org/issues/361862746.

gamebeaker avatar Sep 08 '24 14:09 gamebeaker

@Franzeeee Updated version (1.0.0.0) has been submitted to Firefox and Chrome stores. Firefox version is available now. Chrome might be available in a few hours to 21 days.

dteviot avatar Sep 26 '24 08:09 dteviot