e621ng
e621ng copied to clipboard
Source alternate handlers & cleanup: FA, DA, IB, Twitter, Weasyl
As discussed in this thread here: https://e621.net/forum_topics/34730
I've improved the FurAffinity source handler, and added source handlers for Deviantart, Twitter, Inkbunny, and Weasyl.
I've added to the Source::Alternate
handlers, and added new ones. I've add unit tests for them, and added a fix script to update a large amount of broken sources.
Full list of the checks added:
- Base: Convert known secure domains to https [also in fix script]
- Deviantart: Add gallery link via submission link
- Deviantart: Convert old format links (artist.deviantart.com) to new format (deviantart.com/artist) [also in fix script]
- Furaffinity: Convert old/broken cdn links (d.facdn.net, d2.facdn.net) to new domain (d.furaffinity.net) [also in fix script]
- Furaffinity: Convert
/full/
links to/view/
links [also in fix script] - Furaffinity: Remove
?upload-successful
queries from URLs [also in fix script] - Furaffinity: Remove comment anchors
#cid:
from URLs [also in fix script] - Inkbunny: Remove page anchors
#pictop
from URLs [also in fix script] - Twitter: Convert mobile links (mobile.twitter.com) to main domain (twitter.com) [also in fix script]
- Twitter: Convert FixTweet links (fxtwitter.com, ayytwitter.com, vxtwitter.com, etc) to main domain (twitter.com) [also in fix script]
- Twitter: Remove tracking data from links [also in fix script]
- Twitter: Convert old direct links (
file.jpg:orig
) to new format (file?format=jpg&name=orig
) [also in fix script] - Weasyl: Add gallery links from submission links
And other changes:
- Moved
force_https
check after parsing URL, to allow base handler to force https on other domains without a specific handler - Moved source length truncation to post.rb, so all source links get truncated, not just unhandled ones
- Updated direct image link parser in FurAffinity source handler
- Stopped FurAffinity source handler from duplicating direct link
- Fixed adding direct_url to list of sources
- Moved null source strategy test into
tests/unit/sources/strategies
so that alternates tests can go in a sibling directory - Converting some post tests to use https scheme. Fixing one of them
I've tested this with the included unit tests, and deployed a local instance with these changes and ensured new sources get modified as planned. Also tried spinning up a local instance on the master
branch and running the fix script, which behaved as designed.
Running the full suite of tests, there are a bunch of failures, but all look unrelated to this.
1319 runs, 2322 assertions, 32 failures, 10 errors, 0 skips
I checked a couple that looked like they could be relevant, but they were not. Fixed one of them anyway.