e621ng icon indicating copy to clipboard operation
e621ng copied to clipboard

Source alternate handlers & cleanup: FA, DA, IB, Twitter, Weasyl

Open Deer-Spangle opened this issue 1 year ago • 34 comments

As discussed in this thread here: https://e621.net/forum_topics/34730

I've improved the FurAffinity source handler, and added source handlers for Deviantart, Twitter, Inkbunny, and Weasyl. I've added to the Source::Alternate handlers, and added new ones. I've add unit tests for them, and added a fix script to update a large amount of broken sources.

Full list of the checks added:

  • Base: Convert known secure domains to https [also in fix script]
  • Deviantart: Add gallery link via submission link
  • Deviantart: Convert old format links (artist.deviantart.com) to new format (deviantart.com/artist) [also in fix script]
  • Furaffinity: Convert old/broken cdn links (d.facdn.net, d2.facdn.net) to new domain (d.furaffinity.net) [also in fix script]
  • Furaffinity: Convert /full/ links to /view/ links [also in fix script]
  • Furaffinity: Remove ?upload-successful queries from URLs [also in fix script]
  • Furaffinity: Remove comment anchors #cid: from URLs [also in fix script]
  • Inkbunny: Remove page anchors #pictop from URLs [also in fix script]
  • Twitter: Convert mobile links (mobile.twitter.com) to main domain (twitter.com) [also in fix script]
  • Twitter: Convert FixTweet links (fxtwitter.com, ayytwitter.com, vxtwitter.com, etc) to main domain (twitter.com) [also in fix script]
  • Twitter: Remove tracking data from links [also in fix script]
  • Twitter: Convert old direct links (file.jpg:orig) to new format (file?format=jpg&name=orig) [also in fix script]
  • Weasyl: Add gallery links from submission links

And other changes:

  • Moved force_https check after parsing URL, to allow base handler to force https on other domains without a specific handler
  • Moved source length truncation to post.rb, so all source links get truncated, not just unhandled ones
  • Updated direct image link parser in FurAffinity source handler
  • Stopped FurAffinity source handler from duplicating direct link
  • Fixed adding direct_url to list of sources
  • Moved null source strategy test into tests/unit/sources/strategies so that alternates tests can go in a sibling directory
  • Converting some post tests to use https scheme. Fixing one of them

I've tested this with the included unit tests, and deployed a local instance with these changes and ensured new sources get modified as planned. Also tried spinning up a local instance on the master branch and running the fix script, which behaved as designed. Running the full suite of tests, there are a bunch of failures, but all look unrelated to this. 1319 runs, 2322 assertions, 32 failures, 10 errors, 0 skips I checked a couple that looked like they could be relevant, but they were not. Fixed one of them anyway.

Deer-Spangle avatar Aug 06 '22 14:08 Deer-Spangle