Add support for https://ranobes.net/

[samdiggidy]

[Seven0492]

Is there any news on this?

I might go looking for alternative sites, but ranobes has been my number 1 site for finding novels which have been stubbed from Royal Road so far, mainly because I like the interface, which, unfortunately, is increasingly common. (not my best phrasing, I'll be honest)

Or, is there any way I could contribute?

Granted, I'm not the best 'coder' out there, learned recently, but I've tried my hands at a few minor projects, mostly in Rust.

If there's a.. 'framework', or template, for lack of better words, to setup new sites for scraping, all I'd need is a documentation link and I'd figure it out, eventually. And when I figure it out I'll do a pull request.

~~Now, all I need is to find where the damned documentation is. (will update when I find it)~~

Edit: Found what I was looking for.

[Seven0492]

Well, it won't be too easy since ranobes ~~.net~~ .top (ranobes.net redirects to ranobes.top, usually) implements a very, very rudimentary form of CAPTCHA, that's so simple it can't be bypassed without breaking my head over it. (it doesn't do any kind of bot checks before prompting with a CAPTCHA, that means faking user-agents and the like won't cut it)

Perhaps the most sensible solution is just to use a headless browser to brute-force it, I tested and there's no limit to how much you can spam the boxes wrong until it hits, ironically.

But, the thing is, I don't know if dragging in additional dependencies would be received in a good way from the maintainer, so.. not sure how I want to proceed here.

[Seven0492]

@JimmXinu sorry for the ping, but what do you think about this?

I don't know what's your stance regarding sites who implement anti-bot measures, is it free for all to implement as long as it works, or do you reject them because they might drag in additional dependencies?

And, if you perceive pinging as rude, I'm sorry for the inconvenience, I didn't know, just thought you didn't reread every single issue in your repo consistently, which would be kind off inhuman.

At least I wouldn't unless I have a very good reason to.

Edit: Oh, and, unrelated question, I couldn't find any 'How to get started contributing' resources, or simply a 'CONTRIBUTING' file in the root of the project, as is standard for some. Is this intentional, or lack of interest?

I can 'scrape' (:wink:) pull requests, issues and the mobileread forum for information (which I've already started to do), but it doesn't really seem.. accessible, without someone to show you the ropes.

I git cloned and explored a bit, I get how adapters are structured, how they're imported and stuff.. but when it gets out of theory and I look at making my own.. I got the base template but it's not exactly clear what I should mess with, modify, and/or should ignore.

[JimmXinu]

A few points:

• Since you specifically mention 'stubbing', I previously commented in an issue about royalroad:

  A nontechnical issue is the FanFicFare is meant for fanfiction. If an author has removed their content to make money on it, I'm not interested in helping work around that.

• Translated novels sites are generally considerably more complex and javascript intensive than most fanfiction sites. I will admit that part of that is simply that they tend to be newer.

• I, personally, have no interest in translated novels sites. Either reading them or supporting them. I try to keep the ones we already have working, but I remove them now if becomes difficult.

IMO, at this point, downloading translated novel sites should be the job of some other tool. A quick search of github for download novels finds several candidates. Maybe you could look into one of those?

So far, I've left all these 'add site XYZ' issues open so I don't get repeats for the same sites. Perhaps it's time I make it official project policy that I'm not interested in adding them.

[Seven0492]

Thanks for the quick response! Much appreciate it.

Yeah, though that might have been an issue:

A nontechnical issue is the FanFicFare is meant for fanfiction. If an author has removed their content to make money on it, I'm not interested in helping work around that.

Honestly, I think it's a really nuanced subject concerning ethics, but I can respect that.

Yeah, I think I'll stick to what works with this plugin, or, if I really needed to I'd look into that, but I mainly wanted it because I was familiar with it, not because it's a good or ethical platform, so I'm willing to switch.

I think that'd have been a huge time saver:

So far, I've left all these 'add site XYZ' issues open so I don't get repeats for the same sites. Perhaps it's time I make it official project policy that I'm not interested in adding them.

I went and checked on the link your posted on the swank issue, but if contributions for these were gonna be implemented was left to speculation.

In any case, thanks for being patient with me, not everyone would be.

[Lagicrus]

@JimmXinu just wondering, if the community made a PR to add support for say this site or others you are personally not interested in adding. Would that be an option you'd allow?

[JimmXinu]

At this point I would probably lean against it for several reasons:

Copyright

Fanfiction authors, by the very nature of their works would have a hard time bringing copyright claims against anyone else.

Some of the translation sites claim to have relationships with the original publishers that make them legitimate distributors. Others don't and are presumably running their own risks. Either way, translated works appear to me to carry more legal risk than fanfiction.

I have rejected some sites before for hosting content I'm reasonably sure should be considered copyrighted in my country, at least. And that was based on searching for just one title that I happened to know had come into the US.

#440 has some additional past discussion on the topic.

Javascript / Complex Interactions

These tend to be more 'modern' sites using lots of javascript and complex browser/site interactions making it difficult or even practically impossible for FFF to handle. webnovel.com support was removed for that reason (#843).

(I don't know if ranobes.top in particular meets that description.)

Long Term Support

It's an unfortunate fact that code contributors don't usually stick around long term and then I end up doing the maintenance. And I don't have much personal motivation to maintain support for these sites.

[Lagicrus]

Hm, that makes sense, thanks for getting back to me 👍

Was mostly wondering as it was a focal point of multiple sources, so in theory adding support for one site Vs many would be an easier job. But I digress, thanks again