xbvr icon indicating copy to clipboard operation
xbvr copied to clipboard

Added SLR scraper for VRLatina/HoloGirlsVR/VRPFilms/RealityLovers

Open CoolMintChocolate opened this issue 3 years ago • 8 comments

Adding SLR as an alternative source for VRLatina scenes, because there are scenes that are only on SLR and not on VRLatina website. If the user chooses to use both the SLR scraper and the dedicated VRLatina scraper, the scenes that come from SLR will have vrlatina-slr prefix in its id, while the scenes from the dedicated scraper will have its existing vrlatina prefix.

I had to change the SLR scraper a bit to make this prefix happen, in the current code base it's a convention that SLR scraper always uses the scraper Id in the scene Id, which will lead to vrlatina being used for both SLR & dedicated scraper, which might lead to ID duplications.

Edited: I've added support for HoloGirlsVR/VRPFilms/RealityLovers together with this PR, because they fundamentally do the same thing code-wise.

CoolMintChocolate avatar May 10 '21 19:05 CoolMintChocolate

How does the second scraper show up in XBVR? Ideally, the scraper table should contain two entries, "VRLatinas" and "VRLatinas (SLR)", both with a separate toggle and the ability to force update and delete.

Unfortunately, right now the force update/delete is buggy (fixed in #351 and #479) so please make sure this PR is compatible to those two PRs.

lockedmouse avatar May 11 '21 10:05 lockedmouse

I fixed the typo in the PR, thanks for the suggestion.

Here's how the two scrapers looks side by side: Screenshot

For the force-update/delete issue, I'm not sure what I can do on my side to make it work. I'm following the convention in the SLR scrapers, so if those don't work mine won't either, but it sounds like a bigger change is needed which is outside the scope for this PR?

CoolMintChocolate avatar May 11 '21 13:05 CoolMintChocolate

Oh, then my concern was unwarranted; this looks fine. 👍

I'm a little concerned that the scenes are prefixed by vrlatina- and vrlatina-slr-, so it will be difficult to query for scenes of the dedicated VRLatinas scraper without also getting SLR scenes. Maybe change it to vrlatina-main- or similar?

lockedmouse avatar Jun 06 '21 06:06 lockedmouse

I wouldn't be opposed to that at all - the only possible concern I have is that we might have to migrate existing data to new studio id, right? I'm not overly familiar with the database structure at the moment, so that's just a guess. If we do have to migrate data, is there some code in the current code base that did something similar before so I can repurpose it for this PR?

CoolMintChocolate avatar Jun 06 '21 15:06 CoolMintChocolate

@lockedmouse @crwxaj I have added migration path for existing scenes scraped by the existing VRLatina scraper. From now on scenes scraped by that scraper will have vrlatina-site- prefix, while scenes scraped by SLR scraper will have vrlatina-slr- prefix. I have also tested that deleting scenes scraped by one scraper does not affect scenes scraped by the other scraper.

CoolMintChocolate avatar Sep 16 '21 17:09 CoolMintChocolate

I've also added HoloGirlsVR/VRPFilms/RealityLovers SLR scrapers to this PR as well, they fundamentally do the same thing as the VRLatina SLR scraper

CoolMintChocolate avatar Sep 17 '21 00:09 CoolMintChocolate

In the migrations, actions also need to be updated to refer to scenes with their new IDs. Also scene's ID changes it needs to be re-indexed, so you could kick off the indexing at the end of the migration. See #586 for an example.

ghost avatar Oct 04 '21 04:10 ghost

Thanks for pointing that out @alexnsfw. I've made the changes necessary to address your concerns.

CoolMintChocolate avatar Oct 10 '21 15:10 CoolMintChocolate

Closing this PR since this feature is now available out of the box

CoolMintChocolate avatar Feb 15 '23 03:02 CoolMintChocolate