Removing deleted resources

[irfan-dahir]

Jikan needs to be able to keep track of pending entries and make sure they're not viewable in search.

Additionally, entries may be removed by MAL which end up giving a 404 where-as they still show up in search and as a resource.

References: https://discord.com/channels/460491088004907029/475889826852503554/994158995264786532 https://discord.com/channels/460491088004907029/989255412358451240/990855325043261440

[pushrbx]

We might want to have versioning on the Ids we are downloading at the time of running the anime:indexer manga:indexer, and the next time it runs, take a diff, and remove the elements from the db which has been removed from the id cache.

[irfan-dahir]

@pushrbx That's a very solid idea. I think that may be the best approach 👍

[pushrbx]

I imagine it like this:

private function fetchMalIds() : array
{
  // ...
  $existingStuff = Storage::get('indexer/anime_mal_id.json');
  if ($existingStuff) {
    $a = json_decode($existingStuff, true);
    $b = json_decode(
            file_get_contents('https://raw.githubusercontent.com/seanbreckenridge/mal-id-cache/master/cache/anime_cache.json'),
            true
        );
    $b = $b['sfw'] + $b['nsfw'];
    $diff = array_diff($a, $b);
    remove_diff_from_db($diff);
  }
  // ...
}

[irfan-dahir]

That's a suitable approach. I think it would be best to implement this as a separate artisan command for this and add it to the scheduler as a daily task.

e.g php artisan indexer:anime-remove-deleted, php artisan indexer:manga-remove-deleted