Detach GameVault from strict coupling to RAWG Database, Implement Agnostic Metadata Approach, Implement IGDB as the new Default

[Alfagun74]

Description: Essential Predecessor Tickets:

• #252
• #161
• https://github.com/Phalcode/gamevault-app/issues/172

Problem: Our system currently relies solely on the RAWG Videogame Database. This limits our flexibility and scalability because we're tightly bound to RAWG's data structure. This makes it difficult to incorporate other metadata sources or adapt to changes in user needs.

Solution: This feature request suggests a complete redesign of how we integrate metadata. Instead of being tied to RAWG, we want to create a more flexible system that can work with various metadata databases. Here's what we plan to do:

1. Detach from RAWG: We'll rework the code so it doesn't rely directly on RAWG anymore, freeing us from its specific data format.

2. Create a Flexible Data Model: We'll design a versatile data structure that can handle metadata from different sources like Steam (#252), IGDB, and user-generated data (#161). This means figuring out a common structure that can accommodate all these sources.

3. Manage Metadata Elements: We'll establish ways to handle important metadata like Tags, Genres, Publishers, Stores, and Platforms within our new data model.

4. Smooth Migration: We'll create a detailed plan to shift from RAWG's data to our new system seamlessly. This involves moving data over, making sure everything stays intact, and minimizing any disruptions.

5. API for Database Integration: We'll look into making an API that allows users to add new metadata providers through server plugins (#140). This gives users the power to expand the system with new data sources as they see fit.

Open Questions:

• What should the new metadata model look like to handle various sources effectively?
• How can we best manage Tags, Genres, Publishers, Stores, and Platforms within the new data model for maximum flexibility?
• What steps do we need to take to transition smoothly from RAWG's data without causing any problems?
• Can we realistically create an API for users to add more database providers, and if so, what's the best way to do it?

[Alfagun74]

I can not and do not want to spoil too much, but as mentioned in the recent blog post, there are ongoing partner-contract-negotiations with IGDB/Twitch.

[Alfagun74]

Update: Their legal team needs approx. ~ 3-4 weeks to prepare the final contract.

We will use this time to plan how we are going to pull off this major change exactly.

[Alfagun74]

We need to seperate filedata (and in future config #148) from Metadata providers data. For example like:

Game File Entity

id: #
created_at: #
updated_at: #
deleted_at: #
entity_version: #
title: #
version: #
release_date: #
file_path: #
size: #
early_access: #

Game Metadata Entity

id: #Comes from DB
created_at: #Comes from DB
updated_at: #Comes from DB
deleted_at: #Comes from DB
entity_version: #Comes from DB
meta_source: "IGDB"
meta_title: "Cyberpunk 2077"
meta_description: "Cyberpunk 2077 is an bla bla bla..."
meta_release_date: "2020-12-10" 
meta_box_image: One2One Image Relation
meta_background_images: One2Many Image Relation
meta_links: One2Many Link Relation (Urls, Socials etc)
meta_rating: 69
meta_average_playtime: 2000
meta_tags: Many2Many Tags Relation
meta_genres: Many2Many Genres Relation
meta_developers: Many2Many Developers Relation
meta_stores: Many2Many Stores Relation #(JoinTable could include a storelink for a game)
meta_platforms: Many2Many Platforms Relation #(JoinTable could include a release date for a specific platform)

These Meta Objects could come from various sources like "CONFIG", "IGDB", "PLUGIN_VNDB", "IMDB", "USER". Then be ordered via config so the server knows which properties take priority. ["USER", "CONFIG", "IGDB", everything else] would be the default.

It would take first custom entered data of users to fullfill #161.
Then it would take the config file data to fullfill #148
Then it would take any other metadata like IGDBs for example.

[Alfagun74]

We're officially partnered with IGDB!

[Alfagun74]

After a lot of thoughts and work on this i have set my goal to the following workflow:

Workflow Overview

• File Indexing and Initial Game Creation

  • The FileService continues indexing new files and creates a "container-like" "GameVaultGame" object.

• Metadata Retrieval Task

  • After indexing, the indexer triggers the good ol' "recache" operation to search metadata for new GameVaultGames.
  • For each game, the indexer sends a "findMetadata" task to the MetadataService.

• MetadataService and Providers

  • The MetadataService acts as an orchestrator where metadata provider plugins (either built-in or plug-in) register with a certain priority.

• IGDB Provider

  • A new default provider "IGDB" will be shipped with GameVault.

• AbstractMetadataProvider Implementation

  • Each metadata provider implements the AbstractMetadataProvider class, which includes functionalities like searching for games or updating existing data.

• Executing Metadata Searches

  • Upon receiving a "findMetadata" task, the MetadataService sequentially calls all search functions on its registered metadata providers based on priority.

• Filtering and Storing Metadata

  • The results are filtered using the existing "getBestMatchingGame" logic to determine the best match per provider.

• Merging Metadata

  • A new & almost final metadata object, "gamevault," is created with the provider_id of the gamevault game and populated with the retrieved metadata in a process called mergeMetadata.

• Search and Access

  • The "gamevault" metadata objects are exposed through search functionalities in APIs, making them accessible and searchable.

• User-Modified Metadata

  • When a user edits a game, a "user" metadata object is created with the provider_id of the gamevault game.

• Overwriting Metadata

  • During each mergeMetadata, the almost final "gamevault" result gets partially overwritten with the data in the "user" metadata.

• Periodic Metadata Updates

  • A service periodically checks the metadata and, if a hash mismatch occurs, triggers the provider's "update" function. This fetches new metadata, updates the "provider" object, and calls mergeMetadata for affected games.

[Alfagun74]

Implemented in v13

[Alfagun74]

will be completed via https://github.com/Phalcode/gamevault-backend/issues/161