Looking to download whole database for libretro project

[i30817]

Basically i want a competitor to have images.

Retroplay whdload set provides a set of dats; which on their whdload subset (not hd or beta whdloads unfortunately), often have a variant entry in your database.

Getting the 'x_name' and 'variant_name' derived from the filenames of these sets with their dats is straightforward, just a question of replacing '_' by ', ', replacing the first token (name) by 'WHDLoad' and there you have the 'variant_name' etc.

Now i'm looking for a way that doesn't hit your database to get your info and maybe images, to get the 'real' name of the game based on that property to get the parent_uuid and get it that way, and maybe associate a image to it (either from your downloable pack or trying to tie in the 'standard' name to the amiga images libretro-db already has).

Can one download your db for this? And do you have any advice, ie, is x_name or variant_name enough for this?

Maybe i should just scrape in python?

[FrodeSolheim]

Please don't run a HTML scraper at least :) If you can code in python, you can quite easily extract the information from the local copies of the database that FS-UAE Launcher downloads (i.e. Amiga.sqlite). That's the entire database. It consists of a table with uuid and a binary blob of the json data for that entry.

See GameDatabase.py from FS-UAE Launcher for more code, but some relevant snippets:

    cursor.execute(
        "SELECT uuid, data FROM game WHERE uuid = ?",
        (sqlite3.Binary(unhexlify(game_uuid.replace("-", ""))),),
    )
    row = cursor.fetchone()

and to get a JSON document from the compressed data:

    data = zlib.decompress(row[1])
    data = data.decode("UTF-8")
    doc = json.loads(data)

[FrodeSolheim]

I might create a whdload_archivename field in the database to register what the archive name is, compatible with Retroplay's naming...

[FrodeSolheim]

I write a small scripts with dumps the entire local database (e.g. Amiga.sqlite) to stdout in JSON format:

#!/usr/bin/env python3

from binascii import hexlify
import json
import sqlite3
import sys
import zlib


def main():
    databasepath = sys.argv[1]
    database = sqlite3.connect(databasepath)
    cursor = database.cursor()
    cursor.execute("SELECT uuid, data FROM game")
    print("{", end="")
    count = 0
    for row in cursor:
        s = hexlify(row[0]).decode("ASCII")
        uuid = f"{s[0:8]}-{s[8:12]}-{s[12:16]}-{s[16:20]}-{s[20:32]}"
        if row[1]:
            data = zlib.decompress(row[1])
            jsonstr = data.decode("UTF-8")
            print("," if count > 0 else "")
            print(f'    "{uuid}": {jsonstr}', end="")
            count += 1
    print("\n}")
    print(f"Exported {count} entries", file=sys.stderr)


if __name__ == "__main__":
    main()

[FrodeSolheim]

I'm uploading a whdload_archive field for current WHDload variants. Better to use than x_name, which is for debugging purposes and for identification when the variant is not associated with a game entry.

[i30817]

Ok that's super cool. Do you happen to have a link to wget the Amiga.sqlite database?

[FrodeSolheim]

The Amiga.sqlite database isn't available anywhere, it is created locally by FS-UAE Launcher. It is possible to download the entries from openretro.org via the same way that the Launcher does when it updates Amiga.sqlite, but the API isn't currently documented (apart from by the source code in FS-UAE Launcher).