soweego: link Wikidata to large catalogs

soweego is a pipeline that connects Wikidata to large-scale third-party catalogs.

soweego is the only system that makes statisticians, epidemiologists, historians, and computer scientists agree. Why? Because it performs record linkage, data matching, and entity resolution at the same time. Too easy, they all seem to be synonyms!

Oh, soweego also embeds Machine Learning and advocates for Linked Data.

Official Project Pages

soweego is made possible thanks to the Wikimedia Foundation:

• https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego
• https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego_2

Documentation

https://soweego.readthedocs.io/

Highlights

• Run the whole pipeline, or
• use the command line;
• import large catalogs into a SQL database;
• gather live Wikidata datasets;
• connect them to target catalogs via rule-based and supervised linkers;
• upload links to Wikidata and Mix'n'match;
• synchronize Wikidata to imported catalogs;
• enrich Wikidata items with relevant statements.

Get Ready

Install Docker and Compose, then enter soweego:

$ git clone -b v1.1 https://github.com/Wikidata/soweego.git
$ cd soweego
$ ./docker/run.sh
Building soweego
...

root@70c9b4894a30:/app/soweego#

Now it's too late to get out!

Run the Pipeline

Piece of cake:

:/app/soweego# python -m soweego run CATALOG

Pick CATALOG from discogs, imdb, or musicbrainz.

These steps are executed by default:

1. import the target catalog into a local database;
2. link Wikidata to the target with a supervised linker;
3. synchronize Wikidata to the target.

Results are in /app/shared/results.

Use the Command Line

You can launch every single soweego action with CLI commands:

:/app/soweego# python -m soweego
Usage: soweego [OPTIONS] COMMAND [ARGS]...

  Link Wikidata to large catalogs.

Options:
  -l, --log-level <TEXT CHOICE>...
                                  Module name followed by one of [DEBUG, INFO,
                                  WARNING, ERROR, CRITICAL]. Multiple pairs
                                  allowed.
  --help                          Show this message and exit.

Commands:
  importer  Import target catalog dumps into a SQL database.
  ingester  Take soweego output into Wikidata items.
  linker    Link Wikidata items to target catalog identifiers.
  run       Launch the whole pipeline.
  sync      Sync Wikidata to target catalogs.

Just two things to remember:

1. you can always get --help;
2. each command may have sub-commands.

Contribute

The best way is to import a new catalog. Please also have a look at the guidelines.

License

The source code is under the terms of the GNU General Public License, version 3.