edition manga db

[joelbrownstein]

Hi @havok2063, can you look at creating multiple manga databases, to keep each MPL separate, i.e. split manga into editions (manga_dr15, manga_mpl8, ...). Otherwise the database is going to be too difficult to backup ever again and having a smaller database would make queries faster anyway. This would also make the sdssdb product more transparent on which manga releases are available as connection strings.

[havok2063]

@joelbrownstein Refactoring the database is a rather large project that entails large changes on the server, database, and software. While ultimately this is what we ought to do, and I agree, this might be out of scope currently. We also did not get much feedback at the Ensenada meeting that this is how we should be spending our time.

A stopgap solution would be to break up our largest tables, modelspaxel, spaxel, and rssfiber into table partitions. This solution maintains a single "virtual" table that is queryable that is actually constructed from many smaller partitioned tables. This solution would make backing up and restoring the database much faster since it's only dealing with smaller tables, and also would require no software changes or changes in how one queries the database in psql. In particular, breaking the tables up along pk ranges which correspond to data releases would improve both the loading, since adding a new release amounts to only adding a new partition table, and the querying, since it only needs to search/access one partition rather than the full monolithic table.

I have tasked Sue at JHU to make this a priority after she restores the recent dump we made, ideally to prepare this for MPL-9.