addons-server icon indicating copy to clipboard operation
addons-server copied to clipboard

Use db_backup + management commands to implement data dump/load/seed

Open KevinMind opened this issue 5 months ago • 1 comments

Relates to: mozilla/addons#15020 Unblocks: https://github.com/mozilla/addons-server/pull/22663

Description

  • migrate data backup management to use django manmagement commands and db-backup under the hood
  • add test coverage for the behaviour
  • add synchronization of the storage volume with the database snapshot as those two data sources are tightly coupled.

Context

In order to include data initialization into the make up flow, we need a better way to dump/load/seed data. This PR moves the mechanism for exporting/importing data to use the db-backup lib orchestrated by management commands, includes test coverage for these behaviours and removes the concept from our make files.

db-backup is a light wrapper around django database connectors to use mysql commands for dumping/loading. This side steps issues we have with using native django dump/load data commands. Before this patch, we had a db connector and could implement this logic ourselves.. but looking at this lib it is well maintained, has a lot of activity and is very transparent looking through the source code. I cannot imagine implementing something much different then what they already have.

This unblocks the initialization part because the command for initializing data in make up will use seeding (which uses dumping) and loading in order to make sure the database is in the "correct" state based on the the arguments passed by the developer.

This can be extracted as a separate PR to make review and testing easier as this part is relatively low impact.

From this PR we also get synchronization of the ./storage directory with the database backup which squashes a whole range of bugs related to mismatches from db file references and the underlying file system.

Testing

  1. run a data dump.
make data_dump

Expect a directory with the timestamp name is added to ./backups. It should include a file db.sql and a storage file storage.tar which is an exact replica of the ./storage directory.

Optionally pass an argument ARGS=--name foo to put the data in a specific named directory.

  1. run data dump force

Assuming you dumped data to a directory with the name "foo" if you try to dump data there again, it will fail. You can make it pass by adding ARGS="--name foo --force" which will clean the foo directory before dumping.

  1. Seed data

Now run the command

make data_seed

This should reset your database to the initial data.

Note. any data you previously had in your DB will now be gone. Expect no add-ons for the local_admin user.

  1. Load data

You should be able to load data from a directory by passing the directory name. Use the name from step 1.

Note, this should restore the data you had originally

make data_load ARGS=--name foo

If you don't pass a name or pass an invalid name the command will raise.

Checklist

  • [X] Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
  • [X] Successfully verified the change locally.
  • [X] The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
  • [ ] Add before and after screenshots (Only for changes that impact the UI).
  • [X] Add or update relevant docs reflecting the changes made.

KevinMind avatar Sep 23 '24 11:09 KevinMind