pontoon icon indicating copy to clipboard operation
pontoon copied to clipboard

Update: added migration script for migration of historic data

Open RafaelJohn9 opened this issue 1 year ago • 8 comments

Fix #2196

It migrates data from the previous years up to (2020/1/ 7, 9:25:11:829125)

RafaelJohn9 avatar Aug 16 '24 12:08 RafaelJohn9

Hello @mathjazz I was trying a way not to hardcode the actiontypes but right now they are just leading to Attribute errors. Is there a way to achieve this or should I just hardcode the action types

RafaelJohn9 avatar Aug 21 '24 03:08 RafaelJohn9

Hello @mathjazz made the changes that you recommended, Looking forward to your review

RafaelJohn9 avatar Aug 22 '24 11:08 RafaelJohn9

Fix changes you requested :)

RafaelJohn9 avatar Aug 24 '24 13:08 RafaelJohn9

done 👍

RafaelJohn9 avatar Aug 27 '24 03:08 RafaelJohn9

Fixed them 👍

RafaelJohn9 avatar Aug 27 '24 13:08 RafaelJohn9

We'll need to optimize the code further, because it still doesn't complete. I have some ideas and I'll do a bit of testing on stage first, so we don't run in circles for too long.

I was thinking a potential reason why it may be slow is due to creating the objects in bulk at the end instead of single processing each? Though this is just a theory, I'll have to prove it first, then I get back to you

RafaelJohn9 avatar Aug 28 '24 06:08 RafaelJohn9

We'll need to optimize the code further, because it still doesn't complete.

I'm curious: what is killing the migration? Is it a setting on Heroku, a setting in Django? Can it be overridden?

flodolo avatar Aug 29 '24 13:08 flodolo

I'm curious: what is killing the migration? Is it a setting on Heroku, a setting in Django? Can it be overridden?

Could be a setting, but I don't think this is the right approach, at least not right now.

It took at least 30 minutes before the process was killed, and that's with bulk_create() action commented out the. Such long migrations are risky and most likely require app downtime.

We should try to make the code faster first.

mathjazz avatar Aug 29 '24 13:08 mathjazz

@RafaelJohn9 r? https://github.com/mozilla/pontoon/pull/3299/commits/7ac05b9eca910665231788f0bdcc5e82c14d5bbc

Without this change, create_at was always set to "now", i.e. to the date and time when the migration was executed: https://docs.djangoproject.com/en/5.1/ref/models/fields/#django.db.models.DateField.auto_now_add

mathjazz avatar Sep 24 '24 18:09 mathjazz