Support for customized migration methods

[heqingy]

Is your feature request related to a problem? Please describe.

Currently, Bytebase directly runs DDL to apply the schema change. However, this is risky for large tables due to the lock contention and intensive disk I/O etc.

For some large tables, the migration is usually done online with some 3rd party tools like PT-OSC to minimize the performance impact, and this is not supported yet.

Describe the solution you'd like

Support customized migration script so that users are allowed to use tools like PT-OSC. Looks like it's feasible by abstracting the "driver" in schema update task executor. https://github.com/bytebase/bytebase/blob/main/server/task_executor_schema_update.go#L151

Additional context

Add any other context or screenshots about the feature request here.

[tianzhou]

Thanks for opening the issue and even pointing the exact place!

This is a good one. Hopefully, we can deliver it soon.

[tianzhou]

@heqingy since the large table DDL will take a long time. How do you track the progress currently? And what kind of interface do you expect from us to track those progress?

[heqingy]

@tianzhou We are using PT-OSC and it prints out the percentage progress. It does the migration by copying data from the old table to the new table, and by estimating the number of rows copied/left, the progress can be calculated.

I guess the communication can be done with socket/FILE if you are going to support this feature. When the user is using some customized tools, they can write to that file / socket to report the progress.