[Bug]: AMS start error after executing upgrade.sql to update tables in database

[Jzjsnow]

What happened?

After executing the latest upgrade.sql to update the tables in MySQL, starting AMS will fail. The detailed issue is as follows:

1. SQL syntax errors in upgrade.sql: No such column current_change_snapshot_id exists.

2. Unknown column bucket_id in field list:

3. java.lang.IllegalStateException: Expected a string but was BEGIN_OBJECT

Affects Versions

master

What table formats are you seeing the problem on?

No response

What engines are you seeing the problem on?

AMS

How to reproduce

Execute the latest upgrade.sql in mysql and start ams.

Relevant log output

Anything else

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[Jzjsnow]

Bug 1 and 2 can be resolved in #3816. But 3 appears to have inconsistent data results after the update:

The table_config column information in table table_runtime inserted from my old tableRuntime table is in JSON format:

{ "expireSnapshotEnabled": true, "snapshotTTLMinutes": 720, "snapshotMinCount": 1, "changeDataTTLMinutes": 10080, "cleanOrphanEnabled": false, "orphanExistingMinutes": 2880, "deleteDanglingDeleteFilesEnabled": true, "optimizingConfig": { "enabled": true, "targetQuota": 0.5, "optimizerGroup": "flinkgroup-new", "maxExecuteRetryCount": 5, "targetSize": 134217728, "maxTaskSize": 134217728, "maxFileCount": 10000, "openFileCost": 4194304, "fragmentRatio": 8, "minTargetSizeRatio": 0.75, "minorLeastFileCount": 12, "minorLeastInterval": 3600000, "majorDuplicateRatio": 0.1, "fullTriggerInterval": -1, "fullRewriteAllFiles": true, "filter": null, "baseHashBucket": 4, "baseRefreshInterval": -1, "hiveRefreshInterval": -1, "minPlanInterval": 60000 }, "expiringDataConfig": { "enabled": false, "expirationField": null, "expirationLevel": "PARTITION", "retentionTime": 0, "dateTimePattern": "yyyy-MM-dd", "numberDateFormat": "TIMESTAMP_MS", "baseOnRule": "LAST_COMMIT_TIME" }, "tagConfiguration": { "autoCreateTag": false, "tagFormat": "'tag-'yyyyMMdd", "triggerPeriod": "DAILY", "triggerOffsetMinutes": 0, "maxDelayMinutes": 60, "tagMaxAgeMs": -1 } }

While the table_config column information inserted in the new run is in map format, e.g.:

{ "self-optimizing.group": "flinkgroup-new", "upsert-enabled": "false", "write-format": "parquet", "write.parquet.compression-codec": "snappy" }

It seems the old tableRuntime table information may require some adjustments before it can be inserted into the new table. @baiyangtx PTAL

[liaoyt]

UPDATE table_runtime SET table_config = table_config::jsonb - 'optimizingConfig' - 'expiringDataConfig' - 'tagConfiguration'; if use default setting, after executing this sql, server can start