apache
[pipeline-connector] Ability to provide a schema replacement
In our usage scenario, the source schema of mysql and the target schema of starrocks are often different, but the table names and column names are consistent, and this pr provides this capability. Maintain a route configuration as follows:
route:
- source-table: test.[\S]*
sink-table: test02.<>
Represents the table below the test will be transferred to the test02, where the symbol <> the representative table name is unchanged, just change the schema. For example, the test.table would be replaced with the test02.table. And maintain a route configuration as follows:
route:
- source-table: test.[\S]*
sink-table: test02.s02_<>
Represents the table below the test will be transferred to the test02, Replace the original table name with the <> symbol. For example, the test.table would be replaced with the test02.s02_table. And the user can define the replacement symbols by themselves as follows:
route:
- source-table: mrtdb.[\S]*
sink-table: mrtdb_sd.sd_s<>
replace-symbol: s<>
Represents the table below the test will be transferred to the test02, Replace the original table name with the s<> symbol.
Hi, @WholeWorld-Timothy this idea looks good to me. Can we support adding table name prefix and table name suffix based on this placeholder, And can we expand this to modifying the schema?
The current mode is a support for modification mode, and I think it is a good idea to add a table name prefix or a table name suffix based on this placeholder, so I can try it out.
My local compilation and testing are correct, so the error in check may not be caused by the code? How can rerun these check ? I click rerun in github desktop doesn t work
Thanks for your contribution, please rebase the master branch and I'll take a look on the CI failure later.
Besides, there are some doc modifications which seems to be unnecessary, could you please revert them back?
Hi @WholeWorld-Timothy, could you please rebase this PR with latest master branch and address the comments above?
cc @whhe
