paimon icon indicating copy to clipboard operation
paimon copied to clipboard
verified publisher icon apache

[Bug] how to sync data from mysql to paimon

Open homesickjava opened this issue 2 years ago • 7 comments

Search before asking

  • [X] I searched in the issues and found nothing similar.

Paimon version

05-SNAPSHOT

Compute Engine

flink

Minimal reproduce step

  1. run paimon-flink-action.jar
  2. use hive to query paimon table
  3. could not get data from paimon table

image

I have 8898 rows of data in source mysql table image

BUT there's no data sync into paimon table image

What doesn't meet your expectations?

I want to get sync data from source mysql table

Anything else?

No response

Are you willing to submit a PR?

  • [X] I'm willing to submit a PR!

homesickjava avatar Jul 13 '23 06:07 homesickjava

I guess you need to provide more info on your job and make sure the configuration in mysqlcdc source and paimon sink are correct.

EMsnap avatar Jul 13 '23 06:07 EMsnap

I have mysqlcdc depended jar in $FLINK_HOME/lib image

and I use flink to run paimon-flink-action.jar to synchronizing datas , and my command is : ./flink run -Dclassloader.check-leaked-classloader=false paimon-flink-action-0.4-SNAPSHOT.jar mysql-sync-table --warehouse hdfs://pcosmo-gyos03:8020/user/root/warehouse --database paimon_db --table paimon_sync_machine_baseinfo --mysql-conf hostname=10.206.32.3 --mysql-conf port=9006 --mysql-conf username=root --mysql-conf password=123456 --mysql-conf database-name=paimontest --mysql-conf table-name='machine_baseinfo' --catalog-conf metastore=hive --catalog-conf uri=thrift://10.206.32.3:9083 --table-conf bucket=4 --table-conf changelog-producer=input --table-conf sink.parallelism=4

I use both hive and flink sql-client to query paimon data of paimon_sync_machine_baseinfo . however, I got empty set of the quering, how can I synchronizing the data from mysql to paimon ~~

homesickjava avatar Jul 13 '23 08:07 homesickjava

it has data synchronizing info, however, when I use flink-sql to query, I could not get any records the script that I use: ./flink run -Dclassloader.check-leaked-classloader=false paimon-flink-action-0.4-SNAPSHOT.jar mysql-sync-table --warehouse hdfs://pcosmo-gyos03:8020/user/root/warehouse --database paimon_db --table sync_machine_baseinfo_addcolumn --mysql-conf hostname=10.206.32.3 --mysql-conf port=9006 --mysql-conf username=root --mysql-conf password=123456 --mysql-conf database-name=paimontest --mysql-conf table-name='machine_baseinfo'

image

homesickjava avatar Jul 13 '23 23:07 homesickjava

+1，我碰到了一样的问题。通过 paimon flink action 进行 mysql 整库同步 环境：数据库：mysql 8.0，cdc：3.3，paimon：1.0.1，flink：1.20.1 启动命令： ./bin/flink run
./paimon-flink-action-1.0.1.jar
mysql_sync_database --warehouse "oss://----------/paimon" --database "ods_erp" --mode combined --type_mapping to-nullable,char-to-string,bigint-unsigned-to-bigint --mysql_conf hostname=--------------------------- --mysql_conf username=-------- --mysql_conf password=----------- --mysql_conf database-name=zx_erp --mysql_conf server-time-zone=Asia/Shanghai --mysql_conf scan.startup.mode=initial --catalog_conf metastore=filesystem --catalog_conf warehouse=oss://---------------/paimon --catalog_conf fs.oss.endpoint=oss-cn-hangzhou-internal.aliyuncs.com --catalog_conf fs.oss.accessKeyId=------------------------ --catalog_conf fs.oss.accessKeySecret=----------------------- --table_conf bucket=1 --table_conf changelog-producer=input --table_conf sink.parallelism=1./bin/flink run
./paimon-flink-action-1.0.1.jar
mysql_sync_database --warehouse "oss://----------/paimon" --database "ods_erp" --mode combined --type_mapping to-nullable,char-to-string,bigint-unsigned-to-bigint --mysql_conf hostname=--------------------------- --mysql_conf username=-------- --mysql_conf password=----------- --mysql_conf database-name=zx_erp --mysql_conf server-time-zone=Asia/Shanghai --mysql_conf scan.startup.mode=initial --catalog_conf metastore=filesystem --catalog_conf warehouse=oss://---------------/paimon --catalog_conf fs.oss.endpoint=oss-cn-hangzhou-internal.aliyuncs.com --catalog_conf fs.oss.accessKeyId=------------------------ --catalog_conf fs.oss.accessKeySecret=----------------------- --table_conf bucket=1 --table_conf changelog-producer=input --table_conf sink.parallelism=1

wcc1433 avatar Mar 07 '25 03:03 wcc1433

截取部分日志： 06 12:56:59,592 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - Read list of available databases 2025-03-06 12:56:59,597 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - list of available databases is: [zx_erp] 2025-03-06 12:56:59,678 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - Read list of available tables in each database 2025-03-06 12:56:59,685 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_book_platform' for further processing 2025-03-06 12:56:59,686 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_book_platform_account' for further processing 2025-03-06 12:56:59,686 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_ding_talk_dept' for further processing 2025-03-06 12:56:59,687 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_ding_talk_dept_user_relation' for further processing 2025-03-06 12:56:59,687 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_ding_talk_user' for further processing 2025-03-06 12:56:59,687 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_gdt_account' for further processing 2025-03-06 12:56:59,687 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_gdt_account_change_record' for further processing 2025-03-06 12:56:59,687 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_platform_order' for further processing 2025-03-06 12:56:59,687 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_quick_app_account' for further processing 2025-03-06 12:56:59,687 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_quick_app_account_change_record' for further processing 2025-03-06 12:56:59,688 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_resource_change_record' for further processing 2025-03-06 12:56:59,688 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_resource_change_record2' for further processing 2025-03-06 12:56:59,688 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_api' for further processing 2025-03-06 12:56:59,688 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_auth_account' for further processing 2025-03-06 12:56:59,688 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_book' for further processing 2025-03-06 12:56:59,688 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_book_ratio' for further processing 2025-03-06 12:56:59,688 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_book_type' for further processing 2025-03-06 12:56:59,689 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_business_group' for further processing 2025-03-06 12:56:59,689 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_company' for further processing 2025-03-06 12:56:59,689 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_company_menu_relation' for further processing 2025-03-06 12:56:59,689 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_company_platform_relation' for further processing 2025-03-06 12:56:59,689 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_config' for further processing 2025-03-06 12:56:59,690 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_dept' for further processing 2025-03-06 12:56:59,690 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_game_user_group' for further processing 2025-03-06 12:56:59,691 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_game_user_group_relation' for further processing 2025-03-06 12:56:59,691 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_gateway_white_url' for further processing 2025-03-06 12:56:59,692 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_ipv4' for further processing 2025-03-06 12:56:59,692 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_login_log' for further processing 2025-03-06 12:56:59,692 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_menu' for further processing 2025-03-06 12:56:59,693 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_menu_api_relation' for further processing 2025-03-06 12:56:59,693 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_module' for further processing 2025-03-06 12:56:59,693 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_msg_group' for further processing 2025-03-06 12:56:59,694 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_msg_group_log' for further processing 2025-03-06 12:56:59,778 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_period' for further processing 2025-03-06 12:56:59,779 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_platform' for further processing 2025-03-06 12:56:59,779 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_post' for further processing 2025-03-06 12:56:59,779 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_post_menu_relation' for further processing 2025-03-06 12:56:59,779 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_resource' for further processing 2025-03-06 12:56:59,779 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_resource_change_record' for further processing 2025-03-06 12:56:59,779 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_resource_change_record2' for further processing 2025-03-06 12:56:59,779 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_resource_group' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_resource_group_relation' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_user' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_user_company_relation' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_user_group' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_user_group_relation' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_user_param' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_user_post_relation' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_user_resource_relation' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.t_sys_user_resource_tag' for further processing 2025-03-06 12:56:59,780 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.temp' for further processing 2025-03-06 12:56:59,781 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - 'zx_erp.temp2' is filtered out of table capturing 2025-03-06 12:56:59,781 INFO org.apache.flink.cdc.connectors.mysql.source.utils.TableDiscoveryUtils [] - including table 'zx_erp.tmp_order' for further processing 2025-03-06 12:56:59,880 INFO io.debezium.jdbc.JdbcConnection [] - Connection gracefully closed 2025-03-06 12:56:59,888 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_book_platform into chunks... 2025-03-06 12:56:59,980 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager pekko.tcp://[email protected]:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000) 2025-03-06 12:56:59,990 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration 2025-03-06 12:57:00,085 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager [email protected]://[email protected]:6123/user/rpc/jobmanager_2 for job e83b46d05ed86c91ef51e84fc4047e64. 2025-03-06 12:57:00,178 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registered job manager [email protected]://[email protected]:6123/user/rpc/jobmanager_2 for job e83b46d05ed86c91ef51e84fc4047e64. 2025-03-06 12:57:00,185 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager [email protected]://[email protected]:6123/user/rpc/jobmanager_2 for job e83b46d05ed86c91ef51e84fc4047e64. 2025-03-06 12:57:00,190 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registered job manager [email protected]://[email protected]:6123/user/rpc/jobmanager_2 for job e83b46d05ed86c91ef51e84fc4047e64. 2025-03-06 12:57:00,383 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000. 2025-03-06 12:57:00,389 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Received resource requirements from job e83b46d05ed86c91ef51e84fc4047e64: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}] 2025-03-06 12:57:00,580 INFO org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager [] - Matching resource requirements against available resources. Missing resources: Job e83b46d05ed86c91ef51e84fc4047e64 ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1} Current resources: (none) 2025-03-06 12:57:00,886 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - need request 1 new workers, current worker number 0, declared worker number 1 2025-03-06 12:57:00,888 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (92.444mb (96935027 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead 2025-03-06 12:57:00,890 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requesting new worker with resource spec WorkerResourceSpec {cpuCores=0.5, taskHeapSize=25.600mb (26843542 bytes), taskOffHeapSize=0 bytes, networkMemSize=64.000mb (67108864 bytes), managedMemSize=230.400mb (241591914 bytes), numSlots=1}, current pending count: 1. 2025-03-06 12:57:01,088 INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: [] 2025-03-06 12:57:01,282 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'kubernetes.service-account' instead of key 'kubernetes.taskmanager.service-account' 2025-03-06 12:57:01,489 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Creating new TaskManager pod with name flink-application-paimon-action-erp-taskmanager-1-1 and resource <1024,0.5>. 2025-03-06 12:57:03,591 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Pod flink-application-paimon-action-erp-taskmanager-1-1 is created. 2025-03-06 12:57:03,893 INFO org.apache.flink.kubernetes.KubernetesResourceManagerDriver [] - Received new TaskManager pod: flink-application-paimon-action-erp-taskmanager-1-1 2025-03-06 12:57:03,894 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Requested worker flink-application-paimon-action-erp-taskmanager-1-1 with resource spec WorkerResourceSpec {cpuCores=0.5, taskHeapSize=25.600mb (26843542 bytes), taskOffHeapSize=0 bytes, networkMemSize=64.000mb (67108864 bytes), managedMemSize=230.400mb (241591914 bytes), numSlots=1}. 2025-03-06 12:57:06,394 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_book_platform is 1.2831 according to the min split key 1, max split key 68 and approximate row count 53 2025-03-06 12:57:06,394 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_book_platform is 1.2831, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:06,394 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_book_platform, the approximate row count is 53, the chunk size is 8096, the dynamic chunk size is 10387 2025-03-06 12:57:06,396 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_book_platform into 1 chunks, time cost: 6507ms. 2025-03-06 12:57:06,396 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_book_platform_account into chunks... 2025-03-06 12:57:06,878 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_book_platform_account is 42.8113 according to the min split key 601, max split key 20336 and approximate row count 461 2025-03-06 12:57:06,879 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_book_platform_account is 42.8113, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:06,880 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_book_platform_account, the approximate row count is 461, the chunk size is 8096, the dynamic chunk size is 346600 2025-03-06 12:57:06,880 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_book_platform_account into 1 chunks, time cost: 483ms. 2025-03-06 12:57:06,880 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_ding_talk_dept into chunks... 2025-03-06 12:57:07,100 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_ding_talk_dept is 1.8542 according to the min split key 1, max split key 89 and approximate row count 48 2025-03-06 12:57:07,101 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_ding_talk_dept is 1.8542, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:07,178 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_ding_talk_dept, the approximate row count is 48, the chunk size is 8096, the dynamic chunk size is 15011 2025-03-06 12:57:07,179 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_ding_talk_dept into 1 chunks, time cost: 299ms.

--------------------------------------- 分界线 --------------------------------------------------

2025-03-06 12:57:07,179 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_ding_talk_dept_user_relation into chunks... 2025-03-06 12:57:07,291 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_ding_talk_dept_user_relation is 1692996.7569 according to the min split key 1, max split key 932841213 and approximate row count 551 2025-03-06 12:57:07,292 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_ding_talk_dept_user_relation is 1692996.7569, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:07,293 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table zx_erp.t_ding_talk_dept_user_relation, the chunk size is 8096 from null 2025-03-06 12:57:07,405 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - ChunkSplitter has split 0 chunks for table zx_erp.t_ding_talk_dept_user_relation 2025-03-06 12:57:07,406 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_ding_talk_dept_user_relation into 1 chunks, time cost: 226ms. 2025-03-06 12:57:07,406 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_ding_talk_user into chunks... 2025-03-06 12:57:07,697 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_ding_talk_user is 3.9456 according to the min split key 1, max split key 580 and approximate row count 147 2025-03-06 12:57:07,697 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_ding_talk_user is 3.9456, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:07,698 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_ding_talk_user, the approximate row count is 147, the chunk size is 8096, the dynamic chunk size is 31943 2025-03-06 12:57:07,698 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_ding_talk_user into 1 chunks, time cost: 292ms. 2025-03-06 12:57:07,698 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_gdt_account into chunks... 2025-03-06 12:57:07,882 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_gdt_account is 2.0202 according to the min split key 1, max split key 14004 and approximate row count 6932 2025-03-06 12:57:07,882 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_gdt_account is 2.0202, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:07,883 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_gdt_account, the approximate row count is 6932, the chunk size is 8096, the dynamic chunk size is 16355 2025-03-06 12:57:07,883 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_gdt_account into 1 chunks, time cost: 185ms. 2025-03-06 12:57:07,883 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_gdt_account_change_record into chunks... 2025-03-06 12:57:07,983 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_gdt_account_change_record is 1.8162 according to the min split key 1, max split key 14820 and approximate row count 8160 2025-03-06 12:57:07,984 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_gdt_account_change_record is 1.8162, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:07,984 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_gdt_account_change_record, the approximate row count is 8160, the chunk size is 8096, the dynamic chunk size is 14703 2025-03-06 12:57:07,985 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_gdt_account_change_record into 2 chunks, time cost: 102ms. 2025-03-06 12:57:07,985 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_platform_order into chunks... 2025-03-06 12:57:08,390 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_platform_order is 6.003 according to the min split key 369715, max split key 430752 and approximate row count 10168 2025-03-06 12:57:08,391 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_platform_order is 6.003, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:08,391 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_platform_order, the approximate row count is 10168, the chunk size is 8096, the dynamic chunk size is 48600 2025-03-06 12:57:08,391 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_platform_order into 2 chunks, time cost: 406ms. 2025-03-06 12:57:08,391 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_quick_app_account into chunks... 2025-03-06 12:57:08,579 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_quick_app_account is 2.7676 according to the min split key 1, max split key 1298 and approximate row count 469 2025-03-06 12:57:08,580 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_quick_app_account is 2.7676, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:08,580 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_quick_app_account, the approximate row count is 469, the chunk size is 8096, the dynamic chunk size is 22406 2025-03-06 12:57:08,580 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_quick_app_account into 1 chunks, time cost: 189ms. 2025-03-06 12:57:08,580 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_quick_app_account_change_record into chunks... 2025-03-06 12:57:08,699 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_quick_app_account_change_record is 2.7877 according to the min split key 1, max split key 1352 and approximate row count 485 2025-03-06 12:57:08,700 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_quick_app_account_change_record is 2.7877, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:08,700 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_quick_app_account_change_record, the approximate row count is 485, the chunk size is 8096, the dynamic chunk size is 22569 2025-03-06 12:57:08,700 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_quick_app_account_change_record into 1 chunks, time cost: 119ms. 2025-03-06 12:57:08,700 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_resource_change_record into chunks... 2025-03-06 12:57:08,982 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_resource_change_record is 1.1977 according to the min split key 1, max split key 30691 and approximate row count 25625 2025-03-06 12:57:08,983 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_resource_change_record is 1.1977, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:08,983 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_resource_change_record, the approximate row count is 25625, the chunk size is 8096, the dynamic chunk size is 9696 2025-03-06 12:57:08,983 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_resource_change_record into 4 chunks, time cost: 283ms. 2025-03-06 12:57:08,984 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_resource_change_record2 into chunks... 2025-03-06 12:57:09,178 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_resource_change_record2 is 6.1792 according to the min split key 3, max split key 60663 and approximate row count 9817 2025-03-06 12:57:09,179 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_resource_change_record2 is 6.1792, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:09,179 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_resource_change_record2, the approximate row count is 9817, the chunk size is 8096, the dynamic chunk size is 50026 2025-03-06 12:57:09,179 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_resource_change_record2 into 2 chunks, time cost: 195ms. 2025-03-06 12:57:09,179 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_sys_api into chunks... 2025-03-06 12:57:09,211 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_sys_api is 2.3151 according to the min split key 6, max split key 1864 and approximate row count 803 2025-03-06 12:57:09,212 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_sys_api is 2.3151, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:09,212 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_sys_api, the approximate row count is 803, the chunk size is 8096, the dynamic chunk size is 18743 2025-03-06 12:57:09,278 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_sys_api into 1 chunks, time cost: 99ms. 2025-03-06 12:57:09,279 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_sys_auth_account into chunks... 2025-03-06 12:57:09,379 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_sys_auth_account is 1.7976931348623157E308, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:09,380 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table zx_erp.t_sys_auth_account, the chunk size is 8096 from null 2025-03-06 12:57:09,484 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - ChunkSplitter has split 0 chunks for table zx_erp.t_sys_auth_account 2025-03-06 12:57:09,484 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_sys_auth_account into 1 chunks, time cost: 205ms. 2025-03-06 12:57:09,484 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_sys_book into chunks... 2025-03-06 12:57:09,687 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_sys_book is 2.1977 according to the min split key 1, max split key 3147 and approximate row count 1432 2025-03-06 12:57:09,688 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_sys_book is 2.1977, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:09,688 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_sys_book, the approximate row count is 1432, the chunk size is 8096, the dynamic chunk size is 17792 2025-03-06 12:57:09,689 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_sys_book into 1 chunks, time cost: 205ms. 2025-03-06 12:57:09,689 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Start splitting table zx_erp.t_sys_book_ratio into chunks... 2025-03-06 12:57:09,715 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The distribution factor of table zx_erp.t_sys_book_ratio is 1.273 according to the min split key 1, max split key 527 and approximate row count 414 2025-03-06 12:57:09,715 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - The actual distribution factor for table zx_erp.t_sys_book_ratio is 1.273, the lower bound of evenly distribution factor is 0.05, the upper bound of evenly distribution factor is 1000.0 2025-03-06 12:57:09,716 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use evenly-sized chunk optimization for table zx_erp.t_sys_book_ratio, the approximate row count is 414, the chunk size is 8096, the dynamic chunk size is 10306 2025-03-06 12:57:09,716 INFO org.apache.flink.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner [] - Split table zx_erp.t_sys_book_ratio into 1 chunks, time cost: 27ms.

wcc1433 avatar Mar 07 '25 03:03 wcc1433

查看日志可以看到所有表都正常扫描到而且在 piamon 中成功创建了对应的表。但是只有分界线之前表数据正常同步了（表 zx_erp.t_book_platform、zx_erp.t_book_platform_account、zx_erp.t_ding_talk_dept 的数据同步成功了。）分界线之后的表感觉像是卡住或者被过滤了，下面是被卡住的第一张表 zx_erp.t_ding_talk_dept_user_relation 的表结构： CREATE TABLE t_ding_talk_dept_user_relation ( dept_id bigint(20) unsigned NOT NULL, userid varchar(64) NOT NULL, PRIMARY KEY (dept_id,userid) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

wcc1433 avatar Mar 07 '25 03:03 wcc1433

paimon action 1.2.0整库同步遇到类似的问题 总共400多张表 初始化建表都完成了 但是同步数据的时候 只同步了其中的几张表 且没有完成同步 mysql_sync_database --warehouse hdfs://huiyun-nameservice-ha/user/flink_1.2/paimon --database test_db1 --table_prefix ods_ --table_suffix _cdc --mysql_conf hostname=172.18.8.214 --mysql_conf username=yuntai --mysql_conf password=YunTai@123!# --mysql_conf database-name=newworld_prod --type_mapping tinyint1-not-bool,bigint-unsigned-to-bigint --table_conf bucket=4 --table_conf changelog-producer=input --table_conf sink.parallelism=4

Gaeas avatar Oct 10 '25 09:10 Gaeas