全量同步切thunk的时候,直接启用数据同步

Viewed 23

SELECT MAX(order_id) FROM (SELECT order_id FROM shard.order WHERE order_id >= 1650896595254848768 ORDER BY order_id ASC LIMIT 100000) AS T

麻烦问下,flink-cdc导全量数据的时候,是否能直接启动数据同步,因为切完数据ChunkSplitter has split的时候,其实数据都在内存里了,直接开始导,总体时间会大幅缩减;
2024-07-18 04:28:05,065 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650895789369676032
2024-07-18 04:28:06,794 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650896595254848768
2024-07-18 04:28:08,553 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650900142218367232
2024-07-18 04:28:10,127 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650900825713749249
2024-07-18 04:28:11,633 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650901504998059264
2024-07-18 04:28:13,171 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650902174576749824
2024-07-18 04:28:14,772 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650902849138282753
2024-07-18 04:28:16,493 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650903540779015936
2024-07-18 04:28:18,719 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650904218477866240
2024-07-18 04:28:20,235 INFO com.ververica.cdc.connectors.mysql.source.assigners.MySqlChunkSplitter [] - Use unevenly-sized chunks for table shard.order, the chunk size is 100000 from 1650904906293403393

1 Answers

建议可以先阅读以下博文:https://flink-learning.org.cn/article/detail/3ebe9f20774991c4d5eeb75a141d9e1e