在doris2.0.14使用restore snapshot将doris1.2.8版本备份的数据从hdfs仓库还原出来报错

Viewed 58

背景:有两套doris集群,生产环境1.2.8版本,测试环境2.0.14版本。需要将生产环境大量数据同步至测试环境中,使用方案大致是从生产环境将数据backup snapshot到hdfs仓库中。在测试环境中使用restore snapshot从hdfs仓库拉下来实现离线数据同步。

生产环境备份成功截图:
image.png

测试环境restore还原失败:
image.png

完整的TaskErrMsg内容(其中10.96.70.0/24是我的测试环境地址,10.96.74.0/24是我的生产环境地址):
[54445246: (10.96.70.87)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_100604881/__part_100604880/__idx_100604882/__100604883/02000000005a7101d346cf11afc989792018c6436862419d_0.dat.35e9c7e1dbb8b47f4daaecda8af29a28: (10.96.70.87:8000), Invalid method name: 'fileSize'], [54445247: (10.96.70.87)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_57834279/__part_57834278/__idx_57834280/__57834285/0200000002ba8f190545d4c443264fc31065db939acaba8f_0.dat.863340316bd65c8441f785f649530ae6: (10.96.70.87:8000), Invalid method name: 'fileSize'], [54445256: (10.96.70.89)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_108108026/__part_108108025/__idx_108108027/__108108028/020000000140f5c68c4ee952e76a3f9ddcc86ec488361792_0.dat.0966e8cee3f7d87088e85d267d8f17df: (10.96.70.89:8000), Invalid method name: 'fileSize'], [54445257: (10.96.70.89)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_103827626/__part_103827625/__idx_103827627/__103827628/0200000003b88ded1240a02161d9af5cf89fcbaae3a10b9e_0.dat.0b4b256cd0c6f6751e2c30ebfc643f6b: (10.96.70.89:8000), Invalid method name: 'fileSize'], [54445254: (10.96.70.94)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_103827626/__part_103827625/__idx_103827627/__103827632/0200000003b88dee1240a02161d9af5cf89fcbaae3a10b9e_0.dat.03ab0d8a47022606f668535364199328: (10.96.70.89:8000), Invalid method name: 'fileSize'], [54445255: (10.96.70.89)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_129206371/__part_129206370/__idx_129206372/__129206377/02000000006ae9b88d44b5bb5d2ea60cfea926e98057db96_0.dat.c7a776e28216f7a69d5c25210d982766: (10.96.70.89:8000), Invalid method name: 'fileSize'], [54445252: (10.96.70.94)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_100604881/__part_100604880/__idx_100604882/__100604883/02000000005a7101d346cf11afc989792018c6436862419d_0.dat.35e9c7e1dbb8b47f4daaecda8af29a28: (10.96.70.89:8000), Invalid method name: 'fileSize'], [54445253: (10.96.70.94)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_57834279/__part_57834278/__idx_57834280/__57834285/0200000002ba8f190545d4c443264fc31065db939acaba8f_0.dat.863340316bd65c8441f785f649530ae6: (10.96.70.89:8000), Invalid method name: 'fileSize'], [54445250: (10.96.70.88)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_57834279/__part_57834278/__idx_57834280/__57834285/0200000002ba8f190545d4c443264fc31065db939acaba8f_0.dat.863340316bd65c8441f785f649530ae6: (10.96.70.87:8000), Invalid method name: 'fileSize'], [54445251: (10.96.70.88)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_103827626/__part_103827625/__idx_103827627/__103827628/0200000003b88ded1240a02161d9af5cf89fcbaae3a10b9e_0.dat.0b4b256cd0c6f6751e2c30ebfc643f6b: (10.96.70.87:8000), Invalid method name: 'fileSize'], [54445248: (10.96.70.87)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_103827626/__part_103827625/__idx_103827627/__103827628/0200000003b88ded1240a02161d9af5cf89fcbaae3a10b9e_0.dat.0b4b256cd0c6f6751e2c30ebfc643f6b: (10.96.70.87:8000), Invalid method name: 'fileSize'], [54445249: (10.96.70.88)[THRIFT_RPC_ERROR]failed to get file size of path hdfs://10.96.74.4:8020/doris_view_tables/__palo_repository_doris_view_tables/__ss_ods_add_bdt_backup_20241022/__ss_content/__db_57834227/__tbl_100604881/__part_100604880/__idx_100604882/__100604883/02000000005a7101d346cf11afc989792018c6436862419d_0.dat.35e9c7e1dbb8b47f4daaecda8af29a28: (10.96.70.87:8000), Invalid method name: 'fileSize']

1 Answers

将hdfs仓库换成minio之后,不会产生这个问题。可以正常在目标库进行restore