【已解决】2.0.2版本查询hive catalog, 报错 get file split failed for table

Viewed 52

Doris2.0.2版本,在搜索 hive catalog时报错。查询sql:

select * from hive.flowai4_20231215.dwd_uid_bscode_szip_td_mb limit 10;

按下方sql单独查询每个分区是没问题的,去分区条件去掉就报错。

select * from hive.flowai4_20231215.dwd_uid_bscode_szip_td_mb where day_id='20240301' limit 10;

也尝试过重启,更新catalog元信息,无效。
refresh catalog hive;


详细报错信息:

2024-03-28 21:25:49,034 WARN (mysql-nio-pool-0|481) [HiveScanNode.getSplits():203] get file split failed for table: dwd_uid_bscode_szip_td_mb
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.doris.datasource.HMSClientException: failed to get partition for table dwd_uid_bscode_szip_td_mb in db flowai4_20231215 with value [day_id=20231201/hour=14/seg_id_10min=2835684,……{省略,上万个分区路径}, day_id=20240328/hour=19/seg_id_10min=2852710]. reason: shade.doris.hive.org.apache.thrift.transport.TTransportException: null
        at com.google.common.cache.LocalCache.loadAll(LocalCache.java:4129) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.cache.LocalCache.getAll(LocalCache.java:4082) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:5028) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.datasource.hive.HiveMetaStoreCache.getAllPartitions(HiveMetaStoreCache.java:581) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.hive.HiveMetaStoreCache.getAllPartitionsWithCache(HiveMetaStoreCache.java:563) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.HiveScanNode.getPartitions(HiveScanNode.java:172) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.HiveScanNode.getSplits(HiveScanNode.java:198) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.FileQueryScanNode.createScanRangeLocations(FileQueryScanNode.java:259) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.FileQueryScanNode.doFinalize(FileQueryScanNode.java:218) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.FileQueryScanNode.finalizeForNereids(FileQueryScanNode.java:210) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.util.Utils.execWithUncheckedException(Utils.java:65) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalFileScan(PhysicalPlanTranslator.java:501) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalFileScan(PhysicalPlanTranslator.java:208) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalFileScan.accept(PhysicalFileScan.java:92) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalLimit(PhysicalPlanTranslator.java:1496) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalLimit(PhysicalPlanTranslator.java:208) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalLimit.accept(PhysicalLimit.java:154) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:261) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:208) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute.accept(PhysicalDistribute.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalLimit(PhysicalPlanTranslator.java:1496) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalLimit(PhysicalPlanTranslator.java:208) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalLimit.accept(PhysicalLimit.java:154) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:343) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:208) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalResultSink.accept(PhysicalResultSink.java:70) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:235) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:141) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:554) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:432) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:583) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222-ea]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222-ea]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222-ea]

Xnip2024-03-28_21-33-43.jpg

1 Answers

【问题状态】已处理
【问题处理】确认为已知fix了的问题,可以升级至最新的稳定生产版本2.0.7