通过select into outfile方式导出csv文件到S3后再通过select * from s3方式读取文件内容时报错

Viewed 72

使用的版本为2.0.5
1、首先通过select into outfile导出表数据到csv文件
表数据:
mysql> select * from test_insert;
+------------------------------------------+----------+
| k1 | k2 |
+------------------------------------------+----------+
| 99999999999999999999999999999999.999999 | 3.000000 |
| -99999999999999999999999999999999.999999 | 4.000000 |
+------------------------------------------+----------+
2 rows in set (0.03 sec)
导出数据到csv文件:
mysql> select * from test_insert
-> into outfile "s3://backup-restore-s3/export/my_file_"
-> format as csv
-> properties
-> (
-> "s3.endpoint" = "http://cos.hz.test.com:9000",
-> "s3.access_key" = "cz7sPhiTNk5GtzAIflPC",
-> "s3.secret_key" = "Al6bML8jI3vZf9ZrocnZMQpXUVWAs5NRSigu1FcL",
-> "s3.region" = "ap-hz",
-> "use_path_style" = "true"
-> );
+------------+-----------+----------+-------------------------------------------------------------------------+
| FileNumber | TotalRows | FileSize | URL |
+------------+-----------+----------+-------------------------------------------------------------------------+
| 1 | 2 | 99 | s3://backup-restore-s3/export/my_file_2e68acca0684f1a-b2f7efd2331270d0_ |
+------------+-----------+----------+-------------------------------------------------------------------------+
1 row in set (0.04 sec)
2、通过select方式来查看导出文件的内容:
mysql> select * from s3("uri" = "https://cos.hz.test.com:9000/backup-restore-s3/export/my_file_2e68acca0684f1a-b2f7efd2331270d0_0.csv",
-> "s3.access_key"= "cz7sPhiTNk5GtzAIflPC",
-> "s3.secret_key" = "Al6bML8jI3vZf9ZrocnZMQpXUVWAs5NRSigu1FcL",
-> "format" = "csv",
-> "use_path_style" = "true");
ERROR 1105 (HY000): errCode = 2, detailMessage = parse file failed, path = s3://backup-restore-s3/export/my_file_2e68acca0684f1a-b2f7efd2331270d0_0.csv

fe日志中的报错信息:
2024-03-26 11:13:14,724 WARN (mysql-nio-pool-665|457125) [StmtExecutor.executeByLegacy():810] execute Exception. stmt[268420, 5b031d44ea394284-b7b1cb55d10c7901]
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = parse file failed, path = s3://backup-restore-s3/export/my_file_2e68acca0684f1a-b2f7efd2331270d0_0.csv
at org.apache.doris.tablefunction.ExternalFileTableValuedFunction.parseFile(ExternalFileTableValuedFunction.java:165) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.tablefunction.S3TableValuedFunction.(S3TableValuedFunction.java:103) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.tablefunction.TableValuedFunctionIf.getTableFunction(TableValuedFunctionIf.java:50) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.analysis.TableValuedFunctionRef.(TableValuedFunctionRef.java:47) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.analysis.CUP$SqlParser$actions.case911(SqlParser.java:32234) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.analysis.CUP$SqlParser$actions.CUP$SqlParser$do_action(SqlParser.java:10018) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.analysis.SqlParser.do_action(SqlParser.java:2731) ~[doris-fe.jar:1.2-SNAPSHOT]
at java_cup.runtime.lr_parser.parse(lr_parser.java:584) ~[jflex-1.4.3.jar:?]
at org.apache.doris.common.util.SqlParserUtils.getStmt(SqlParserUtils.java:51) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.parseByLegacy(StmtExecutor.java:1062) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:909) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:703) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:475) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:443) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:584) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:841) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.doris.common.UserException: errCode = 2, detailMessage = S3TvfBroker list path exception. path=s3://backup-restore-s3/export/my_file_2e68acca0684f1a-b2f7efd2331270d0_0.csv, err: errCode = 2, detailMessage = S3TvfBroker list path failed. path=s3://backup-restore-s3/export/my_file_2e68acca0684f1a-b2f7efd2331270d0_0.csv,msg=errors while get file status getFileStatus on s3://backup-restore-s3/export/my_file_2e68acca0684f1a-b2f7efd2331270d0_0.csv: com.amazonaws.SdkClientException: Unable to execute HTTP request: Unsupported or unrecognized SSL message: Unable to execute HTTP request: Unsupported or unrecognized SSL message
at org.apache.doris.common.util.BrokerUtil.parseFile(BrokerUtil.java:99) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.tablefunction.ExternalFileTableValuedFunction.parseFile(ExternalFileTableValuedFunction.java:163) ~[doris-fe.jar:1.2-SNAPSHOT]

2 Answers

麻烦这样测试下哈

  1. 检查下对象地址是否正确
  2. "use_path_style" = "false"

按提示重新测试的结果如下:
补充说明下,因为前面的导出的文件删除了,重新通过select into outfile生成了新的文件,截图中的提示文件名跟问题描述中的存在一点出入

补充url信息: