【已解决】glue catalog不能正常访问

Viewed 145
catalog 2.0

按照文档配置了glue catalog

CREATE CATALOG glue PROPERTIES (
    "type"="iceberg",
    "iceberg.catalog.type" = "glue",
    "glue.endpoint" = "https://glue.us-east-1.amazonaws.com",
    "glue.access_key" = "ak",
    "glue.secret_key" = "sk"
);

运行如下SQL:
SWITCH glue; SHOW DATABASES;

错误：

SQL Error [1105] [HY000]: SdkClientException, msg: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profiles=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profiles=[])): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]

He Zhao3
4 Answers

报错： Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain

从报错来看可能是环境变量未设置，可以先按以下方式排查下：

  1. 检查环境变量：确保AWS_ACCESS_KEY_ID和AWS_SECRET_ACCESS_KEY环境变量已经设置并且是正确的。

  2. 检查AWS配置文件：查看~/.aws/credentials文件，确保它存在并且包含正确的凭证信息。

  3. 检查IAM角色：如果你在EC2上运行，确保实例角色有正确的权限，并且SDK配置为使用实例角色凭证。

可参考：https://stackoverflow.com/questions/68864095/unable-to-load-credentials-from-any-of-the-providers-in-the-chain-awscredentials

阿渊@SelectDB2896

【内容补齐】目前信息还没法定位具体原因哈，请问有更全的异常堆栈不（如下日志补齐）
【问题状态】当前已解决
【解决方式】如下

  1. aws的ec2配置ak sk和aws profile环境；
  2. 在fe.conf和be.conf中加这两个环境变量、然后重启集群
    AWS_ACCESS_KEY_ID=ak
    AWS_SECRET_ACCESS_KEY=sk
张彬华4550

你得doris具体版本是多少？

morningman222

2024-02-29 05:25:17,920 WARN (heartbeat mgr|28) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: FRONTEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), name: fe_5b61c459_d209_4484_a0ba_868301b88f2a, version: null, queryPort: 0, rpcPort: 0, replayedJournalId: 0
2024-02-29 05:25:22,922 WARN (heartbeat mgr|28) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: FRONTEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), name: fe_5b61c459_d209_4484_a0ba_868301b88f2a, version: null, queryPort: 0, rpcPort: 0, replayedJournalId: 0
2024-02-29 05:25:27,924 WARN (heartbeat mgr|28) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: FRONTEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), name: fe_5b61c459_d209_4484_a0ba_868301b88f2a, version: null, queryPort: 0, rpcPort: 0, replayedJournalId: 0
2024-02-29 05:25:32,926 WARN (heartbeat mgr|28) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: FRONTEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), name: fe_5b61c459_d209_4484_a0ba_868301b88f2a, version: null, queryPort: 0, rpcPort: 0, replayedJournalId: 0
2024-02-29 05:25:37,928 WARN (heartbeat mgr|28) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: FRONTEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), name: fe_5b61c459_d209_4484_a0ba_868301b88f2a, version: null, queryPort: 0, rpcPort: 0, replayedJournalId: 0
2024-02-29 05:25:42,931 WARN (heartbeat mgr|28) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: FRONTEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), name: fe_5b61c459_d209_4484_a0ba_868301b88f2a, version: null, queryPort: 0, rpcPort: 0, replayedJournalId: 0
2024-02-29 05:25:47,933 WARN (heartbeat mgr|28) [HeartbeatMgr.runAfterCatalogReady():140] get bad heartbeat response: type: FRONTEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), name: fe_5b61c459_d209_4484_a0ba_868301b88f2a, version: null, queryPort: 0, rpcPort: 0, replayedJournalId: 0
2024-02-29 05:25:48,250 WARN (mysql-nio-pool-36|19835) [StmtExecutor.executeByLegacy():798] execute Exception. stmt[543, 78e9c625bc3044f8-87a5a572981ba2ae]
2024-02-29 05:25:48,250 WARN (mysql-nio-pool-36|19835) [StmtExecutor.executeByLegacy():798] execute Exception. stmt[543, 78e9c625bc3044f8-87a5a572981ba2ae]
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profiles=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profiles=[])): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102) ~[sdk-core-2.17.257.jar:?]
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:117) ~[auth-2.17.257.jar:?]
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) ~[auth-2.17.257.jar:?]
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:123) ~[auth-2.17.257.jar:?]
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.resolveCredentials(AwsExecutionContextBuilder.java:184) ~[aws-core-2.17.257.jar:?]
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:120) ~[aws-core-2.17.257.jar:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) ~[aws-core-2.17.257.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) ~[sdk-core-2.17.257.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175) ~[sdk-core-2.17.257.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.17.257.jar:?]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.17.257.jar:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[aws-core-2.17.257.jar:?]
at software.amazon.awssdk.services.glue.DefaultGlueClient.getDatabases(DefaultGlueClient.java:5731) ~[glue-2.17.257.jar:?]
at org.apache.iceberg.aws.glue.GlueCatalog.listNamespaces(GlueCatalog.java:464) ~[iceberg-aws-1.1.0.jar:?]
at org.apache.iceberg.catalog.SupportsNamespaces.listNamespaces(SupportsNamespaces.java:74) ~[iceberg-api-1.1.0.jar:?]
at org.apache.doris.datasource.iceberg.IcebergGlueExternalCatalog.listDatabaseNames(IcebergGlueExternalCatalog.java:66) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.init(ExternalCatalog.java:255) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.iceberg.IcebergExternalCatalog.init(IcebergExternalCatalog.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.makeSureInitialized(ExternalCatalog.java:178) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.getDbNames(ExternalCatalog.java:345) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ShowExecutor.handleShowDb(ShowExecutor.java:770) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ShowExecutor.execute(ShowExecutor.java:278) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleShow(StmtExecutor.java:2124) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:766) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:459) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:429) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:583) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_391]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_391]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_391]

He Zhao3