X2Doris运行任务报错java.lang.NoClassDefFoundError: org/apache/http/client/config/RequestConfig

Viewed 32

使用X2Doris导数据的时候Spark任务报错:
24/12/12 09:37:51 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: org/apache/http/client/config/RequestConfig
java.lang.NoClassDefFoundError: org/apache/http/client/config/RequestConfig
at com.selectdb.x2doris.connector.doris.endpoint.RestEndpoint.send(RestEndpoint.java:98)
at com.selectdb.x2doris.connector.doris.endpoint.RestEndpoint.getSchema(RestEndpoint.java:271)
at com.selectdb.x2doris.connector.doris.util.DorisUtils$.discoverSchemaFromFe(DorisUtils.scala:247)
at com.selectdb.x2doris.connector.doris.util.DorisUtils$.discoverSchema(DorisUtils.scala:232)
at com.selectdb.x2doris.connector.doris.reader.DorisRelation.lazySchema$lzycompute(DorisRelation.scala:54)
at com.selectdb.x2doris.connector.doris.reader.DorisRelation.lazySchema(DorisRelation.scala:54)
at com.selectdb.x2doris.connector.doris.reader.DorisRelation.schema(DorisRelation.scala:58)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:403)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at com.selectdb.x2doris.DorisDataLoader$.dorisToDoris(DorisDataLoader.scala:179)
at com.selectdb.x2doris.DorisDataLoader$.main(DorisDataLoader.scala:67)
at com.selectdb.x2doris.DorisDataLoader.main(DorisDataLoader.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:694)
Caused by: java.lang.ClassNotFoundException: org.apache.http.client.config.RequestConfig
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 19 more
24/12/12 09:37:51 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoClassDefFoundError: org/apache/http/client/config/RequestConfig
at com.selectdb.x2doris.connector.doris.endpoint.RestEndpoint.send(RestEndpoint.java:98)
at com.selectdb.x2doris.connector.doris.endpoint.RestEndpoint.getSchema(RestEndpoint.java:271)
at com.selectdb.x2doris.connector.doris.util.DorisUtils$.discoverSchemaFromFe(DorisUtils.scala:247)
at com.selectdb.x2doris.connector.doris.util.DorisUtils$.discoverSchema(DorisUtils.scala:232)
at com.selectdb.x2doris.connector.doris.reader.DorisRelation.lazySchema$lzycompute(DorisRelation.scala:54)
at com.selectdb.x2doris.connector.doris.reader.DorisRelation.lazySchema(DorisRelation.scala:54)
at com.selectdb.x2doris.connector.doris.reader.DorisRelation.schema(DorisRelation.scala:58)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:403)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at com.selectdb.x2doris.DorisDataLoader$.dorisToDoris(DorisDataLoader.scala:179)
at com.selectdb.x2doris.DorisDataLoader$.main(DorisDataLoader.scala:67)
at com.selectdb.x2doris.DorisDataLoader.main(DorisDataLoader.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:694)
Caused by: java.lang.ClassNotFoundException: org.apache.http.client.config.RequestConfig
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 19 more
)

1 Answers

spark的版本问题,使用x2doris的scala对应的2.12的版本就可以了(内置了spark)