执行Spark任务查询数据结果为空

提问者:帅平 问题分类:大数据
CDH环境提交spark-sql查询任务,结果数据为空,日志报org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy
并提示21304, Task is Failed,errorMsg: UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
执行Spark任务查询数据结果为空
1 个回答
一闪一闪亮晶晶べ
一闪一闪亮晶晶べ
未配置CDH native 相关的依赖包路径,在/opt/spark-2.4.3-bin-hadoop2.6/conf中添加spark-defaults.conf内容如下
spark.driver.extraLibraryPath=/home/cloudera/parcel/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/lib/native
spark.executor.extraLibraryPath=/home/cloudera/parcel/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/lib/native
spark.yarn.am.extraLibraryPath=/home/cloudera/parcel/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/lib/native
发布于:9个月前 (10-09) IP属地:四川省
我来回答