alluxio
alluxio copied to clipboard
Alluxio
java.io.IOException No FileSystem for scheme: alluxio
Alluxio Version:
2.9.3 version
我想试用alluxio 来帮助加速trino,所以开始接触alluxio. 步骤如下： 1.我先按照这里的入门使用，在linux 系统搭建一台alluxio
已经成功搭建，可以访问 https://docs.alluxio.io/os/user/stable/cn/overview/Getting-Started.html 2.再参考这个文档，到了第三部，我创建这个hive 表时，会报如下错误 https://docs.alluxio.io/os/user/stable/en/compute/Trino.html
文件已经关联上
命令和内容如下： Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/hive-common-1.1.0-cdh5.16.2.jar!/hive-log4j.properties WARNING: Hive CLI is deprecated and migration to Beeline is recommended.
hive> CREATE TABLE u_user ( > userid INT, > age INT, > gender CHAR(1), > occupation STRING, > zipcode STRING) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY '|' > STORED AS TEXTFILE > LOCATION 'alluxio://cdh5:19999/ml-100k'; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
我希望能够创建hive表和alluxio关联上
希望能得到回复，谢谢
have you put the alluxio-client-2.9.3.jar to the lib directory of hive installation? @jiang320
and then, you should restart hive meta service and hive server2 service one after another.
你好，我将包放入原本正常运行的hive的包中，再重启hive,,通过命令行登入hive的过程，报错了，如下
[root@cdh3 ~]# sudo -u impala hive
Exception in thread "main" java.lang.UnsupportedClassVersionError: alluxio/hadoop/FileSystem : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:363)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2767)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:358)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(TableMapReduceUtil.java:853)
at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addHBaseDependencyJars(TableMapReduceUtil.java:781)
at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.run(MapreduceDependencyClasspathTool.java:59)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70)
Exception in thread "main" java.lang.UnsupportedClassVersionError: alluxio/hadoop/FileSystem : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:363)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2767)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.hive.common.FileUtils.getJarFilesByPath(FileUtils.java:850)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2891)
at org.apache.hadoop.hive.conf.HiveConf.
it seems that jvm version is incompatible. what's your jvm version to run hive ? firstly unify the version of jvm you used to compile alluxio and the one to run hive.
你好，我将包放入原本正常运行的hive的包中，再重启hive,,通过命令行登入hive的过程，报错了，如下 [root@cdh3 ~]# sudo -u impala hive Exception in thread "main" java.lang.UnsupportedClassVersionError: alluxio/hadoop/FileSystem : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:363) at java.util.ServiceLoader$1.next(ServiceLoader.java:445) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2767) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:358) at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(TableMapReduceUtil.java:853) at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addHBaseDependencyJars(TableMapReduceUtil.java:781) at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.run(MapreduceDependencyClasspathTool.java:59) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70) Exception in thread "main" java.lang.UnsupportedClassVersionError: alluxio/hadoop/FileSystem : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:363) at java.util.ServiceLoader$1.next(ServiceLoader.java:445) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2767) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.hive.common.FileUtils.getJarFilesByPath(FileUtils.java:850) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2891) at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:2800) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:80) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:64) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:650) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:634) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141) 没放之前，没有这个问题
看看Hadoop2和Hadoop3版本是不是匹配，Alluxio的包有Hadoop2和Hadoop3不同的版本，另外可能需要对齐一下Java的版本，build和运行的Java版本尽量保持一致
${ALLUIXO_HOME}/client/build alluxio-2.9.3-hadoop2-client.jar alluxio-2.9.3-hadoop3-client.jar
I have the same problem with Trino (v442) and the hive catalog. I tried the alluxio-2.9.3-client.jar (softlink to client/build/alluxio-2.9.3-hadoop2-client.jar), as well as the shaded client from maven central, the client-jar from the assembly folder (.tar.gz), several jars with "client" in its name from maven central with version "311". No success. Tested with Alluxio standalone version with short-circuit and OpenJDK21 in the Trino container.
With alluxio-shaded-client-2.9.3.jar added Trino v433 works and v434 and later is not working.
@jiang320 @realknorke I have managed to solve this problem:
Instead of copying the Alluxio client jar file alluxio-<version>-client.jar to the path ${Trino_HOME}/plugin/hive-hadoop2/, you need to copy it to the path ${Trino_HOME}/plugin/hive/hdfs/.
I have created an issue for this: https://github.com/Alluxio/alluxio/issues/18612
