poseidon icon indicating copy to clipboard operation
poseidon copied to clipboard

使用hdfs的bug(将mock_local置为false)

Open tammypi opened this issue 8 years ago • 8 comments

1.dist/index-0.1/lib里的docformat,index,indexmeta的jar包分别为(build.sh生成的): index-0.1.jar,indexmeta-0.1.jar,docmeta-0.1.jar 但是start.sh却写着: cmd="$HADOOP jar ${current_dir}/lib/docmeta-1.0-SNAPSHOT.jar ... cmd="$HADOOP jar ${current_dir}/lib/index-1.0-SNAPSHOT.jar ... cmd="$HADOOP jar ${current_dir}/lib/indexmeta-1.0-SNAPSHOT.jar ...

2.start.sh在某些情况下时无法正确的将dist/index-0.1/lib中的jar包加入HADOOP_CLASSPATH,导致hadoop jar docmeta-1.0.jar...命令会报错,no class found meta/MetaConfigured。

3.hadoop jar docmeta-1.0.jar....会报如下错误: java.io.IOException: Incomplete HDFS URI, no host: hdfs:/// at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:355) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:520) at meta.MetaConfigured.run(MetaConfigured.java:97) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at meta.DocMetaConfigured.main(DocMetaConfigured.java:22) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 但etc/test.json里的namenode是配了的。所以不知道为什么会报no host。

tammypi avatar Dec 30 '16 10:12 tammypi

如果是初次试用,请参考快速开始,用demo.sh和mock_start.sh

liwei-ch avatar Dec 30 '16 10:12 liwei-ch

用demo.sh和mock_start.sh,只是提供体验本地的mapreduce任务,无需代建hadoop系统; 如果是和已有的hadoop系统结合试用,需要对start.sh脚本进行改造

liwei-ch avatar Dec 30 '16 10:12 liwei-ch

就是要和hdfs一起用,然后发现有报错。

tammypi avatar Jan 03 '17 00:01 tammypi

如果要使用hadoop集群,需要修改哪些,可不可以提供一个文档。

jwongo avatar Jan 06 '17 02:01 jwongo

可以,我们稍后提供一个文档和示例

liwei-ch avatar Jan 06 '17 05:01 liwei-ch

我也遇到这个问题,期待早日解决

wubo2cyp avatar Jan 12 '17 02:01 wubo2cyp

遇到同样的问题,期望早日解决

jwongo avatar Jan 13 '17 12:01 jwongo

使用hdfs部署:https://github.com/Qihoo360/poseidon/blob/master/docs/run_with_hadoop.md

dunixd avatar Jan 16 '17 08:01 dunixd