after yarn starts

[mahsa-frj]

trafficstars

Hi, i start yarn in my servers, and when i use jps command, the output shows me resourcemanager and nodemanager is running, now i have one question, how to test hadoop works true with glusterfs? please help me. thanks for your reply.

[jayunit100]

simple. just do hadoop fs -ls / and you should see all the directories in gluster's root.

[mahsa-frj]

thanks for your reply, @jayunit100 when i use above command ( hadoop fs -ls / ) i get this output:

15/06/10 11:31:11 INFO glusterfs.GlusterVolume: Initializing gluster volume..
15/06/10 11:31:11 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS
15/06/10 11:31:11 INFO glusterfs.GlusterFileSystem: Initializing GlusterFS,  CRC disabled.
15/06/10 11:31:11 INFO glusterfs.GlusterFileSystem: GIT INFO={git.commit.id.abbrev=f0fee73, [email protected], git.commit.message.full=Merge pull request #122 from childsb/getfattrparse

Refactor and cleanup the BlockLocation parsing code, git.commit.id=f0fee73c336ac19461d5b5bb91a77e05cff73361, git.commit.message.short=Merge pull request #122 from childsb/getfattrparse, git.commit.user.name=bradley childs, git.build.user.name=Unknown, git.commit.id.describe=GA-12-gf0fee73, git.build.user.email=Unknown, git.branch=master, git.commit.time=31.03.2015 @ 00:36:46 IRDT, git.build.time=09.06.2015 @ 12:00:02 IRDT}
15/06/10 11:31:11 INFO glusterfs.GlusterFileSystem: GIT_TAG=GA
15/06/10 11:31:11 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS
15/06/10 11:31:11 INFO glusterfs.GlusterVolume: Initializing gluster volume..
15/06/10 11:31:11 INFO glusterfs.GlusterVolume: Gluster volume: hadoop at : /mnt/hadoop
15/06/10 11:31:11 INFO glusterfs.GlusterVolume: Working directory is : /
15/06/10 11:31:11 INFO glusterfs.GlusterVolume: Write buffer size : 131072
15/06/10 11:31:11 INFO glusterfs.GlusterVolume: Default block size : 67108864
15/06/10 11:31:11 INFO glusterfs.GlusterVolume: Directory list order : fs ordering
15/06/10 11:31:11 INFO glusterfs.GlusterVolume: File timestamp lease significant digits removed : 0
-ls: Fatal internal error
 ``` java.lang.RuntimeException: Error undefined volume:hadoop1 in path: glusterfs://hadoop1/```
at org.apache.hadoop.fs.glusterfs.GlusterVolume.pathToFile(GlusterVolume.java:242)
at org.apache.hadoop.fs.glusterfs.GlusterVolume.getFileStatus(GlusterVolume.java:376)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:265)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1655)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)

it is so important for me to do that, please help me to understand where is my configuration is not correct, i think my core-site.xml's configuration is correct.

[mahsa-frj]

thanks, my problem solved.

[mahsa-frj]

sorry, what is the best tool for testing hadoop's performance?

[jayunit100]

hi @mahsa-frj .

0. There are terasort, teragen, and so on. Those are available as part of your hadoop distribution in a jar file, and are also curated in apache hadoop upstream. They are very easy to run and are the most well known benchmarks for raw hadoop i/o and single job perf testing.

1. If you want a "full" testing solution for your hadoop cluster, you can use ASF BigTop's smoke test libraries, which run these tests for you, alongside other important tests (hadoop client tests, pig tests, hive tests, flume, and so on) (scale is configurable).

2. Additionally the bigtop distribution has a realistic realistic test of the entire hadoop ecosystem (mapreduce->pig->mahout), you can try - its called BigPetStore-Mapreduce application which isalso curated in BigTop : https://github.com/apache/bigtop/tree/master/bigtop-bigpetstore/bigpetstore-mapreduce.

[septianw]

Is this issue solved?