Enable qjournal JN cluster to be set as dfs.namenode.shared.edits.dir to load test JNs

[aswinmprabhu]

Summary

Dynamometer spins up a simulated NN and makes it write edit logs to a local directory instead of a remote qjournal JN cluster because the primary objective of dynamometer was to load test NNs. But this prevents its usage in load testing JNs as well as an NNs performance w.r.t JN edit log sync.

This PR enables JN load testing using dynamometer by setting the property dfs.namenode.shared.edits.dir if the namenode_edits_dir config starts with qjournal://. If this condition is true, the same JN cluster is used for dfs.namenode.shared.edits.dir as well as dfs.namenode.edits.dir. If the condition is false, dfs.namenode.shared.edits.dir is left empty and dfs.namenode.edits.dir is set to the regular local directory with the prefix file://.

All changes are limited to start-component.sh file which is used as a java resource by Client.java, the dynamometer client used to spin up infra.

Testing Done

Ran the following bash:

#!/bin/bash

NN_EDITS_DIR="/grid/tmpfs/name-data"
baseDir="`pwd`/dyno-node"

qjournalprefix="qjournal://"
if [[ "$NN_EDITS_DIR" == $qjournalprefix* ]]; then
  editsDir="$NN_EDITS_DIR"
  sharedEditsDir="$NN_EDITS_DIR"
  editsDirPathPrefix=""
else
  editsDir=${NN_EDITS_DIR:-${baseDir}/name-data}
  sharedEditsDir=""
  editsDirPathPrefix="file://"
fi

read -r -d '' namenodeConfigs <<EOF
  -D dfs.namenode.edits.dir=${editsDirPathPrefix}${editsDir}
  -D dfs.namenode.shared.edits.dir=${sharedEditsDir}
EOF

echo "$namenodeConfigs"

And got the output:

./dynamo.sh
-D dfs.namenode.edits.dir=file:///grid/tmpfs/name-data
  -D dfs.namenode.shared.edits.dir=

While this bash code:

#!/bin/bash

NN_EDITS_DIR="qjournal://test1.g.linkedin.com:8485;test2.g.linkedin.com:8485;test3.g.linkedin.com:8485/test"
baseDir="`pwd`/dyno-node"

qjournalprefix="qjournal://"
if [[ "$NN_EDITS_DIR" == $qjournalprefix* ]]; then
  editsDir="$NN_EDITS_DIR"
  sharedEditsDir="$NN_EDITS_DIR"
  editsDirPathPrefix=""
else
  editsDir=${NN_EDITS_DIR:-${baseDir}/name-data}
  sharedEditsDir=""
  editsDirPathPrefix="file://"
fi

read -r -d '' namenodeConfigs <<EOF
  -D dfs.namenode.edits.dir=${editsDirPathPrefix}${editsDir}
  -D dfs.namenode.shared.edits.dir=${sharedEditsDir}
EOF

echo "$namenodeConfigs"

Gives the output:

./dynamo.sh
-D dfs.namenode.edits.dir=qjournal://test1.g.linkedin.com:8485;test2.g.linkedin.com:8485;test3.g.linkedin.com:8485/test
  -D dfs.namenode.shared.edits.dir=qjournal://test1.g.linkedin.com:8485;test2.g.linkedin.com:8485;test3.g.linkedin.com:8485/test