hdfs-deprecated

hdfs-deprecated copied to clipboard

Hello,

i've deployed the HDFS on DCOS cluster from mesosphere/universe repo with updated mesos-site.xml for my cluster, but it seems that framework ignores:

  <property>
    <name>mesos.hdfs.data.dir</name>
    <description>The primary data directory in HDFS</description>
    <value>/var/lib/mesos/slave/hdfs/data</value>
  </property>

The data dir is still /var/lib/hdfs/data

Can you point me on what am i doing wrong, or its a bug? Thanks!

Any chance you typo'd the name of mesos-site.xml?

Hi @tymofii-polekhin, you should set this variable before compiling. See step 1 in building hdfs-mesos 1) Customize configuration in conf/*-site.xml. All configuration files updated here will be used by the scheduler and also bundled with the executors.

to clarify, i followed this steps:

1. download mesos-site.xml from this repo
2. changed the properties to what i need
3. dcos package install --cli hdfs
4. dcos hdfs config-gen mesos-site.xml
5. dcos package install --options=options.json hdfs

Ah, thanks. This appears to be an issue on DCOS. We are hoping to ultimately add command line args and remove the config-gen command. I believe there is an issue with the latest changes and the config-gen command.

As a workaround, to update the config you should download the tarball and update the mesos-site.xml under the conf directory. You should run ./bin/build-hdfs. You should then upload the new tarball and point to your URL by overriding hdfs/scheduler-uris by using the options.json with "http://your-uri/hdfs-mesos-0.1.4.tgz", "https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz".

Great! Will test this workaround and update here on results Thanks!

Perfect, thx!

@tymofii-polekhin Hi Tymofii, any update here?

This problem still persists. I tried either to deploy the latest code from master branch or the latest package release, which is hdfs-mesos-0.1.4.

@dmitryvakulovych, did you follow the steps I mentioned above including modifying conf/mesos-site.xml and rebuilding the code and hosting the tarball? Note you would need to specify the options.json when installing with the correct tarball and hosted JRE, see the configurable options here: https://github.com/mesosphere/universe/blob/version-1.x/repo/packages/H/hdfs/2/config.json#L13

@elingg I was able to mount data dir using other path but to do that, i had to download source code and change following files: i've replaced "/var/lib/hdfs' /var/lib/mesos/slave

Binary file ../changed_params_hdfs/hdfs-commons/build/classes/main/org/apache/mesos/hdfs/config/HdfsFrameworkConfig.class matches ../changed_params_hdfs/hdfs-commons/src/main/java/org/apache/mesos/hdfs/config/HdfsFrameworkConfig.java: return getConf().get("mesos.hdfs.data.dir", "/var/lib/mesos/slave"); ../changed_params_hdfs/hdfs-scheduler/src/test/java/org/apache/mesos/hdfs/config/HdfsFrameworkConfigSpec.groovy: value == "/var/lib/mesos/slave" ../changed_params_hdfs/hdfs-scheduler/src/test/java/org/apache/mesos/hdfs/config/HdfsFrameworkConfigSpec.groovy: /var/lib/mesos/slave ../changed_params_hdfs/build/cache/hdfs-mesos-executor-0.1.5/etc/hadoop/mesos-site.xml: /var/lib/mesos/slave ../changed_params_hdfs/build/hdfs-mesos-0.1.5/etc/hadoop/mesos-site.xml: /var/lib/mesos/slave ../changed_params_hdfs/conf/mesos-site.xml: /var/lib/mesos/slave

Would you please advice how we can do such things in simplest way.

Btw i' had to unpack hdfs-mesos-executor-0.1.5.tgz and change params as well only then i were able to get success with it/

Thanks @pkutishchev

If you are downloading the existing binary and making the modifications and not rebuilding the code, you would need to change:../changed_params_hdfs/build/cache/hdfs-mesos-executor-0.1.5/etc/hadoop/mesos-site.xml: /var/lib/mesos/slave ../changed_params_hdfs/build/hdfs-mesos-0.1.5/etc/hadoop/mesos-site.xml: /var/lib/mesos/slave ../changed_params_hdfs/conf/mesos-site.xml: /var/lib/mesos/slave

A simpler approach would be to change only one file conf/mesos-site.xml and then recompile the code (./bin/build-hdfs).

Note that @kensipe is working on making the configuration simpler in DCOS for these sorts of things! Configuration is an area that we need to keep working on improving.