hdfs-deprecated

hdfs-deprecated copied to clipboard

Upgrade of HDFS Framework

2

Create process for upgrade. Ensure that the framework upgrades successfully without data loss, etc.

elingg

Design Discussion of Dead Node Timeout

1

For data nodes, we do not need to recover them in the case that there is a sufficient number of data nodes. There are a couple of options here regarding...

elingg

Configurable number of data nodes

2

The user would like to configure the number of data nodes used.

elingg

Package scheduler into a Docker image

1

Dockerize it!

adam-mesos

Request the appropriate port for NN, DN, and JN's

elingg

If a JN/NN/ZK task dies due to a slave being lost, it is never started.

1

This is because there are datanodes on every slave so there isn't any room to start another JN/NN/ZK when a slave is detached. We should instead be shutting down a...

abhay-agarwal

Looks like SchedulerConf is bloating the build with all of hadoop-commons just for a few classes. ``` $ grep -rn "org.apache.hadoop" src/ src/main/java/org/apache/mesos/hdfs/config/SchedulerConf.java:4:import org.apache.hadoop.conf.Configuration; src/main/java/org/apache/mesos/hdfs/config/SchedulerConf.java:5:import org.apache.hadoop.conf.Configured; src/main/java/org/apache/mesos/hdfs/config/SchedulerConf.java:6:import org.apache.hadoop.fs.Path; src/test/java/org/apache/mesos/hdfs/TestScheduler.java:4:import org.apache.hadoop.conf.Configuration;...

adam-mesos

Some parameters, such as `dfs.namenode.heartbeat.recheck-interval` are outdated and don't exist in the [2.5.0 spec](https://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml)

abhay-agarwal

Add fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey to mesos-site.xml and include them in core-site.xml

4

Add configuration parameters for AWS key and secret to be used in core-site.xml. Resulting XML in core-site.xml should look like this: ``` XML fs.s3n.awsAccessKeyId {AWS_ACCESS_KEY} fs.s3n.awsSecretAccessKey {AWS_SECRET_KEY} ```

rolandomanrique

Include fs.defaultFS and fs.default.name in core-site.xml

2

Include fs.defaultFS and fs.default.name in core-site.xml to allow dfs commands without providing full site name in the command. Running `hdfs dfs -ls /` instead of `hdfs dfs -ls hdfs://hdfs/` Xml...

rolandomanrique