docker-hadoop
docker-hadoop copied to clipboard
Wrong hadoop version on tag 3.1.3
Using the tag 2.0.0-hadoop3.1.3-java8 bash
, the actual installed version is 3.2.1
and not 3.1.3
.
~/docker-hive$ docker run -it bde2020/hadoop-namenode:2.0.0-hadoop3.1.3-java8 bash
Configuring core
- Setting fs.defaultFS=hdfs://bc96e1de2c94:8020
Configuring hdfs
- Setting dfs.namenode.name.dir=file:///hadoop/dfs/name
Configuring yarn
Configuring httpfs
Configuring kms
Configuring mapred
Configuring for multihomed network
root@bc96e1de2c94:/# echo $PATH
/opt/hadoop-3.2.1/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root@784684f7199c:/bin# which hdfs
/opt/hadoop-3.2.1/bin//hdfs
Also, run.sh
fails to run as hdfs
cannot be found, presumably related to this different version being installed.
$ docker logs 765
Configuring core
- Setting hadoop.proxyuser.hue.hosts=*
- Setting fs.defaultFS=hdfs://namenode:8020
- Setting hadoop.http.staticuser.user=root
- Setting hadoop.proxyuser.hue.groups=*
Configuring hdfs
- Setting dfs.namenode.datanode.registration.ip-hostname-check=false
- Setting dfs.webhdfs.enabled=true
- Setting dfs.permissions.enabled=false
- Setting dfs.namenode.name.dir=file:///hadoop/dfs/name
Configuring yarn
- Setting yarn.timeline-service.enabled=true
- Setting yarn.resourcemanager.system-metrics-publisher.enabled=true
- Setting yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
- Setting yarn.log.server.url=http://historyserver:8188/applicationhistory/logs/
- Setting yarn.resourcemanager.fs.state-store.uri=/rmstate
- Setting yarn.timeline-service.generic-application-history.enabled=true
- Setting yarn.log-aggregation-enable=true
- Setting yarn.resourcemanager.hostname=resourcemanager
- Setting yarn.resourcemanager.resource_tracker.address=resourcemanager:8031
- Setting yarn.timeline-service.hostname=historyserver
- Setting yarn.resourcemanager.scheduler.address=resourcemanager:8030
- Setting yarn.resourcemanager.address=resourcemanager:8032
- Setting yarn.nodemanager.remote-app-log-dir=/app-logs
- Setting yarn.resourcemanager.recovery.enabled=true
Configuring httpfs
Configuring kms
Configuring mapred
Configuring for multihomed network
Formatting namenode name directory: /hadoop/dfs/name
/run.sh: line 16: /bin/hdfs: No such file or directory
/run.sh: line 19: /bin/hdfs: No such file or directory
you can download the code, load the 3.1.3 branch then run make and build the image yourself locally then launch it as a workaround
This should be already fixed. @Data-drone did you have the same issue?
was just suggesting a workaround. I haven't used the 2.0.0-hadoop3.1.3-java8 version