docker-hadoop-spark-workbench icon indicating copy to clipboard operation
docker-hadoop-spark-workbench copied to clipboard

Published 20 hours ago • verified publisher icon big-data-europe

There are 1 datanode(s) running and 1 node(s) are excluded in this operation

Open lokinell opened this issue 6 years ago • 13 comments

Hi, when I try to write parquet file into HDFS, I have below issue:

File /data_set/hello/crm_last_month/2534cb7a-fc07-401e-bdd3-2299e7e657ea.parquet could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1733) at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2496) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:828) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:506) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

    at org.apache.hadoop.ipc.Client.call(Client.java:1475)
    at org.apache.hadoop.ipc.Client.call(Client.java:1412)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at com.sun.proxy.$Proxy288.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy289.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1455)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1251)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)

lokinell avatar Mar 06 '18 12:03 lokinell

@lokinell hi!

Can you check logs of datanode/namenode and copy paste it here.

How to check logs: docker ps --> copy container id for your namenode/datanode docker logs -f containerid

Most likely there is a problem with datanode startup, and it has not been registered with namenode.

earthquakesan avatar Mar 14 '18 13:03 earthquakesan

请问最后怎么解决的

shicanLeft avatar Aug 30 '18 08:08 shicanLeft

@shicanLeft if you have the same problem, please share the logs from your namenode/datanode

earthquakesan avatar Sep 04 '18 15:09 earthquakesan

@earthquakesan I have the same problem, Below is my namenode log

125 2018-10-12 17:00:19,363 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30008 milliseconds 126 2018-10-12 17:00:19,363 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 127 2018-10-12 17:00:32,733 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 172.17.230.44 128 2018-10-12 17:00:32,733 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 129 2018-10-12 17:00:32,733 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 1 130 2018-10-12 17:00:32,733 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Num ber of syncs: 2 SyncTimes(ms): 59 131 2018-10-12 17:00:32,753 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Num ber of syncs: 3 SyncTimes(ms): 79 132 2018-10-12 17:00:32,754 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /home/yuwen/data/hadoopdata/dfs/name/current/edits_inprogress_00000000000000 00001 -> /home/yuwen/data/hadoopdata/dfs/name/current/edits_0000000000000000001-0000000000000000002 133 2018-10-12 17:00:32,757 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 3 134 2018-10-12 17:00:35,756 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.03s at 0.00 KB/s 135 2018-10-12 17:00:35,757 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000002 size 352 bytes. 136 2018-10-12 17:00:35,787 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 0 137 2018-10-12 17:00:49,370 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30007 milliseconds 138 2018-10-12 17:00:49,370 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 139 2018-10-12 17:01:19,378 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30008 milliseconds 140 2018-10-12 17:01:19,379 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 141 2018-10-12 17:01:34,783 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 31 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 49 142 2018-10-12 17:01:34,901 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /hello.txt. BP-241542120-172.17.230.44-1539334719222 blk_1073741825_1001{blockUCState=UNDER_CONSTRU CTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-4a550f28-2167-4160-a778-c3765e3225e9:NORMAL:172.17.230.44:50010|RBW]]} 143 2018-10-12 17:01:49,384 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30006 milliseconds 144 2018-10-12 17:01:49,384 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 145 2018-10-12 17:01:56,016 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], s toragePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on o rg.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy 146 2018-10-12 17:01:56,017 WARN org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication =1, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 147 2018-10-12 17:01:56,017 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[DISK ], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavail ableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 148 2018-10-12 17:01:56,017 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:yuwen (auth:SIMPLE) cause:java.io.IOException: File /hello.txt could only be r eplicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. 149 2018-10-12 17:01:56,018 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 114.247.222.18:50286 Call#3 Ret ry#0 150 java.io.IOException: File /hello.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. 151 at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1595) 152 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3287) 153 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:677) 154 at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:213) 155 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:485) 156 at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) 157 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) 158 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) 159 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) 160 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) 161 at java.security.AccessController.doPrivileged(Native Method) 162 at javax.security.auth.Subject.doAs(Subject.java:415) 163 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) 164 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) 165 2018-10-12 17:02:19,393 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30009 milliseconds 166 2018-10-12 17:02:19,393 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).

dogourd avatar Oct 12 '18 09:10 dogourd

I have a strange problem pretty much like describe above.

When I use Hue I can upload files (as long as they are below 64MB), but when I use HDFS from outside in, it fails. HUE 20170924.csv is uploaded using Hue filebrowser and works fine 20170926.csv is uploaded from my PC using hdfs dfs -put and does not work There logs are here: 18/11/22 19:46:58 INFO namenode.FSEditLog: Number of transactions: 38 Total time for transactions(ms): 21 Number of transactions batched in Syncs: 1 Number of syncs: 25 SyncTimes(ms): 190 18/11/22 19:46:58 INFO hdfs.StateChange: DIR* completeFile: /20170924.csv.tmp is closed by DFSClient_NONMAPREDUCE_-177179472_46 18/11/22 19:46:58 INFO hdfs.StateChange: BLOCK* allocate blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e11bbcf-fdab-4c81-baf7-82d5b7f29f30:NORMAL:172.19.0.6:50010|RBW]]} for /20170924.csv.tmp 18/11/22 19:46:58 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 172.19.0.6:50010 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e11bbcf-fdab-4c81-baf7-82d5b7f29f30:NORMAL:172.19.0.6:50010|RBW]]} size 0 18/11/22 19:46:58 INFO hdfs.StateChange: DIR* completeFile: /20170924.csv.tmp is closed by DFSClient_NONMAPREDUCE_-1394979784_47

HDFS 18/11/22 19:53:10 INFO namenode.FSEditLog: Number of transactions: 46 Total time for transactions(ms): 22 Number of transactions batched in Syncs: 1 Number of syncs: 31 SyncTimes(ms): 210 18/11/22 19:53:10 INFO hdfs.StateChange: BLOCK* allocate blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e11bbcf-fdab-4c81-baf7-82d5b7f29f30:NORMAL:172.19.0.6:50010|RBW]]} for /20170926.csv.COPYING 18/11/22 19:53:31 WARN blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy 18/11/22 19:53:31 WARN protocol.BlockStoragePolicy: Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=1, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 18/11/22 19:53:31 WARN blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 18/11/22 19:53:31 INFO ipc.Server: IPC Server handler 8 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.1.6:65100 Call#7 Retry#0 java.io.IOException: File /20170926.csv.COPYING could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1628) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3121) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3045) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:493) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)

gotthardsen avatar Nov 22 '18 20:11 gotthardsen

check your docker version. in version 18.03 everything is ok. i had the same problem with docker 18.06 and after downgrading to 18.03 the problem was solved.

kuchksara avatar Nov 26 '18 07:11 kuchksara

check your docker version. in version 18.03 everything is ok. i had the same problem with docker 18.06 and after downgrading to 18.03 the problem was solved.

Mine was 18.09, but I downgrade to 18.03 likes suggested. Unfortunately I get the same result as described above.

I might need to clarify that I'm running windows and have a Hyper-V VM running Ubuntu 18.04 Server, and on this I run docker with this projects docker-compose.yml. On my Windows I have hadoop downloaded (not running) to get to do hadoop fs -copyFromLocal into the VM IP:8020. I have modified the compose file with port 8020:8020 I have no problem running hdfs dfs -ls / or hdfs dfs -cat /file.csv from my windows machine when /file.csv is uploaded with Hue

gotthardsen avatar Nov 26 '18 09:11 gotthardsen

i think this problem is caused by the fact that datanode is not started an permissions, you can start from these two aspects. i meet this problem not in the docker environment, but i solved it from these two directions

sept44 avatar Aug 26 '19 14:08 sept44

How did you solve it in the end? I have the same problem.

marcosfpr avatar Feb 20 '21 16:02 marcosfpr

How did you solve it in the end? I have the same problem.

21/05/20 03:10:59 INFO namenode.FSEditLog: Number of transactions: 87 Total time for transactions(ms): 20 Number of transactions batched in Syncs: 27 
21/05/20 03:10:59 INFO hdfs.StateChange: BLOCK* allocate blk_1073741836_1012, replicas=192.168.96.5:50010 for /upload/test.csv
21/05/20 03:11:20 WARN blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], sock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.had
21/05/20 03:11:20 WARN protocol.BlockStoragePolicy: Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replonFallbacks=[], replicationFallbacks=[ARCHIVE]})
21/05/20 03:11:20 WARN blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[DISKewBlock=true) All required storage types are unavailable:  unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], cr
21/05/20 03:11:20 INFO ipc.Server: IPC Server handler 1 on 8020, call Call#6 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.
java.io.IOException: File /upload/test.csv could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1733)
	at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2496)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:828)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:5
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

pursue-wind avatar May 20 '21 03:05 pursue-wind

any updates on this?

shivanshkaushikk avatar Apr 01 '22 16:04 shivanshkaushikk

any updates on this?

mohocp avatar Apr 29 '22 21:04 mohocp

The reason of this exception is that client cannot connect to the DATANODE so u need to modify two things:

  1. docker-compose.yml
  2. /etc/hosts

First, in docker-compose.yml, specify hostnames and ports for both for datanode and namenode, like this:

namenode:
    image: bde2020/hadoop-namenode:1.1.0-hadoop2.8-java8
    container_name: namenode
    hostname: NameNode
    volumes:
      - ./data/namenode:/hadoop/dfs/name
    environment:
      - CLUSTER_NAME=test
    env_file:
      - ./hadoop.env
    ports:
      - 50070:50070
      - 9000:9000
      - 8020:8020
  datanode:
    image: bde2020/hadoop-datanode:1.1.0-hadoop2.8-java8
    container_name: datanode
    hostname: DataNode
    depends_on:
      - namenode
    volumes:
      - ./data/datanode:/hadoop/dfs/data
    env_file:
      - ./hadoop.env
    ports:
      - 50075:50075
      - 50010:50010
      - 50020:50020

the port 50010 and 50020 are used for datanode to communicate, while the 8020 for namenode is used for namenode to communicate too.

After this, then specify the port mapping in /etc/hosts (if your os is linux or mac), like this:

xxx.xx.xx.xx NameNode
xxx.xx.xx.xx DataNode

"xxx.xx.xx.xx" is the ip address of your server machine (if u are using your own computer, then can set it to localhost) After these two steps, the client(your computer) should be able to connect to datanode container!

cristianoKaKa avatar Aug 05 '22 09:08 cristianoKaKa