incubator-uniffle

incubator-uniffle copied to clipboard

Provide optimized default values when a field in config file is missing, so it's easier for inexperienced users to try and experiment with this project.

For example: If the rss.storage.type field is missing , it may defaults to MEMORY_LOCALFILE_HDFS. Currently, it throws NullPointerException: Name is null.

In summary, let's try make the default settings work out of the box. Please refer to how easy it is to setup a single node cluster for HDFS or Spark.

Ok, we can give more configurations a default value.

Make sense

@kaijchen @zuston @jerqi When I deploy Coordinator Server from master , it throws [ERROR] 2022-10-13 11:22:04,116 main CoordinatorServer <init> - Errors on initializing coordinator server. java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011) at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006) at org.apache.uniffle.coordinator.ApplicationManager.getStorageHost(ApplicationManager.java:278) at org.apache.uniffle.coordinator.ApplicationManager.addRemoteStorageMetrics(ApplicationManager.java:263) at org.apache.uniffle.coordinator.ApplicationManager.refreshRemoteStorage(ApplicationManager.java:109) at org.apache.uniffle.coordinator.ClientConfManager.updateClientConfInternal(ClientConfManager.java:130) at org.apache.uniffle.coordinator.ClientConfManager.init(ClientConfManager.java:73) at org.apache.uniffle.coordinator.ClientConfManager.<init>(ClientConfManager.java:58) at org.apache.uniffle.coordinator.CoordinatorServer.initialization(CoordinatorServer.java:142) at org.apache.uniffle.coordinator.CoordinatorServer.<init>(CoordinatorServer.java:61) at org.apache.uniffle.coordinator.CoordinatorServer.main(CoordinatorServer.java:79) How to solve the problem？

Please attach your coordinator.conf file @jiangbiao910 The conf of rss.coordinator.dynamicClientConf.enabled default value is false.

@zuston conf shoud be set
rss.coordinator.remote.storage.path hdfs://nameservice1/tmp/path1,hdfs://nameservice1/tmp/path2 no rss.coordinator.remote.storage.path hdfs:///tmp/path1,hdfs:///tmp/path2

Ok, we can give more configurations a default value.

@jerqi Hi, I have a question , there are 3 conf file can set rss.storage.type ， (1) spark-default.conf as uniffle client (2) $RSS_HOME/conf/server.conf （3 $RSS_HOME/conf/server.conf where can I set the path on hdfs if rss.storage.type is MEMORY_LOCALFILE_HDFS ? rss.storage.basePath is local disk path the shuffle server be deployed ?