OpenMLDB sbin deploy can't set HADOOP_CONF

and spark extra conf, e.g. hdfs-site.xml, needs to add in spark/conf before deploy. -- HADOOP_CONF_DIR设置为openmldb/conf应该可以？但taskmanager读取需要start.sh时有环境，sbin不会传去host里 just deploy conf for fix config options

More

stat-all.sh没有recover工具时最好failed，除非force执行，不然用户很容易重启后表有问题，但当时无法察觉 start-all.sh里的recovering没有进度，前台也没法看，没有多开shell时很麻烦 recovering本来就是在tablet、nameserver都启动后的操作，在线集群可以独立recovering，不做不必要的stop

yarn job如果日志出现connect 0.0.0.0:8032，说明yarn-site.xml没有正确load。使用了默认yarn地址。

Mar 22 '23 10:03 vagetablechicken

I label it as a high priority issue

Mar 23 '23 03:03 lumianph

after c19aa7f901958dcce1ae99b50acdc739147e37ed, taskmanager can set HADOOP_CONF_DIR/USER_NAME when submit spark job. But docs need update:

cp all hive/hdfs conf to spark/conf(deploy will copy it to tm hosts)
set hadoop.conf.dir in taskmanager properties(taskmanager read in tm hosts, can be the exists conf dir in host, e.g. /opt/hadoop/conf), deploy-node doesn't need to have conf files.
system ENV which taskmanager can load, only onebox deploy cluster can use it. sbin won't pass env to tm hosts, except SPARK_HOME
[ ] hadoop conf dir, hadoop user name
[ ] hive conf cp to hadoop_conf_dir? or HIVE_HOME?

Jan 31 '24 05:01 vagetablechicken

sbin deploy can't set HADOOP_CONF_DIR for taskmanager