roryqi

Should we use Spark's metrics instead of log?

> > Should we use Spark's metrics instead of log? > > This may be also valid for MR and Tez. BTW, the task metrics for low shuffle-server don't exist....

> So how to meet the requirement if using metrics ? @jerqi Can you give some ideas if you want Spark metrics system allows user to add extra metrics. You...

ping @zhengchenyu

ShuffleHandleInfo seems immutable, We can't change it.

> > ShuffleHandleInfo seems immutable, We can't change it. > > ShuffleHandleInfo can't change in executor beacuse it is a broadcast value. but we can add a new server to...

I'm not familiar with object storage. Could you give me more inputs? @hiboyang @pspoerri @melodyyangaws @zhaohc10 @LantaoJin @yuyang733

cc @xianjingfeng @zuston Could we finish this issue together?

> **Regarding upload to S3**: As long as you use the Apache HDFS S3A adapter you can stream data to an object store. However you can only append as long...

> Yeah, a lot of small index files will not work well in object storage like S3. Maybe good idea to store in other places. Or is it possible to...