littlexyw

> @littlexyw Thanks for this PR. Your description is clear and makes sense. Can you provide some worker metrics for this PR, especially for the scenario in which the worker...

> For > > https://github.com/apache/celeborn/blob/a2110568f279c2714600b31e05708b7cb76e9936/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/TierWriter.scala#L317-L330 > > We should not call `MemoryManager.instance.releaseMemoryFileStorage(buf.readableBytes())` in L323? > > https://github.com/apache/celeborn/blob/a2110568f279c2714600b31e05708b7cb76e9936/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/TierWriter.scala#L323 When OOM is thrown, buf.readableBytes should be calculated as if no exception was...

> Merge to main(v0.6.0). Could you also backport this pr to branch-0.5 @littlexyw ? OK

@dongjoon-hyun Could you please review this PR when you have time? Many Thanks.

> Thank you for pinging me, @littlexyw . > > This needs to proceed with Apache Spark 4.2.0 as an improvement because Apache Spark provides no `spark-env.sh` file by default....