celeborn [CELEBORN-2063] Parallelize the create partition writer in handleReserveSlots to speed up the reserveSlots RPC process time

What changes were proposed in this pull request?

Parallelize the create partition writer in handleReserveSlots to speed up the reserveSlots RPC process time。

Why are the changes needed?

The creation of partition writer in handleReserveSlots could use parallelize way to speed up the reserveSlots RPC process time.

Does this PR introduce any user-facing change?

Introduce celeborn.worker.writer.create.parallel.enabled, celeborn.worker.writer.create.parallel.threads and eleborn.worker.writer.create.parallel.timeout to config parallelize the creation of file writer.

How was this patch tested?

CI.

Jul 24 '25 12:07 SteNicholas

Ping @turboFei, @RexXiong, @FMX.

Jul 25 '25 07:07 SteNicholas

Ping @RexXiong, @turboFei, @zaynt4606. I have addressed above comments.

Aug 12 '25 04:08 SteNicholas

Plz share some stats about this PR. I was wondering if this PR is helpful for the Spark job's e2e time.

Aug 27 '25 02:08 FMX

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

Sep 24 '25 08:09 github-actions[bot]

This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.

Oct 15 '25 08:10 github-actions[bot]