incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

[Improvement] Default value optimization for 0.8

Open jerqi opened this issue 1 year ago • 6 comments

Code of Conduct

Search before asking

  • [X] I have searched in the issues and found no similar issues.

What would you like to be improved?

We have developed features to improve the performance of Uniffle. But most of them are turned off by default. Now we will prefer to turn on some config options by default.

How should we improve?

No response

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

jerqi avatar May 17 '23 05:05 jerqi

Maybe we can make these parameters as default value. @leixm These config options are related to you. Could you raise three pull requests to modify them one by one? (1)

rss.server.single.buffer.flush.enabled true
rss.server.single.buffer.flush.threshold 128M

(2)

rss.coordinator.select.partition.strategy CONTINUOUS

(3)

rss.server.max.concurrency.of.per-partition.write 30

jerqi avatar May 28 '23 09:05 jerqi

Maybe we can make these parameters as default value. @leixm These config options are related to you. Could you raise three pull requests to modify them one by one? (1)

rss.server.single.buffer.flush.enabled true
rss.server.single.buffer.flush.threshold 128M

(2)

rss.coordinator.select.partition.strategy CONTINUOUS

(3)

rss.server.max.concurrency.of.per-partition.write 30

Sure, i will raise PRs later.

leixm avatar Jul 24 '23 07:07 leixm

I will make the client compress type as ZSTD by default. cc @jerqi

zuston avatar Aug 23 '23 01:08 zuston

I will make the client compress type as ZSTD by default. cc @jerqi

It's ok for me if we have verified in the production environment.

jerqi avatar Aug 23 '23 02:08 jerqi

I will make the client compress type as ZSTD by default. cc @jerqi

It's ok for me if we have verified in the production environment.

ZSTD has been applied in our production for 1 year by default

zuston avatar Aug 23 '23 02:08 zuston

I will make the client compress type as ZSTD by default. cc @jerqi

It's ok for me if we have verified in the production environment.

ZSTD has been applied in our production for 1 year by default

OK.

jerqi avatar Aug 23 '23 02:08 jerqi