velox icon indicating copy to clipboard operation
velox copied to clipboard

Supporting multi cache prefixes

Open jackylee-ch opened this issue 10 months ago • 1 comments

We are using velox and gluten in Spark, and Spark supports multiple local directories to hold temporary data, which is very important to us since a single disk‘ size is small. This PR attempts to expand the capability of SSDCache in multi-disk scenarios, then we can expanding the amount of cache data.

jackylee-ch avatar Mar 28 '24 06:03 jackylee-ch

Deploy Preview for meta-velox canceled.

Name Link
Latest commit a363e300f485b726331ba48d3ff63b5ab7debda9
Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/661d0fdb54d5780007022863

netlify[bot] avatar Mar 28 '24 06:03 netlify[bot]

Could you review this pr? @gggrace14 cc @PHILO-HE @rui-mo

jackylee-ch avatar Apr 11 '24 12:04 jackylee-ch

@jackylee-ch does the fix impact the configurations? Looks without the fix it's configured as a single file path, with the fix we can configure with multiple paths with "," delimitation, right? If so can you update the config description and document on necessary?

FelixYBW avatar Apr 19 '24 06:04 FelixYBW

@jackylee-ch does the fix impact the configurations? Looks without the fix it's configured as a single file path, with the fix we can configure with multiple paths with "," delimitation, right? If so can you update the config description and document on necessary?

Sorry for late response. This fix won't impat the configurations in velox, however it does affact the configuration spark.gluten.sql.columnar.backend.velox.ssdCachePath in gluten.

jackylee-ch avatar Apr 25 '24 10:04 jackylee-ch