velox
velox copied to clipboard
Supporting multi cache prefixes
We are using velox and gluten in Spark, and Spark supports multiple local directories to hold temporary data, which is very important to us since a single disk‘ size is small. This PR attempts to expand the capability of SSDCache in multi-disk scenarios, then we can expanding the amount of cache data.
Deploy Preview for meta-velox canceled.
Name | Link |
---|---|
Latest commit | a363e300f485b726331ba48d3ff63b5ab7debda9 |
Latest deploy log | https://app.netlify.com/sites/meta-velox/deploys/661d0fdb54d5780007022863 |
Could you review this pr? @gggrace14 cc @PHILO-HE @rui-mo
@jackylee-ch does the fix impact the configurations? Looks without the fix it's configured as a single file path, with the fix we can configure with multiple paths with "," delimitation, right? If so can you update the config description and document on necessary?
@jackylee-ch does the fix impact the configurations? Looks without the fix it's configured as a single file path, with the fix we can configure with multiple paths with "," delimitation, right? If so can you update the config description and document on necessary?
Sorry for late response. This fix won't impat the configurations in velox, however it does affact the configuration spark.gluten.sql.columnar.backend.velox.ssdCachePath
in gluten.