velox
velox copied to clipboard
facebookincubator
Fix loss of precision in timestamp when spilling
Currently we use a presto compatible serializer when spilling which serializes timestamp to millisecond precision. This results in loss of precision for velox timestamp type since velox stores it in nanosecond precision. This patch ensures that precision is maintained when spilling occurs. It introduces a new 'Options' input parameter to the VectorSerde interface which is a map of key-value pairs. This allows the PrestoSerializer to support serialization with both precisions by allowing the caller to toggle between the two via the input parameter.
Additionally, this also removes a hack inside the test suite which by default downgrades timestamp precision to millisecond when comparing results in order to be compatible with duckDB which is used to verify correctness in some tests.
Test Plan Added unit tests
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
Deploy Preview for meta-velox canceled.
| Name | Link |
|---|---|
| Latest commit | a95534409e0450e43e085c2dabbc9e7f38d98227 |
| Latest deploy log | https://app.netlify.com/sites/meta-velox/deploys/630fd7b04bff8b00096aa6bb |
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@bikramSingh91 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.