cudf
cudf copied to clipboard
Refactor parquet writer benchmarks with nvbench
Description
Issue #10941
Example benchmark Results
parquet_write_encode
[0] Quadro RTX 8000
data_type | cardinality | run_length | Samples | CPU Time | Noise | GPU Time | Noise | bytes_per_second | peak_memory_usage | encoded_file_size |
---|---|---|---|---|---|---|---|---|---|---|
INTEGRAL | 0 | 1 | 13x | 1.100 s | 2.13% | 1.100 s | 2.13% | 487932662 | 2.146 GiB | 506.565 MiB |
INTEGRAL | 1000 | 1 | 37x | 371.301 ms | 3.51% | 371.290 ms | 3.51% | 1445960381 | 2.770 GiB | 165.810 MiB |
INTEGRAL | 0 | 32 | 6x | 95.736 ms | 0.33% | 95.727 ms | 0.33% | 5608374209 | 2.770 GiB | 27.592 MiB |
INTEGRAL | 1000 | 32 | 183x | 67.311 ms | 2.25% | 67.304 ms | 2.25% | 7976843257 | 2.770 GiB | 14.369 MiB |
FLOAT | 0 | 1 | 13x | 1.094 s | 1.57% | 1.094 s | 1.57% | 490681898 | 1.100 GiB | 510.303 MiB |
FLOAT | 1000 | 1 | 53x | 245.566 ms | 1.58% | 245.553 ms | 1.58% | 2186376800 | 1.765 GiB | 110.206 MiB |
FLOAT | 0 | 32 | 159x | 74.142 ms | 2.54% | 74.134 ms | 2.54% | 7241937929 | 1.765 GiB | 23.587 MiB |
FLOAT | 1000 | 32 | 266x | 45.006 ms | 3.69% | 44.999 ms | 3.69% | 11930657028 | 1.765 GiB | 9.888 MiB |
DECIMAL | 0 | 1 | 33x | 426.241 ms | 1.20% | 426.228 ms | 1.20% | 1259587153 | 1.039 GiB | 141.641 MiB |
DECIMAL | 1000 | 1 | 111x | 109.277 ms | 3.73% | 109.266 ms | 3.73% | 4913426291 | 1.145 GiB | 44.820 MiB |
DECIMAL | 0 | 32 | 309x | 37.947 ms | 3.60% | 37.940 ms | 3.60% | 14150565744 | 1.145 GiB | 8.327 MiB |
DECIMAL | 1000 | 32 | 371x | 32.174 ms | 4.67% | 32.167 ms | 4.67% | 16690275220 | 1.145 GiB | 6.669 MiB |
TIMESTAMP | 0 | 1 | 14x | 1.047 s | 2.11% | 1.047 s | 2.11% | 512870450 | 1.178 GiB | 462.140 MiB |
TIMESTAMP | 1000 | 1 | 60x | 208.567 ms | 2.25% | 208.555 ms | 2.25% | 2574239221 | 1.474 GiB | 92.808 MiB |
TIMESTAMP | 0 | 32 | 162x | 71.909 ms | 1.82% | 71.901 ms | 1.82% | 7466791943 | 1.474 GiB | 20.855 MiB |
TIMESTAMP | 1000 | 32 | 296x | 40.141 ms | 3.10% | 40.134 ms | 3.10% | 13376977353 | 1.474 GiB | 8.718 MiB |
DURATION | 0 | 1 | 14x | 1.010 s | 2.36% | 1.010 s | 2.36% | 531706626 | 1.150 GiB | 436.918 MiB |
DURATION | 1000 | 1 | 59x | 208.890 ms | 2.81% | 208.877 ms | 2.81% | 2570271173 | 1.474 GiB | 92.663 MiB |
DURATION | 0 | 32 | 166x | 69.930 ms | 1.94% | 69.922 ms | 1.94% | 7678100086 | 1.474 GiB | 19.551 MiB |
DURATION | 1000 | 32 | 295x | 39.998 ms | 3.72% | 39.991 ms | 3.72% | 13424718570 | 1.474 GiB | 8.541 MiB |
STRING | 0 | 1 | 5x | 1.281 s | 0.45% | 1.281 s | 0.45% | 418985121 | 1.342 GiB | 597.486 MiB |
STRING | 1000 | 1 | 100x | 123.906 ms | 3.22% | 123.895 ms | 3.22% | 4333268264 | 677.964 MiB | 46.473 MiB |
STRING | 0 | 32 | 5x | 1.283 s | 0.22% | 1.283 s | 0.22% | 418593329 | 1.342 GiB | 597.486 MiB |
STRING | 1000 | 32 | 96x | 36.813 ms | 4.16% | 36.806 ms | 4.16% | 14586612568 | 677.964 MiB | 8.504 MiB |
LIST | 0 | 1 | 5x | 1.552 s | 0.09% | 1.552 s | 0.09% | 345842800 | 1.695 GiB | 526.626 MiB |
LIST | 1000 | 1 | 5x | 697.747 ms | 0.23% | 697.734 ms | 0.23% | 769449441 | 2.911 GiB | 175.888 MiB |
LIST | 0 | 32 | 42x | 336.564 ms | 1.01% | 336.555 ms | 1.01% | 1595194403 | 2.911 GiB | 38.433 MiB |
LIST | 1000 | 32 | 45x | 316.764 ms | 0.68% | 316.757 ms | 0.68% | 1694897420 | 2.911 GiB | 25.115 MiB |
STRUCT | 0 | 1 | 5x | 1.236 s | 0.16% | 1.236 s | 0.16% | 434277368 | 1.283 GiB | 569.525 MiB |
STRUCT | 1000 | 1 | 5x | 225.491 ms | 0.36% | 225.478 ms | 0.36% | 2381034954 | 1.324 GiB | 90.699 MiB |
STRUCT | 0 | 32 | 5x | 903.626 ms | 0.21% | 903.615 ms | 0.21% | 594136463 | 1.477 GiB | 409.290 MiB |
STRUCT | 1000 | 32 | 182x | 67.608 ms | 2.69% | 67.601 ms | 2.69% | 7941800457 | 1.324 GiB | 15.399 MiB |
parquet_write_io_compression
[0] Quadro RTX 8000
io | compression | cardinality | run_length | Samples | CPU Time | Noise | GPU Time | Noise | bytes_per_second | peak_memory_usage | encoded_file_size |
---|---|---|---|---|---|---|---|---|---|---|---|
FILEPATH | SNAPPY | 0 | 1 | 4x | 3.939 s | inf% | 3.939 s | inf% | 136302643 | 1.643 GiB | 521.113 MiB |
FILEPATH | SNAPPY | 1000 | 1 | 5x | 1.941 s | 0.49% | 1.941 s | 0.49% | 276656089 | 2.727 GiB | 170.914 MiB |
FILEPATH | SNAPPY | 0 | 32 | 5x | 1.329 s | 0.45% | 1.329 s | 0.45% | 403934692 | 2.722 GiB | 50.835 MiB |
FILEPATH | SNAPPY | 1000 | 32 | 12x | 1.275 s | 0.51% | 1.275 s | 0.51% | 421015682 | 2.727 GiB | 24.365 MiB |
FILEPATH | NONE | 0 | 1 | 7x | 2.378 s | 0.77% | 2.378 s | 0.77% | 225765543 | 1.643 GiB | 529.611 MiB |
FILEPATH | NONE | 1000 | 1 | 7x | 1.262 s | 0.49% | 1.262 s | 0.49% | 425263712 | 2.727 GiB | 180.315 MiB |
FILEPATH | NONE | 0 | 32 | 5x | 1.116 s | 0.30% | 1.116 s | 0.30% | 480884592 | 2.722 GiB | 58.968 MiB |
FILEPATH | NONE | 1000 | 32 | 8x | 1.014 s | 0.50% | 1.014 s | 0.50% | 529606276 | 2.727 GiB | 32.308 MiB |
HOST_BUFFER | SNAPPY | 0 | 1 | 4x | 4.181 s | inf% | 4.181 s | inf% | 128399871 | 1.643 GiB | 521.112 MiB |
HOST_BUFFER | SNAPPY | 1000 | 1 | 6x | 2.026 s | 0.48% | 2.026 s | 0.48% | 264969784 | 2.727 GiB | 170.914 MiB |
HOST_BUFFER | SNAPPY | 0 | 32 | 5x | 1.363 s | 0.41% | 1.363 s | 0.41% | 393913005 | 2.722 GiB | 50.835 MiB |
HOST_BUFFER | SNAPPY | 1000 | 32 | 5x | 1.277 s | 0.43% | 1.277 s | 0.43% | 420459944 | 2.727 GiB | 24.364 MiB |
HOST_BUFFER | NONE | 0 | 1 | 5x | 2.649 s | 0.42% | 2.649 s | 0.42% | 202649168 | 1.643 GiB | 529.611 MiB |
HOST_BUFFER | NONE | 1000 | 1 | 5x | 1.332 s | 0.41% | 1.332 s | 0.41% | 403090403 | 2.727 GiB | 180.315 MiB |
HOST_BUFFER | NONE | 0 | 32 | 5x | 1.151 s | 0.46% | 1.151 s | 0.46% | 466449565 | 2.722 GiB | 58.968 MiB |
HOST_BUFFER | NONE | 1000 | 32 | 13x | 1.039 s | 0.50% | 1.039 s | 0.50% | 516732638 | 2.727 GiB | 32.308 MiB |
VOID | SNAPPY | 0 | 1 | 5x | 3.559 s | 0.62% | 3.559 s | 0.62% | 150867866 | 1.643 GiB | 521.113 MiB |
VOID | SNAPPY | 1000 | 1 | 7x | 1.817 s | 0.47% | 1.817 s | 0.47% | 295405582 | 2.727 GiB | 170.914 MiB |
VOID | SNAPPY | 0 | 32 | 5x | 1.299 s | 0.04% | 1.299 s | 0.04% | 413272964 | 2.722 GiB | 50.836 MiB |
VOID | SNAPPY | 1000 | 32 | 5x | 1.264 s | 0.28% | 1.264 s | 0.28% | 424605071 | 2.727 GiB | 24.364 MiB |
VOID | NONE | 0 | 1 | 5x | 2.003 s | 0.50% | 2.003 s | 0.50% | 268012332 | 1.643 GiB | 529.611 MiB |
VOID | NONE | 1000 | 1 | 5x | 1.127 s | 0.45% | 1.127 s | 0.45% | 476312808 | 2.727 GiB | 180.315 MiB |
VOID | NONE | 0 | 32 | 5x | 1.081 s | 0.47% | 1.081 s | 0.47% | 496747581 | 2.722 GiB | 58.968 MiB |
VOID | NONE | 1000 | 32 | 5x | 999.381 ms | 0.48% | 999.378 ms | 0.48% | 537205288 | 2.727 GiB | 32.308 MiB |
parquet_write_options
[0] Quadro RTX 8000
statistics | compression | file_path | Samples | CPU Time | Noise | GPU Time | Noise | bytes_per_second | peak_memory_usage | encoded_file_size |
---|---|---|---|---|---|---|---|---|---|---|
STATISTICS_NONE | SNAPPY | unused_path.parquet | 3x | 5.961 s | inf% | 5.961 s | inf% | 90067884 | 2.427 GiB | 122.010 MiB |
STATISTICS_NONE | SNAPPY | 3x | 5.962 s | inf% | 5.962 s | inf% | 90054559 | 2.427 GiB | 121.968 MiB | |
STATISTICS_NONE | NONE | unused_path.parquet | 4x | 4.253 s | inf% | 4.253 s | inf% | 126221980 | 2.427 GiB | 141.623 MiB |
STATISTICS_NONE | NONE | 4x | 4.249 s | inf% | 4.249 s | inf% | 126356682 | 2.427 GiB | 141.623 MiB | |
STATISTICS_ROWGROUP | SNAPPY | unused_path.parquet | 3x | 6.011 s | inf% | 6.011 s | inf% | 89314511 | 2.427 GiB | 122.055 MiB |
STATISTICS_ROWGROUP | SNAPPY | 3x | 5.983 s | inf% | 5.983 s | inf% | 89740066 | 2.427 GiB | 122.022 MiB | |
STATISTICS_ROWGROUP | NONE | unused_path.parquet | 4x | 4.282 s | inf% | 4.282 s | inf% | 125372100 | 2.427 GiB | 141.626 MiB |
STATISTICS_ROWGROUP | NONE | 4x | 4.287 s | inf% | 4.287 s | inf% | 125241731 | 2.427 GiB | 141.626 MiB | |
STATISTICS_PAGE | SNAPPY | unused_path.parquet | 3x | 5.976 s | inf% | 5.976 s | inf% | 89837494 | 2.427 GiB | 122.090 MiB |
STATISTICS_PAGE | SNAPPY | 3x | 5.979 s | inf% | 5.979 s | inf% | 89788086 | 2.427 GiB | 121.977 MiB | |
STATISTICS_PAGE | NONE | unused_path.parquet | 4x | 4.290 s | inf% | 4.290 s | inf% | 125138510 | 2.427 GiB | 141.633 MiB |
STATISTICS_PAGE | NONE | 4x | 4.292 s | inf% | 4.292 s | inf% | 125087291 | 2.427 GiB | 141.633 MiB |
parquet_write_num_cols
[0] Quadro RTX 8000
num_cols | Samples | CPU Time | Noise | GPU Time | Noise | bytes_per_second | peak_memory_usage | encoded_file_size |
---|---|---|---|---|---|---|---|---|
8 | 5x | 217.270 ms | 0.13% | 217.262 ms | 0.13% | 2471073081 | 2.648 GiB | 114.635 MiB |
1024 | 5x | 339.592 ms | 0.25% | 339.582 ms | 0.25% | 1580974198 | 2.649 GiB | 145.293 MiB |
parquet_chunked_write
[0] Quadro RTX 8000
num_cols | num_chunks | Samples | CPU Time | Noise | GPU Time | Noise | bytes_per_second | peak_memory_usage | encoded_file_size |
---|---|---|---|---|---|---|---|---|---|
8 | 8 | 5x | 239.509 ms | 0.05% | 239.501 ms | 0.05% | 2241622403 | 338.950 MiB | 115.038 MiB |
1024 | 8 | 5x | 441.931 ms | 0.46% | 441.921 ms | 0.46% | 1214856630 | 339.430 MiB | 158.714 MiB |
8 | 64 | 5x | 458.133 ms | 0.10% | 458.125 ms | 0.10% | 1171887455 | 42.372 MiB | 117.129 MiB |
1024 | 64 | 12x | 1.284 s | 0.80% | 1.284 s | 0.80% | 418236962 | 42.828 MiB | 214.851 MiB |
Checklist
- [x] I am familiar with the Contributing Guidelines.
- [x] New or existing tests cover these changes.
- [x] The documentation is up to date with these changes.
Codecov Report
:exclamation: No coverage uploaded for pull request base (
branch-22.10@6cace8e
). Click here to learn what that means. Patch has no changes to coverable lines.
Additional details and impacted files
@@ Coverage Diff @@
## branch-22.10 #11623 +/- ##
===============================================
Coverage ? 85.89%
===============================================
Files ? 151
Lines ? 23534
Branches ? 0
===============================================
Hits ? 20215
Misses ? 3319
Partials ? 0
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.
:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
rerun tests
rerun tests
@gpucibot merge