fury icon indicating copy to clipboard operation
fury copied to clipboard

[Java] primitive array compression by SIMD

Open chaokunyang opened this issue 1 year ago • 3 comments

Is your feature request related to a problem? Please describe. Currently primitive array are serialized by copy the data buffer directly using sun.misc.Unsafe, this is super fast. Buf if all values in array are small, compression may reduce the data dramatically.

Compression needs to iterate the array at least once, which may be slow.

Describe the solution you'd like

  • Using JDK 17 vector API for fast compression by SIMD.
  • Providec an option to enable/disable this feature.

Additional context #108

chaokunyang avatar May 08 '23 06:05 chaokunyang

Hello, can you describe the solution in more detail?

heliang666s avatar Apr 16 '24 00:04 heliang666s

Hello, can you describe the solution in more detail?

Hi @heliang666s , thanks for the interest about this issue.

We may need to use SIMD api to detect whether all values in array are small value. If all values are small or first n values are small, we should use SIMD to cast it into smaller value into target buffer directly .

chaokunyang avatar Apr 16 '24 02:04 chaokunyang

ok,I will try it.Please assign me thanks~

heliang666s avatar Apr 16 '24 14:04 heliang666s