Memory error when compressing almost 2GB

[nhz2]

Here is a MWE:

#include <stdio.h>
#include <stdint.h>
#include <blosc.h>

#define SIZE 2147483631

int main(){
  
  /* Allocate Input and output data byte buffers*/
  uint8_t *data = malloc(SIZE);
  uint8_t *data_out = malloc(SIZE+BLOSC_MAX_OVERHEAD);
  /* Check if the allocation was successful*/
  if(data == NULL || data_out == NULL){
    printf("Memory allocation failed\n");
    return 1;
  }

  /* Fill the input data buffer  with random bytes*/
  srand(1234);
  for(int i=0;i<SIZE;i++){
    data[i] = (uint8_t)rand();
  }

  int csize = blosc_compress_ctx(5, 1, 1,
                            SIZE, data, data_out, SIZE+BLOSC_MAX_OVERHEAD,
                            "lz4", 0, 1);
  printf("Compression Returned: %d\n", csize);
  return 0;
}

Running this with valgrind I get the following error:

==170491== Invalid write of size 8
==170491==    at 0x1102FE: _mm_storeu_si128 (emmintrin.h:739)
==170491==    by 0x1102FE: copy_16_bytes (fastcopy.c:95)
==170491==    by 0x11080F: chunk_memcpy_unaligned (fastcopy.c:429)
==170491==    by 0x1108F1: fastcopy (fastcopy.c:516)
==170491==    by 0x10B42F: blosc_c (blosc.c:712)
==170491==    by 0x10B881: serial_blosc (blosc.c:833)
==170491==    by 0x10BABE: do_job (blosc.c:911)
==170491==    by 0x10C355: blosc_compress_context (blosc.c:1252)
==170491==    by 0x10C52D: blosc_compress_ctx (blosc.c:1292)
==170491==    by 0x10A6E5: main (noinit.c:24)
==170491==  Address 0x159c8803c is 2,147,483,644 bytes inside a block of size 2,147,483,647 alloc'd
==170491==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==170491==    by 0x10A64C: main (noinit.c:11)
==170491== 
Compression Returned: -1
==170491== 

I think this is what is causing the segfault in https://github.com/JuliaIO/Blosc.jl/issues/91

[FrancescAlted]

Well spotted. We will try to fix this, but if you can beat us in doing this, a PR is more than welcome.