Support for more than 2^32 elements?

[JustinChu]

It seems there is a limit to use a uint32_t for the count parameter of the PackedArray_create() function. Do you foresee any major issues modifying the code downstream to allow for a uint64_t count so I can populate it with more than 2^32 elements?

[gpakosz]

Hello @JustinChu

I never needed more than UINT32_MAX elements. That's why I chose uint32_t instead of size_t. I also recall back then, Microsoft compiler didn't support %zu printf() format specifier and I would have had to use something like

#if defined(_MSCVER)
  #define SIZET_FMT "%Iu"
#elif defined(__GNUC__)
  #define SIZET_FMT "%zu"
#else
  #define SIZET_FMT "%u"
#endif

You should be just fine replacing uint32_t count with uint64_t count or size_t count. If you decide yo use size_t you need #include <stddef.h> in PackedArray.h. In any case you will need to fix -Wformat warnings on printf() calls when compiling the self test and the self bench programs with GCC or Clang.

That being said, at this point I'm undecided on whether it's a change I want to merge.

[JustinChu]

My code only needs to work in a Linux environment using gcc and maybe clang. I'll give a shot and let you know if I have any problems.

Thanks.

[JustinChu]

So far it seems to be okay. I basically changed a bunch of uint32_t to uint64_t https://github.com/JustinChu/PackedArray/commit/bdc91637edb210950a668f280fcabf7a5788cd57. I tested in a few informal cases with no problems so far. I have yet to touch the simd code, however.

Unrelated question (maybe I should add this to another issue). How thread safe is the code? I assume I won't be able to atomically insert values but are at least lookups safe?

Ideally, I would like to be able to do atomic inserts with compare_and_swap (e.g. with atomic built-ins like __sync_bool_compare_and_swap) operations but that doesn't seem like something easily done with a packed array. I wonder if it is possible to use an unaligned pointer in a compare and swap operation...

[NickStrupat]

did you ever get to modifying the SIMD code?

[guinn8]

I found that replacing all uint32_t with uint64_t in PackedArray.c and PackedArray.h got me the extra space I needed. Likely overkill but ensures I don't overflow some uint32_t I forgot about. Thanks for this libary gpakosz, super cool stuff!