32-bit Limits on armv7l Tinker Board

[madhavajay]

I am trying to find a way to memory map 1080p video frames in numpy format into something fast and lightweight and reduce the CPU utilization in reading directly from a video file or images dir for running on a resource constrained system. The system is a Tinker Board with RK3288 4 core armv7l 32 bit CPUs and 2gb of ram.

bcolz is working perfectly for this on my Mac Book Pro. I have a 30 second video which gets converted into about 4gb of nbytes which is only about 1.3gb of cbytes with the default compression which is awesome.

However when I copy this data to my Tinker Board running Raspbian 9 with 4 armv7l 32-bit cores, I get errors.

I assume this is due to the 32bit?

So the big 730 frame 1.3gb dir loads, but instead of reporting a length of 730 it reports a length of like 37. Weird... maybe a memory int overflow? (4541184000 nbytes and 1311997679 cbytes)

So I turned off compression and went for half the video length which is about 350 frames. This dir wont load at all giving the following error:

OverflowError: Python int too large to convert to C long

Of course the arm 32bit system reports:

>>> sys.maxsize
2147483647

And my desktop reports:

>>> a = bcolz.open(rootdir=bcolz_dir)
>>> len(a)
350
>>> a.nbytes
2177280000
>>> a.cbytes
2183506400

So I assume the bytes exceeds the system max so its a 32 bit addressing issue? Is there a solution here? Can I somehow shard these to 2gb and just switch between them or something?

Or is there a better solution to this problem? I tried a single big numpy memmap and it failed to load with some kind of similar error.

Thanks!