ValueError: bytes object is too large when using .sync_cube to save a trend stack

[georgeboldeanu]

Hello! After i compute the trend regression and try to save it with .sync_cube function I receive a strange error in dask client.

Traceback (most recent call last):
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/protocol/core.py", line 109, in dumps
    frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
  
File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/msgpack/__init__.py", line 36, in packb
    return Packer(**kwargs).pack(o)
  
File "msgpack/_packer.pyx", line 294, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 300, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 297, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 202, in msgpack._cmsgpack.Packer._pack

ValueError: bytes object is too large
2024-05-24 10:44:56,253 - distributed.comm.utils - ERROR - bytes object is too large
Traceback (most recent call last):
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/comm/utils.py", line 34, in _to_frames
    return list(protocol.dumps(msg, **kwargs))
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/protocol/core.py", line 109, in dumps
    frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/msgpack/__init__.py", line 36, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 294, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 300, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 297, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 202, in msgpack._cmsgpack.Packer._pack
ValueError: bytes object is too large
2024-05-24 10:44:56,255 - distributed.batched - ERROR - Error in batched write
Traceback (most recent call last):
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/batched.py", line 115, in _background_send
    nbytes = yield coro
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/tornado/gen.py", line 767, in run
    value = future.result()
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/comm/tcp.py", line 264, in write
    frames = await to_frames(
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/comm/utils.py", line 48, in to_frames
    return await offload(_to_frames)
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/utils.py", line 1540, in run_in_executor_with_context
    return await loop.run_in_executor(
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/utils.py", line 1541, in <lambda>
    executor, lambda: context.run(func, *args, **kwargs)
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/comm/utils.py", line 34, in _to_frames
    return list(protocol.dumps(msg, **kwargs))
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/distributed/protocol/core.py", line 109, in dumps
    frames[0] = msgpack.dumps(msg, default=_encode_default, use_bin_type=True)
  File "/home/gebo/py_venv/cophil/lib/python3.10/site-packages/msgpack/__init__.py", line 36, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 294, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 300, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 297, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 264, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 202, in msgpack._cmsgpack.Packer._pack
ValueError: bytes object is too large

From my basic understanding and some google searches its something related to file dimensions. But this behavior persists even when I try to save just the first pair from the stack. The stack is composed from 53x11138x7257 with a dimension per pair with 300mb and got a total dimension of 16 GBs. So, i don't think that the issues is directly related to filesizes.

Somehow, I can tackle this?

System and software version:

Server Specifications: 125 GB RAM, CPU: Intel(R) Xeon(R) Silver 4309Y CPU @ 2.80GHz OS: Ubuntu 22.04.2 LTS (GNU/Linux 6.5.0-28-generic x86_64) PyGMTSAR version: 2024.4.17.post2