Is there a limit on the size of a single serialized object?

[liuxin99]

I got this error:

Traceback (most recent call last):
  File "/user/liuxin/workspace/modelbest-audio-dataset/modelbest_audio_dataset/datasets/passage_level/scripts/wuxi/test_pipeline/simple_demucs_44k_test.py", line 326, in <module>
    main_cli()
  File "/user/liuxin/workspace/modelbest-audio-dataset/modelbest_audio_dataset/datasets/passage_level/scripts/wuxi/test_pipeline/simple_demucs_44k_test.py", line 322, in main_cli
    to_sstable(args.src_partition, args.tgt_partition, args.limit)
  File "/user/liuxin/workspace/modelbest-audio-dataset/modelbest_audio_dataset/datasets/passage_level/scripts/wuxi/test_pipeline/simple_demucs_44k_test.py", line 293, in to_sstable
    raise e
  File "/user/liuxin/workspace/modelbest-audio-dataset/modelbest_audio_dataset/datasets/passage_level/scripts/wuxi/test_pipeline/simple_demucs_44k_test.py", line 277, in to_sstable
    msg_size = len(msg.serialize())
  File "/user/liuxin/miniconda3/envs/sdk_env/lib/python3.10/site-packages/modelbest_sdk/dataset/thrift_wrapper/messages.py", line 286, in serialize
    return serialize(self.to_thrift())
  File "/user/liuxin/miniconda3/envs/sdk_env/lib/python3.10/site-packages/thriftpy2/utils.py", line 14, in serialize
    thrift_object.write(protocol)
  File "/user/liuxin/miniconda3/envs/sdk_env/lib/python3.10/site-packages/thriftpy2/thrift.py", line 158, in write
    oprot.write_struct(self)
  File "/user/liuxin/miniconda3/envs/sdk_env/lib/python3.10/site-packages/thriftpy2/protocol/binary.py", line 419, in write_struct
    write_val(self.trans, TType.STRUCT, obj)
  File "/user/liuxin/miniconda3/envs/sdk_env/lib/python3.10/site-packages/thriftpy2/protocol/binary.py", line 169, in write_val
    write_val(outbuf, f_type, v, f_container_spec)
  File "/user/liuxin/miniconda3/envs/sdk_env/lib/python3.10/site-packages/thriftpy2/protocol/binary.py", line 124, in write_val
    outbuf.write(pack_string(val))
  File "thriftpy2/transport/memory/cymemory.pyx", line 66, in thriftpy2.transport.memory.cymemory.TCyMemoryBuffer.write
  File "thriftpy2/transport/memory/cymemory.pyx", line 38, in thriftpy2.transport.memory.cymemory.TCyMemoryBuffer.c_write
MemoryError: Write to memory error

[aisk]

The error may be raised from here: https://github.com/Thriftpy/thriftpy2/blob/8e82f1c091a9d9db30dc01af25431d06056c3d88/thriftpy2/transport/cybase.pyx#L97-L99

It seems to be caused by a failed malloc, so I guess the real issue is that there wasn’t enough memory available on the server at that time.