weird struct.error when running xpore-diffmod

[pabloacera]

I am running xpore-diffmod with a reasonably large files and I get this error in the middle of the execution. Have you seen this error before?

[pm1122@gadi-hmem-clx-0013 WT2_RNAAB056712]$ /home/249/pm1122/raijin_home_2019-11-15/pm1122/lib/bin/xpore-diffmod --n_processes 5 --config /g/data/xc17/pm1122/yeast/yeast_config_test.yml --save_models
Using the signal of unmodified RNA from /home/249/pm1122/raijin_home_2019-11-15/pm1122/lib/lib/python3.6/site-packages/xpore/diffmod/model_kmer.csv
18 ids to be testing ...
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr1.hdf5 chr1
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr10.hdf5 chr10
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr11.hdf5 chr11
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr12.hdf5 chr12
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr13.hdf5 chr13
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr14.hdf5 chr14
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr15.hdf5 chr15
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr16.hdf5 chr16
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr2.hdf5 chr2
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr3.hdf5 chr3
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr4.hdf5 chr4
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr5.hdf5 chr5
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr6.hdf5 chr6
Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/queues.py", line 240, in _feed
    send_bytes(obj)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 204, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 397, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/scplasm1.hdf5 scplasm1
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr8.hdf5 chr8
/g/data/xc17/pm1122/yeast/diffmod_out_test/models/chr9.hdf5 chr9

I previously ran xpore-diffmod with even larger files and the error happened before processing chromosome 1. To be honest there is not much info about this error online and I have little idea about it, but I am guessing that it has to do with some kind of memory issue or loading of big files? maybe the data.json??

Just in case you can shed some light here! thanks!

[pabloacera]

This is an issue related to the upper limit size of objects share between processes, this issue is fixed in python 3.8 or above. Cheers!

[ploy-np]

@pabloacera Thank you very much. Yes, it was the memory issue. I've encountered this before when I used python 3.7 :(. I've changed the python version required for this package to >=3.8 already. Thanks again!