qbatch
qbatch copied to clipboard
Documentation: There are limits to the "array job magics"
Student of mine was submitting a job with 500000 (long) lines and "-c 4000"
When jobs started, they all failed with:
/var/spool/torque/mom_priv/jobs/43286355-1.gpc-sched-ib0.SC: xrealloc: cannot allocate 18446744072881863172 bytes (249856 bytes allocated)
A quick bit of googling indicates that bash pipes have limits and we were overflowing some buffer somewhere.
(testing right now) using individual jobs resolves this.
We should maybe provide some warning about very very large jobfiles.