qbatch icon indicating copy to clipboard operation
qbatch copied to clipboard

Documentation: There are limits to the "array job magics"

Open gdevenyi opened this issue 7 years ago • 0 comments

Student of mine was submitting a job with 500000 (long) lines and "-c 4000"

When jobs started, they all failed with:

/var/spool/torque/mom_priv/jobs/43286355-1.gpc-sched-ib0.SC: xrealloc: cannot allocate 18446744072881863172 bytes (249856 bytes allocated)

A quick bit of googling indicates that bash pipes have limits and we were overflowing some buffer somewhere.

(testing right now) using individual jobs resolves this.

We should maybe provide some warning about very very large jobfiles.

gdevenyi avatar Jul 11 '17 17:07 gdevenyi