hail icon indicating copy to clipboard operation
hail copied to clipboard

MethodTooLargeException when exporting very wide Table

Open phildarnowsky-broad opened this issue 2 years ago • 2 comments

See https://github.com/broadinstitute/gnomad-browser/issues/914. In the line in question, we attempt to export a Table with ~13,000 columns, and get the following error: is.hail.relocated.org.objectweb.asm.MethodTooLargeException: Method too large: __C19580collect_distributed_array.__m19633split_InsertFields ()V (see above-referenced issue for full stacktrace).

Hail version was 0.2.96-39909e0a396f

phildarnowsky-broad avatar Jun 28 '22 19:06 phildarnowsky-broad

This is a known limitation -- it's hard to fix without significant investment in a backend we don't want to maintain forever. This issue generally pops up when people use tables instead of matrix tables to process wide files. Can you rewrite the GTEx processing script to use import_matrix_table instead of import_table, then process as a matrix?

tpoterba avatar Jun 30 '22 16:06 tpoterba

@tpoterba yes I think that's what we'll do when we come back to this, thanks

phildarnowsky-broad avatar Jul 05 '22 13:07 phildarnowsky-broad