MakeNDArray OOM on stream data

[ehigham]

What happened?

From: https://discuss.hail.is/t/connectionerror-after-mt-aggregate-cols-hl-agg-collect-and-hl-nd-array-in-linear-skat/3839

The following code snippet blows up from OOM. Interestingly, I can only reproduce for _localize=False, indicating we have an problem with our Emit rule for MakeNDArray for data IRs of type TStream.

hl.init()

mt = hl.utils.range_matrix_table(n_rows=7944, n_cols=442075)
covariates = [1.0]

mt = mt.select_cols(covariates=covariates)
covmat = mt.aggregate_cols(
    hl.agg.collect(mt.covariates.map(hl.float)),
    _localize=False,
)

hl.nd.array(covmat).show() # boom

Version

0.2.130

Relevant log output

No response