hail
hail copied to clipboard
[query] map in python should allow requiring memory management per row
If a user writes a map over a local array or stream value, which they know is large, they should be able to force scratch memory to be cleared after each row. This can be an underscored parameter for expert use only.
We would need to add a requires_memory_management_per_row
flag to the StreamMap
node, and expose it in user facing methods which generate a StreamMap
like ArrayExpression.map
. Then the rule in EmitStream
would need to combine this with the child stream's requiresMemoryManagementPerRow
.