IPRTable memory use

[greydot]

At the moment, IPRTable is quite a hungry beast. For example, a table of almost 700k entries without any data occupies at least several hundreds megabytes. This small test [1] peaks at 318 megabytes [2]. The networks list I used is here [3]. For comparison, the same code using Data.Map.Strict requires half that amount [4].

Is there a way to improve this? Another question is whether this is just an IPRTable problem, or are there any other improvements to make?

For example, the 'mask' field could be dropped from AddrRange since it is redundant. Without it and with mlen's type switched to Word8, memory use dropped to 295 megabytes, which is a bit better but clearly not enough.

1: https://gist.github.com/greydot/b61836d8a7991da1bad37ed4ac4b0584 2: > /usr/bin/time -v ./t networks 2>&1 | grep "Maximum res" Maximum resident set size (kbytes): 318756 3: https://gist.githubusercontent.com/greydot/8ad4b7c947d00060cd385402f36bf881/raw/ac645499a4a139ba0926a791cc0a8da52352bf3b/networks 4: > /usr/bin/time -v ./t networks 2>&1 | grep "Maximum res" Maximum resident set size (kbytes): 179364

[kazu-yamamoto]

I would be better to implement Poptrie to improve footprint: https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p57.pdf

[greydot]

@kazu-yamamoto are there any comparisons against the current implementation? From what I understand, Poptrie improves access time but doesn't reduce memory footprint.

[kazu-yamamoto]

Let's try.

Would you send a PR of Word8 first?

And tell me your GHC version.

[greydot]

Do you want just Word8 or do you want mask dropped as well?

[kazu-yamamoto]

Would you make two commits: one for Word8 and the other for mask?

[greydot]

Sure. I'll get to that later today.