mph icon indicating copy to clipboard operation
mph copied to clipboard

Published 20 hours ago • verified publisher icon alecthomas

Builder & mmap

Open raichu opened this issue 11 years ago • 3 comments

A major motivation for using mmap'ed files is whe the data-set does not fit in the memory. Currently, the builder uses a lot of memory.

It would be great to have a Builder that will work under these conditions.

raichu avatar Apr 08 '14 08:04 raichu

You should be able to pass mmapped slices as the inputs to the builder. Does that help your use case? Not that there aren't other inefficiencies though.

alecthomas avatar Apr 08 '14 11:04 alecthomas

@alecthomas - sorry it is not so clear to me what you mean. Could you give a small example? I have the same problem as @raichu where I run out of memory.

gofrailo avatar Jan 23 '15 08:01 gofrailo

Would be nice to see how to build an index for the 35 GB of the Have I been Pwned database.

ceving avatar Apr 05 '24 22:04 ceving