annoy-java icon indicating copy to clipboard operation
annoy-java copied to clipboard

Published 20 hours ago • verified publisher icon spotify

Only supports files up to 2GB

Open a1k0n opened this issue 9 years ago • 4 comments

Because the API for MappedByteBuffer, or whatever it's called, uses Java int for offsets, you cannot address past 2GB. To work around this, annoy-java would have to open multiple windows into the file as separate buffers 2gb apart.

If there is a need for this, please comment here.

a1k0n avatar Jan 31 '15 03:01 a1k0n

This would be nice to have.

joprice avatar Jun 22 '16 23:06 joprice

+1 that this would be nice to have

flylo avatar Feb 28 '17 20:02 flylo

Hi! I am using this library now. I am curious whether files larger than 2GB can be supported now since my index file is around 100GB. Thanks.

Yuanrui1994 avatar Jun 21 '17 17:06 Yuanrui1994

Dear @Yuanrui1994, I've just submitted a PR with a solution to this restriction. It ran successfully in my index (3GB). It would be nice if you also execute some tests in your index of 100GB.

julsal avatar Jul 07 '17 01:07 julsal