Make ID assignment deterministic

[zhaih]

Changes

1. Changes the script that build the binary LFDs and added ID to each line
2. Changes the way of assign IDs in LineFileDocs.java

[mikemccand]

Aha! Now I see #186! EventuallyConsistentMikeException.

You're right -- using AtomicInteger means the id assignment is non-deterministic!

But couldn't we still make it deterministic, in the binary case, by knowing the idBase of each block, and then each thread indexes that block by incrementing its id locally?

And in the text case, I think we hold a lock while reading the file, and we could do a simple id++ (perhaps on a volatile int, though I think because the same lock is held by each thread that does the increment, we may not need the volatile) there?

[zhaih]

Thanks @zhaih!

Do we think this might mean we can turn back on the rearrange step and use multiple threads to build the "for deterministic searching" index?

Yes I hope this is enough. I'll try to revive some of the memory and do the experiment locally and let you know what happened over the weekend!