Compute hash for index incrementally

[edsko]

Right now we always compute the hash for the entire file. This is an optimisation only (and profiling suggests it won't make all that much of a difference). Note that we are now always downloading the .tar.gz, even when we are doing an incremental update, so instead of hashing a 300M file we are hashing a 30M file.