When incrementally updating the index how much should I care about rebuilding the index?

[paulomann]

When incrementally updating the index removing/adding vectors, how much should I care to rebuild the index from time to time? Or only adding/removing vectors in an existing index is enough?

[mcelhennyi]

I am also interested in this answer.