elasticsearch
elasticsearch copied to clipboard
elastic
Allow writing to multiple indices via alias
When making a mapping change which cannot be made in-place, the current standard practice seems to be:
- create a new index with the desired mapping
- multiplex writes to both indices
- use the reindex API to copy old data
- only write to the new index
- delete the old index
Currently, steps 2 & 4 in that process must be done by updating all writers to the index, as "It is an error to index to an alias which points to more than one index." per the docs. Being able to use aliases to manage steps 2 & 4 in this process will make migrations much simpler.
Being able to atomically reindex & update the target of an alias (steps 2-4 above), all as a single step, would also solve this issue but might not be ideal for very large data loads.
This has also been discussed in #6240 but this use case was only really highlighted five years after the issue was closed.
+1 for this feature, it'll be good for reindexing when you have an option to do writes simultaneously to two indices so that the new index will always be in sync with the live index.
The primary usecase of the Reindex API is to allow zero downtime index migrations for mapping changes requiring reindexing.
But the Reindex API and aliases alone do not enable zero-downtime index migrations without potential data loss. To do this correctly requires custom Indexer work so that an Indexer is aware of the reindex and may write to both old and new indices while reindexing is happening.
This could happen here. I realize that forcing aliases to have a single write index is convenient for ILM and Rollover, but could there not be a way to override this for reindexing?
+1. This is an issue whenever doing zero downtime reindexing of mutable data that may be updated while the new index is being generated.
