elasticsearch
elasticsearch copied to clipboard
elastic
Include runtime fields in total fields count
We have a check that enforces the total number of fields needs to be below a certain (configurable) threshold. Runtime fields currently do not contribute to the count. There have been misunderstandings around this, caused by runtime fields not having a corresponding lucene field, but the reason to have the total fields limit is to limit the cluster state size as well as the memory footprint of mappings, which runtime fields certainly contribute to. For this reason we should fix this and count runtime fields as well.
I'm not sure we should do this. The main reason we have this field limit is to protect against mapping explosion from dynamic mappings. Dynamic mappings do not create runtime fields AFAIK (i.e. do not contribute to this explosion), so might not need to be considered for this check. Metadata fields don't contribute to the limit either (_id, _size, _seq_no, _routing, ...), so there is already a disparity between the fields in the system and the limit.
An issue which I have seen with the limit is that it's currently difficult to understand why the given limit is hit (in particular as there is no relation to the number of fields shown/counted by Kibana), which won't be addressed by the proposal here either. More and more people seem to be hitting the limit in cases we no longer consider as "abusive", so perhaps we should look at increasing the default soft-limit instead.
Dynamic mappings do not create runtime fields AFAIK (i.e. do not contribute to this explosion)
dynamic:runtime does :) which is what led me to open this issue. And the fact that we said we would make this change a few times in the past yet we haven't tracked it and forgot about it.
I agree though on opening a broader discussion around the field limit, especially as many consumers end up increasing that limit and it becomes unclear why we have a limit in the first place and what the real limit is. Possibly working on #86440 will help have more insight into what the limit should be nowadays.