feat(sinks elasticsearch): Add compatibility between elasticsearch cl…

[gmicouin]

trafficstars

Add compatibility between the elasticsearch client and elasticsearch server in 8.x and after 7.12 (tested from 7.12)

[bits-bot]

All committers have signed the CLA.

[netlify[bot]]

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	c89a65642aaf6082495388b86b7f0edb294e2667
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/62f1346447fc1c0008af4809

[tobz]

@gmicouin Do you mind providing references to the documentation used to show why this should be valid?

[github-actions[bot]]

Soak Test Results

Baseline: 4611307463ced2d069e806666178a3977265a3bc Comparison: 13ecb127c6fac6364e69892a98edbaa8d7ba3666 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

Changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

experiment	Δ mean	Δ mean %	confidence
datadog_agent_remap_blackhole	-6.3MiB	-9.88	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
datadog_agent_remap_blackhole_acks	482.23KiB	0.74	100.00%	64.0MiB	4.35MiB	90.5KiB	0	0.0678777	64.47MiB	2.57MiB	53.81KiB	0	0.0398669	False	False
splunk_hec_route_s3	75.03KiB	0.39	74.43%	18.79MiB	2.3MiB	47.86KiB	0	0.122264	18.87MiB	2.17MiB	45.44KiB	0	0.115016	False	False
http_pipelines_blackhole_acks	3.68KiB	0.31	90.73%	1.17MiB	91.15KiB	1.86KiB	0	0.0760127	1.17MiB	56.63KiB	1.16KiB	0	0.047075	False	False
syslog_splunk_hec_logs	23.73KiB	0.15	80.50%	15.48MiB	703.32KiB	14.31KiB	0	0.0443546	15.51MiB	559.95KiB	11.42KiB	0	0.0352598	False	False
splunk_hec_to_splunk_hec_logs_noack	37.34KiB	0.15	99.12%	23.8MiB	615.98KiB	12.56KiB	0	0.0252698	23.84MiB	328.96KiB	6.71KiB	0	0.0134746	False	False
socket_to_socket_blackhole	16.0KiB	0.07	68.43%	22.69MiB	556.81KiB	11.37KiB	0	0.0239638	22.7MiB	548.14KiB	11.19KiB	0	0.0235744	False	False
splunk_hec_indexer_ack_blackhole	16.81KiB	0.07	50.10%	23.75MiB	892.67KiB	18.16KiB	0	0.0366991	23.77MiB	834.86KiB	16.99KiB	0	0.0342986	False	False
enterprise_http_to_http	-1.13KiB	-0	12.31%	23.85MiB	252.79KiB	5.16KiB	0	0.01035	23.85MiB	254.33KiB	5.2KiB	0	0.0104135	False	False
syslog_log2metric_humio_metrics	-1.91KiB	-0.01	12.60%	13.13MiB	317.64KiB	6.48KiB	0	0.0236224	13.13MiB	499.01KiB	10.16KiB	0	0.0371159	False	False
syslog_log2metric_splunk_hec_metrics	-5.48KiB	-0.03	19.87%	17.54MiB	738.05KiB	15.04KiB	0	0.0410813	17.54MiB	773.48KiB	15.75KiB	0	0.0430665	False	False
splunk_hec_to_splunk_hec_logs_acks	-7.21KiB	-0.03	22.97%	23.75MiB	843.23KiB	17.15KiB	0	0.0346593	23.75MiB	874.26KiB	17.78KiB	0	0.0359452	False	False
http_to_http_acks	-5.58KiB	-0.03	1.86%	17.22MiB	8.12MiB	169.66KiB	0	0.471379	17.21MiB	8.07MiB	168.32KiB	0	0.468869	True	True
file_to_blackhole	-45.69KiB	-0.05	35.33%	95.34MiB	3.09MiB	64.01KiB	0	0.0323821	95.3MiB	3.67MiB	76.41KiB	0	0.0385205	False	False
http_text_to_http_json	-22.5KiB	-0.06	62.90%	38.33MiB	867.33KiB	17.7KiB	0	0.0220946	38.31MiB	874.7KiB	17.86KiB	0	0.0222951	False	False
http_to_http_json	-28.17KiB	-0.12	97.97%	23.85MiB	343.28KiB	7.01KiB	0	0.0140539	23.82MiB	484.38KiB	9.9KiB	0	0.0198538	False	False
syslog_regex_logs2metric_ddmetrics	-20.08KiB	-0.16	77.82%	12.33MiB	553.95KiB	11.29KiB	0	0.0438668	12.31MiB	586.24KiB	11.94KiB	0	0.0464977	False	False
syslog_humio_logs	-33.77KiB	-0.21	100.00%	16.07MiB	235.58KiB	4.81KiB	0	0.0143111	16.04MiB	167.49KiB	3.43KiB	0	0.0101959	False	False
datadog_agent_remap_datadog_logs_acks	-166.68KiB	-0.26	84.46%	63.67MiB	3.36MiB	70.16KiB	0	0.0527299	63.5MiB	4.52MiB	94.0KiB	0	0.0710954	False	False
http_pipelines_blackhole	-5.12KiB	-0.31	99.76%	1.64MiB	9.83KiB	205.8B	0	0.00586611	1.63MiB	81.94KiB	1.67KiB	0	0.049059	False	False
http_to_http_noack	-78.3KiB	-0.32	99.71%	23.82MiB	611.33KiB	12.5KiB	0	0.0250578	23.74MiB	1.11MiB	23.13KiB	0	0.0467123	False	False
fluent_elasticsearch	-336.5KiB	-0.41	99.99%	79.47MiB	53.47KiB	1.08KiB	0	0.000656879	79.14MiB	4.13MiB	84.78KiB	0	0.052138	False	False
datadog_agent_remap_datadog_logs	-360.01KiB	-0.55	100.00%	63.66MiB	527.54KiB	10.8KiB	0	0.00809135	63.31MiB	4.15MiB	86.52KiB	0	0.0656045	False	False
syslog_loki	-127.1KiB	-0.84	100.00%	14.76MiB	260.4KiB	5.33KiB	0	0.0172294	14.63MiB	727.28KiB	14.78KiB	0	0.0485285	False	False
http_pipelines_no_grok_blackhole	-114.04KiB	-1.02	100.00%	10.96MiB	273.63KiB	5.59KiB	0	0.0243674	10.85MiB	1.08MiB	22.53KiB	0	0.0996629	False	False
datadog_agent_remap_blackhole	-6.3MiB	-9.88	100.00%	63.75MiB	3.93MiB	81.9KiB	0	0.0615684	57.45MiB	6.83MiB	142.65KiB	0	0.118941	False	False

[github-actions[bot]]

Soak Test Results

Baseline: bdff6b8c8a17ef01deb9c8c673fa6bfa4dab37e3 Comparison: c89a65642aaf6082495388b86b7f0edb294e2667 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
datadog_agent_remap_blackhole	1.92MiB	3.28	100.00%	58.52MiB	4.67MiB	97.24KiB	0	0.0797349	60.43MiB	4.18MiB	87.28KiB	0	0.0692127	False	False
datadog_agent_remap_datadog_logs	1.32MiB	2.18	100.00%	60.47MiB	2.79MiB	58.52KiB	0	0.046196	61.79MiB	4.47MiB	93.04KiB	0	0.0722774	False	False
syslog_splunk_hec_logs	337.02KiB	2.1	100.00%	15.65MiB	953.88KiB	19.39KiB	0	0.0595119	15.98MiB	741.19KiB	15.13KiB	0	0.0452899	False	False
splunk_hec_route_s3	125.71KiB	0.64	95.00%	19.1MiB	2.22MiB	46.33KiB	0	0.116383	19.22MiB	2.12MiB	44.35KiB	0	0.110314	False	False
datadog_agent_remap_blackhole_acks	242.21KiB	0.36	97.63%	65.35MiB	4.22MiB	87.94KiB	0	0.0645689	65.59MiB	2.92MiB	61.09KiB	0	0.0445233	False	False
socket_to_socket_blackhole	76.24KiB	0.32	100.00%	23.41MiB	195.7KiB	3.99KiB	0	0.00816285	23.48MiB	150.19KiB	3.07KiB	0	0.00624458	False	False
http_pipelines_blackhole_acks	2.81KiB	0.25	77.57%	1.09MiB	95.71KiB	1.95KiB	0	0.0853736	1.1MiB	61.06KiB	1.25KiB	0	0.0543325	False	False
syslog_regex_logs2metric_ddmetrics	21.1KiB	0.17	69.15%	12.35MiB	715.31KiB	14.57KiB	0	0.0565458	12.37MiB	722.31KiB	14.72KiB	0	0.057004	False	False
splunk_hec_to_splunk_hec_logs_noack	16.69KiB	0.07	84.08%	23.82MiB	476.4KiB	9.72KiB	0	0.0195264	23.84MiB	331.97KiB	6.78KiB	0	0.0135974	False	False
splunk_hec_indexer_ack_blackhole	11.22KiB	0.05	34.89%	23.75MiB	887.26KiB	18.04KiB	0	0.0364809	23.76MiB	836.94KiB	17.03KiB	0	0.0343959	False	False
splunk_hec_to_splunk_hec_logs_acks	10.24KiB	0.04	33.73%	23.76MiB	832.47KiB	16.94KiB	0	0.034215	23.77MiB	798.45KiB	16.25KiB	0	0.0328026	False	False
enterprise_http_to_http	-1.77KiB	-0.01	19.58%	23.85MiB	245.23KiB	5.01KiB	0	0.0100402	23.85MiB	248.15KiB	5.08KiB	0	0.0101608	False	False
file_to_blackhole	-57.16KiB	-0.06	37.97%	95.33MiB	3.79MiB	78.62KiB	0	0.0397746	95.28MiB	4.06MiB	84.41KiB	0	0.0426211	False	False
http_to_http_json	-19.22KiB	-0.08	90.88%	23.85MiB	340.89KiB	6.96KiB	0	0.013957	23.83MiB	439.93KiB	9.0KiB	0	0.0180259	False	False
syslog_loki	-25.8KiB	-0.17	78.51%	14.44MiB	588.36KiB	12.05KiB	0	0.0397695	14.42MiB	834.22KiB	16.96KiB	0	0.0564863	False	False
syslog_humio_logs	-32.72KiB	-0.2	100.00%	15.8MiB	180.53KiB	3.69KiB	0	0.0111564	15.77MiB	188.86KiB	3.87KiB	0	0.011695	False	False
http_pipelines_blackhole	-5.91KiB	-0.37	99.82%	1.56MiB	44.29KiB	926.1B	0	0.0276921	1.56MiB	81.4KiB	1.66KiB	0	0.0510881	False	False
http_to_http_noack	-91.18KiB	-0.37	100.00%	23.84MiB	264.25KiB	5.4KiB	0	0.0108207	23.75MiB	1.04MiB	21.76KiB	0	0.04391	False	False
syslog_log2metric_splunk_hec_metrics	-63.27KiB	-0.37	99.92%	16.83MiB	487.62KiB	9.95KiB	0	0.0282824	16.77MiB	783.55KiB	15.95KiB	0	0.0456141	False	False
syslog_log2metric_humio_metrics	-54.87KiB	-0.43	99.96%	12.59MiB	482.87KiB	9.85KiB	0	0.0374593	12.53MiB	588.31KiB	11.98KiB	0	0.0458343	False	False
fluent_elasticsearch	-371.65KiB	-0.46	100.00%	79.47MiB	53.96KiB	1.09KiB	0	0.000662905	79.11MiB	4.08MiB	83.95KiB	0	0.0515843	False	False
http_text_to_http_json	-279.47KiB	-0.7	100.00%	39.22MiB	796.43KiB	16.26KiB	0	0.0198279	38.94MiB	851.0KiB	17.37KiB	0	0.021335	False	False
datadog_agent_remap_datadog_logs_acks	-561.92KiB	-0.86	100.00%	63.73MiB	3.35MiB	70.06KiB	0	0.0525997	63.18MiB	4.49MiB	93.38KiB	0	0.0709845	False	False
http_pipelines_no_grok_blackhole	-134.03KiB	-1.19	100.00%	11.03MiB	165.77KiB	3.38KiB	0	0.0146788	10.9MiB	1.07MiB	22.18KiB	0	0.0977322	False	False
http_to_http_acks	-332.5KiB	-1.84	83.94%	17.68MiB	8.04MiB	168.18KiB	0	0.454911	17.36MiB	8.0MiB	166.89KiB	0	0.461104	True	True

[github-actions[bot]]

Soak Test Results

Baseline: 4611307463ced2d069e806666178a3977265a3bc Comparison: 1608022b983ca02762fb1cafb41eac2a85a0dfe2 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	7.49KiB	0.62	99.39%	1.18MiB	112.55KiB	2.29KiB	0	0.0930851	1.19MiB	73.14KiB	1.49KiB	0	0.0601143	False	False
http_text_to_http_json	182.19KiB	0.46	100.00%	39.09MiB	875.96KiB	17.88KiB	0	0.0218785	39.27MiB	850.63KiB	17.36KiB	0	0.0211495	False	False
splunk_hec_route_s3	82.38KiB	0.44	78.34%	18.46MiB	2.32MiB	48.22KiB	0	0.125422	18.54MiB	2.2MiB	46.02KiB	0	0.118577	False	False
syslog_loki	40.96KiB	0.28	93.91%	14.22MiB	627.42KiB	12.83KiB	0	0.0430783	14.26MiB	870.0KiB	17.69KiB	0	0.059566	False	False
syslog_splunk_hec_logs	37.89KiB	0.23	95.73%	16.27MiB	753.23KiB	15.33KiB	0	0.0451923	16.31MiB	523.83KiB	10.7KiB	0	0.0313574	False	False
syslog_regex_logs2metric_ddmetrics	26.7KiB	0.21	89.66%	12.38MiB	581.62KiB	11.85KiB	0	0.0458575	12.41MiB	555.84KiB	11.32KiB	0	0.0437329	False	False
datadog_agent_remap_blackhole_acks	66.78KiB	0.11	52.29%	59.41MiB	3.94MiB	82.03KiB	0	0.0662953	59.47MiB	2.19MiB	45.75KiB	0	0.0367535	False	False
splunk_hec_indexer_ack_blackhole	18.59KiB	0.08	55.32%	23.74MiB	880.95KiB	17.91KiB	0	0.0362268	23.76MiB	816.56KiB	16.62KiB	0	0.0335533	False	False
splunk_hec_to_splunk_hec_logs_noack	16.08KiB	0.07	82.77%	23.82MiB	473.0KiB	9.66KiB	0	0.0193872	23.84MiB	330.25KiB	6.74KiB	0	0.0135272	False	False
enterprise_http_to_http	-1.39KiB	-0.01	15.49%	23.85MiB	245.79KiB	5.02KiB	0	0.0100633	23.85MiB	248.16KiB	5.08KiB	0	0.0101609	False	False
socket_to_socket_blackhole	-10.61KiB	-0.05	48.07%	22.72MiB	537.88KiB	10.98KiB	0	0.0231159	22.71MiB	601.07KiB	12.27KiB	0	0.0258433	False	False
splunk_hec_to_splunk_hec_logs_acks	-11.33KiB	-0.05	36.74%	23.76MiB	800.04KiB	16.28KiB	0	0.03287	23.75MiB	846.33KiB	17.22KiB	0	0.034788	False	False
syslog_humio_logs	-18.59KiB	-0.11	100.00%	16.57MiB	135.21KiB	2.76KiB	0	0.00796479	16.56MiB	131.13KiB	2.68KiB	0	0.0077327	False	False
fluent_elasticsearch	-94.2KiB	-0.12	100.00%	79.47MiB	53.28KiB	1.08KiB	0	0.000654543	79.38MiB	1.03MiB	21.13KiB	0	0.0129194	False	False
file_to_blackhole	-115.59KiB	-0.12	68.54%	95.34MiB	3.15MiB	65.3KiB	0	0.0330333	95.23MiB	4.55MiB	94.56KiB	0	0.0477639	False	False
http_to_http_json	-32.86KiB	-0.13	98.99%	23.85MiB	348.28KiB	7.11KiB	0	0.0142601	23.81MiB	519.03KiB	10.6KiB	0	0.0212798	False	False
syslog_log2metric_splunk_hec_metrics	-25.83KiB	-0.14	83.11%	17.86MiB	609.47KiB	12.42KiB	0	0.0333187	17.83MiB	691.37KiB	14.08KiB	0	0.0378495	False	False
datadog_agent_remap_datadog_logs_acks	-131.71KiB	-0.21	71.96%	59.88MiB	3.56MiB	74.42KiB	0	0.0594785	59.75MiB	4.65MiB	96.69KiB	0	0.0777293	False	False
http_to_http_noack	-109.49KiB	-0.45	100.00%	23.85MiB	259.98KiB	5.32KiB	0	0.0106445	23.74MiB	1.12MiB	23.4KiB	0	0.0472742	False	False
http_pipelines_blackhole	-9.69KiB	-0.57	99.99%	1.65MiB	12.32KiB	257.89B	0	0.00730607	1.64MiB	117.37KiB	2.39KiB	0	0.0699935	False	False
http_to_http_acks	-108.59KiB	-0.61	35.31%	17.42MiB	8.14MiB	170.16KiB	0	0.467324	17.31MiB	7.9MiB	164.99KiB	0	0.456474	True	True
syslog_log2metric_humio_metrics	-91.9KiB	-0.68	100.00%	13.15MiB	287.88KiB	5.88KiB	0	0.0213723	13.06MiB	547.31KiB	11.14KiB	0	0.0409122	False	False
datadog_agent_remap_blackhole	-733.48KiB	-1.17	100.00%	61.05MiB	5.43MiB	113.21KiB	0	0.0889237	60.33MiB	5.01MiB	104.56KiB	0	0.083018	False	False
http_pipelines_no_grok_blackhole	-212.43KiB	-1.89	100.00%	10.96MiB	566.04KiB	11.55KiB	0	0.0504449	10.75MiB	1.18MiB	24.57KiB	0	0.109752	False	False
datadog_agent_remap_datadog_logs	-1.45MiB	-2.34	100.00%	62.16MiB	2.58MiB	54.05KiB	0	0.0415071	60.7MiB	4.98MiB	103.79KiB	0	0.0821009	False	False

[gmicouin]

Hi @tobz ! You can find here an example for the python client to enable the compatibility with elastic v8.

And here the code of official python client which replaces the Content-Type and Accept header by "application/vnd.elasticsearch+json;compatible-with=7"

I'm newbie in Rust, perhaps a better way exists to do that ;)

[tobz]

@gmicouin No worries, your change is totally valid.

My main question would be: it sounds like, from the PR description, you tested this against Elasticsearch 7.12, but did you also test it against 8.x? Did it not work with 8.x without this change?

From reading the REST API compatibility documentation, it appears that there also needs to be an Accept header on requests, in addition to Content-Type if there's a post body... but we don't send an Accept header. I'm trying to understand if the documentation is wrong, or if we're setting ourselves up for some incompatibility down the road.

[gmicouin]

@tobz Yes I test this code with Elastic 8.x, and without this change the Elastic server 8.0 does not accept the request.

Yes you're right, I have forgotten the Accept header, because during my tests it was possible to configure it with requests.header.

Do you think it would be possible to configure the Content-Type in the vector.toml ? It would be more generic, what do you think ?

[tobz]

Yes I test this code with Elastic 8.x, and without this change the Elastic server 8.0 does not accept the request.

Yes you're right, I have forgotten the Accept header, because during my tests it was possible to configure it with requests.header.

@gmicouin OK, great. I'd say that this PR should also be setting the Accept header then, as well.

Do you think it would be possible to configure the Content-Type in the vector.toml ? It would be more generic, what do you think ?

It's technically possible, but it's not a change we would make. Elasticsearch only supports a given set of encodings, and we need to ensure the codec we use to generate bulk requests matches what we send in Content-Type, so exposing the ability to override Content-Type just makes it easy to misconfigure the sink.

I think what we'd want to do, if anything, is expose a configuration setting that allows setting the version value that gets used in compatibility mode i.e. compatible_version = 7, which would then get interpolated in the Accept/Content-Type headers.

But like I said above, let's do the change to also update the Accept header. I still need to figure out what our actual ES compatibility guarantees are, to think about whether this current PR is acceptable as-is or if we need to expose something to allow configuring the compatibility headers (the idea about compatible_version).

[gmicouin]

Hi @tobz ! It's a good idea this compatible_version. I have seen how could I do this implementation about the Accept, but I have no knowledge in Rust :s I think we could do something like this

for (header, value) in &self.http_request_config.headers {
  let new_value = &value
  if "Accept".eq(header){
    new_value = &value +"," + "application/vnd.elasticsearch+json;compatible-with=7"
  } else {
    new_value = "application/vnd.elasticsearch+json;compatible-with=7"
  }
  builder = builder.header(&header[..], &new_value);
}

But I know this code doesn't compile, and I don't know how do that :s

[spencergilbert]

If we're considering adding a compatibile_version option, we should consider folding suppress_type_name into that as well as it's used for compatibility reasons as well.

Going even further we should be able to query the cluster on startup and identify the version from that request and self-configure the sink at runtime, rather than having the user configure ahead of time. I haven't fully thought through the implications of this, and it's certainly a larger change.

[gmicouin]

Hi @spencergilbert, Query the cluster to identify the version is a good idea, but I think this can be have limitations. For security reason, the version api is not authorized for all users. (See here)

[StephenWakely]

We are closing this for now in favour of coming up with a better way to handling behaviour appropriately according to a specified or detected version of ElasticSearch.

Ref #10342