vector
                                
                                 vector copied to clipboard
                                
                                    vector copied to clipboard
                            
                            
                            
                        feat(enterprise): add hostname field when reporting a configuration version
To avoid reporting several time the same configuration version for a given host, we need to report the current hostname. This PR just adds the hostname field to the reported data.
Related to https://datadoghq.atlassian.net/browse/OPB-80
Signed-off-by: Jeremie Drouet [email protected]
Deploy Preview for vector-project canceled.
| Name | Link | 
|---|---|
| Latest commit | 51811214d6a3b6398a3df3558c759c73533f5bf0 | 
| Latest deploy log | https://app.netlify.com/sites/vector-project/deploys/631625d2d3135b0008e69714 | 
Without totally being aware of the use-case here, this would likely return the name of Vector's pod when running in kubernetes.
Soak Test Results
Baseline: 05a318729cbcd3733957da7c3f6cf1d9bf70283b Comparison: 6a97e86131a4e0a98861c2fdedc322ab0becde98 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic | 
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_loki | 544.39KiB | 3.8 | 100.00% | 13.98MiB | 469.39KiB | 9.62KiB | 0 | 0.032786 | 14.51MiB | 750.31KiB | 15.25KiB | 0 | 0.0504879 | False | False | 
| socket_to_socket_blackhole | 306.46KiB | 2.23 | 100.00% | 13.41MiB | 111.48KiB | 2.28KiB | 0 | 0.00811715 | 13.71MiB | 127.99KiB | 2.61KiB | 0 | 0.00911596 | False | False | 
| syslog_regex_logs2metric_ddmetrics | 239.9KiB | 1.85 | 100.00% | 12.7MiB | 717.01KiB | 14.61KiB | 0 | 0.0551352 | 12.93MiB | 671.57KiB | 13.7KiB | 0 | 0.0507057 | False | False | 
| datadog_agent_remap_blackhole | 789.16KiB | 1.23 | 100.00% | 62.78MiB | 3.91MiB | 81.49KiB | 0 | 0.062232 | 63.55MiB | 3.07MiB | 64.07KiB | 0 | 0.0483025 | False | False | 
| datadog_agent_remap_blackhole_acks | 735.58KiB | 1.12 | 100.00% | 64.2MiB | 4.51MiB | 93.95KiB | 0 | 0.0702695 | 64.92MiB | 3.09MiB | 64.59KiB | 0 | 0.0475837 | False | False | 
| syslog_splunk_hec_logs | 147.59KiB | 0.84 | 100.00% | 17.21MiB | 880.26KiB | 17.92KiB | 0 | 0.0499436 | 17.35MiB | 769.08KiB | 15.67KiB | 0 | 0.0432734 | False | False | 
| splunk_hec_route_s3 | 164.61KiB | 0.84 | 98.84% | 19.04MiB | 2.28MiB | 47.46KiB | 0 | 0.119675 | 19.2MiB | 2.14MiB | 44.68KiB | 0 | 0.111268 | False | False | 
| http_text_to_http_json | 295.74KiB | 0.77 | 100.00% | 37.56MiB | 837.32KiB | 17.09KiB | 0 | 0.0217643 | 37.85MiB | 894.53KiB | 18.26KiB | 0 | 0.0230742 | False | False | 
| syslog_humio_logs | 129.51KiB | 0.74 | 100.00% | 16.98MiB | 377.5KiB | 7.71KiB | 0 | 0.0217025 | 17.11MiB | 357.71KiB | 7.32KiB | 0 | 0.0204128 | False | False | 
| syslog_log2metric_splunk_hec_metrics | 104.83KiB | 0.54 | 100.00% | 18.94MiB | 583.5KiB | 11.89KiB | 0 | 0.0300729 | 19.05MiB | 731.77KiB | 14.91KiB | 0 | 0.0375118 | False | False | 
| splunk_hec_to_splunk_hec_logs_noack | 32.02KiB | 0.13 | 98.23% | 23.81MiB | 572.88KiB | 11.68KiB | 0 | 0.0234941 | 23.84MiB | 331.19KiB | 6.76KiB | 0 | 0.0135643 | False | False | 
| splunk_hec_to_splunk_hec_logs_acks | 9.34KiB | 0.04 | 31.04% | 23.75MiB | 832.0KiB | 16.92KiB | 0 | 0.0341983 | 23.76MiB | 792.47KiB | 16.13KiB | 0 | 0.0325609 | False | False | 
| http_pipelines_blackhole_acks | 141.69B | 0.01 | 3.78% | 1.15MiB | 114.26KiB | 2.33KiB | 0 | 0.0966561 | 1.15MiB | 86.53KiB | 1.76KiB | 0 | 0.0731906 | False | False | 
| enterprise_http_to_http | -1.14KiB | -0 | 13.04% | 23.85MiB | 239.59KiB | 4.89KiB | 0 | 0.0098096 | 23.85MiB | 241.43KiB | 4.94KiB | 0 | 0.00988522 | False | False | 
| http_pipelines_blackhole | -296.56B | -0.02 | 10.38% | 1.63MiB | 21.34KiB | 446.56B | 0 | 0.0128159 | 1.63MiB | 106.8KiB | 2.18KiB | 0 | 0.0641457 | False | False | 
| splunk_hec_indexer_ack_blackhole | -10.89KiB | -0.04 | 32.11% | 23.75MiB | 892.56KiB | 18.16KiB | 0 | 0.0366905 | 23.74MiB | 935.02KiB | 19.01KiB | 0 | 0.0384531 | False | False | 
| file_to_blackhole | -46.42KiB | -0.05 | 33.21% | 95.34MiB | 3.51MiB | 72.76KiB | 0 | 0.0368059 | 95.29MiB | 3.85MiB | 80.07KiB | 0 | 0.040406 | False | False | 
| http_to_http_json | -33.73KiB | -0.14 | 99.23% | 23.85MiB | 338.6KiB | 6.91KiB | 0 | 0.0138634 | 23.81MiB | 518.86KiB | 10.6KiB | 0 | 0.0212733 | False | False | 
| http_to_http_noack | -58.67KiB | -0.24 | 98.61% | 23.82MiB | 591.9KiB | 12.1KiB | 0 | 0.0242611 | 23.76MiB | 1008.62KiB | 20.55KiB | 0 | 0.0414417 | False | False | 
| fluent_elasticsearch | -277.14KiB | -0.34 | 99.99% | 79.47MiB | 87.57KiB | 1.77KiB | 0 | 0.00107588 | 79.2MiB | 3.53MiB | 72.63KiB | 0 | 0.0446072 | False | False | 
| datadog_agent_remap_datadog_logs_acks | -320.27KiB | -0.49 | 99.57% | 63.87MiB | 2.9MiB | 60.62KiB | 0 | 0.0453624 | 63.56MiB | 4.53MiB | 94.19KiB | 0 | 0.0711825 | False | False | 
| http_pipelines_no_grok_blackhole | -57.05KiB | -0.5 | 98.47% | 11.23MiB | 159.82KiB | 3.26KiB | 0 | 0.0138907 | 11.18MiB | 1.12MiB | 23.29KiB | 0 | 0.100058 | False | False | 
| datadog_agent_remap_datadog_logs | -398.95KiB | -0.61 | 99.98% | 63.65MiB | 1.94MiB | 40.7KiB | 0 | 0.0304863 | 63.26MiB | 4.71MiB | 98.12KiB | 0 | 0.0744682 | False | False | 
| syslog_log2metric_humio_metrics | -100.18KiB | -0.74 | 100.00% | 13.25MiB | 383.13KiB | 7.82KiB | 0 | 0.0282232 | 13.16MiB | 567.02KiB | 11.54KiB | 0 | 0.0420798 | False | False | 
| http_to_http_acks | -593.06KiB | -3.1 | 99.40% | 18.7MiB | 7.07MiB | 147.78KiB | 0 | 0.377822 | 18.13MiB | 7.51MiB | 156.9KiB | 0 | 0.414472 | True | True | 
Without totally being aware of the use-case here, this would likely return the name of Vector's pod when running in kubernetes.
@spencergilbert is there a way to get that from the environment variable? I've the impression that it's the same as the HOSTNAME 🤔
The goal of this hostname field is just to identify an instance of vector with it's version when reporting a new configuration to the obs pipeline
If they're using the default Helm chart - we inject the actual node's name as a variable: https://github.com/vectordotdev/helm-charts/blob/develop/charts/vector/templates/_pod.tpl#L53-L56
This seems to be the generally suggested method, otherwise there's API calls to kubernetes infrastructure or weird attempts at parsing machine-id.
Soak Test Results
Baseline: f64bceffe36cf987b23bd541bebc5ef8498d8171 Comparison: 2dfa7bf29700c9dde7b82780292097647f85446c Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic | 
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_blackhole | 1.23MiB | 1.99 | 100.00% | 61.63MiB | 4.54MiB | 94.63KiB | 0 | 0.0736738 | 62.85MiB | 2.84MiB | 59.27KiB | 0 | 0.0451477 | False | False | 
| splunk_hec_route_s3 | 119.47KiB | 0.63 | 92.06% | 18.54MiB | 2.33MiB | 48.55KiB | 0 | 0.125696 | 18.66MiB | 2.28MiB | 47.73KiB | 0 | 0.122412 | False | False | 
| datadog_agent_remap_blackhole_acks | 213.92KiB | 0.36 | 95.16% | 57.5MiB | 4.14MiB | 86.11KiB | 0 | 0.0719129 | 57.71MiB | 3.15MiB | 65.73KiB | 0 | 0.0544883 | False | False | 
| http_pipelines_blackhole_acks | 2.06KiB | 0.17 | 57.57% | 1.17MiB | 95.33KiB | 1.94KiB | 0 | 0.0793111 | 1.18MiB | 83.07KiB | 1.69KiB | 0 | 0.068994 | False | False | 
| splunk_hec_to_splunk_hec_logs_noack | 22.04KiB | 0.09 | 92.48% | 23.82MiB | 507.1KiB | 10.35KiB | 0 | 0.0207895 | 23.84MiB | 333.0KiB | 6.8KiB | 0 | 0.0136396 | False | False | 
| splunk_hec_to_splunk_hec_logs_acks | 5.43KiB | 0.02 | 17.90% | 23.75MiB | 846.33KiB | 17.21KiB | 0 | 0.0347889 | 23.76MiB | 820.17KiB | 16.69KiB | 0 | 0.0337063 | False | False | 
| enterprise_http_to_http | -746.53B | -0 | 8.22% | 23.85MiB | 246.93KiB | 5.04KiB | 0 | 0.0101103 | 23.85MiB | 242.12KiB | 4.95KiB | 0 | 0.00991333 | False | False | 
| splunk_hec_indexer_ack_blackhole | -10.81KiB | -0.04 | 33.58% | 23.76MiB | 842.98KiB | 17.15KiB | 0 | 0.0346457 | 23.75MiB | 887.1KiB | 18.04KiB | 0 | 0.036475 | False | False | 
| file_to_blackhole | -35.02KiB | -0.04 | 23.09% | 95.33MiB | 4.04MiB | 83.83KiB | 0 | 0.0424097 | 95.29MiB | 4.08MiB | 84.92KiB | 0 | 0.0428169 | False | False | 
| http_to_http_json | -24.52KiB | -0.1 | 95.70% | 23.84MiB | 350.72KiB | 7.16KiB | 0 | 0.014361 | 23.82MiB | 478.19KiB | 9.78KiB | 0 | 0.0196003 | False | False | 
| syslog_humio_logs | -21.58KiB | -0.12 | 94.11% | 17.03MiB | 531.33KiB | 10.85KiB | 0 | 0.0304534 | 17.01MiB | 174.21KiB | 3.57KiB | 0 | 0.00999755 | False | False | 
| fluent_elasticsearch | -160.76KiB | -0.2 | 100.00% | 79.47MiB | 53.7KiB | 1.09KiB | 0 | 0.000659736 | 79.32MiB | 1.48MiB | 30.51KiB | 0 | 0.018685 | False | False | 
| http_to_http_noack | -58.36KiB | -0.24 | 99.85% | 23.85MiB | 260.06KiB | 5.32KiB | 0 | 0.0106483 | 23.79MiB | 861.06KiB | 17.55KiB | 0 | 0.035341 | False | False | 
| syslog_regex_logs2metric_ddmetrics | -44.15KiB | -0.34 | 98.90% | 12.78MiB | 646.65KiB | 13.17KiB | 0 | 0.0493896 | 12.74MiB | 554.52KiB | 11.3KiB | 0 | 0.0424958 | False | False | 
| syslog_splunk_hec_logs | -84.19KiB | -0.49 | 99.99% | 16.7MiB | 823.31KiB | 16.75KiB | 0 | 0.0481463 | 16.61MiB | 683.61KiB | 13.94KiB | 0 | 0.0401748 | False | False | 
| syslog_log2metric_splunk_hec_metrics | -99.46KiB | -0.54 | 100.00% | 18.14MiB | 687.26KiB | 14.01KiB | 0 | 0.0369996 | 18.04MiB | 863.32KiB | 17.57KiB | 0 | 0.0467284 | False | False | 
| http_text_to_http_json | -281.58KiB | -0.68 | 100.00% | 40.58MiB | 717.13KiB | 14.64KiB | 0 | 0.0172524 | 40.31MiB | 682.02KiB | 13.92KiB | 0 | 0.0165197 | False | False | 
| http_pipelines_no_grok_blackhole | -86.29KiB | -0.75 | 99.99% | 11.23MiB | 71.38KiB | 1.46KiB | 0 | 0.00620422 | 11.15MiB | 1.05MiB | 21.84KiB | 0 | 0.0940477 | False | False | 
| http_pipelines_blackhole | -14.81KiB | -0.88 | 100.00% | 1.65MiB | 12.07KiB | 252.68B | 0 | 0.00713843 | 1.64MiB | 133.62KiB | 2.72KiB | 0 | 0.0797373 | False | False | 
| datadog_agent_remap_datadog_logs_acks | -588.23KiB | -0.91 | 100.00% | 63.47MiB | 3.2MiB | 66.86KiB | 0 | 0.0503693 | 62.9MiB | 4.52MiB | 94.1KiB | 0 | 0.0718625 | False | False | 
| http_to_http_acks | -226.98KiB | -1.21 | 64.49% | 18.26MiB | 8.44MiB | 176.57KiB | 0 | 0.462176 | 18.04MiB | 8.17MiB | 170.45KiB | 0 | 0.452953 | True | True | 
| datadog_agent_remap_datadog_logs | -937.6KiB | -1.43 | 100.00% | 64.05MiB | 1.13MiB | 23.68KiB | 0 | 0.0176214 | 63.14MiB | 4.48MiB | 93.26KiB | 0 | 0.0709258 | False | False | 
| syslog_log2metric_humio_metrics | -248.33KiB | -1.86 | 100.00% | 13.03MiB | 205.8KiB | 4.2KiB | 0 | 0.0154227 | 12.79MiB | 498.74KiB | 10.16KiB | 0 | 0.0380851 | False | False | 
| syslog_loki | -368.53KiB | -2.53 | 100.00% | 14.22MiB | 348.5KiB | 7.13KiB | 0 | 0.0239262 | 13.86MiB | 688.56KiB | 14.0KiB | 0 | 0.0485003 | False | False | 
| socket_to_socket_blackhole | -691.35KiB | -2.78 | 100.00% | 24.25MiB | 162.76KiB | 3.32KiB | 0 | 0.00655208 | 23.58MiB | 182.51KiB | 3.73KiB | 0 | 0.00755773 | False | False | 
Soak Test Results
Baseline: c402e6ed94400959dbc298d4b342dcb0842089d4 Comparison: b673e40f2e9f0454ab9e89431a318dedea39d770 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic | 
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_text_to_http_json | 2.42MiB | 6.51 | 100.00% | 37.11MiB | 931.79KiB | 19.02KiB | 0 | 0.0245128 | 39.53MiB | 1.06MiB | 22.2KiB | 0 | 0.026861 | False | False | 
| socket_to_socket_blackhole | 1.27MiB | 5.73 | 100.00% | 22.17MiB | 136.67KiB | 2.79KiB | 0 | 0.00601926 | 23.44MiB | 218.86KiB | 4.47KiB | 0 | 0.00911653 | False | False | 
| splunk_hec_route_s3 | 539.99KiB | 2.79 | 100.00% | 18.87MiB | 2.21MiB | 46.1KiB | 0 | 0.117266 | 19.4MiB | 2.12MiB | 44.36KiB | 0 | 0.109312 | False | False | 
| datadog_agent_remap_blackhole_acks | 800.64KiB | 1.2 | 100.00% | 64.98MiB | 4.36MiB | 90.77KiB | 0 | 0.0670471 | 65.77MiB | 2.95MiB | 61.68KiB | 0 | 0.0448287 | False | False | 
| datadog_agent_remap_datadog_logs_acks | 671.56KiB | 1.03 | 100.00% | 63.51MiB | 3.2MiB | 66.96KiB | 0 | 0.0504302 | 64.17MiB | 4.55MiB | 94.73KiB | 0 | 0.0709027 | False | False | 
| datadog_agent_remap_blackhole | 642.12KiB | 0.98 | 100.00% | 64.06MiB | 4.09MiB | 85.27KiB | 0 | 0.0638665 | 64.69MiB | 3.6MiB | 75.02KiB | 0 | 0.0556086 | False | False | 
| http_pipelines_blackhole_acks | 6.41KiB | 0.53 | 98.54% | 1.19MiB | 104.78KiB | 2.13KiB | 0 | 0.0857673 | 1.2MiB | 75.06KiB | 1.53KiB | 0 | 0.0611238 | False | False | 
| datadog_agent_remap_datadog_logs | 286.42KiB | 0.44 | 99.73% | 63.98MiB | 1.08MiB | 22.57KiB | 0 | 0.0168332 | 64.26MiB | 4.45MiB | 92.56KiB | 0 | 0.0691719 | False | False | 
| syslog_regex_logs2metric_ddmetrics | 41.22KiB | 0.32 | 97.91% | 12.74MiB | 610.87KiB | 12.45KiB | 0 | 0.0468229 | 12.78MiB | 627.14KiB | 12.77KiB | 0 | 0.0479187 | False | False | 
| syslog_loki | 47.11KiB | 0.3 | 99.53% | 15.19MiB | 310.34KiB | 6.35KiB | 0 | 0.0199412 | 15.24MiB | 758.29KiB | 15.41KiB | 0 | 0.0485783 | False | False | 
| syslog_splunk_hec_logs | 34.66KiB | 0.2 | 90.50% | 16.81MiB | 734.88KiB | 14.96KiB | 0 | 0.0426719 | 16.85MiB | 705.89KiB | 14.38KiB | 0 | 0.0409061 | False | False | 
| http_to_http_acks | 29.48KiB | 0.16 | 9.46% | 17.96MiB | 8.61MiB | 180.06KiB | 0 | 0.479383 | 17.99MiB | 8.18MiB | 170.68KiB | 0 | 0.454611 | True | True | 
| splunk_hec_to_splunk_hec_logs_noack | 15.76KiB | 0.06 | 81.70% | 23.82MiB | 475.26KiB | 9.7KiB | 0 | 0.0194795 | 23.84MiB | 332.21KiB | 6.78KiB | 0 | 0.0136077 | False | False | 
| syslog_humio_logs | 2.08KiB | 0.01 | 23.71% | 16.93MiB | 244.27KiB | 4.99KiB | 0 | 0.0140865 | 16.93MiB | 232.29KiB | 4.75KiB | 0 | 0.0133939 | False | False | 
| enterprise_http_to_http | -1.23KiB | -0.01 | 13.50% | 23.85MiB | 246.11KiB | 5.02KiB | 0 | 0.0100766 | 23.85MiB | 254.92KiB | 5.21KiB | 0 | 0.0104377 | False | False | 
| splunk_hec_indexer_ack_blackhole | -6.98KiB | -0.03 | 21.64% | 23.76MiB | 862.22KiB | 17.55KiB | 0 | 0.0354324 | 23.75MiB | 904.2KiB | 18.4KiB | 0 | 0.0371684 | False | False | 
| file_to_blackhole | -40.74KiB | -0.04 | 31.10% | 95.34MiB | 3.16MiB | 65.52KiB | 0 | 0.0331438 | 95.3MiB | 3.74MiB | 77.91KiB | 0 | 0.039275 | False | False | 
| splunk_hec_to_splunk_hec_logs_acks | -12.01KiB | -0.05 | 37.40% | 23.76MiB | 829.53KiB | 16.88KiB | 0 | 0.0340932 | 23.74MiB | 882.65KiB | 17.95KiB | 0 | 0.0362944 | False | False | 
| http_to_http_json | -31.06KiB | -0.13 | 98.60% | 23.85MiB | 357.13KiB | 7.29KiB | 0 | 0.0146228 | 23.81MiB | 505.75KiB | 10.33KiB | 0 | 0.0207346 | False | False | 
| http_pipelines_blackhole | -2.85KiB | -0.17 | 77.94% | 1.65MiB | 8.58KiB | 179.79B | 0 | 0.00509214 | 1.64MiB | 113.92KiB | 2.32KiB | 0 | 0.0677099 | False | False | 
| fluent_elasticsearch | -173.99KiB | -0.21 | 100.00% | 79.47MiB | 52.83KiB | 1.07KiB | 0 | 0.00064901 | 79.3MiB | 1.49MiB | 30.58KiB | 0 | 0.0187379 | False | False | 
| http_to_http_noack | -103.94KiB | -0.43 | 100.00% | 23.85MiB | 251.08KiB | 5.14KiB | 0 | 0.0102806 | 23.74MiB | 1.11MiB | 23.06KiB | 0 | 0.0465681 | False | False | 
| syslog_log2metric_splunk_hec_metrics | -98.38KiB | -0.52 | 100.00% | 18.6MiB | 594.55KiB | 12.12KiB | 0 | 0.0312154 | 18.5MiB | 827.53KiB | 16.84KiB | 0 | 0.043673 | False | False | 
| http_pipelines_no_grok_blackhole | -172.95KiB | -1.53 | 100.00% | 11.06MiB | 269.98KiB | 5.51KiB | 0 | 0.0238341 | 10.89MiB | 1.1MiB | 22.96KiB | 0 | 0.10123 | False | False | 
| syslog_log2metric_humio_metrics | -964.09KiB | -7.43 | 100.00% | 12.66MiB | 564.81KiB | 11.53KiB | 0 | 0.0435471 | 11.72MiB | 1.0MiB | 20.94KiB | 0 | 0.085673 | False | False | 
Soak Test Results
Baseline: 4611307463ced2d069e806666178a3977265a3bc Comparison: 6c38648d552a414897bfc99497ff70de4f921554 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic | 
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_log2metric_splunk_hec_metrics | 804.22KiB | 4.49 | 100.00% | 17.49MiB | 619.12KiB | 12.61KiB | 0 | 0.0345595 | 18.28MiB | 820.77KiB | 16.71KiB | 0 | 0.0438472 | False | False | 
| syslog_loki | 507.48KiB | 3.37 | 100.00% | 14.73MiB | 428.0KiB | 8.75KiB | 0 | 0.0283761 | 15.22MiB | 759.85KiB | 15.45KiB | 0 | 0.0487379 | False | False | 
| syslog_humio_logs | 546.15KiB | 3.22 | 100.00% | 16.58MiB | 218.75KiB | 4.47KiB | 0 | 0.0128788 | 17.12MiB | 113.19KiB | 2.32KiB | 0 | 0.00645643 | False | False | 
| http_to_http_acks | 539.04KiB | 3.02 | 97.30% | 17.42MiB | 8.08MiB | 168.97KiB | 0 | 0.464016 | 17.94MiB | 8.41MiB | 175.56KiB | 0 | 0.468749 | True | True | 
| syslog_splunk_hec_logs | 459.39KiB | 2.73 | 100.00% | 16.43MiB | 682.02KiB | 13.88KiB | 0 | 0.0405206 | 16.88MiB | 600.32KiB | 12.24KiB | 0 | 0.0347192 | False | False | 
| syslog_regex_logs2metric_ddmetrics | 306.28KiB | 2.44 | 100.00% | 12.23MiB | 633.19KiB | 12.9KiB | 0 | 0.0505332 | 12.53MiB | 668.22KiB | 13.61KiB | 0 | 0.0520562 | False | False | 
| http_text_to_http_json | 929.34KiB | 2.32 | 100.00% | 39.06MiB | 982.94KiB | 20.06KiB | 0 | 0.0245675 | 39.97MiB | 875.05KiB | 17.87KiB | 0 | 0.0213745 | False | False | 
| socket_to_socket_blackhole | 348.57KiB | 1.45 | 100.00% | 23.47MiB | 144.7KiB | 2.95KiB | 0 | 0.00601851 | 23.82MiB | 153.54KiB | 3.14KiB | 0 | 0.00629489 | False | False | 
| splunk_hec_route_s3 | 203.1KiB | 1.03 | 99.84% | 19.2MiB | 2.21MiB | 46.1KiB | 0 | 0.115341 | 19.4MiB | 2.13MiB | 44.64KiB | 0 | 0.109968 | False | False | 
| datadog_agent_remap_blackhole_acks | 621.79KiB | 0.93 | 100.00% | 65.2MiB | 4.71MiB | 98.03KiB | 0 | 0.0722115 | 65.81MiB | 3.24MiB | 67.72KiB | 0 | 0.049203 | False | False | 
| datadog_agent_remap_blackhole | 423.84KiB | 0.65 | 99.99% | 63.99MiB | 4.4MiB | 91.73KiB | 0 | 0.0688138 | 64.4MiB | 2.51MiB | 52.44KiB | 0 | 0.0389897 | False | False | 
| datadog_agent_remap_datadog_logs_acks | 389.89KiB | 0.61 | 99.87% | 62.08MiB | 3.47MiB | 72.55KiB | 0 | 0.0559007 | 62.46MiB | 4.68MiB | 97.48KiB | 0 | 0.0749616 | False | False | 
| datadog_agent_remap_datadog_logs | 234.01KiB | 0.36 | 99.54% | 64.12MiB | 329.54KiB | 6.75KiB | 0 | 0.00501812 | 64.35MiB | 3.95MiB | 82.31KiB | 0 | 0.0613747 | False | False | 
| http_pipelines_blackhole_acks | 1.82KiB | 0.16 | 48.94% | 1.15MiB | 103.91KiB | 2.12KiB | 0 | 0.088453 | 1.15MiB | 87.94KiB | 1.79KiB | 0 | 0.0747407 | False | False | 
| splunk_hec_to_splunk_hec_logs_noack | 9.42KiB | 0.04 | 60.80% | 23.83MiB | 423.71KiB | 8.65KiB | 0 | 0.0173616 | 23.84MiB | 333.13KiB | 6.8KiB | 0 | 0.0136448 | False | False | 
| splunk_hec_indexer_ack_blackhole | 4.01KiB | 0.02 | 12.15% | 23.75MiB | 915.72KiB | 18.63KiB | 0 | 0.0376482 | 23.75MiB | 906.98KiB | 18.45KiB | 0 | 0.0372827 | False | False | 
| enterprise_http_to_http | -542.86B | -0 | 5.98% | 23.85MiB | 244.95KiB | 5.0KiB | 0 | 0.0100292 | 23.85MiB | 243.9KiB | 4.99KiB | 0 | 0.00998649 | False | False | 
| splunk_hec_to_splunk_hec_logs_acks | -310.56B | -0 | 0.99% | 23.75MiB | 848.26KiB | 17.26KiB | 0 | 0.0348671 | 23.75MiB | 843.5KiB | 17.16KiB | 0 | 0.0346721 | False | False | 
| file_to_blackhole | -62.92KiB | -0.06 | 47.33% | 95.34MiB | 3.1MiB | 64.27KiB | 0 | 0.0325129 | 95.28MiB | 3.65MiB | 75.8KiB | 0 | 0.0382582 | False | False | 
| fluent_elasticsearch | -141.85KiB | -0.17 | 100.00% | 79.47MiB | 52.22KiB | 1.06KiB | 0 | 0.000641602 | 79.33MiB | 1.49MiB | 30.74KiB | 0 | 0.0187928 | False | False | 
| http_to_http_json | -45.74KiB | -0.19 | 99.92% | 23.85MiB | 339.04KiB | 6.92KiB | 0 | 0.0138814 | 23.8MiB | 572.18KiB | 11.67KiB | 0 | 0.0234708 | False | False | 
| http_to_http_noack | -88.05KiB | -0.36 | 99.98% | 23.84MiB | 410.88KiB | 8.4KiB | 0 | 0.0168293 | 23.75MiB | 1.06MiB | 22.09KiB | 0 | 0.0445786 | False | False | 
| http_pipelines_no_grok_blackhole | -52.06KiB | -0.46 | 97.05% | 11.0MiB | 209.67KiB | 4.28KiB | 0 | 0.0186057 | 10.95MiB | 1.13MiB | 23.52KiB | 0 | 0.103129 | False | False | 
| http_pipelines_blackhole | -10.88KiB | -0.64 | 100.00% | 1.65MiB | 11.08KiB | 231.84B | 0 | 0.00656038 | 1.64MiB | 112.68KiB | 2.3KiB | 0 | 0.0671475 | False | False | 
| syslog_log2metric_humio_metrics | -340.16KiB | -2.63 | 100.00% | 12.65MiB | 251.66KiB | 5.14KiB | 0 | 0.019424 | 12.32MiB | 545.28KiB | 11.1KiB | 0 | 0.0432218 | False | False | 
Soak Test Results
Baseline: 8e2201bcfd7b7ec460d783ed8b8489cafef55902 Comparison: f3c63d701517bcc9b1a5860e393963254229f4c2 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic | 
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| datadog_agent_remap_datadog_logs | 1.24MiB | 2.03 | 100.00% | 61.33MiB | 3.44MiB | 72.11KiB | 0 | 0.0560867 | 62.57MiB | 4.1MiB | 85.45KiB | 0 | 0.0655507 | False | False | 
| datadog_agent_remap_blackhole_acks | 1.26MiB | 1.98 | 100.00% | 63.58MiB | 5.39MiB | 112.26KiB | 0 | 0.0847641 | 64.84MiB | 2.43MiB | 50.84KiB | 0 | 0.0374652 | False | False | 
| socket_to_socket_blackhole | 427.95KiB | 1.82 | 100.00% | 22.97MiB | 319.58KiB | 6.52KiB | 0 | 0.013584 | 23.39MiB | 400.05KiB | 8.17KiB | 0 | 0.0167009 | False | False | 
| http_text_to_http_json | 430.04KiB | 1.06 | 100.00% | 39.52MiB | 1.0MiB | 20.99KiB | 0 | 0.0254025 | 39.94MiB | 909.06KiB | 18.56KiB | 0 | 0.0222236 | False | False | 
| datadog_agent_remap_datadog_logs_acks | 524.03KiB | 0.83 | 99.99% | 61.94MiB | 4.74MiB | 98.84KiB | 0 | 0.0764527 | 62.45MiB | 4.43MiB | 92.25KiB | 0 | 0.0709466 | False | False | 
| http_pipelines_blackhole_acks | 7.75KiB | 0.62 | 99.54% | 1.22MiB | 119.3KiB | 2.43KiB | 0 | 0.0953748 | 1.23MiB | 61.81KiB | 1.26KiB | 0 | 0.0491108 | False | False | 
| splunk_hec_route_s3 | 99.83KiB | 0.53 | 85.02% | 18.33MiB | 2.38MiB | 49.46KiB | 0 | 0.129537 | 18.43MiB | 2.32MiB | 48.53KiB | 0 | 0.125846 | False | False | 
| syslog_splunk_hec_logs | 70.33KiB | 0.43 | 99.84% | 15.99MiB | 818.24KiB | 16.65KiB | 0 | 0.0499731 | 16.06MiB | 725.23KiB | 14.79KiB | 0 | 0.0441032 | False | False | 
| syslog_log2metric_splunk_hec_metrics | 18.84KiB | 0.11 | 49.40% | 16.95MiB | 893.59KiB | 18.21KiB | 0 | 0.0514818 | 16.97MiB | 1.04MiB | 21.69KiB | 0 | 0.061307 | False | False | 
| splunk_hec_to_splunk_hec_logs_acks | 26.24KiB | 0.11 | 74.93% | 23.75MiB | 843.14KiB | 17.15KiB | 0 | 0.0346605 | 23.78MiB | 741.04KiB | 15.09KiB | 0 | 0.0304306 | False | False | 
| splunk_hec_to_splunk_hec_logs_noack | 17.07KiB | 0.07 | 84.90% | 23.82MiB | 476.49KiB | 9.72KiB | 0 | 0.0195303 | 23.84MiB | 335.14KiB | 6.84KiB | 0 | 0.0137269 | False | False | 
| syslog_humio_logs | 9.65KiB | 0.06 | 46.03% | 15.96MiB | 544.72KiB | 11.12KiB | 0 | 0.0333145 | 15.97MiB | 543.43KiB | 11.12KiB | 0 | 0.0332156 | False | False | 
| splunk_hec_indexer_ack_blackhole | 13.33KiB | 0.05 | 39.56% | 23.75MiB | 913.68KiB | 18.58KiB | 0 | 0.037568 | 23.76MiB | 874.2KiB | 17.79KiB | 0 | 0.0359248 | False | False | 
| enterprise_http_to_http | 1.34KiB | 0.01 | 14.42% | 23.84MiB | 255.25KiB | 5.21KiB | 0 | 0.0104518 | 23.85MiB | 254.89KiB | 5.21KiB | 0 | 0.0104366 | False | False | 
| file_to_blackhole | -49.14KiB | -0.05 | 36.02% | 95.32MiB | 3.4MiB | 70.43KiB | 0 | 0.0356639 | 95.28MiB | 3.75MiB | 77.87KiB | 0 | 0.0393054 | False | False | 
| datadog_agent_remap_blackhole | -84.17KiB | -0.13 | 62.08% | 63.45MiB | 3.53MiB | 73.73KiB | 0 | 0.0556765 | 63.36MiB | 2.92MiB | 61.0KiB | 0 | 0.0461236 | False | False | 
| http_pipelines_blackhole | -2.19KiB | -0.13 | 54.27% | 1.67MiB | 80.76KiB | 1.65KiB | 0 | 0.047112 | 1.67MiB | 119.44KiB | 2.44KiB | 0 | 0.0697679 | False | False | 
| http_to_http_json | -39.56KiB | -0.16 | 99.73% | 23.84MiB | 351.06KiB | 7.17KiB | 0 | 0.0143756 | 23.8MiB | 542.26KiB | 11.06KiB | 0 | 0.0222411 | False | False | 
| syslog_regex_logs2metric_ddmetrics | -24.48KiB | -0.19 | 83.87% | 12.55MiB | 588.39KiB | 11.99KiB | 0 | 0.0457737 | 12.53MiB | 624.6KiB | 12.72KiB | 0 | 0.0486831 | False | False | 
| http_to_http_noack | -59.21KiB | -0.24 | 98.57% | 23.82MiB | 605.52KiB | 12.37KiB | 0 | 0.0248194 | 23.76MiB | 1018.17KiB | 20.74KiB | 0 | 0.0418347 | False | False | 
| fluent_elasticsearch | -405.71KiB | -0.5 | 100.00% | 79.47MiB | 53.32KiB | 1.08KiB | 0 | 0.000655028 | 79.08MiB | 4.29MiB | 88.24KiB | 0 | 0.0542886 | False | False | 
| syslog_loki | -164.08KiB | -1.1 | 100.00% | 14.6MiB | 603.92KiB | 12.35KiB | 0 | 0.040393 | 14.44MiB | 863.25KiB | 17.55KiB | 0 | 0.0583789 | False | False | 
| http_to_http_acks | -225.67KiB | -1.26 | 66.55% | 17.47MiB | 7.9MiB | 165.11KiB | 0 | 0.452026 | 17.25MiB | 7.92MiB | 165.54KiB | 0 | 0.459378 | True | True | 
| http_pipelines_no_grok_blackhole | -141.2KiB | -1.27 | 100.00% | 10.82MiB | 299.3KiB | 6.11KiB | 0 | 0.0270056 | 10.68MiB | 977.45KiB | 19.89KiB | 0 | 0.0893339 | False | False | 
| syslog_log2metric_humio_metrics | -180.79KiB | -1.38 | 100.00% | 12.77MiB | 281.18KiB | 5.74KiB | 0 | 0.0215056 | 12.59MiB | 516.34KiB | 10.51KiB | 0 | 0.0400457 | False | False | 
Soak Test Results
Baseline: 2fd8a51cbe1813cc2afd3e27d05268e5173f367c Comparison: 8b4f61512e25d3ca5072cff5c102989f71fb4435 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic | 
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| syslog_humio_logs | 814.31KiB | 5.06 | 100.00% | 15.72MiB | 1.14MiB | 23.93KiB | 0 | 0.0728127 | 16.51MiB | 721.91KiB | 14.79KiB | 0 | 0.0426858 | False | False | 
| http_text_to_http_json | 541.68KiB | 1.35 | 100.00% | 39.08MiB | 964.46KiB | 19.69KiB | 0 | 0.0240957 | 39.61MiB | 720.62KiB | 14.72KiB | 0 | 0.0177632 | False | False | 
| http_pipelines_blackhole_acks | 13.21KiB | 1.07 | 100.00% | 1.21MiB | 110.36KiB | 2.25KiB | 0 | 0.089078 | 1.22MiB | 88.51KiB | 1.8KiB | 0 | 0.0706904 | False | False | 
| http_to_http_acks | 125.87KiB | 0.71 | 41.63% | 17.26MiB | 7.67MiB | 160.44KiB | 0 | 0.444568 | 17.38MiB | 7.87MiB | 164.34KiB | 0 | 0.452612 | True | True | 
| socket_to_socket_blackhole | 89.67KiB | 0.38 | 100.00% | 22.93MiB | 128.1KiB | 2.61KiB | 0 | 0.00545492 | 23.02MiB | 145.0KiB | 2.96KiB | 0 | 0.00615109 | False | False | 
| syslog_log2metric_humio_metrics | 25.62KiB | 0.19 | 97.56% | 12.93MiB | 189.96KiB | 3.88KiB | 0 | 0.0143431 | 12.96MiB | 525.39KiB | 10.7KiB | 0 | 0.0395927 | False | False | 
| splunk_hec_to_splunk_hec_logs_noack | 4.28KiB | 0.02 | 32.27% | 23.83MiB | 378.95KiB | 7.74KiB | 0 | 0.0155248 | 23.84MiB | 332.42KiB | 6.79KiB | 0 | 0.0136165 | False | False | 
| datadog_agent_remap_datadog_logs_acks | 9.03KiB | 0.01 | 5.69% | 59.98MiB | 3.97MiB | 82.82KiB | 0 | 0.0660965 | 59.99MiB | 4.59MiB | 95.53KiB | 0 | 0.0764634 | False | False | 
| enterprise_http_to_http | 705.88B | 0 | 7.58% | 23.84MiB | 252.03KiB | 5.14KiB | 0 | 0.0103198 | 23.85MiB | 249.23KiB | 5.1KiB | 0 | 0.010205 | False | False | 
| http_pipelines_blackhole | -165.24B | -0.01 | 7.82% | 1.7MiB | 10.47KiB | 219.15B | 0 | 0.00601083 | 1.7MiB | 79.86KiB | 1.63KiB | 0 | 0.0458362 | False | False | 
| splunk_hec_to_splunk_hec_logs_acks | -5.69KiB | -0.02 | 18.47% | 23.75MiB | 837.22KiB | 17.03KiB | 0 | 0.034414 | 23.75MiB | 855.68KiB | 17.4KiB | 0 | 0.0351808 | False | False | 
| splunk_hec_indexer_ack_blackhole | -11.14KiB | -0.05 | 35.03% | 23.76MiB | 835.02KiB | 16.99KiB | 0 | 0.0343112 | 23.75MiB | 869.74KiB | 17.69KiB | 0 | 0.035754 | False | False | 
| splunk_hec_route_s3 | -31.06KiB | -0.16 | 36.21% | 18.89MiB | 2.27MiB | 47.29KiB | 0 | 0.120129 | 18.86MiB | 2.2MiB | 46.04KiB | 0 | 0.116654 | False | False | 
| http_to_http_json | -40.27KiB | -0.16 | 99.79% | 23.85MiB | 339.49KiB | 6.93KiB | 0 | 0.0139001 | 23.81MiB | 543.18KiB | 11.09KiB | 0 | 0.0222767 | False | False | 
| fluent_elasticsearch | -134.41KiB | -0.17 | 100.00% | 79.47MiB | 53.13KiB | 1.07KiB | 0 | 0.000652711 | 79.34MiB | 1.31MiB | 26.92KiB | 0 | 0.0164731 | False | False | 
| syslog_regex_logs2metric_ddmetrics | -25.8KiB | -0.2 | 90.42% | 12.33MiB | 594.7KiB | 12.11KiB | 0 | 0.0471099 | 12.3MiB | 473.27KiB | 9.65KiB | 0 | 0.0375669 | False | False | 
| file_to_blackhole | -337.57KiB | -0.35 | 98.25% | 95.35MiB | 3.51MiB | 72.45KiB | 0 | 0.0367562 | 95.02MiB | 5.91MiB | 122.11KiB | 0 | 0.0621888 | False | False | 
| datadog_agent_remap_blackhole | -217.1KiB | -0.35 | 96.71% | 60.96MiB | 4.27MiB | 88.99KiB | 0 | 0.0700998 | 60.75MiB | 2.36MiB | 49.28KiB | 0 | 0.0388422 | False | False | 
| syslog_splunk_hec_logs | -60.85KiB | -0.36 | 99.60% | 16.42MiB | 790.49KiB | 16.1KiB | 0 | 0.0470025 | 16.36MiB | 671.41KiB | 13.7KiB | 0 | 0.0400669 | False | False | 
| http_to_http_noack | -112.11KiB | -0.46 | 100.00% | 23.85MiB | 250.7KiB | 5.13KiB | 0 | 0.0102647 | 23.74MiB | 1.13MiB | 23.59KiB | 0 | 0.0476597 | False | False | 
| syslog_loki | -91.57KiB | -0.6 | 100.00% | 14.94MiB | 309.04KiB | 6.33KiB | 0 | 0.0201974 | 14.85MiB | 741.68KiB | 15.08KiB | 0 | 0.048765 | False | False | 
| datadog_agent_remap_datadog_logs | -418.65KiB | -0.66 | 100.00% | 62.3MiB | 733.49KiB | 15.01KiB | 0 | 0.0114946 | 61.89MiB | 3.97MiB | 82.75KiB | 0 | 0.0641592 | False | False | 
| datadog_agent_remap_blackhole_acks | -427.73KiB | -0.68 | 99.96% | 61.74MiB | 4.41MiB | 91.7KiB | 0 | 0.0713396 | 61.32MiB | 3.8MiB | 79.22KiB | 0 | 0.0619466 | False | False | 
| syslog_log2metric_splunk_hec_metrics | -221.0KiB | -1.19 | 100.00% | 18.14MiB | 589.35KiB | 12.01KiB | 0 | 0.0317228 | 17.92MiB | 820.86KiB | 16.71KiB | 0 | 0.0447167 | False | False | 
| http_pipelines_no_grok_blackhole | -136.77KiB | -1.22 | 100.00% | 10.97MiB | 314.49KiB | 6.42KiB | 0 | 0.0279956 | 10.83MiB | 1.16MiB | 24.05KiB | 0 | 0.106605 | False | False | 
Soak Test Results
Baseline: 2fd8a51cbe1813cc2afd3e27d05268e5173f367c Comparison: 51811214d6a3b6398a3df3558c759c73533f5bf0 Total Vector CPUs: 4
Explanation
A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.
The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.
No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:
Fine details of change detection per experiment.
| experiment | Δ mean | Δ mean % | confidence | baseline mean | baseline stdev | baseline stderr | baseline outlier % | baseline CoV | comparison mean | comparison stdev | comparison stderr | comparison outlier % | comparison CoV | erratic | declared erratic | 
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| http_text_to_http_json | 720.3KiB | 1.8 | 100.00% | 39.03MiB | 1.0MiB | 20.93KiB | 0 | 0.0256458 | 39.73MiB | 740.54KiB | 15.12KiB | 0 | 0.0181971 | False | False | 
| syslog_log2metric_humio_metrics | 161.48KiB | 1.23 | 100.00% | 12.78MiB | 530.42KiB | 10.82KiB | 0 | 0.0405267 | 12.94MiB | 533.16KiB | 10.85KiB | 0 | 0.0402395 | False | False | 
| http_pipelines_blackhole_acks | 6.78KiB | 0.56 | 98.68% | 1.19MiB | 111.23KiB | 2.26KiB | 0 | 0.0912098 | 1.2MiB | 75.26KiB | 1.53KiB | 0 | 0.0613685 | False | False | 
| socket_to_socket_blackhole | 103.24KiB | 0.44 | 100.00% | 22.93MiB | 105.48KiB | 2.15KiB | 0 | 0.00449103 | 23.03MiB | 162.53KiB | 3.32KiB | 0 | 0.00688932 | False | False | 
| splunk_hec_to_splunk_hec_logs_noack | 37.69KiB | 0.15 | 99.24% | 23.8MiB | 609.79KiB | 12.43KiB | 0 | 0.0250156 | 23.84MiB | 327.21KiB | 6.68KiB | 0 | 0.0134025 | False | False | 
| enterprise_http_to_http | -1.02KiB | -0 | 10.83% | 23.85MiB | 261.12KiB | 5.33KiB | 0 | 0.010691 | 23.85MiB | 255.8KiB | 5.23KiB | 0 | 0.0104735 | False | False | 
| datadog_agent_remap_blackhole_acks | -11.47KiB | -0.02 | 8.95% | 62.01MiB | 4.09MiB | 85.22KiB | 0 | 0.0659674 | 62.0MiB | 2.68MiB | 56.09KiB | 0 | 0.0432343 | False | False | 
| splunk_hec_to_splunk_hec_logs_acks | -6.29KiB | -0.03 | 20.56% | 23.75MiB | 827.45KiB | 16.83KiB | 0 | 0.0340096 | 23.75MiB | 851.52KiB | 17.32KiB | 0 | 0.0350077 | False | False | 
| file_to_blackhole | -96.23KiB | -0.1 | 58.30% | 95.34MiB | 3.36MiB | 69.72KiB | 0 | 0.0352676 | 95.25MiB | 4.61MiB | 95.88KiB | 0 | 0.0483988 | False | False | 
| splunk_hec_indexer_ack_blackhole | -25.43KiB | -0.1 | 66.84% | 23.76MiB | 861.87KiB | 17.54KiB | 0 | 0.0354167 | 23.73MiB | 956.75KiB | 19.45KiB | 0 | 0.0393568 | False | False | 
| http_to_http_json | -32.75KiB | -0.13 | 99.08% | 23.85MiB | 336.38KiB | 6.87KiB | 0 | 0.0137732 | 23.81MiB | 515.89KiB | 10.54KiB | 0 | 0.0211518 | False | False | 
| splunk_hec_route_s3 | -25.64KiB | -0.14 | 29.33% | 18.3MiB | 2.34MiB | 48.69KiB | 0 | 0.127693 | 18.27MiB | 2.28MiB | 47.67KiB | 0 | 0.12462 | False | False | 
| syslog_regex_logs2metric_ddmetrics | -25.84KiB | -0.2 | 86.58% | 12.68MiB | 658.3KiB | 13.4KiB | 0 | 0.0507078 | 12.65MiB | 532.51KiB | 10.86KiB | 0 | 0.0411 | False | False | 
| fluent_elasticsearch | -187.81KiB | -0.23 | 100.00% | 79.47MiB | 52.81KiB | 1.07KiB | 0 | 0.000648745 | 79.29MiB | 1.67MiB | 34.29KiB | 0 | 0.021012 | False | False | 
| http_to_http_noack | -103.3KiB | -0.42 | 100.00% | 23.84MiB | 403.71KiB | 8.25KiB | 0 | 0.0165361 | 23.74MiB | 1.13MiB | 23.6KiB | 0 | 0.0476766 | False | False | 
| datadog_agent_remap_datadog_logs_acks | -289.6KiB | -0.45 | 99.31% | 62.55MiB | 2.63MiB | 55.07KiB | 0 | 0.0420586 | 62.27MiB | 4.41MiB | 91.81KiB | 0 | 0.0708137 | False | False | 
| http_pipelines_blackhole | -8.01KiB | -0.46 | 99.80% | 1.71MiB | 40.91KiB | 856.54B | 0 | 0.0234138 | 1.7MiB | 120.41KiB | 2.45KiB | 0 | 0.0692315 | False | False | 
| http_pipelines_no_grok_blackhole | -56.43KiB | -0.5 | 98.70% | 11.01MiB | 260.93KiB | 5.33KiB | 0 | 0.0231344 | 10.96MiB | 1.06MiB | 22.06KiB | 0 | 0.0966683 | False | False | 
| syslog_loki | -90.47KiB | -0.6 | 100.00% | 14.68MiB | 323.39KiB | 6.62KiB | 0 | 0.0215115 | 14.59MiB | 738.92KiB | 15.02KiB | 0 | 0.0494499 | False | False | 
| datadog_agent_remap_blackhole | -520.19KiB | -0.83 | 100.00% | 61.04MiB | 3.55MiB | 74.05KiB | 0 | 0.0581322 | 60.53MiB | 2.97MiB | 61.9KiB | 0 | 0.049007 | False | False | 
| syslog_splunk_hec_logs | -143.72KiB | -0.86 | 100.00% | 16.35MiB | 707.77KiB | 14.41KiB | 0 | 0.0422654 | 16.21MiB | 636.02KiB | 12.96KiB | 0 | 0.0383094 | False | False | 
| syslog_humio_logs | -171.63KiB | -0.99 | 100.00% | 16.97MiB | 137.25KiB | 2.8KiB | 0 | 0.00789822 | 16.8MiB | 124.14KiB | 2.54KiB | 0 | 0.00721511 | False | False | 
| syslog_log2metric_splunk_hec_metrics | -208.73KiB | -1.12 | 100.00% | 18.22MiB | 543.33KiB | 11.07KiB | 0 | 0.0291085 | 18.02MiB | 694.37KiB | 14.14KiB | 0 | 0.0376208 | False | False | 
| datadog_agent_remap_datadog_logs | -818.45KiB | -1.32 | 100.00% | 60.66MiB | 2.16MiB | 45.32KiB | 0 | 0.0356441 | 59.86MiB | 4.03MiB | 83.97KiB | 0 | 0.0673137 | False | False | 
| http_to_http_acks | -458.07KiB | -2.58 | 95.22% | 17.34MiB | 7.96MiB | 166.42KiB | 0 | 0.459106 | 16.89MiB | 7.7MiB | 160.84KiB | 0 | 0.456034 | True | True | 
Just noting that this could still result in publishing without a hostname if the hostname couldn't be detected. Is that a hard requirement for OP? If so, we could probably error if the hostname couldn't be determined. We have #14250 to cover this more generally within Vector.
This feature has been put on hold for now so we can close this for now I guess.
Closing per https://github.com/vectordotdev/vector/pull/13703#issuecomment-1271310613