vector icon indicating copy to clipboard operation
vector copied to clipboard

fix(docker-logs source): old messages that are dropped should be a component_error

Open StephenWakely opened this issue 3 years ago • 4 comments

Ref #14411

When receiving a message from docker that is older than the previous one, this message was being dropped and a trace message was emitted.

I would argue that this should be a component error, and this PR makes it so.

It's not 100% clear to me under what circumstances this error would occur, so I am open to arguments as to why this should remain a trace - in which case I will update to add a comment with that reason.

Signed-off-by: Stephen Wakely [email protected]

StephenWakely avatar Sep 16 '22 11:09 StephenWakely

Deploy Preview for vector-project canceled.

Name Link
Latest commit 802d5a1a57a80d212b4ef45a208cde2bacfbbab5
Latest deploy log https://app.netlify.com/sites/vector-project/deploys/632c35537d4d6b000840595e

netlify[bot] avatar Sep 16 '22 11:09 netlify[bot]

But it still looks like we process it the same otherwise. thinking I guess I'm not totally convinced it is an Error case shrug

No. If you note when we hit that point there is a return None;. This breaks out of creating the event completely. None values get filtered out.

StephenWakely avatar Sep 21 '22 10:09 StephenWakely

Soak Test Results

Baseline: 9cf1ea9b08ed745e3872c1cc81757f6078c82419 Comparison: 4f5960f5fcc559e4ddc47819e42acbdda7a3af78 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
socket_to_socket_blackhole 253.92KiB 1.08 100.00% 22.89MiB 549.6KiB 11.22KiB 0 0.0234449 23.14MiB 217.32KiB 4.44KiB 0 0.00917114 False False
http_pipelines_blackhole_acks 9.95KiB 0.8 99.90% 1.21MiB 116.83KiB 2.38KiB 0 0.0944206 1.22MiB 92.02KiB 1.87KiB 0 0.0737749 False False
syslog_loki 108.75KiB 0.73 100.00% 14.49MiB 433.22KiB 8.87KiB 0 0.0291819 14.6MiB 748.57KiB 15.22KiB 0 0.0500575 False False
syslog_log2metric_humio_metrics 47.0KiB 0.36 99.99% 12.9MiB 268.71KiB 5.48KiB 0 0.0203378 12.95MiB 503.59KiB 10.25KiB 0 0.0379796 False False
splunk_hec_indexer_ack_blackhole 7.05KiB 0.03 21.86% 23.76MiB 896.77KiB 18.25KiB 0 0.0368563 23.76MiB 868.85KiB 17.68KiB 0 0.0356985 False False
splunk_hec_to_splunk_hec_logs_noack 4.5KiB 0.02 33.35% 23.83MiB 386.08KiB 7.88KiB 0 0.0158164 23.84MiB 335.29KiB 6.85KiB 0 0.0137334 False False
http_pipelines_blackhole 232.99B 0.01 6.72% 1.62MiB 58.43KiB 1.19KiB 0 0.0351275 1.62MiB 118.81KiB 2.42KiB 0 0.0714174 False False
enterprise_http_to_http 1.05KiB 0 11.56% 23.85MiB 249.73KiB 5.1KiB 0 0.0102256 23.85MiB 251.4KiB 5.14KiB 0 0.0102935 False False
splunk_hec_to_splunk_hec_logs_acks 873.07B 0 2.81% 23.76MiB 842.11KiB 17.13KiB 0 0.0346112 23.76MiB 841.32KiB 17.12KiB 0 0.0345777 False False
http_to_http_acks -5.97KiB -0.03 2.00% 17.61MiB 8.09MiB 169.1KiB 0 0.459282 17.6MiB 8.07MiB 168.33KiB 0 0.45821 True True
file_to_blackhole -55.48KiB -0.06 39.74% 95.34MiB 3.38MiB 70.13KiB 0 0.035476 95.28MiB 3.86MiB 80.21KiB 0 0.0404589 False False
http_to_http_json -42.2KiB -0.17 99.87% 23.85MiB 335.86KiB 6.86KiB 0 0.013749 23.81MiB 548.59KiB 11.2KiB 0 0.0224962 False False
fluent_elasticsearch -168.52KiB -0.21 100.00% 79.47MiB 55.55KiB 1.12KiB 0 0.000682474 79.31MiB 1.46MiB 30.03KiB 0 0.0184025 False False
datadog_agent_remap_blackhole_acks -143.56KiB -0.23 79.00% 61.08MiB 4.5MiB 93.65KiB 0 0.0736225 60.94MiB 3.15MiB 65.88KiB 0 0.05172 False False
http_to_http_noack -69.5KiB -0.28 99.85% 23.84MiB 408.11KiB 8.34KiB 0 0.0167158 23.77MiB 992.74KiB 20.23KiB 0 0.0407784 False False
datadog_agent_remap_datadog_logs_acks -244.26KiB -0.38 96.68% 62.3MiB 3.44MiB 71.88KiB 0 0.0552354 62.06MiB 4.29MiB 89.3KiB 0 0.0690997 False False
datadog_agent_remap_blackhole -287.79KiB -0.46 99.76% 61.68MiB 3.56MiB 74.3KiB 0 0.0577422 61.4MiB 2.83MiB 59.1KiB 0 0.0461213 False False
splunk_hec_route_s3 -151.07KiB -0.78 97.99% 19.01MiB 2.21MiB 45.94KiB 0 0.11599 18.87MiB 2.19MiB 45.93KiB 0 0.116311 False False
datadog_agent_remap_datadog_logs -540.66KiB -0.84 100.00% 62.83MiB 299.12KiB 6.12KiB 0 0.00464837 62.3MiB 4.18MiB 87.03KiB 0 0.0670678 False False
syslog_regex_logs2metric_ddmetrics -122.06KiB -0.95 100.00% 12.51MiB 575.08KiB 11.72KiB 0 0.0448857 12.39MiB 560.86KiB 11.43KiB 0 0.0441968 False False
http_pipelines_no_grok_blackhole -138.46KiB -1.25 100.00% 10.84MiB 93.72KiB 1.91KiB 0 0.00844227 10.7MiB 1.07MiB 22.36KiB 0 0.100317 False False
syslog_splunk_hec_logs -226.76KiB -1.33 100.00% 16.6MiB 714.94KiB 14.56KiB 0 0.0420574 16.38MiB 609.67KiB 12.43KiB 0 0.0363495 False False
syslog_humio_logs -252.72KiB -1.49 100.00% 16.58MiB 109.32KiB 2.23KiB 0 0.00643698 16.33MiB 129.63KiB 2.65KiB 0 0.00774826 False False
syslog_log2metric_splunk_hec_metrics -294.92KiB -1.62 100.00% 17.81MiB 699.63KiB 14.25KiB 0 0.0383597 17.52MiB 776.76KiB 15.81KiB 0 0.0432885 False False
http_text_to_http_json -2.25MiB -5.81 100.00% 38.68MiB 884.51KiB 18.05KiB 0 0.0223285 36.43MiB 842.69KiB 17.2KiB 0 0.0225847 False False

github-actions[bot] avatar Sep 21 '22 12:09 github-actions[bot]

But it still looks like we process it the same otherwise. thinking I guess I'm not totally convinced it is an Error case shrug

No. If you note when we hit that point there is a return None;. This breaks out of creating the event completely. None values get filtered out.

Ah, I missed the return None; there. :+1:

neuronull avatar Sep 21 '22 14:09 neuronull