IntegrationTests.SmokeTests.PrometheusExporter [FAIL] Did not expect any exceptions after 30s, but found System.Net.Http.HttpRequestException with message "Connection refused"

[pellared]

Potentially flaky test

• https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7500128904?check_suite_focus=true
• https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7513623116?check_suite_focus=true

[pellared]

Now it fails like this:

• https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7553815148?check_suite_focus=true
• https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7556338386?check_suite_focus=true

Did not expect any exceptions after 30s, but found System.Net.Http.HttpRequestException with message "Connection refused"

[pellared]

This one looks interesting: https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7549934722?check_suite_focus=true

IntegrationTests.SmokeTests.PrometheusExporter [FAIL]
  Failed IntegrationTests.SmokeTests.PrometheusExporter [30 s]
Error Message:
   Did not expect any exceptions after 30s, but found Xunit.Sdk.XunitException with message "Expected content "
# HELP process_runtime_dotnet_gc_collections_count Number of garbage collections that have occurred since process start.
# TYPE process_runtime_dotnet_gc_collections_count counter
process_runtime_dotnet_gc_collections_count{generation="gen2"} 0 1658960659966
process_runtime_dotnet_gc_collections_count{generation="gen1"} 0 1658960659966
process_runtime_dotnet_gc_collections_count{generation="gen0"} 0 1658960659966
# HELP process_runtime_dotnet_gc_allocations_size_bytes Count of bytes allocated on the managed GC heap since the process start. .NET objects are allocated from this heap. Object allocations from unmanaged languages such as C/C++ do not use this heap.
# TYPE process_runtime_dotnet_gc_allocations_size_bytes counter
process_runtime_dotnet_gc_allocations_size_bytes 3127832 1658960659966
# HELP process_runtime_dotnet_monitor_lock_contention_count The number of times there was contention when trying to acquire a monitor lock since the process start. Monitor locks are commonly acquired by using the lock keyword in C#, or by calling Monitor.Enter() and Monitor.TryEnter().
# TYPE process_runtime_dotnet_monitor_lock_contention_count counter
process_runtime_dotnet_monitor_lock_contention_count 3 1658960659966
# HELP process_runtime_dotnet_thread_pool_threads_count The number of thread pool threads that currently exist.
# TYPE process_runtime_dotnet_thread_pool_threads_count gauge
process_runtime_dotnet_thread_pool_threads_count 2 1658960659966
# HELP process_runtime_dotnet_thread_pool_completed_items_count The number of work items that have been processed by the thread pool since the process start.
# TYPE process_runtime_dotnet_thread_pool_completed_items_count counter
process_runtime_dotnet_thread_pool_completed_items_count 99 1658960659966
# HELP process_runtime_dotnet_thread_pool_queue_length The number of work items that are currently queued to be processed by the thread pool.
# TYPE process_runtime_dotnet_thread_pool_queue_length gauge
process_runtime_dotnet_thread_pool_queue_length 0 1658960659966
# HELP process_runtime_dotnet_timer_count The number of timer instances that are currently active. Timers can be created by many sources such as System.Threading.Timer, Task.Delay, or the timeout in a CancellationSource. An active timer is registered to tick at some point in the future and has not yet been canceled.
# TYPE process_runtime_dotnet_timer_count gauge
process_runtime_dotnet_timer_count 10 1658960659966
# HELP process_runtime_dotnet_assemblies_count The number of .NET assemblies that are currently loaded.
# TYPE process_runtime_dotnet_assemblies_count gauge
process_runtime_dotnet_assemblies_count 69 1658960659966
# HELP process_runtime_dotnet_exceptions_count Count of exceptions that have been thrown in managed code, since the observation started. The value will be unavailable until an exception has been thrown after OpenTelemetry.Instrumentation.Runtime initialization.
# TYPE process_runtime_dotnet_exceptions_count counter
" to contain "TYPE MyFruitCounter counter"."

Notice other failures as well

This one also looks strange: https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7549934722?check_suite_focus=true

Same here: https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7549934642?check_suite_focus=true

What happened? Some problems with the machine?

[pjanotti]

The failure below is due to it missing the expected string, but, in this case it seems related to be missing the expected metric https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7648224878?check_suite_focus=true#step:4:3005

[pjanotti]

There seems to be a synchronization problem with the test in this failure here

https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7639823504?check_suite_focus=true#step:4:3004

https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/runs/7639820988?check_suite_focus=true#step:4:3005

It seems to get its answer after the first reply from the server, but that reply didn't have content yet.