[BUG] Exception/error log on application stop

[aheubusch]

APM Agent version

1.20.0

Environment

Windows 10, .NET 7, apm-server 7.17.6

Describe the bug

When the application stops Elastic.Apm writes an error log because of an InvalidOperationException in PayloadSenderV2 (see Actual Behavior).

To Reproduce

Steps to reproduce the behavior:

1. Prepare a working apm-server (7.17.6)
2. Start minimal reproduce application
3. Wait a few seconds
4. CTRL+C to stop application

Program.cs

using Elastic.Apm.NetCoreAll;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var builder = Host.CreateDefaultBuilder(args)
    .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); })
    .ConfigureAppConfiguration(config =>
    {
        config.AddInMemoryCollection(new Dictionary<string, string>
        {
            ["ElasticApm:ServerUrl"] = "http://localhost:8200",
            ["ElasticApm:CentralConfig"] = "false", // Remove error at startup
        });
    })
    .UseAllElasticApm();

await builder.RunConsoleAsync();

class Startup
{
    public void ConfigureServices(IServiceCollection services) { }
    public void Configure(IApplicationBuilder app) { }
}

ElasticApm_Exception_On_Stop.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Elastic.Apm.NetCoreAll" Version="1.20.0" />
  </ItemGroup>

</Project>

Expected behavior

No exception and/or error log.

Actual behavior

info: Elastic.Apm[0]
      ********************************************************************************
info: Elastic.Apm[0]
      Elastic APM .NET Agent, version: 1.20.0+8b1b6c83522b19dfbcc415f7ef62059e24fc1398
info: Elastic.Apm[0]
      Process ID: 42248
info: Elastic.Apm[0]
      Process Name: ElasticApm_Exception_On_Stop
info: Elastic.Apm[0]
      Command line arguments: 'D:\src_test\ElasticApm_Exception_On_Stop\ElasticApm_Exception_On_Stop\bin\Debug\net7.0\ElasticApm_Exception_On_Stop.dll'
info: Elastic.Apm[0]
      Operating System: Microsoft Windows 10.0.19044
info: Elastic.Apm[0]
      CPU architecture: X64
info: Elastic.Apm[0]
      Host: WS013W1064
info: Elastic.Apm[0]
      Time zone: (UTC+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna
info: Elastic.Apm[0]
      Runtime: .NET 7.0.3
info: Elastic.Apm[0]
      ********************************************************************************
info: Elastic.Apm[0]
      Agent Configuration (via 'Elastic.Apm.Extensions.Hosting.Config.MicrosoftExtensionsConfig'):
info: Elastic.Apm[0]
       - central_config: 'false' (Microsoft.Extensions.Configuration)
info: Elastic.Apm[0]
       - log_level: 'Error' (default)
info: Elastic.Apm[0]
       - server_url: 'http://localhost:8200' (Microsoft.Extensions.Configuration)
info: Elastic.Apm[0]
      {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service name. The service name will be automatically discovered.
info: Elastic.Apm[0]
      {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service name. The automatically discovered service name is ElasticApm_Exception_On_Stop
info: Elastic.Apm[0]
       - service_name: 'ElasticApm_Exception_On_Stop' (default)
info: Elastic.Apm[0]
      {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service version. The service version will be automatically discovered.
info: Elastic.Apm[0]
      {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service version. The automatically discovered service version is 1.0.0
info: Elastic.Apm[0]
       - service_version: '1.0.0' (default)
info: Elastic.Apm[0]
      ********************************************************************************
info: Elastic.Apm[0]
      {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service name. The service name will be automatically discovered.
info: Elastic.Apm[0]
      {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service name. The automatically discovered service name is ElasticApm_Exception_On_Stop
info: Elastic.Apm[0]
      {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service version. The service version will be automatically discovered.
info: Elastic.Apm[0]
      {AbstractConfigurationReader (MicrosoftExtensionsConfig)} The agent was started without a service version. The automatically discovered service version is 1.0.0
info: Elastic.Apm[0]
      {Service} Detected agent activation method: nuget
info: Elastic.Apm[0]
      CreateHttpClientHandler - SslProtocols: Tls12
info: Elastic.Apm[0]
      {CentralConfigurationFetcher} Central configuration feature is DISABLED because CentralConfig option's value is False (default value is True)
info: Elastic.Apm[0]
      {MetricsCollector} Collecting metrics in 30000 milliseconds interval
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: D:\src_test\ElasticApm_Exception_On_Stop\ElasticApm_Exception_On_Stop\bin\Debug\net7.0
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
fail: Elastic.Apm[0]
      {BackendCommComponentBase (PayloadSenderV2)} WorkLoop Current thread: `ElasticApmPayloadSenderV2' (managed ID: 11)
      System.InvalidOperationException: The source completed without providing data to receive.
         at System.Threading.Tasks.Dataflow.Internal.Common.InitializeStackTrace(Exception exception)
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Dataflow.DataflowBlock.Receive[TOutput](ISourceBlock`1 source, TimeSpan timeout, CancellationToken cancellationToken)
         at Elastic.Apm.Report.PayloadSenderV2.ReceiveBatch()
         at Elastic.Apm.Report.PayloadSenderV2.WorkLoopIteration()
         at Elastic.Apm.BackendComm.BackendCommComponentBase.WorkLoop()