Ability to programmatically control config settings

[ejsmith]

My environment in aspnetcore is set to staging, but I want to change it to something more specific for Elastic APM. I know that I can control it via environment variables, but I need to set it programmatically based on other settings. It appears that you do not allow just passing a configuration object into UseElasticApm and force me to pass an IConfiguration object. Seems like it would be very useful to be able to read settings from IConfiguration and then be able to additionally change something programmatically before I give it to the UseElasticApm method and start the instrumentation process.

[vhatsura]

Hey, @ejsmith

You're right, at the moment Elastic.Apm doesn't allow to pass a configuration object instead of IConfiguration. The reason for it is pretty simple, Elastic.Apm implements a concept of configuration readers when the only source of configuration can be specified. So, maybe it make sense to introduce additional configuration reader: from key-value pairs or another data structure that is stored in memory.

In addition, I believe it's possible to achieve your goal without any changes in Elastic.Apm. The only one thing that you need to add in-memory configuration provider, which can provide values based on various assumptions: environment, where code is executed, values from a configuration file or whatever. It would be nice to hear your thoughts regarding it.

In the end, at least one enhancement makes sense in case of agent configuration for ASP.NET Core : allow users to pass custom implementation of the IConfigurationReader interface for more advanced scenarios like in your case.

It would be interesting to hear the opinion of @gregkalapos, too.

[ejsmith]

Yeah, I could possibly do something with InMemoryConfiguration, except my programmatic config relies on other config sources. So I would need to build my configuration multiple times. It's not horrible, but it just seems pretty strange that I can't programmatically set config for Elastic APM.

[gregkalapos]

I agree with the recommendation from @vhatsura - the whole idea of passing an IConfiguration was that you can use any config source through that and the agent will read those out based on the key that you also find in our docs (see "IConfiguration or Web.config key").

To this point:

In the end, at least one enhancement makes sense in case of agent configuration for ASP.NET Core : allow users to pass custom implementation of the IConfigurationReader interface for more advanced scenarios like in your case.

I see a couple of issues with this. First of all, we are not very happy with that interface in its current form - it's getting very big and the really bad thing is that technically we break it every time we introduce a new config currently (all implementors break in that case). So the future of that interface is a bit uncertain. We could do a lot here - like make it a class with default implementations (default interface method would limit runtime support, so that probably won't help :( ), or do it with some KeyValuePair<string,string> solution - so the point here is that I'm a bit reluctant to expose that interface right now, because it may go away or may changes radically in the future - this'd happen typically on a major version bump (like in agent Version 2 or so).

But once we take care of that then from the API stability point of view, we could expose it over the UseElasticApm (and UseAllElasticApm) methods. This would complicate the API and users may be surprised and confused by accepting both IConfugration from Microsoft.Extensions.Configuration and IConfigurationReader from Elastic APM.

The reason I say this is I wanna make sure we do more good than harm with this. If it's the case, we can go for it.

Therefore I'd like to understand the issue @ejsmith has a bit more.

Could you elaborate on this please a bit to make sure I fully get it?

Yeah, I could possibly do something with InMemoryConfiguration, except my programmatic config relies on other config sources. So I would need to build my configuration multiple times.

If I understand correctly then accepting an IConfigurationReader in the UseElasticApm does not really make a difference here - or I miss the point, which is very much possible :)

The way this should work is that you can do something like this:

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
	WebHost.CreateDefaultBuilder(args)
		.ConfigureAppConfiguration((hostingContext, config) =>
		{
			config.AddInMemoryCollection(GetInMemoryConfig()); // use the additional config for the APM Agent
			config.AddCommandLine(args); // also use command line args as config source
			config.AddJsonFile("appsettings.json"); // read everything else from appsettings.json
                        //...other config sources potentially
		})
		.UseStartup<Startup>();

private static Dictionary<string, string> GetInMemoryConfig()
	=> // in real world - read the config value from the real config source (like file, from an HTTP endpoint, or something else)
		new Dictionary<string, string> { { "ElasticApm:Environment", "MyEnvironment" } };

And then register the agent with:

app.UseAllElasticApm(Configuration);

With this the agent will set the environment to the value that you return in the GetInMemoryConfig method into the ElasticApm:Environment. So I don't really see how this would be simpler by accepting an IConfigurationReader in app.UseAllElasticApm - I feel you'd have the same code in the GetInMemoryConfig method except you'd fill an IConfigurationReader implementation instead of a Dictionary<string,string>.

Or isn't that the case?

[ejsmith]

I mean that I am using config values to derive other config values and in this particular case I am trying to set the service environment setting to something more specific. This is what I ended up doing which I think is less than ideal:

var config = new ConfigurationBuilder()
    .SetBasePath(Directory.GetCurrentDirectory())
    .AddYamlFile("appsettings.yml", optional: false, reloadOnChange: true)
    .AddYamlFile($"appsettings.{environment}.yml", optional: true, reloadOnChange: true)
    .AddEnvironmentVariables()
    .AddEnvironmentVariables("ASPNETCORE_")
    .AddCommandLine(args)
    .Build();

if (config.GetSection("ElasticApm").Exists()) {
    var options = AppOptions.ReadFromConfiguration(config);
    
    var dynamicConfig = new Dictionary<string, string>();
    dynamicConfig["ElasticApm:ServiceName"] = "Contacts Web";
    if (!String.IsNullOrEmpty(options.AppScope))
        dynamicConfig["ElasticApm:Environment"] = options.AppScope;
    
    config = new ConfigurationBuilder()
        .AddConfiguration(config)
        .AddInMemoryCollection(dynamicConfig)
        .Build();
}

I think that it is very common for a user to want to use all default config for Elastic APM and we might just want to use an overload to set the secret with just this app.UseAllElasticApm("mysecret") and then in addition to that I think you should create a strongly typed config object that just has the values that can be set through configuration on the client and you should be able to use that like this:

var config = AgentConfiguraton.ReadFromConfiguration(myAspNetConfig);
config.Environment = "My Overridden Environment Name";
app.UseAllElasticApm(config);

or

var config = AgentConfiguraton.ReadFromEnvironment(myAspNetConfig);
config.Environment = "My Overridden Environment Name";
app.UseAllElasticApm(config);

So basically, you have a APM config object that contains all the things that can be set and you allow it to be passed into the UseAllElasticApm method. That config object can be populated many ways including programmatically. You can add new config options without breaking anyone. You can have several different built in ways of reading the config options from the ASP.NET configuration system or environment variables and then you could keep your same current app.UseAllElasticApm(myAspNetConfig) that simply just calls the AgentConfiguraton.ReadFromConfiguration(myAspNetConfig) to get the agent configuration.

[ejsmith]

By the way, those methods ReadFromConfiguration and ReadFromEnvironment could just be extension methods and anyone else could create other variations as well.

The bottom line is that it is very cumbersome in a lot of situations to require me to put all of my configuration inside of the ASP.NET configuration system. It also presumes that I even use that configuration system which I've seen a lot of apps that don't. Being able to easily programmatically configure the APM agent seems essential to me and gives ultimate flexibility and it is the same thing that pretty much all similar tools do.

[vhatsura]

I see a couple of issues with this. First of all, we are not very happy with that interface in its current form - it's getting very big and the really bad thing is that technically we break it every time we introduce a new config currently (all implementors break in that case). So the future of that interface is a bit uncertain. We could do a lot here - like make it a class with default implementations (default interface method would limit runtime support, so that probably won't help :( ), or do it with some KeyValuePair<string,string> solution - so the point here is that I'm a bit reluctant to expose that interface right now, because it may go away or may changes radically in the future - this'd happen typically on a major version bump (like in agent Version 2 or so).

I totally agree with you, IConfigurationReader is a very tricky interface, but I was speaking from a user perspective: Elastic.Apm exposes public interface (not only IConfigurationReader), which I can implement, but, on the other hand, I cannot change the default implementation by mine realization. Although, I also agreed on the following statement:

This would complicate the API and users may be surprised and confused by accepting both IConfugration from Microsoft.Extensions.Configuration and IConfigurationReader from Elastic APM.

[xitaocrazy]

Hello guys, I would like to share my experience and how I solved this type of problem, it's nothing special, but it helped me to configure APM programmatically

First, I created a class with all the APM constants, I really don't know if the agent already provides something like this.

public static class ElasticApmConstants
{
	public static string ServiceName => "ElasticApm:ServiceName";
	public static string ServiceNodeName => "ElasticApm:ServiceNodeName";
	public static string ServiceVersion => "ElasticApm:ServiceVersion";
	public static string Environment => "ElasticApm:Environment";
	public static string TransactionSampleRate => "ElasticApm:TransactionSampleRate";
	public static string TransactionMaxSpans => "ElasticApm:TransactionMaxSpans";
	public static string CentralConfig => "ElasticApm:CentralConfig";
	public static string SanitizeFieldNames => "ElasticApm:SanitizeFieldNames";
	public static string GlobalLabels => "ElasticApm:GlobalLabels";
	public static string ServerUrls => "ElasticApm:ServerUrls";
	public static string SecretToken => "ElasticApm:SecretToken";
	public static string ApiKey => "ElasticApm:ApiKey";
	public static string VerifyServerCert => "ElasticApm:VerifyServerCert";
	public static string FlushInterval => "ElasticApm:FlushInterval";
	public static string MaxBatchEventCount => "ElasticApm:MaxBatchEventCount";
	public static string MaxQueueEventCount => "ElasticApm:MaxQueueEventCount";
	public static string MetricsInterval => "ElasticApm:MetricsInterval";
	public static string DisableMetrics => "ElasticApm:DisableMetrics";
	public static string CaptureBody => "ElasticApm:CaptureBody";
	public static string CaptureBodyContentTypes => "ElasticApm:CaptureBodyContentTypes";
	public static string CaptureHeaders => "ElasticApm:CaptureHeaders";
	public static string ApplicationNamespaces => "ElasticApm:ApplicationNamespaces";
	public static string ExcludedNamespaces => "ElasticApm:ExcludedNamespaces";
	public static string StackTraceLimit => "ElasticApm:StackTraceLimit";
	public static string SpanFramesMinDuration => "ElasticApm:SpanFramesMinDuration";
	public static string LogLevel => "ElasticApm:LogLevel";
}

And I have created an extension method that receives IConfiguration and a dictionary with my settings defined on the fly.

public static IApplicationBuilder UseElasticApm(this IApplicationBuilder builder, IConfiguration configuration, Dictionary<string, string> myConfig)
{
	myConfig.ForEach(x => configuration[x.Key] = x.Value);
	return builder.UseAllElasticApm(configuration);
}

In my Configure method, I call my own extension by passing my list of settings, so I override the IConfiguration and send it to APM, of course, I have to use the properties in ElasticApmConstants as keys in my dictionary.

[ArveSystad]

Just wanna chime in and say that some way to inject a custom IConfigurationProvider (or whatever form it takes in the future) would be a very, very welcome change.

In our scenario, we're on a long journey from old to new on a distributed system, where we have components that are ASPNET Full Framework, ASPNET Core and Windows Services (both hosted inside an NServicebus process and as separate generic hosts, the "modern way"). Some of the Windows services host ASPNET Core APIs on top of that. So for the Windows Services I need to do manual setup (which is fine), and then duplicate that same configuration into the ASPNETCore config files that are deployed along side it, which cannot be used in any standard way from the service as of now (NServiceBus hosted process, so no appsettings.json support out of the box).

I would like to use the auto instrumentation where possible (so for example let it hook up to HTTP requests honoring all config etc that is set), but at the same time be able to provide my own configuration implementation once, and only once. As of now, this doesn't seem possible, since for example app.UseElasticApm() doesn't accept any way to inject the static config, and the PayloadSenderV2 then ends up with a convention based configuration even if you've set the config on the Agent earlier.

It claims to be a singleton and stopping subsequent setup-attempts (It even reports The singleton APM agent has already been instantiated and can no longer be configured. Reusing existing instance.), which it does, but that doesn't protect from misconfiguration if AgentComponents are newed up with a separate configuration set in app.UseElasticApm().