[Keycloak Integration] How to setup database connection for Keycloak when deploying to Azure

[ekomsctr]

Hello,

i'm trying to deploy my Aspire solution to the Azure cloud. I'm having some trouble when configuring Keycloak environment variables to connect to the deployed Azure PostgreSql (also a resource of the same solution).

I made a simple extension method for the KeycloakResource, which accepts the AzurePostgresFlexibleServerDatabaseResource i'm trying to configure:

public static IResourceBuilder<KeycloakResource> WithDatabase(this IResourceBuilder<KeycloakResource> builder, IResourceBuilder<AzurePostgresFlexibleServerDatabaseResource> source)
{
    builder.WithEnvironment(async context =>
     {
         if (context.ExecutionContext.IsPublishMode)
         {
             context.EnvironmentVariables.Add("TEST_KC_DB", "");
             context.EnvironmentVariables.Add("TEST_KC_DB_PASSWORD", "");
             context.EnvironmentVariables.Add("TEST_KC_DB_URL", "");
             context.EnvironmentVariables.Add("TEST_KC_DB_USERNAME", "");
             context.EnvironmentVariables.Add("TEST_KC_HOSTNAME", "");
         }
         else
         {
             var dbConnection = await source.Resource.ConnectionStringExpression.GetValueAsync(CancellationToken.None);
             var connStringBuilder = new Npgsql.NpgsqlConnectionStringBuilder(dbConnection);
             var keycloakDb = source.Resource;

             context.EnvironmentVariables.Add("KC_DB", "postgres");
             context.EnvironmentVariables.Add("KC_DB_PASSWORD", connStringBuilder.Password);
             context.EnvironmentVariables.Add("KC_DB_URL", $"jdbc:postgresql://pgsql:5432/{keycloakDb.DatabaseName}");
             context.EnvironmentVariables.Add("KC_DB_USERNAME", connStringBuilder.Username);
             context.EnvironmentVariables.Add("KC_HOSTNAME", connStringBuilder.Host);
         }
     });

    return builder;
}

For local development, i've retrieved successfully the connection string at runtime, and everything runs smoothly, but i'm having trouble doing the same for when the solution gets deployed (using Visual Studio publish profile at this moment).

For reference, here are the configured resources in the AppHost:

KeycloakResource

var keycloak = builder.AddKeycloak("keycloak", port: 8080)
    .WithDataVolume()
    .WithAnnotation(new CommandLineArgsCallbackAnnotation(args =>
    {
        if (!builder.ExecutionContext.IsPublishMode)
        {
            args.Add("--log-level=DEBUG");
            args.Add("--log-console-color=true");
            args.Add("--log=console,file");
            args.Add("--features=scripts");
        }
    }))
    .WithExternalHttpEndpoints()
    .WithProxyEdgeConfiguration()
    .WithDatabase(keycloakdb)
    .WithLifetime(ContainerLifetime.Persistent)
    .WaitFor(keycloakdb);

AzurePostgresFlexibleServerDatabaseResource

var sql = builder.ExecutionContext.IsPublishMode ?
    builder.AddAzurePostgresFlexibleServer("pgsql").WithPasswordAuthentication() :
    builder.AddAzurePostgresFlexibleServer("pgsql").RunAsContainer(container =>
    {
        container
            .WithLifetime(ContainerLifetime.Persistent)
            .WithDataVolume(isReadOnly: false)
            .WithPgWeb(pgWeb => pgWeb.WithExternalHttpEndpoints().PublishAsContainer());
    });

var keycloakdb = sql.AddDatabase("keycloakdb", "KeycloakDb")
    .WithCreateCommand(true);

Any tips on improving this implementation and how to get the parameters i need when publishing?

Thank you in advance, Roberto

[davidfowl]

Let me know if this helps https://github.com/dotnet/docs-aspire/issues/2340#issuecomment-2566057406

[ekomsctr]

Let me know if this helps dotnet/docs-aspire#2340 (comment)

Hi David,

i will look into it and let you know ASAP! But atleast in the "local environment", is that the right way to go or should i still use endpoints when working with a container? I'm looking for the most "standard" way, and building the connection string (like a did in the WithDatabase extension method) didn't convince me enough to not classify it as a "workaround".

Thanks in advance!

[davidfowl]

If you use the right expression it'll work in both cases.

[ekomsctr]

So, reading the article you linked surely helped me understand better how to use the endpoints, so it's a great piece! Unfortunately, that didn't help (or better, didn't go deep enough) for my topic, or better, maybe the Aspire Azure Postgres Flexible Server resource documentation needs some more guidance. Anyway, i'll summarize where i've been going:

Aspire Resource Configuration

//  Database credential secrets
var dbUsername = builder.AddParameter("DatabaseUsername", true);
var dbPassword = builder.AddParameter("DatabasePassword", true);

//  Declare the database endpoint reference
EndpointReference postgresEndpoint = null;

//  Azure Postgres resource; run locally in a container with PgWeb, published on Azure using password authentication
var postgres = builder.ExecutionContext.IsPublishMode ?
    builder.AddAzurePostgresFlexibleServer("postgres")
        .WithPasswordAuthentication(dbUsername, dbPassword) :
    builder.AddAzurePostgresFlexibleServer("postgres")
        .WithPasswordAuthentication(dbUsername, dbPassword)
        .RunAsContainer(container =>
        {
            container
                .WithLifetime(ContainerLifetime.Persistent)
                .WithDataVolume()
                .WithPgWeb();

            postgresEndpoint = container.GetEndpoint("tcp");
        });

KeyCloakResource WithDatabase extension method

public static IResourceBuilder<KeycloakResource> WithDatabase(this IResourceBuilder<KeycloakResource> builder, EndpointReference? postgresEndpoint = null, IResourceBuilder<ParameterResource>? dbUsername = null, IResourceBuilder<ParameterResource>? dbPassword = null)
{
    builder.WithAnnotation(new CommandLineArgsCallbackAnnotation(args =>
    {
        if (builder.ApplicationBuilder.ExecutionContext.IsRunMode)
        {
            args.Add("--log-level=DEBUG");
            args.Add("--log-console-color=true");
            args.Add("--features=scripts");
        }
        else
        {
            args.Add("start-dev");
            args.Add("--log-console-color=true");
            args.Add("--proxy-headers=xforwarded");
            args.Add("--features=scripts");
        }
    }));

    if (dbUsername != null)
        builder.WithEnvironment("KC_DB_USERNAME", dbUsername);
    if (dbPassword != null)
        builder.WithEnvironment("KC_DB_PASSWORD", dbPassword);
    if (postgresEndpoint != null)
        builder.WithEnvironment("KC_DB_URL", postgresEndpoint);
    builder.WithEnvironment("KC_HOSTNAME", "localhost");
    builder.WithEnvironment("KC_DB", "postgres");

    return builder;
}

However, i can't find a way to pass the endpoint when not running as container, since neither IResourceBuilder<AzurePostgresFlexibleServerDatabaseResource> nor IResourceBuilder<AzurePostgresFlexibleServerResource> expose the GetEndpoint() method.

[ekomsctr]

Any insight on how to handle this? In any case, even locally, the only endpoint available is the TCP endpoint, it would be nice to get a way (if available, maybe not everything when working locally, but ideally everything when deploying on Azure) to retrieve these settings:

[davidfowl]

The one thing here missing is that the azure Postgres flexible server resource does not expose the host name of the postgres server as a separate value you can use in keycloak.

You can manipulate the bicep so that the hostname is exposed (we should probably just do that natively) and then you can pass that reference to the environment variable of keycloak. Instead of taking an EndpointReference for the postgresEndpoint make it a ReferenceExpression. That'll make it easier to switch between an endpoint or bicep output.

var dbUsername = builder.AddParameter("DatabaseUsername", true);
var dbPassword = builder.AddParameter("DatabasePassword", true);

ReferenceExpression? hostEndpoint = null;

var pg = builder.AddAzurePostgresFlexibleServer("azpg")
                    .WithPasswordAuthentication(dbUsername, dbPassword)
                    .ConfigureInfrastructure(infra =>
                    {
                        // Get the postgres flexible server resource
                        var pg = infra.GetProvisionableResources().OfType<PostgreSqlFlexibleServer>().Single();

                        // Add the host name as an output to the bicep module
                        infra.Add(new ProvisioningOutput("hostname", typeof(string))
                        {
                            Value = pg.FullyQualifiedDomainName
                        });
                    })
                    .RunAsContainer(c =>
                    {
                        hostEndpoint = ReferenceExpression.Create($"{c.Resource.PrimaryEndpoint}");
                    });

hostEndpoint ??= ReferenceExpression.Create($"{pg.GetOutput("hostname")}");

var db = pg.AddDatabase("db");

// Wire up keycloak now...

[ekomsctr]

The one thing here missing is that the azure Postgres flexible server resource does not expose the host name of the postgres server as a separate value you can use in keycloak.

You can manipulate the bicep so that the hostname is exposed (we should probably just do that natively) and then you can pass that reference to the environment variable of keycloak. Instead of taking an EndpointReference for the postgresEndpoint make it a ReferenceExpression. That'll make it easier to switch between an endpoint or bicep output.

var dbUsername = builder.AddParameter("DatabaseUsername", true); var dbPassword = builder.AddParameter("DatabasePassword", true);

ReferenceExpression? hostEndpoint = null;

var pg = builder.AddAzurePostgresFlexibleServer("azpg") .WithPasswordAuthentication(dbUsername, dbPassword) .ConfigureInfrastructure(infra => { // Get the postgres flexible server resource var pg = infra.GetProvisionableResources().OfType<PostgreSqlFlexibleServer>().Single();

                    // Add the host name as an output to the bicep module
                    infra.Add(new ProvisioningOutput("hostname", typeof(string))
                    {
                        Value = pg.FullyQualifiedDomainName
                    });
                })
                .RunAsContainer(c =>
                {
                    hostEndpoint = ReferenceExpression.Create($"{c.Resource.PrimaryEndpoint}");
                });

hostEndpoint ??= ReferenceExpression.Create($"{pg.GetOutput("hostname")}");

var db = pg.AddDatabase("db");

// Wire up keycloak now...

Nice, thank you for the insight! Perhaps, this same solution would work both when running locally and when deploying to Azure?

Edit: i see how that would work when running locally (missed the RunAsContainer part), but how will hostEndpoint be populated when deploying to Azure? (i'm using the publish feature of Visual Studio 2022)

[davidfowl]

This code:

hostEndpoint ??= ReferenceExpression.Create($"{pg.GetOutput("hostname")}");

[ekomsctr]

This code:

hostEndpoint ??= ReferenceExpression.Create($"{pg.GetOutput("hostname")}");

Oh ok, so let me clarify if i understood this correctly: if i'm in RunMode, the hostEndpoint variable is allocated from the PrimaryEndpoint of the postgres container, meanwhile if the context is in PublishMode, the variable gets allocated from the Bicep output (which we added in the ConfigureInfrastructure method (but only if the hostEndpoint variable is null, which means the RunAsContainer method should have not run).

Is this right?

This way, i shouldn't even need to do a check for the environment, as this would work both in RunMode and PublishMode, right?

Thank you in advance! This has been really helpful

[davidfowl]

You got it!

[davidfowl]

This change should make it easier https://github.com/dotnet/aspire/pull/11051 to resolve the host username password and db.

[ekomsctr]

Hello, is this already available in .NET Aspire 9.4.2?

[davidfowl]

Is https://github.com/dotnet/aspire/pull/11051 available? Or are you asking about something else?

[ekomsctr]

Is #11051 available? Or are you asking about something else?

Yes, the "simplified" retrieval of the Azure Postgres instance parameters like HostName, UserName, Password, etc..

[davidfowl]

It’s in the next version 9.5.

You can look at the milestone of the PR.

[ekomsctr]

It’s in the next version 9.5.

You can look at the milestone of the PR.

Thank you very much!