AKS
AKS copied to clipboard
Azure
Fail to load Azure AD Authentication configuration when AzureAD:xxx env var are mounted from KeyVault CSI SecretStore
Describe the bug I have mounted a CSI secret store from my KeyVault. It works for all attributes, except AzureAD:Domain, AzureAD:TenantId, AzureAD:Instance. When the AzureAD Authentication loads, it seems to use a file:///mnt/xxx/yyy instead of the KeyVaulkt secret value.
I got the error :
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: \u0027file:///mnt/xxx-kv/AzureAdInstance/mnt/xxx-kv/AzureAdTenantId/v2.0/.well-known/openid-configuration\u0027.
at Microsoft.IdentityModel.Protocols.ConfigurationManager\u00601.GetConfigurationAsync(CancellationToken cancel)
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
at Microsoft.AspNetCore.Authentication.AuthenticationHandler\u00601.AuthenticateAsync()
at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication\u00601 application)
If I deactivate Swagger, I got the same error :
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: \u0027file:///mnt/xxx-kv/AzureAdInstance/mnt/xxx-kv/AzureAdTenantId/v2.0/.well-known/openid-configuration\u0027.
at Microsoft.IdentityModel.Protocols.ConfigurationManager\u00601.GetConfigurationAsync(CancellationToken cancel)
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
at Microsoft.AspNetCore.Authentication.AuthenticationHandler\u00601.AuthenticateAsync()
at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication\u00601 application)
If instead of the mounted values I put hard coded values for AzureAD__Domain, AzureAD__TenantId, AzureAD__Instance, it works fine.
To Reproduce Steps to reproduce the behavior:
- Mount a CSI Secret Store
- Use an ASPNET 5.0 application with AzureAD Authentication enabled
- Run the application in AKS 1.24.6
Expected behavior The AzureAD Authentication loads.
Manifest Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-deployment
namespace: dev
spec:
replicas: 1
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
serviceAccountName: workload-identity-dev-sa
containers:
- name: front
image: xxx.azurecr.io/xxx:yyy
ports:
- containerPort: 8080
env:
- name: "AzureAd__ClientId"
value: "8fe7d5bc-xxx-xxx-xxx-3d44be83cb20"
- name: "AzureAd__Instance"
value: "/mnt/xxx-kv/AzureAdInstance"
- name: "AzureAd__Domain"
value: "/mnt/xxx-kv/AzureAdDomain"
- name: "AzureAd__TenantId"
value: "/mnt/xxx-kv/AzureAdTenantId"
- name: "InstallationServiceBusConnectionString"
value: "/mnt/xxx-kv/installationCoreServiceBusConnectionString"
volumeMounts:
- name: xxxName
mountPath: /mnt/xxx-kv
readOnly: true
imagePullPolicy: Always
volumes:
- name: xxxName
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: xxx-secret-provider
CSI Secret Store
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: xxx-secret-provider
namespace: dev
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "false"
clientID: "xxx"
keyvaultName: "xxx"
objects: |
array:
- |
objectName: AzureAdInstance
objectType: secret
objectVersion: ""
- |
objectName: AzureAdDomain
objectType: secret
objectVersion: ""
- |
objectName: AzureAdTenantId
objectType: secret
objectVersion: ""
- |
objectName: installationCoreServiceBusConnectionString
objectType: secret
objectVersion: ""
tenantId: "xxx"
Startup.cs
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
IdentityModelEventSource.ShowPII = true;
app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
});
}
public virtual void ConfigureServices(IServiceCollection services)
{
services.AddMicrosoftIdentityWebApiAuthentication(Configuration);
services.Configure<JwtBearerOptions>(JwtBearerDefaults.AuthenticationScheme,
options => {
options.RequireHttpsMetadata = false;
}
);
services
.AddControllers()
.AddJsonOptions(opt => opt.JsonSerializerOptions.Converters.Add(new JsonStringEnumConverter()));
var sliceAssembly = typeof(BackMeActivationTrackingController).Assembly;
services.AddControllers()
.PartManager.ApplicationParts.Add(new AssemblyPart(sliceAssembly));
}
Environment (please complete the following information):
- Kubernetes version 1.24.6
I have mounted a CSI secret store from my KeyVault. It works for all attributes, except AzureAD:Domain, AzureAD:TenantId, AzureAD:Instance. When the AzureAD Authentication loads, it seems to use a file:///mnt/xxx/yyy instead of the KeyVaulkt secret value.
@mleneveut Could you clarify what the question is here? Is the CSI driver failing to mount the required secrets from AKV? If the CSI driver is mounting all the secrets as expected and you confirmed the values look fine in the mount, but the application is having issues consuming the secret, it might be good to check the code/reach out to the SDK folks.
@aramase Thanks for your response. The CSI is working fine for some secrets. I only have this problem with the Azure AD configuration.
Instead of giving the value of the secret to the configuration, it seems to give the path of the file containing the secret : the configuration setup constructs this URL : file:///mnt/xxx-kv/AzureAdInstance/mnt/xxx-kv/AzureAdTenantId/v2.0/.well-known/openid-configuration
As shown in the first post, I don't give anything special to the Azure AD Configuration, just :
services.AddMicrosoftIdentityWebApiAuthentication(Configuration);
services.Configure<JwtBearerOptions>(JwtBearerDefaults.AuthenticationScheme,
options => {
options.RequireHttpsMetadata = false;
}
);
If I set hard coded values in the Kubernetes yml for AzureAD__Instance and AzureAD_TenantID, it works fine.
Could you redirect me to the SDK folks if you think it is not in this github ? Thanks.
![microsoft-github-policy-service[bot] avatar](https://avatars.githubusercontent.com/in/95686?v=4 "Published by a pub.dev verified publisher"){kind=small}