Not able to setup agic with service principal

[mohanrao]

I am trying to install the agic with service principal . I am getting the following errors in the agic log

2020-08-13T23:10:49.762686468Z ERROR: logging before flag.Parse: I0813 23:10:49.762518 1 utils.go:115] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL 2020-08-13T23:10:49.791614629Z E0813 23:10:49.789874 1 context.go:198] Error fetching AGIC Pod (This may happen if AGIC is running in a test environment). Error: pods "ingress-azure-cd8f7c4f7-2m2b4" is forbidden: User "system:serviceaccount:default:ingress-azure" cannot get resource "pods" in API group "" in the namespace "default" 2020-08-13T23:10:49.791641229Z I0813 23:10:49.789987 1 main.go:128] Appication Gateway Details: Subscription="xxxxxxxxxxxx" Resource Group="xxxxxxxxxx" Name="xxxxxxxxxxxx" 2020-08-13T23:10:49.791646829Z I0813 23:10:49.790012 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:10:49.791650429Z I0813 23:10:49.790158 1 retry.go:33] Retrying in 10s 2020-08-13T23:10:49.791653729Z I0813 23:10:49.790178 1 httpserver.go:57] Starting API Server on :8123 2020-08-13T23:10:59.790530792Z I0813 23:10:59.790371 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:10:59.790565292Z I0813 23:10:59.790487 1 retry.go:33] Retrying in 10s 2020-08-13T23:11:09.790946623Z I0813 23:11:09.790792 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:11:09.791054523Z I0813 23:11:09.790958 1 retry.go:33] Retrying in 10s 2020-08-13T23:11:19.791323586Z I0813 23:11:19.791166 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:11:19.791347886Z I0813 23:11:19.791278 1 retry.go:33] Retrying in 10s 2020-08-13T23:11:29.791639443Z I0813 23:11:29.791453 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:11:29.791667243Z I0813 23:11:29.791572 1 retry.go:33] Retrying in 10s 2020-08-13T23:11:39.791907558Z I0813 23:11:39.791769 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:11:39.791949958Z I0813 23:11:39.791891 1 retry.go:33] Retrying in 10s 2020-08-13T23:11:49.792201895Z I0813 23:11:49.792025 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:11:49.792222595Z I0813 23:11:49.792124 1 retry.go:33] Retrying in 10s 2020-08-13T23:11:59.792500517Z I0813 23:11:59.792305 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:11:59.792531017Z I0813 23:11:59.792453 1 retry.go:33] Retrying in 10s 2020-08-13T23:12:09.792938693Z I0813 23:12:09.792690 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:12:09.792971793Z I0813 23:12:09.792863 1 retry.go:33] Retrying in 10s 2020-08-13T23:12:19.793237586Z I0813 23:12:19.793043 1 auth.go:37] Creating authorizer from file referenced by environment variable: /etc/Azure/Networking-AppGW/auth/armAuth.json 2020-08-13T23:12:19.896421216Z E0813 23:12:19.896260 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:12:19.896447716Z I0813 23:12:19.896306 1 retry.go:33] Retrying in 10s 2020-08-13T23:12:29.917820817Z E0813 23:12:29.917668 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:12:29.917850117Z I0813 23:12:29.917689 1 retry.go:33] Retrying in 10s 2020-08-13T23:12:39.970482196Z E0813 23:12:39.970268 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:12:39.970514396Z I0813 23:12:39.970289 1 retry.go:33] Retrying in 10s 2020-08-13T23:12:50.022996512Z E0813 23:12:50.022840 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:12:50.023029912Z I0813 23:12:50.022865 1 retry.go:33] Retrying in 10s 2020-08-13T23:13:00.060863419Z E0813 23:13:00.060745 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:13:00.060891719Z I0813 23:13:00.060765 1 retry.go:33] Retrying in 10s 2020-08-13T23:13:10.114209512Z E0813 23:13:10.114034 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:13:10.114244412Z I0813 23:13:10.114056 1 retry.go:33] Retrying in 10s 2020-08-13T23:13:20.171392053Z E0813 23:13:20.171229 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:13:20.171428253Z I0813 23:13:20.171252 1 retry.go:33] Retrying in 10s 2020-08-13T23:13:30.221371298Z E0813 23:13:30.221161 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:13:30.221400698Z I0813 23:13:30.221183 1 retry.go:33] Retrying in 10s 2020-08-13T23:13:40.268931522Z E0813 23:13:40.268766 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:13:40.268963722Z I0813 23:13:40.268789 1 retry.go:33] Retrying in 10s 2020-08-13T23:13:50.320222305Z E0813 23:13:50.320015 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:13:50.320253505Z I0813 23:13:50.320039 1 retry.go:33] Retrying in 10s 2020-08-13T23:14:00.378074726Z E0813 23:14:00.377916 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:14:00.378098026Z I0813 23:14:00.377941 1 retry.go:33] Retrying in 10s 2020-08-13T23:14:10.425220826Z E0813 23:14:10.424972 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:14:10.425258326Z I0813 23:14:10.424994 1 retry.go:33] Retrying in 10s 2020-08-13T23:14:20.478527208Z E0813 23:14:20.478339 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:14:20.478550109Z I0813 23:14:20.478361 1 retry.go:33] Retrying in 10s 2020-08-13T23:14:30.501584868Z E0813 23:14:30.501444 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:14:30.501613568Z I0813 23:14:30.501466 1 retry.go:33] Retrying in 10s 2020-08-13T23:14:40.521278630Z E0813 23:14:40.521126 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:14:40.521306530Z I0813 23:14:40.521150 1 retry.go:33] Retrying in 10s 2020-08-13T23:14:50.569248548Z E0813 23:14:50.569080 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:14:50.569286448Z I0813 23:14:50.569101 1 retry.go:33] Retrying in 10s 2020-08-13T23:15:00.590000960Z E0813 23:15:00.589859 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:15:00.590033160Z I0813 23:15:00.589880 1 retry.go:33] Retrying in 10s 2020-08-13T23:15:10.610708108Z E0813 23:15:10.610578 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:15:10.610731908Z I0813 23:15:10.610600 1 retry.go:33] Retrying in 10s 2020-08-13T23:15:20.657969843Z E0813 23:15:20.657842 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:15:20.658006043Z I0813 23:15:20.657864 1 retry.go:33] Retrying in 10s 2020-08-13T23:15:30.675335283Z E0813 23:15:30.675179 1 client.go:170] Code="ErrorApplicationGatewayUnexpectedStatusCode" Message="Unexpected status code '401' while performing a GET on Application Gateway." InnerError="network.ApplicationGatewaysClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."" 2020-08-13T23:15:30.675367883Z I0813 23:15:30.675205 1 retry.go:33] Retrying in 10s

Also in the documentation it is not clear how to create the json for existing service principal user

[mohanrao]

After debugging the code it seems the json what we created/extracted from the local ./azure seems wrong.

@akshaysngupta can you point a right direction on how to create the json for existing service principal .

I couldn't find a mechanism how to create the rbac json file for existing service principal

[mohanrao]

Found out the json content need should be below like below

{ "clientId": "{{ clientid/spid }}", "clientSecret": "{{ sp_secret _password }}", "subscriptionId": "{{ subscription_id }}", "tenantId": "{{ tenant_id }}", "activeDirectoryEndpointUrl": "https://login.microsoftonline.com", "resourceManagerEndpointUrl": "https://management.azure.com/", "activeDirectoryGraphResourceId": "https://graph.windows.net/", "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/", "galleryEndpointUrl": "https://gallery.azure.com/", "managementEndpointUrl": "https://management.core.windows.net/" }

adding this to documentation would be helpful for others

[GenesisCoast]

It's also worth noting that the secretJSON needs to be encoded using UTF-8. Which can be done in Windows Powershell like this:

[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($ServicePrincipalJSON))

[Vishal2696]

Here is how I formed the Json manually for an existing service principal in Linux

cat > spjson << EOF
{
	"clientId": "${APP_URL}",
	"clientSecret": "${AZ_PASS}",
	"subscriptionId": "${SUBSCRIPTION_ID}",
	"tenantId": "${TENANT}",
	"activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
	"resourceManagerEndpointUrl": "https://management.azure.com/",
	"activeDirectoryGraphResourceId": "https://graph.windows.net/",
	"sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
	"galleryEndpointUrl": "https://gallery.azure.com/",
	"managementEndpointUrl": "https://management.core.windows.net/"
}
EOF
encryptedvalue=$(cat spjson | base64 -w0)
helm install ........ --set armAuth.secretJSON=${encryptedvalue}                      

[runes83]

Any updates on how to fix this?

[akshaysngupta]

@runes83 Adding to @vishal8k, let me know if the following help:

# create a new service principal and format the output so that it can be used with SDK. Generate base64 of the output.
# you can also use an existing service principal as well.
az ad sp create-for-rbac --sdk-auth > sp.json

# here is how the file looks:
cat sp.json
{
	"clientId": "${APP_URL}",
	"clientSecret": "${AZ_PASS}",
	"subscriptionId": "${SUBSCRIPTION_ID}",
	"tenantId": "${TENANT}",
	"activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
	"resourceManagerEndpointUrl": "https://management.azure.com/",
	"activeDirectoryGraphResourceId": "https://graph.windows.net/",
	"sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
	"galleryEndpointUrl": "https://gallery.azure.com/",
	"managementEndpointUrl": "https://management.core.windows.net/"
}

# convert to base64 to use with AGIC
encodedJson=$(cat sp.json | base64 -w0)

# If you have an existing deployment provide the encoded output when installing AGIC.
helm upgrade \
<helm-release-name>  \
 application-gateway-kubernetes-ingress/ingress-azure \
--reuse-values \
--set armAuth.type=servicePrincipal \
--set armAuth.secretJSON=${encodedJson}

@vishal8k Thanks for jumping in!

[Xat59]

It would be nice to document the way to get the secretJSON from an existing service principal.

[runes83]

@akshaysngupta Is this a work around or the way to do it? Msi works find on my other clusters but not in dev. The solution above looks a little "hacky"

[akanieski]

It appears --sdk-auth is "Deprecated"? Is that the case? Is AGIC prepped to use the oidc based alternative?

[joshusrytls]

I still seem to be running into this issue as well :\