Cannot connect new kubernetes cluster using Azure Arc

[radderz]

Describe the issue or the bug I have run through all the initial steps to connect my dev microk8s single node cluster running on Ubuntu Server 22.04 but I am getting an error saying that the network test is failing.

This is a brand new server with just microk8s with the core dns package installed. there is no web proxy or filtering of any of the servers outbound network traffic.

The server can connect to the outside world fine and can resolve the url it is saying is failing.

Setup Process

# pre connect commands:

# install microk8s
sudo snap install microk8s --classic
sudo snap install kubectl --classic
sudo usermod -a -G microk8s markr
sudo microk8s enable dns

sudo usermod -a -G microk8s markr
sudo chown -f -R markr ~/.kube
newgrp microk8s

# create config file in default location
microk8s config view > ~/.kube/config

# install azure cli
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# add extensions
az extension add --name connectedk8s

# register providers
az provider register --namespace Microsoft.Kubernetes
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ExtendedLocation

# wait for registered status
az provider show -n Microsoft.Kubernetes -o table
az provider show -n Microsoft.KubernetesConfiguration -o table
az provider show -n Microsoft.ExtendedLocation -o table

# login to azure
az login --use-device-code
az account set --subscription "xxxxxx"

#  connect the cluster
az connectedk8s connect --name "test-kube-ubuntu-22.04" --resource-group "kubernetes-arc-test" --location "australiaeast" --correlation-id "xxxxx"

Connection Error

markr@ubuntu-2204:~/.kube$ az connectedk8s connect --name "bh-kube-ubuntu-22.04" --resource-group "bh-kubernetes-arc-mark" --location "australiaeast" --correlation-id "c18ab9d0-685e-48e7-ab55-12588447b0ed"
This operation might take a while...

The outbound network connectivity check has failed for the endpoint - https://australiaeast.obo.arc.azure.com:8084/
This will affect the "cluster-connect" feature. If you are planning to use "cluster-connect" functionality , please ensure outbound connectivity to the above endpoint.

Error: We found an issue with outbound network connectivity from the cluster to the endpoints required for onboarding.
Please ensure to meet the following network requirements 'https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements?tabs=azure-cloud'
If your cluster is behind an outbound proxy server, please ensure that you have passed proxy parameters during the onboarding of your cluster.
For more details visit 'https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli#connect-using-an-outbound-proxy-server'

The pre-check result logs logs have been saved at this path:/home/markr/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024 .
These logs can be attached while filing a support ticket for further assistance.

One or more pre-onboarding diagnostic checks failed and hence not proceeding with             cluster onboarding. Please resolve them and try onboarding again.
markr@ubuntu-2204:~/.kube$ cd /home/markr/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024
markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$ dir
cluster_diagnostic_checks_job_log.txt  cluster_diagnostic_checks_pod_description.txt  diagnoser_output.txt  dns_check.txt  outbound_network_connectivity_check_for_cluster_connect.txt  outbound_network_connectivity_check_for_onboarding.txt
markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$ cat outbound_network_connectivity_check_for_onboarding.txt
Response code 000
We found an issue with Outbound network connectivity from the cluster required for onboarding.markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$ cat dns_check.txt
DNS Result:Server:10.152.183.10  Address:10.152.183.10#53    Name:kubernetes.default.svc.cluster.local  Address: 10.152.183.1  Checking outbound connectivity for Cluster Connect Pre-check Endpoint. This is an optional endpoint required only for cluster-connect functionality
Cluster DNS check passed successfully.markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$ cat cluster_diagnostic_checks_job_log.txt
Executing DNS and outbound connectivity check
DNS Result:Server:              10.152.183.10
Address:        10.152.183.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.152.183.1
Checking outbound connectivity for Cluster Connect Pre-check Endpoint. This is an optional endpoint required only for cluster-connect functionality
Outbound Network Connectivity Check for Cluster Connect : https://australiaeast.obo.arc.azure.com:8084/ : 000
Checking outbound connectivity for Onboarding Pre-check Endpoint. This is a mandatory endpoint.
Outbound Network Connectivity Check for Onboarding Result : https://mcr.microsoft.com : 000
markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$ cat diagnoser_output.txt
Error: We found an issue with outbound network connectivity from the cluster to the endpoints required for onboarding.
Please ensure to meet the following network requirements 'https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements?tabs=azure-cloud'
If your cluster is behind an outbound proxy server, please ensure that you have passed proxy parameters during the onboarding of your cluster.
For more details visit 'https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli#connect-using-an-outbound-proxy-server'

markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$ cat outbound_network_connectivity_check_for_cluster_connect.txt
Response code 000
Outbound connectivity failed for the endpoint:https://australiaeast.obo.arc.azure.com:8084/ ,this is an optional endpoint needed for cluster-connect feature.markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$
markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$ ping australiaeast.obo.arc.azure.com
PING k8connectcdmanaged.australiaeast.cloudapp.azure.com (20.37.196.249) 56(84) bytes of data.
64 bytes from 20.37.196.249 (20.37.196.249): icmp_seq=1 ttl=115 time=26.7 ms
^C
--- k8connectcdmanaged.australiaeast.cloudapp.azure.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 26.667/26.667/26.667/0.000 ms
markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$ curl https://australiaeast.obo.arc.azure.com:8084
markr@ubuntu-2204:~/.azure/pre_onboarding_check_logs/bh-kube-ubuntu-22.04-Thu-May--2-01.00.18-2024$

[microsoft-github-policy-service[bot]]

Hi radderz! Thank you for opening this issue. We appreciate your contribution and welcome you to our community! We are glad to have you here and to have your input on the Azure Arc Jumpstart.

[radderz]

I forgot to mention that I am using an "owner" user for the dev subscription

[likamrat]

Hi @radderz, since this is not a Jumpstart issue I will go ahead and close this issue for now. For a core product issue or feature request/feedback, please create an official Azure support ticket or general feedback request.