Grafana was not created

[r-moiseev]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

• Tectonic version (release or commit hash): 1.8.4-tectonic.2
• Terraform version (terraform version): 0.10.7
• Platform (aws|azure|openstack|metal): metal

What happened?

When new cluster is started /grafana path returns 503 Service Unavailable. There is Ingress named tectonic-monitoring-ingress that has /grafaba path points to tectonic-monitoring-auth-grafana service. But this service is missing. Also there is no grafana deployment

What you expected to happen?

Grafana to be available

How to reproduce it (as minimally and precisely as possible)?

Provision new cluster with terraform (not UI).

There is my terraform.tfvars:

tectonic_license_path = "./license.txt"
tectonic_pull_secret_path = "./pull_secret.json"

tectonic_admin_email = "***"
tectonic_admin_password = "***"

tectonic_base_domain = "unused"

tectonic_cluster_cidr = "10.2.0.0/16"

tectonic_cluster_name = "dev-cluster"

tectonic_container_linux_version = "latest"

tectonic_networking = "canal"

tectonic_kube_apiserver_service_ip = "10.3.0.1"

tectonic_kube_dns_service_ip = "10.3.0.10"

tectonic_kube_etcd_service_ip = "10.3.0.15"

tectonic_metal_controller_domain = "master.cluster.***.com"

tectonic_metal_controller_domains = [
  "master.cluster.***.com"
]

tectonic_metal_controller_macs = [
  "56:00:01:53:11:c1"
]

tectonic_metal_controller_names = [
  "master"
]
tectonic_metal_ingress_domain = "tectonic.cluster.***.com"

tectonic_metal_matchbox_ca = "-----BEGIN CERTIFICATE-----***-----END CERTIFICATE-----"

tectonic_metal_matchbox_client_cert = "-----BEGIN CERTIFICATE-----***-----END CERTIFICATE-----\n"

tectonic_metal_matchbox_client_key = "-----BEGIN RSA PRIVATE KEY-----***----END RSA PRIVATE KEY-----\n"

tectonic_metal_matchbox_http_url = "http://matchbox.dev.***.com:8080"

tectonic_metal_matchbox_rpc_endpoint = "matchbox.dev.***.com:8081"

tectonic_worker_count = "2"

tectonic_metal_worker_domains = [
  "node1.cluster.***.com",
  "node2.cluster.***.com"
]

tectonic_metal_worker_macs = [
  "56:00:01:53:11:cd",
  "56:00:01:54:53:50"
]

tectonic_metal_worker_names = [
  "node1",
  "node2"
]

tectonic_service_cidr = "10.3.0.0/16"

tectonic_ssh_authorized_key =  "ssh-rsa ***"

Anything else we need to know?

Somehow I am able to see embedded panels on tectonic dashboard

[ant31]

Hi,

The tectonic-monitoring-auth-grafana svc should be created or recreated automatically if deleted by the tectonic-prometheus-operator. Looks like it failed to create it in your cluster, could provide the log of the tectonic-prometheus-operator pod?

kubectl logs  -n tectonic-system  prometheus-operator-$PODID 

[r-moiseev]

@ant31 There is lot of text but I found one message that was repeated many times

E0110 19:04:04.987832       5 operator.go:577] Sync "tectonic-system/k8s" failed: updating statefulset failed: StatefulSet.apps "prometheus-k8s" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.

[r-moiseev]

Updating to 1.8.4-tectonic.3 fixed the problem

[r-moiseev]

nope it was working during update, but after it was broken again

[robszumski]

Can you share some logs for the pods after the update?

[brianredbeard]

FYI @robszumski this appears to still be a problem - https://github.com/coreos/tectonic-installer/blob/track-1/config.tf#L114-L115

NOTE: The URL displayed in the hyperlink is for the purpose of referencing the branch while the link points at the current (as of 2017-01-19) commit.