terraform-k8s icon indicating copy to clipboard operation
terraform-k8s copied to clipboard

Introduce better logging

Open rverma-dev opened this issue 4 years ago • 2 comments

Following the guide here

Terraform workspaces created but no run configuration.

Exception:

2021-05-29T00:38:51.072Z	INFO	terraform-k8s	Checking outputs	{"Organization": "nslhub", "WorkspaceID": "ws-fkYap2hAd8ogTkxp", "RunID": ""}
2021-05-29T00:38:51.094Z	INFO	terraform-k8s	Updated outputs	{"Organization": "nslhub", "WorkspaceID": "ws-fkYap2hAd8ogTkxp"}
2021-05-29T00:38:51.094Z	INFO	terraform-k8s	Updating secrets	{"name": "greetings-outputs"}
2021-05-29T00:38:51.094Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Workspace","namespace":"ci","name":"greetings","uid":"220cacfe-7f6c-4717-9258-ad02aec80e87","apiVersion":"app.terraform.io/v1alpha1","resourceVersion":"16909471"}, "reason": "WorkspaceEvent", "message": "Updated outputs for run "}
2021-05-29T00:38:51.564Z	INFO	terraform-k8s	Starting module backed run	{"Organization": "nslhub", "Name": "greetings", "Namespace": "ci"}
2021-05-29T00:38:51.775Z	ERROR	controller	Reconciler error	{"controller": "workspace-controller", "name": "greetings", "namespace": "ci", "error": "resource not found"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:246
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

This makes it very tough to understand the issue. Also observed

Status:
  Config Version ID:  
  Run ID:             
  Run Status:         
  Workspace ID:       ws-fkYap2hAd8ogTkxp

rverma-dev avatar May 29 '21 00:05 rverma-dev

The issue starts resurfacing again,

2021-06-12T07:59:02.740Z	ERROR	terraform-k8s	Failed to update output status	{"error": "workspaces.app.terraform.io \"s3-logs-ap-south-1\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
github.com/hashicorp/terraform-k8s/workspacehelper.(*WorkspaceHelper).reconcileWorkspace
	/workspace/workspacehelper/workspace_helper.go:142
github.com/hashicorp/terraform-k8s/workspacehelper.(*WorkspaceHelper).Reconcile
	/workspace/workspacehelper/workspace_helper.go:442
github.com/hashicorp/terraform-k8s/controllers.(*WorkspaceReconciler).Reconcile
	/workspace/controllers/workspace_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

On further debugging its failing here https://github.com/hashicorp/terraform-k8s/blob/973ac76005280a5aff9656fcb39a436dcda481c4/workspacehelper/workspace_helper.go#L141

There is no way to identify the cause of this. It seems it is failing when updating the workspace id in workspace resource.

rverma-dev avatar Jun 12 '21 08:06 rverma-dev

@rverma-nsl Did you ever figure out what was the root cause of your first issue? Facing the same but can't figure out what the issue is...

mottetm avatar Nov 18 '21 15:11 mottetm