operator: logs instance unable to start because "JMES expression is required"

[uhthomas]

I'm using the manifests generated by Grafana as explained in this issue here: https://github.com/grafana/agent/issues/3373. I'll also post a snippet of what they are exactly for fairness.

❯ k -n grafana-agent logs grafana-agent-logs-fnmj4
ts=2023-03-28T15:40:58.366491317Z caller=server.go:191 level=info msg="server listening on addresses" http=[::]:8080 grpc=127.0.0.1:12346 http_tls_enabled=false grpc_tls_enabled=false
ts=2023-03-28T15:40:58.367071063Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config"
ts=2023-03-28T15:40:58.367204621Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set"
ts=2023-03-28T15:40:58.370284368Z caller=promtail.go:123 level=info component=logs logs_config=grafana-agent/grafana-agent msg="Reloading configuration file" md5sum=3ec0b63b32942ba3fca65c219a1ad81a
ts=2023-03-28T15:40:58.37082161Z caller=main.go:72 level=error msg="error creating the agent server entrypoint" err="unable to apply config for grafana-agent/grafana-agent: unable to create logs instance: failed to make file target manager: invalid json stage config: JMES expression is required"

Manifests:

#LogsInstanceList: items: [{
	spec: {
		clients: [{
			externalLabels: cluster: "unwind"
			url: "http://loki-gateway.loki.svc/loki/api/v1/push"
		}]
		podLogsNamespaceSelector: {}
		podLogsSelector: matchLabels: {}
	}
}]

#PodLogsList: items: [{
	metadata: name: "\(#Name)-kubernetes-logs"
	spec: {
		namespaceSelector: any: true
		pipelineStages: [{cri: {}}]
		relabelings: [{
			sourceLabels: ["__meta_kubernetes_pod_node_name"]
			targetLabel: "__host__"
		}, {
			action: "labelmap"
			regex:  "__meta_kubernetes_pod_label_(.+)"
		}, {
			action: "replace"
			sourceLabels: ["__meta_kubernetes_namespace"]
			targetLabel: "namespace"
		}, {
			action: "replace"
			sourceLabels: ["__meta_kubernetes_pod_name"]
			targetLabel: "pod"
		}, {
			action: "replace"
			sourceLabels: ["__meta_kubernetes_container_name"]
			targetLabel: "container"
		}, {
			replacement: "/var/log/pods/*$1/*.log"
			separator:   "/"
			sourceLabels: [
				"__meta_kubernetes_pod_uid",
				"__meta_kubernetes_pod_container_name",
			]
			targetLabel: "__path__"
		}]
		selector: matchLabels: {}
	}
}]

[uhthomas]

Okay, again, took a little while to fully understand what was going on but we got there I believe.

There was an old PodLogs resource in the namespace which had:

spec: pipelineStages: [{json: {}}]

Which is this. The configuration was invalid.

I'm not sure if preventing the daemonset from starting at all is the right thing to do in this situation. As mentioned before, there must be some way to increase visibility for these sorts of issues.