"Unable to get pod logs: [object Object]" when trying to view logs for pods

[hookenz]

Before you start please confirm the following.

• [x] Yes, I've searched similar issues on GitHub.
• [x] Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

Went to look at the Portainer Agent Pod

Unable to get logs. Unable to get pod logs: [object Object]

I tried another pod

Unable to get logs. Unable to get pod logs: [object Object]

It's broken.

Expected Behavior

I should have some logs

Actual Behavior

As above

Steps to Reproduce

Install Portainer and add kubernetes via the agent.

Portainer logs or screenshots

This in the browser:

Looks to be something to do with certificate verification. That wasn't an issue on older versions.

For example, looking at calico-cube-controller

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Get \"https://10.0.1.1:10250/containerLogs/kube-system/calico-kube-controllers-759cd8b574-hghzf/calico-kube-controllers?tailLines=1000\": tls: failed to verify certificate: x509: certificate is valid for 192.168.167.214, 172.17.0.1, 172.18.0.1, not 10.0.1.1",
  "code": 500
}

Or looking at the log from nginx process running in my default namespace

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Get \"https://10.0.1.1:10250/containerLogs/default/nginx-64b645f47-5wzw8/nginx?tailLines=1000\": tls: failed to verify certificate: x509: certificate is valid for 192.168.167.214, 172.17.0.1, 172.18.0.1, not 10.0.1.1",
  "code": 500
}

Portainer version

2.28.1

Portainer Edition

Business Edition (BE/EE) with 5NF / 3NF license

Platform and Version

Kubernetes

OS and Architecture

Ubuntu 22.04

Browser

Firefox

What command did you use to deploy Portainer?

Portainer under docker with microk8s installed alongside and portainer agent

Additional Information

No response

[hookenz]

Thanks @samdulam

edited: /var/snap/microk8s/current/certs/csr.conf.template

Added 10.0.1.1 as an IP.

Ran sudo microk8s refresh-certs -e ca.crt sudo microk8s refresh-certs -e server.crt sudo microk8s refresh-certs -e front-proxy-client.crt

Restarted microk8s.

Problem persists.

[hookenz]

@samdulam I've fixed it.

So when you see this error:

> kubectl logs pod/serverkit-0
Error from server: Get "https://10.0.1.1:10250/containerLogs/default/serverkit-0/serverkit": tls: failed to verify certificate: x509: certificate is valid for 192.168.167.214, 172.20.0.1, 172.19.0.1, 172.23.0.1, 172.22.0.1, 172.21.0.1, 172.18.0.1, 172.24.0.1, 172.17.0.1, not 10.0.1.1

The error is caused by the host-access addon. https://microk8s.io/docs/addon-host-access

Here are the options.

1. Disable it if not required. sudo microk8s disable host-access
2. Alter the kubelet and add the node IP to the kubelet options. Specifically in this case --node-ip=192.168.167.214

Note: when host-access was enabled I tried --node-ip=10.0.1.1 but for me it didn't function correctly? don't know why. Maybe I didn't have all the required ducks in a row.

You might need to restart microk8s for the first option, definitely the second option requires a restart.

Either of those solutions should fix it.

There is no need to update the csr.conf.template file with the 10.0.1.1 IP. In fact that seemed to stop it working.
I had to remove my prior entry for some unknown reason.

Once all this is done both the logs and console access was restored.

@jamescarppe - Might be useful to add to your microk8s knowledge base.

https://portal.portainer.io/knowledge/microk8s-known-issues