agent icon indicating copy to clipboard operation
agent copied to clipboard

Portainer Agent: Unable to retrieve local agent IP address (no swarm)

Open JuezFenix opened this issue 3 years ago • 11 comments

Using version 2.0.0, 1.5.1 or latest same issue here.

2021/01/12 14:31:32 [INFO] [main] [message: Agent running on Docker platform]

2021/01/12 14:31:32 [ERROR] [main,docker] [message: Unable to retrieve local agent IP address] [error: Error: No such container: da585a09ce2a]

2021/01/12 14:31:58 [WARN] [os,options] [message: the CAP_HOST_MANAGEMENT environment variable is deprecated and will likely be removed in a future version of Portainer agent]

Command used

docker run -d -p 9001:9001 --name portainer_agent --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/volumes:/var/lib/docker/volumes portainer/agent

Docker inspect

[

{
    "Id": "50e1c9e4f031a581af7e58c592674e7f01d06c17681766146fb269a0974fd487",
    "Created": "2021-01-12T14:29:39.949019852Z",
    "Path": "./agent",
    "Args": [],
    "State": {
        "Status": "restarting",
        "Running": true,
        "Paused": false,
        "Restarting": true,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 1,
        "Error": "",
        "StartedAt": "2021-01-12T14:41:36.85960428Z",
        "FinishedAt": "2021-01-12T14:41:36.943743551Z"
    },
    "Image": "sha256:5b96aa0902cb3584c223e512b3488e4db66f4e43f642a66ba78ac58047d9df10",
    "ResolvConfPath": "/var/lib/docker/containers/50e1c9e4f031a581af7e58c592674e7f01d06c17681766146fb269a0974fd487/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/50e1c9e4f031a581af7e58c592674e7f01d06c17681766146fb269a0974fd487/hostname",
    "HostsPath": "/var/lib/docker/containers/50e1c9e4f031a581af7e58c592674e7f01d06c17681766146fb269a0974fd487/hosts",
    "LogPath": "/var/lib/docker/containers/50e1c9e4f031a581af7e58c592674e7f01d06c17681766146fb269a0974fd487/50e1c9e4f031a581af7e58c592674e7f01d06c17681766146fb269a0974fd487-json.log",
    "Name": "/portainer_agent",
    "RestartCount": 21,
    "Driver": "overlay2",
    "Platform": "linux",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "docker-default",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": [
            "/var/lib/docker/volumes:/var/lib/docker/volumes",
            "/var/run/docker.sock:/var/run/docker.sock"
        ],
        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "NetworkMode": "bridge",
        "PortBindings": {
            "9001/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "9001"
                }
            ]
        },
        "RestartPolicy": {
            "Name": "always",
            "MaximumRetryCount": 0
        },
        "AutoRemove": false,
        "VolumeDriver": "",
        "VolumesFrom": null,
        "CapAdd": [
            "AUDIT_WRITE",
            "CHOWN",
            "DAC_OVERRIDE",
            "FOWNER",
            "FSETID",
            "KILL",
            "MKNOD",
            "NET_BIND_SERVICE",
            "NET_RAW",
            "SETFCAP",
            "SETGID",
            "SETPCAP",
            "SETUID",
            "SYS_CHROOT"
        ],
        "CapDrop": [
            "AUDIT_CONTROL",
            "BLOCK_SUSPEND",
            "DAC_READ_SEARCH",
            "IPC_LOCK",
            "IPC_OWNER",
            "LEASE",
            "LINUX_IMMUTABLE",
            "MAC_ADMIN",
            "MAC_OVERRIDE",
            "NET_ADMIN",
            "NET_BROADCAST",
            "SYSLOG",
            "SYS_ADMIN",
            "SYS_BOOT",
            "SYS_MODULE",
            "SYS_NICE",
            "SYS_PACCT",
            "SYS_PTRACE",
            "SYS_RAWIO",
            "SYS_RESOURCE",
            "SYS_TIME",
            "SYS_TTY_CONFIG",
            "WAKE_ALARM"
        ],
        "Capabilities": null,
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": [],
        "GroupAdd": null,
        "IpcMode": "private",
        "Cgroup": "",
        "Links": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "SecurityOpt": null,
        "UTSMode": "",
        "UsernsMode": "",
        "ShmSize": 67108864,
        "Runtime": "runc",
        "ConsoleSize": [
            0,
            0
        ],
        "Isolation": "",
        "CpuShares": 0,
        "Memory": 0,
        "NanoCpus": 0,
        "CgroupParent": "",
        "BlkioWeight": 0,
        "BlkioWeightDevice": [],
        "BlkioDeviceReadBps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteIOps": null,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "Devices": [],
        "DeviceCgroupRules": null,
        "DeviceRequests": null,
        "KernelMemory": 0,
        "KernelMemoryTCP": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "OomKillDisable": false,
        "PidsLimit": null,
        "Ulimits": null,
        "CpuCount": 0,
        "CpuPercent": 0,
        "IOMaximumIOps": 0,
        "IOMaximumBandwidth": 0,
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    },
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/312d107ce83e7b6f547466fdbd39937939ba7167069b4624a1f56417aa50aa7c-init/diff:/var/lib/docker/overlay2/b9dfc5270f68580dc1c3812c03c2b135ecce86bb4596c8aa57ebca76d97a85fc/diff:/var/lib/docker/overlay2/1956b7a72aa204510f5e25138e26da23719926180fe60e03c3aa1afa1978d816/diff:/var/lib/docker/overlay2/69469acedf9139119f0b218be231f57272864d3415290b5c24a703916dd7a72b/diff:/var/lib/docker/overlay2/85ffaa4d193a6bb133607eb77bb242d310262b7f6b8de8a7f1214c57ae3db7c9/diff",
            "MergedDir": "/var/lib/docker/overlay2/312d107ce83e7b6f547466fdbd39937939ba7167069b4624a1f56417aa50aa7c/merged",
            "UpperDir": "/var/lib/docker/overlay2/312d107ce83e7b6f547466fdbd39937939ba7167069b4624a1f56417aa50aa7c/diff",
            "WorkDir": "/var/lib/docker/overlay2/312d107ce83e7b6f547466fdbd39937939ba7167069b4624a1f56417aa50aa7c/work"
        },
        "Name": "overlay2"
    },
    "Mounts": [
        {
            "Type": "bind",
            "Source": "/var/lib/docker/volumes",
            "Destination": "/var/lib/docker/volumes",
            "Mode": "",
            "RW": true,
            "Propagation": "rslave"
        },
        {
            "Type": "bind",
            "Source": "/var/run/docker.sock",
            "Destination": "/var/run/docker.sock",
            "Mode": "",
            "RW": true,
            "Propagation": "rprivate"
        }
    ],
    "Config": {
        "Hostname": "da585a09ce2a",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "9001/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "CAP_HOST_MANAGEMENT=1"
        ],
        "Cmd": null,
        "Image": "portainer/agent:1.5.1",
        "Volumes": {
            "/var/lib/docker/volumes": {},
            "/var/run/docker.sock": {}
        },
        "WorkingDir": "/app",
        "Entrypoint": [
            "./agent"
        ],
        "OnBuild": null,
        "Labels": {}
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "eef7bbe92f4f236145bd6db34d736948aa68db67a3e7f7887f45daf230c5fcdb",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {},
        "SandboxKey": "/var/run/docker/netns/eef7bbe92f4f",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "bridge": {
                "IPAMConfig": {},
                "Links": null,
                "Aliases": null,
                "NetworkID": "0cde22fda2bd5b7f3f6063e219a87587d501a2d335516ace59f188a286f9bcfb",
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "",
                "DriverOpts": null
            }
        }
    }
}

]

JuezFenix avatar Jan 12 '21 14:01 JuezFenix

@JuezFenix do you mind running the agent with debug enabled and share the logs? Add the -e LOG_LEVEL=DEBUG flag when running the agent as a container.

deviantony avatar Jan 12 '21 19:01 deviantony

Updating this, I have noticed that this error only occurs when the container is edited or updated, but not when it is created from 0.

JuezFenix avatar Feb 22 '21 15:02 JuezFenix

I regularly have the same problem. Apparently it always occurs when containrrr/watchtower has initiated an update of the portainer container. I will try it again with -e LOG_LEVEL=DEBUG enabled.

EDIT: I just saw that I still have userns_mode: "host" turned on. I'll try again without...

svengo avatar Jul 14 '21 10:07 svengo

@svengo having the same issue with containrrr/watchtower

2021/07/16 15:52:30 [INFO] [main] [message: Agent running on Docker platform]
2021/07/16 15:52:30 [ERROR] [main,docker] [message: Unable to retrieve local agent IP address] [error: Error: No such container: eed324ab7823]
root@apollo:/usr/local/etc/portainer# cat DOCKER-README-agent.txt.sh
#!/bin/bash
docker run -d \
        -p 9001:9001 \
        --network=portainer \
        --name=portainer-agent \
        --restart=unless-stopped \
        -e LOG_LEVEL=DEBUG \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v /usr/local/etc/portainer/data:/data \
        portainer/agent

root@apollo:/usr/local/etc/portainer# cat DOCKER-README.txt.sh
#!/bin/bash
docker run -d \
        -p 9002:9000 \
        --name=portainer-apollo \
        --restart=unless-stopped \
        --network=portainer \
        --dns=192.168.178.90 \
        --dns=192.168.178.50 \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v /usr/local/etc/portainer/data:/data \
        portainer/portainer-ce

I also get since last release:

root@apollo:/usr/local/etc/portainer# docker logs -f portainer-agent
2021/07/23 08:27:18 [INFO] [main] [message: Agent running on Docker platform]
2021/07/23 08:27:18 [DEBUG] [main,configuration] [Member tags: &{AgentPort:9001 EdgeKeySet:false NodeName:apollo DockerConfiguration:{EngineStatus:1 Leader:false NodeRole:0} KubernetesConfiguration:{}}]
2021/07/23 08:27:18 [DEBUG] [docker] [ip_address: 172.19.0.3] [network_name: portainer] [message: Retrieving IP address from container network]
2021/07/23 08:27:19 [INFO] [http] [server_addr: 0.0.0.0] [server_port: 9001] [secured: true] [api_version: 2.6.1] [message: Starting Agent API server]
2021/07/23 08:27:20 http: TLS handshake error from 192.168.178.100:1232: EOF
2021/07/23 08:27:35 http: TLS handshake error from 192.168.178.100:1410: EOF
2021/07/23 08:27:50 http: TLS handshake error from 192.168.178.100:1534: EOF
2021/07/23 08:28:05 http: TLS handshake error from 192.168.178.100:1692: EOF
2021/07/23 08:28:20 http: TLS handshake error from 192.168.178.100:1840: EOF

last lines repeating.

jobenvil avatar Jul 23 '21 09:07 jobenvil

@jobenvil The problem does not occur anymore after disabling userns_mode: "host".

svengo avatar Jul 24 '21 14:07 svengo

@svengo Thanks for pinging back. Not sure why do you assume that I have userns_mode: "host" enabled 🤷. I just did run the two commands above.

jobenvil avatar Jul 24 '21 17:07 jobenvil

i have the same issue as its tracked here: https://github.com/portainer/agent/issues/203 . I don not use userns_mode: "host". But i also use watchtower.

Trufax avatar Aug 07 '21 17:08 Trufax

Wondering if this has been fixed? Just noticed I've been getting same problem with my agent container, also updated via watchtower

den7007 avatar Feb 09 '22 05:02 den7007

Setting the following in the compose file fixed the issue for me:


hostname: portaineragent
container_name: portaineragent

Trufax avatar Feb 09 '22 13:02 Trufax

Setting the following in the compose file fixed the issue for me:


hostname: portaineragent
container_name: portaineragent

Thank you, this worked for me!

den7007 avatar Feb 09 '22 14:02 den7007

Seems like adding hostname: portaineragent really fixed this issue for me (had container_name already set). Really weird solution, maybe this is a bug in watchtower?

MCWertGaming avatar Mar 09 '22 08:03 MCWertGaming