agent fails to communicate with nginx listen sockets with proxy protocol enabled

[defanator]

Bug Overview

Agent is unable to communicate with nginx listeners that are configured with proxy protocol enabled, e.g. in a configuration like this:

server {
    listen [::]:8888 default_server ipv6only=on proxy_protocol;
    listen 8888 default_server proxy_protocol;

    [..]
    location = /stub_status {
        stub_status on;
    }

    location /api {
        api write=on;
    }
    [..]
}

Expected Behavior

Depending on a scenario, the following behavioral alternatives could be expected:

1. If there are other listeners available to reach out to API/status endpoints, use those instead.
2. If there are no other listeners, agent would seed proxy protocol payload while talking to specific sockets e.g. like curl does with --haproxy-protocol option (https://curl.se/docs/manpage.html#--haproxy-protocol).

Steps to Reproduce the Bug

1. Set up nginx or nginx-plus with proxy_protocol listeners (see the above example).
2. Install the agent.
3. Run the agent.
4. Check the nginx error log for entries like these:

2025/02/11 15:23:32 [error] 3881802#3881802: *889720 broken header: "GET /api HTTP/1.1" while reading PROXY protocol, client: ::1, server: [::]:8888
2025/02/11 15:23:32 [error] 3881803#3881803: *889721 broken header: "GET /api HTTP/1.1" while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:8888
2025/02/11 15:23:34 [error] 3881802#3881802: *889722 broken header: "GET /api HTTP/1.1" while reading PROXY protocol, client: ::1, server: [::]:8888
2025/02/11 15:23:34 [error] 3881802#3881802: *889723 broken header: "??/????I?In??Y?$?2??ↅK??? ?j,???p;ƫa?R?K?6Aijܜѝ?L^V?,?0?+?/̨̩??̪?????????$?(?#?'?" while reading PROXY protocol, client: ::1, server: [::]:8888
2025/02/11 15:23:34 [error] 3881803#3881803: *889724 broken header: "GET /api HTTP/1.1" while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:8888
2025/02/11 15:23:34 [error] 3881803#3881803: *889725 broken header: "??T;"%uB????(????#???9??," while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:8888

5. Check agent log for entries like these:

time="2025-02-11T15:23:02Z" level=warning msg="Unable to perform Stub Status API GET request: Get \"http://[::1]:8888/stub_status\": EOF"
time="2025-02-11T15:23:02Z" level=warning msg="Unable to perform Stub Status API GET request: Get \"http://127.0.0.1:8888/stub_status\": EOF"
time="2025-02-11T15:23:02Z" level=warning msg="Unable to perform NGINX Plus API GET request: Get \"http://[::1]:8888/api\": EOF"
time="2025-02-11T15:23:02Z" level=warning msg="Unable to perform NGINX Plus API GET request: Get \"http://127.0.0.1:8888/api\": EOF"

Environment Details

• Target deployment platform: EC2 VM, local VM
• Target OS: Ubuntu 22.04.5 LTS
• Version of this project or specific commit: nginx-agent 2.39.0~jammy
• Version of any relevant project languages: irrelevant
• nginx version: nginx-plus-r33-p2

Additional Context

No response

[nginx-seanmoloney]

Hi @defanator , thank you for creating this issue, I will send this on to the team for investigation.

Sean

[defanator]

For the reference - https://github.com/nginx/nginx-prometheus-exporter/pull/979.

[nginx-seanmoloney]

~Hi @defanator, this issue has been address in Agent v2/v3.~ Apologies, @defanator looking at this again it seems like there is an issue with how we handle the proxy protocol.