Admin not working in standalone mode

[rdavison]

Description:

I set up a standalone deployment using containers by following the instructions at https://gateway.envoyproxy.io/docs/tasks/operations/standalone-deployment-mode/

Then I tried exposing port 19000 to get access to the admin interface, but it doesn't seem to work.

What issue is being seen? Describe what should be happening instead of the bug, for example: The expected value isn't returned, etc.

$ curl --verbose http://0.0.0.0:19000/
*   Trying 0.0.0.0:19000...
* Connected to 0.0.0.0 (0.0.0.0) port 19000
> GET / HTTP/1.1
> Host: 0.0.0.0:19000
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

Repro steps:

Include sample requests, environment, etc. All data and inputs required to reproduce the bug.

# docker-compose.yaml

services:
  envoy-gateway:
    image: envoyproxy/gateway:v1.3.2
    ports:
      - 8888:8888
      - 19000:19000
    volumes:
      - "./volumes/envoy-gateway:/tmp/envoy-gateway"
    command:
      - "server"
      - "--config-path"
      - "/tmp/envoy-gateway/standalone.yaml"

# standalone.conf

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
gateway:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
provider:
  type: Custom
  custom:
    resource:
      type: File
      file:
        paths: ["/tmp/envoy-gateway/config"]
    infrastructure:
      type: Host
      host: {}
logging:
  level:
    default: debug
extensionApis:
  enableBackend: true

Note: If there are privacy concerns, sanitize the data prior to sharing.

Environment:

Include the environment like gateway version, envoy version and so on.

See above

Logs:

Include the access logs and the Envoy logs.

No logs printed

[arkodg]

the port is set to 0, which makes listener port selection dynamic https://github.com/envoyproxy/gateway/blob/c55b814d1326a046e664ba60b1e058c14f21cb40/internal/infrastructure/host/proxy_infra.go#L73 cc @shawnh2

[shawnh2]

yes, you can find its admin address in the output log

[arkodg]

@shawnh2 any reason why it's not fixed ?

[rdavison]

Correct me if I'm wrong, but afaict the problem isn't that the admin mode doesn't work, but that I can't specify a stable port number to it. This isn't really a problem in k8s because you can easily set up port forwarding, but it's not as straightforward if you're trying to launch standalone mode through docker, since you have to know the port up front, unless you want to open a whole range of ports.

[arkodg]

+1 to fixing it to 19000, and making it configurable in a future iteration

[kivra-ahmwar]

I deployed envoy gateway in kubernetes with helm and i the pods don't even listen to any port that points to admin. Im running version 1.3.2. https://gateway.envoyproxy.io/v1.3/troubleshooting/envoy-proxy-admin-interface/ These docs are not correct. These are the only ports the pods listen to:

        ports:
        - containerPort: 18000
          name: grpc
          protocol: TCP
        - containerPort: 18001
          name: ratelimit
          protocol: TCP
        - containerPort: 18002
          name: wasm
          protocol: TCP
        - containerPort: 19001
          name: metrics
          protocol: TCP

[arkodg]

@kivra-ahmwar this issue is for non k8s, in k8s the admin port is exposed only on localhost, refer to https://gateway.envoyproxy.io/docs/troubleshooting/envoy-proxy-admin-interface/ for more info

[kivra-ahmwar]

Previously i was following that guide but it doesn't work when i try to port-forward the deployment. However, i tried port-forwarding the pod itself and that worked much better. Ignore my previous post :D

[shawnh2]

@shawnh2 any reason why it's not fixed ?

If we fix admin port to 19000, the envoyproxy won't start due to cannot bind '127.0.0.1:19000': Address already in use error. The address 127.0.0.1:19000 is occupied by envoy-gateway's admin server (used for pprof).

Cannot fix this port to another number, it will successfully starts one envoyproxy, but if you are trying to start another one, same Address already in use error will also occur.

[arkodg]

ah, thanks for explaining, can we pick a dedicated port for standalone mode - 19004 maybe ? cc @envoyproxy/gateway-reviewers @envoyproxy/gateway-maintainers

[arkodg]

sounds like there 2 issues - CP admin server and DP admin server use the same port and based on the current design are in the same network namespace, so they have to be set to different ports

the other issue is if more proxies are being created in the same network namespace (because a new Gateway is added), then they will stop working, so fixing it to a specific port will block the creation of more DPs

[shawnh2]

Correct me if I'm wrong, but afaict the problem isn't that the admin mode doesn't work, but that I can't specify a stable port number to it. This isn't really a problem in k8s because you can easily set up port forwarding, but it's not as straightforward if you're trying to launch standalone mode through docker, since you have to know the port up front, unless you want to open a whole range of ports.

Thanks @rdavison for sharing this.

And yes @arkodg . So I initially set admin port to a dynamic one, simplest approach.

Due to the case @rdavison described, we may need to reconsider expose a port-range config. For users who launch standalone through docker, we can limit admin port in a specific range.

[arkodg]

this could be solved by adding an API field to edit admin port within EnvoyProxy API so its per Gateway

[codefromthecrypt]

For devex, a known port or zero works best, and is most familiar. Known port allows integration in docker to work, for example having prometheus scrape config. Ack this is not related to k8s, but it is helpful to have a way to configure this vs heuristically scraping files.

A separate setting for range could be made, though I worry if it would be racy, unless there's some new form of bind that does that or some other feature in Envoy that doesn't loop (racy) to find an open port. Love to know the solution if there is one.

[arkodg]

discussed this in the community meeting this week, jotting down the takeaways

• top level port field in Envoy Proxy spec https://gateway.envoyproxy.io/docs/api/extension_types/#envoyproxyspec that allows modifying admin port and other default port values https://gateway.envoyproxy.io/docs/install/install-helm/#envoyproxy in the future, makes sense
• a top level admin field with a sub field called port also makes sense in Envoy Proxy, which may useful for users like @nezdolik who want to add more modifications to the admin listener like a authn/authz policy and expose it on 0.0.0.0 https://github.com/envoyproxy/gateway/issues/3565

both approaches work, marking this with help-wanted