envoy icon indicating copy to clipboard operation
envoy copied to clipboard
verified publisher icon envoyproxy

Error when setting up Envoy as a forward proxy.

Open ahcognmm opened this issue 1 year ago • 3 comments

Error when setting up Envoy as a forward proxy.

Hi there, i want to use envoy as a proxy to handle all out going traffic. This is my set up:

Envoy config:

admin:
  address:
    socket_address:
      protocol: TCP
      address: 127.0.0.1
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    listener_filters:
    - name: envoy.filters.listener.tls_inspector
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /dev/stdout
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: dynamic_forward_proxy_cluster
          http_filters:
          - name: envoy.filters.http.dynamic_forward_proxy
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
              dns_cache_config:
                name: dynamic_forward_proxy_cache_config
                dns_lookup_family: V4_ONLY
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: dynamic_forward_proxy_cluster
    lb_policy: CLUSTER_PROVIDED
    connect_timeout: 5s
    http2_protocol_options: {}
    cluster_type:
      name: envoy.clusters.dynamic_forward_proxy
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
        dns_cache_config:
          name: dynamic_forward_proxy_cache_config
          dns_lookup_family: V4_ONLY
        allow_insecure_cluster_options: true
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          tls_params:
            tls_minimum_protocol_version: TLSv1_2
            tls_maximum_protocol_version: TLSv1_3
          alpn_protocols:
            - h2
            - http/1.1
          validation_context:
            trusted_ca: {filename: /etc/ssl/certs/ca-certificates.crt}

And my iptables run:

iptables -t nat -N PROXY_INIT_OUTPUT
iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN
iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port 10000
iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT

When I try something like curl google.com it returns this:

[2024-05-17 08:16:56.274][17][trace][connection] [source/common/network/connection_impl.cc:474] [Tags: "ConnectionId":"0"] raising connection event 2
[2024-05-17 08:16:56.274][17][trace][connection] [source/common/network/connection_impl.cc:619] [Tags: "ConnectionId":"0"] socket event: 3
[2024-05-17 08:16:56.274][17][trace][connection] [source/common/network/connection_impl.cc:742] [Tags: "ConnectionId":"0"] write ready
[2024-05-17 08:16:56.274][17][trace][connection] [source/common/network/connection_impl.cc:659] [Tags: "ConnectionId":"0"] read ready. dispatch_buffered_data=0
[2024-05-17 08:16:56.274][17][trace][connection] [source/common/network/raw_buffer_socket.cc:25] [Tags: "ConnectionId":"0"] read returns: 73
[2024-05-17 08:16:56.274][17][trace][connection] [source/common/network/raw_buffer_socket.cc:39] [Tags: "ConnectionId":"0"] read error: Resource temporarily unavailable, code: 0
[2024-05-17 08:16:56.274][17][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"0"] current connecting state: false
[2024-05-17 08:16:56.275][1][debug][upstream] [source/extensions/clusters/dynamic_forward_proxy/cluster.cc:300] Adding host info for google.com:443
[2024-05-17 08:16:56.275][1][debug][upstream] [source/extensions/clusters/dynamic_forward_proxy/cluster.cc:279] adding new dfproxy cluster host 'google.com:443'
[2024-05-17 08:16:56.275][1][debug][upstream] [source/common/upstream/upstream_impl.cc:458] transport socket match, socket default selected for host with address 142.251.175.138:443
[2024-05-17 08:16:56.275][10][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][11][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][17][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][13][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][11][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][17][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][13][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][10][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][15][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][22][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][15][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][25][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][1][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][1][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][25][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][18][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][28][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][18][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][28][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][32][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][32][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][24][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][22][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][24][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][29][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster dynamic_forward_proxy_cluster added 1 removed 0
[2024-05-17 08:16:56.275][29][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1548] re-creating local LB for TLS cluster dynamic_forward_proxy_cluster
[2024-05-17 08:16:56.275][17][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"2"] current connecting state: true
[2024-05-17 08:16:56.275][17][debug][connection] [source/common/network/connection_impl.cc:1021] [Tags: "ConnectionId":"2"] connecting to 142.251.175.138:443
[2024-05-17 08:16:56.275][17][debug][connection] [source/common/network/connection_impl.cc:1040] [Tags: "ConnectionId":"2"] connection in progress
[2024-05-17 08:16:56.276][17][trace][connection] [source/common/network/connection_impl.cc:619] [Tags: "ConnectionId":"2"] socket event: 2
[2024-05-17 08:16:56.276][17][trace][connection] [source/common/network/connection_impl.cc:742] [Tags: "ConnectionId":"2"] write ready
[2024-05-17 08:16:56.276][17][debug][connection] [source/common/network/connection_impl.cc:751] [Tags: "ConnectionId":"2"] connected
[2024-05-17 08:16:56.276][17][trace][connection] [source/common/tls/ssl_handshaker.cc:93] [Tags: "ConnectionId":"2"] ssl error occurred while read: WANT_READ
[2024-05-17 08:16:56.276][15][trace][connection] [source/common/network/connection_impl.cc:474] [Tags: "ConnectionId":"3"] raising connection event 2
[2024-05-17 08:16:56.276][15][trace][connection] [source/common/network/connection_impl.cc:619] [Tags: "ConnectionId":"3"] socket event: 3
[2024-05-17 08:16:56.276][15][trace][connection] [source/common/network/connection_impl.cc:742] [Tags: "ConnectionId":"3"] write ready
[2024-05-17 08:16:56.276][15][trace][connection] [source/common/network/connection_impl.cc:659] [Tags: "ConnectionId":"3"] read ready. dispatch_buffered_data=0
[2024-05-17 08:16:56.276][15][trace][connection] [source/common/network/raw_buffer_socket.cc:25] [Tags: "ConnectionId":"3"] read returns: 247
[2024-05-17 08:16:56.276][15][trace][connection] [source/common/network/raw_buffer_socket.cc:39] [Tags: "ConnectionId":"3"] read error: Resource temporarily unavailable, code: 0
[2024-05-17 08:16:56.276][15][trace][connection] [source/common/network/connection_impl.cc:534] [Tags: "ConnectionId":"3"] writing 145 bytes, end_stream false
[2024-05-17 08:16:56.276][15][trace][connection] [source/common/network/connection_impl.cc:534] [Tags: "ConnectionId":"3"] writing 11 bytes, end_stream false
[2024-05-17T08:16:56.276Z] "- - HTTP/1.1" 400 DPE 0 11 0 - "-" "-" "-" "-" "-"

I'm just a newcomer with 1 week of reading documents. Can any one point out why I got this errors and how to fix it. I have already tried to google but nothing can help.

ahcognmm avatar May 17 '24 09:05 ahcognmm

@phlax might be able to help, or to ping someone who is.

ravenblackx avatar May 17 '24 14:05 ravenblackx

the problem/solution is not immediately obvious to me - but i have limited dfp knowledge

cc @alyssawilk @mattklein123 as codeowners

@wbpcode might also have some idea

phlax avatar May 17 '24 14:05 phlax

Not sure about your iptables stuff but here is a simplified config that I've been using for ages with all the custom access log and tracing stuff elided. I update my git, rust apt configs etc to leverage localhost:9904 as a proxy and it works well. You can export HTTPS_PROXY to point to this for adhoc. I never worked out how to have this work with HTTP/3

admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 9903
static_resources:
  clusters:
  - cluster_type:
      name: envoy.clusters.dynamic_forward_proxy
      typed_config:
        '@type': type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
        allow_coalesced_connections: true
        dns_cache_config:
          dns_lookup_family: ALL
          name: dynamic_forward_proxy_cache_config
    connect_timeout: 2s
    dns_lookup_family: ALL
    lb_policy: CLUSTER_PROVIDED
    name: dynamic_forward_proxy_cluster
  listeners:
  - additional_addresses:
    - address:
        socket_address:
          address: ::1
          port_value: 9904
    address:
      socket_address:
        address: 127.0.0.1
        port_value: 9904
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: AUTO
          http2_protocol_options:
            allow_connect: true
          http_filters:
          - name: envoy.filters.http.dynamic_forward_proxy
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
              dns_cache_config:
                dns_lookup_family: ALL
                name: dynamic_forward_proxy_cache_config
          - name: envoy.filters.http.router
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - domains:
              - '*'
              name: local_service
              routes:
              - match:
                  prefix: /
                route:
                  cluster: dynamic_forward_proxy_cluster
              - match:
                  connect_matcher: {}
                route:
                  cluster: dynamic_forward_proxy_cluster
                  upgrade_configs:
                  - connect_config: {}
                    upgrade_type: CONNECT
          stat_prefix: dynamic_forward_proxy_upgrade
    name: dynamic_forward_proxy_upgrade
    traffic_direction: OUTBOUND

moderation avatar May 17 '24 20:05 moderation

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jun 17 '24 00:06 github-actions[bot]

Not sure about your iptables stuff but here is a simplified config that I've been using for ages with all the custom access log and tracing stuff elided. I update my git, rust apt configs etc to leverage localhost:9904 as a proxy and it works well. You can export HTTPS_PROXY to point to this for adhoc. I never worked out how to have this work with HTTP/3

Sorry for late reply. But it doesn't work for me. I want config envoy as a transparent proxy, which handle all routed traffic via iptables . I dont want manually config like curl -x localhost:9904 google.com , i want curl google.com still going through proxy.

ahcognmm avatar Jun 18 '24 03:06 ahcognmm

@wbpcode do you have any ideas?

ahcognmm avatar Jul 04 '24 08:07 ahcognmm

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Aug 03 '24 12:08 github-actions[bot]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

github-actions[bot] avatar Aug 10 '24 12:08 github-actions[bot]