Make configurable Envoy listener socketOptions to allow DSCP marking for QoS purposes

[egerkke]

Hi Ladies and Gents,

DSCP (differentiated services code point) is a mechanism used for classifying/prioritizing network traffic on IP networks DSCP_wiki. Since Contour is used as O&M external interface provider for microservices at our customer, request arrived to support DSCP marking. Therefore all microservice O&M HTTP interface DSCP marking can be done in a unified place.

DiffServ uses 6-bit differentiated services code point (DSCP) in the 8-bit differentiated services field (DS field) in the IP header for packet classification purposes (last 2 bits are reserved - CU). Please see picture with details:

Example Envoy IPv4 listener configuration to set CS2 selection via Envoy Socket Option

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 100.109.96.4, port_value: 10000 }
    socket_options:
    - level: 0           #  0 for IPPROTO_IP
      name: 1            #  1 for IP_TOS
      int_value: 64
      state: STATE_LISTENING

int_value 64 will be converted into 8 bits: 0 1 0 0 0 0 0 0 and used to fill the given IP header section of DS according to the picture. Since the last 2 bits are reserved, this will end up in configuring 0 1 0 0 0 0 as Class Selector that is CS2 according to Class Selector mapping

This Class Selector can also be captured in IP packets received from Envoy:

Captured packets: dscp_CS2_settings.pcapng.tar.gz

Example Envoy IPv6 listener configuration:

        socket_options:   #  this values need to be double checked!
        - level: 41        #  41 for IPPROTO_IPv6
          name: 67         #  67 for IPV6_TCLASS
          int_value: 32 
          state: STATE_LISTENING 

Example to configure both IPv4 and IPv6 with different DSCP values:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: "::", port_value: 10000, ipv4_compat: true }
    socket_options:
    - level: 0
      name: 1
      int_value: 32
      state: STATE_LISTENING
    - level: 41
      name: 67
      int_value: 64
      state: STATE_LISTENING

Reproduction steps (for Ubuntu): For reproduction purposes the following configuration of Envoy can be used: base_envoy_config_socket_options_CS2.txt

Create a kind network:

docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1400 --opt com.docker.network.bridge.name=docker1 --subnet 100.109.100.0/20 kind

Start Envoy container:

docker run -it --network kind --ip 100.109.96.4 --mount type=bind,source=<PLACEOFTHECONFIGURATIONFILE>base_envoy_config_socket_options_CS2.txt,target=/config.yaml envoyproxy/envoy-dev:latest /usr/local/bin/envoy -c config.yaml --service-cluster front-proxy

Copy go, myserver.go.tar.gz (simple echo application) into container, install tcpdump and start myserver.go:

export ENVOY_CONTAINER=$(docker container ls -l -q)
docker cp /usr/local/go $ENVOY_CONTAINER:/usr/local/
docker cp myserver.go $ENVOY_CONTAINER:/tmp
docker exec -it $ENVOY_CONTAINER bash
export PATH=$PATH:/usr/local/go/bin
cd /tmp
apt-get update
apt-get install -y tcpdump
go run myserver.go

Start tcpdump in another shell:

export ENVOY_CONTAINER=$(docker container ls -l -q)
docker exec -it $ENVOY_CONTAINER bash
cd /tmp
tcpdump -i any -w dscp

Create direct entry in /etc/hosts file that targets the Envoy container:

100.109.96.4   primary.com

In another shell execute curl command:

curl primary.com:10000

Results can be copied and analyzed:

export ENVOY_CONTAINER=$(docker container ls -l -q)
docker cp $ENVOY_CONTAINER:/tmp/dscp .

Other notes:

• Request would be to make configurable int_value via Contour for IPv4 and IPv6 use cases as well
• Different Class Selectors shall be able to be configured on the same listener for IPv4 and IPv6
• Configurability for specific Class Selectors (e.g.: only to support CS1, CS2 and CS3) shall not be limited, since the Class Selectors can be changed in the future.
• Configuring int_value 32 ended up in 0 0 1 0 0 0 as DSCP value that verified to be Class Selector 1 by tcpdump file
• IPv6 listener configuration parameters haven't been verified yet. Once this feature request accepted, can be tried out as well.
• TCP KeepAlive is also implemented via socketOption: 2638, might be interesting to have some general function to configure socket options

[youngnick]

Contour currently configures two types of listeners, the basic HTTP one, that does routing based on HTTP header Hostname, and the HTTPS ones, that route based on the SNI of the TLS connection.

Just to be clear, you're looking for Envoy to set the TCP Socket options across all listeners, or do you want this configurable at the per-vhost level? Because of the above structure, we'd only be able to do per-vhost for HTTPS listeners.

The TCP socket options are to be set on the downstream, towards-the-client socket, correct?

We may be able to implement this, but I would like to ask: Is Contour's Envoys the right place to do this? It will set the DiffServ info for the hop between Envoy and whatever downstream, closer-to-the-internet loadbalancer you have, and I'm not sure if cloud load balancers will respect the DiffServ bits. If you're running on a bare-metal loadbalancer that you control, then I understand a bit more.

[egerkke]

Hi @youngnick !

Many thanks for your reply!

Just to be clear, you're looking for Envoy to set the TCP Socket options across all listeners

No. We are not interested in configuring the listeners of admin or metrics interfaces for instance. Just towards clients that send requests to microservices via Contour.

Because of the above structure, we'd only be able to do per-vhost for HTTPS listeners

This "HTTPS" word confuses me a little. Pure HTTP listeners - belonging for vhost defined in HTTPProxy - cannot be updated with these socket options?

The TCP socket options are to be set on the downstream, towards-the-client socket, correct?

Yes

Is Contour's Envoys the right place to do this?

Good point and was studied for a very long time. The conclusion was this is the preferred solution.

[sunjayBhatia]

Because of the above structure, we'd only be able to do per-vhost for HTTPS listeners

This "HTTPS" word confuses me a little. Pure HTTP listeners - belonging for vhost defined in HTTPProxy - cannot be updated with these socket options?

since this is a socket level option and not something that can be configured per filter chain/vhost, and we only have the two ingress listen sockets (one for HTTP and one for HTTPS+TLS passthrough), this is an all or nothing configuration per listen socket

[youngnick]

Oh, of course, thanks @sunjayBhatia.

Just to be clear, you're looking for Envoy to set the TCP Socket options across all listeners

No. We are not interested in configuring the listeners of admin or metrics interfaces for instance. Just towards clients that send requests to microservices via Contour.

I wasn't clear here. What I'm trying to get at is that we need to figure out where we will put this configuration, and since Contour has two listeners, HTTP and HTTPS, the config will most likely need to live in the configuration file or CRD.

The other thing to determine is whether we need to specify different DiffServ details for HTTP and HTTPS. It doesn't seem like this would be useful to me.

What that leaves us with is a configuration setup added into our CRD and configuration file, that will let you configure these parameters, and have them applied to all sockets opened on the Contour Envoys. Does that describe what you want @egerkke?

[egerkke]

Hi Gents,

Sorry for the late response, I was on short vacation. Many thanks for your replies!

The other thing to determine is whether we need to specify different DiffServ details for HTTP and HTTPS. It doesn't seem like this would be useful to me.

I need to check if such use case cannot happen at our customer and will come back.

What that leaves us with is a configuration setup added into our CRD and configuration file, that will let you configure these parameters, and have them applied to all sockets opened on the Contour Envoys. Does that describe what you want @egerkke?

Yes

[youngnick]

Thanks @egerkke, if you could just confirm that you don't need different DiffServ details between HTTP and HTTPS, we'll get this prioritized.

[egerkke]

Hi @youngnick,

It is confirmed that we don't need different DiffServ details between HTTP and HTTPS, as this is not valid use case for us. Many thanks for your support!

BR. Gergo

[github-actions[bot]]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

[github-actions[bot]]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

[egerkke]

Don't close the issue please, we are trying to get resource to implement it.

[hobbytp]

F.Y.I. To support updating socket_options in dual stack scenarios, the envoy proxy community has already implemented several enhancements which include updating socket_option in runtime, supporting multiple addresses for the same listener, supporting socket_options for different multiple addresses under the same listener.

The last PR for listener was just merged: listener: enable socket_options for multiple addresses#24210

[github-actions[bot]]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

[egerkke]

Trying to get help in implementing the feature.

[github-actions[bot]]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

[github-actions[bot]]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack