Intermittent 404 on v2.2.8

[beranPro]

Hi guys...

Do you want to request a feature or report a bug?

Bug

What did you do?

i'm trying to passthrough a https tls connection through the keycloak container via TCP. I'm doing because i want to use client authentication via x509 certificates and i didn't get it up and running via the https router.

What did you expect to see?

i stable working connection

What did you see instead?

I see a "part time" working connection. The first requests works fine but after some request i get a 404 in the browser. Curl is working fine.

It's the same issue like https://github.com/containous/traefik/issues/7020

The only difference is that i'm using the tcp router.

Output of traefik version: (What version of Traefik are you using?)

/ # traefik version
Version:      2.2.8
Codename:     chevrotin
Go version:   go1.14.6
Built:        2020-07-28T15:46:03Z
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

traefik.yml

log:
  level: DEBUG

serversTransport:
  insecureSkipVerify: true

#accessLog: {}

api:
  dashboard: true

providers:
  docker:
    swarmMode: true
    exposedByDefault: false
    swarmModeRefreshSeconds: 30
    network: web
  file:
    directory: /dynamic/
    watch: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

docker-compose.yml

version: '3.8'

services:
  keycloak:
    image: jboss/keycloak
    ports:
      - target: 8443
        published: 8443
        protocol: tcp
        mode: ingress
    volumes:
      - ./https:/etc/x509/https
      - ./tls:/etc/x509/tls
    environment:
      - KEYCLOAK_LOGLEVEL=DEBUG
    deploy:
      replicas: 1
      placement:
        constraints:
          - "node.hostname==host"
      labels:
        - "traefik.enable=true"
        - "traefik.tcp.routers.tcp_keycloak.entrypoints=websecure"
        - "traefik.tcp.routers.tcp_keycloak.tls.passthrough=true"
        - "traefik.tcp.routers.tcp_keycloak.rule=HostSNI(`auth.mydomain.de`)"
        - "traefik.tcp.services.tcp_keycloak.loadbalancer.server.port=8443"
    networks:
      - web
    command: ["-b", "0.0.0.0", "-Dkeycloak.profile.feature.docker=enabled", "-Dkeycloak.profile.feature.upload_scripts=enabled"]

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

sometime i find some rare error debug messages

rocket_traefik.1.oxpixxbainkn@srvedv128    | time="2020-08-03T07:10:12Z" level=debug msg="Error while setting deadline: set tcp 10.222.1.58:43522: use of closed network connection"
rocket_traefik.1.oxpixxbainkn@srvedv128    | time="2020-08-03T07:10:37Z" level=debug msg="Error while setting deadline: set tcp 10.222.1.58:43784: use of closed network connection"
rocket_traefik.1.oxpixxbainkn@srvedv128    | time="2020-08-03T07:11:40Z" level=debug msg="Error while setting deadline: set tcp 10.222.1.58:44280: use of closed network connection"

[jawabuu]

Hey @beranPro Can you check if the issue disappears when;

1. You use curl (Just saw you already did this.)
2. You use Firefox
3. Use Browser with http/1 chrome --disable-http2

[david-forster10]

We're experiencing the same issue when using Traefik 2.9.9.

When using curl and a browser with http/2 disabled it becomes a reliable connection and traefik routes as expected. When re-enabling http/2 on the browser, the intermittent 404s return.

[everyx]

I also encountered the same problem. I used terminating TLS in Varnish with traefik. Referring to https://github.com/traefik/traefik/issues/9983, I have stopped using wildcard certificates, but this problem still occasionally occurs when making HTTP/2 requests, and I tested traefik 3.0 and the problem still exists.

[nmengin]

Hey @everyx ,

Thanks for reaching out.

Could you provide a minimal reproducible case (for instance, full Kubernetes manifest to reproduce the issue)? In the meantime, if any community member can help us find verified steps to reproduce and fix the issue if possible, we would love the help.

[everyx]

Hey @everyx ,

Thanks for reaching out.

Could you provide a minimal reproducible case (for instance, full Kubernetes manifest to reproduce the issue)? In the meantime, if any community member can help us find verified steps to reproduce and fix the issue if possible, we would love the help.

@nmengin This seems to be random, and I can't give steps that will definitely reproduce it, so I can't provide a minimal reproducible case. When I have this kind of problem, it seems to be after I update the service, Maybe traefik keep the previous TCP connection for HTTP/2, but the background service was shut down?