openshift
USHIFT-4378: introduce ingress customization fields
@eslutsky: This pull request references USHIFT-4378 which is a valid jira issue.
Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.
In response to this:
Which issue(s) this PR addresses:
Closes #<Issue Number>
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.
Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all
![openshift-ci[bot] avatar](https://avatars.githubusercontent.com/in/91280?v=4 "Published by a pub.dev verified publisher"){kind=small}
the default maxConnections should be 50000, not 5000, not sure it is expected. Others look good to me, thanks.
1.
% oc -n openshift-ingress rsh router-default-64785749b4-zz84t
sh-5.1$ env | grep -i conn
ROUTER_MAX_CONNECTIONS=5000
sh-5.1$
sh-5.1$ cat haproxy.config | grep -i maxconn
maxconn 5000
maxconn 5000
sh-5.1$
2. in config.yaml.default, the default of maxConnections is 50000
# maxConnections defines the maximum number of simultaneous
# connections that can be established per HAProxy process.
# Increasing this value allows each ingress controller pod to
# handle more connections but at the cost of additional
# system resources being consumed.
# Permitted values are: empty, 0, -1, and the range
# 2000-2000000.
# If this field is empty or 0, the IngressController will use
# the default value of 50000, but the default is subject to
# change in future releases.
Configure the custom parameters with 2 times of the default values, restart the microshift service(sudo systemctl restart microshift), then check the router-default deployment, haproxy.conf and env under a router pod. All are expected except connectTimeout and reloadInterval
1.
sh-5.1# cat /etc/microshift/config.yaml | grep connectTimeout:
connectTimeout: 10s
sh-5.1#
sh-5.1$ grep "timeout connect" haproxy.config
timeout connect 5s
sh-5.1$
sh-5.1$ env | grep CONN
ROUTER_MAX_CONNECTIONS=100000
sh-5.1$
2.
sh-5.1# cat /etc/microshift/config.yaml | grep reloadInterval:
reloadInterval: 10s
sh-5.1#
% oc -n openshift-ingress get deployment router-default -oyaml | grep -A1 RELOAD_INTERVAL
- name: RELOAD_INTERVAL
value: 5s
sh-5.1$ env | grep RELOAD_INTERVAL
RELOAD_INTERVAL=5s
sh-5.1$
@ShudiLi , connectTimeout and reloadInterval wasnt meant to be exposed as ingress configuration option, we inheritted them because we used IngressControllerTuningOptions from the ingress operator API @pacevedom , it looks like we have to duplicate that struct into our code base.
@eslutsky some timeout parameters weren't updated in the haproxy.config file, thanks.
1.
% oc -n openshift-ingress get deployment -oyaml | grep -A1 TIMEOUT
- name: ROUTER_DEFAULT_CLIENT_TIMEOUT
value: 2m0s
- name: ROUTER_CLIENT_FIN_TIMEOUT
value: 1m0s
- name: ROUTER_DEFAULT_SERVER_TIMEOUT
value: 2m0s
- name: ROUTER_DEFAULT_SERVER_FIN_TIMEOUT
value: 2m0s
- name: ROUTER_DEFAULT_TUNNEL_TIMEOUT
value: 2h0m0s
2. in the router pod
sh-5.1$ env | grep -i timeout
ROUTER_DEFAULT_TUNNEL_TIMEOUT=2h0m0s
ROUTER_DEFAULT_SERVER_TIMEOUT=2m0s
ROUTER_DEFAULT_CLIENT_TIMEOUT=2m0s
ROUTER_DEFAULT_SERVER_FIN_TIMEOUT=2m0s
ROUTER_CLIENT_FIN_TIMEOUT=1m0s
sh-5.1$
3. in the router pod, check the timeout parameters, "timeout tunnel", "timeout server", "timeout client", "timeout server-fin", and "timeout client-fin" were the default values, weren't updated.
sh-5.1$ cat haproxy.config | grep -i "timeout "
stats timeout 2m
timeout connect 5s
timeout client 30s
timeout client-fin 1s
timeout server 30s
timeout server-fin 1s
timeout http-request 10s
timeout http-keep-alive 300s
# Long timeout for WebSocket connections.
timeout tunnel 1h
sh-5.1$
good catch !
looks like timeSpecPattern is not accepting all the metav1.Duration variables like 2m0s,so it has to be a single value ie:
2h0m0s should be just 2h , we will have to use some other validation type. not sure how openshift accept those..
good catch ! looks like timeSpecPattern is not accepting all the metav1.Duration variables like 2m0s,so it has to be a single value ie: 2h0m0s should be just
2h, we will have to use some other validation type. not sure how openshift accept those..
i've checked again against the operator code, and we were missing the durationToHAProxyTimespec function
@eslutsky Thanks for the fixed, I did test today and found httpEmptyRequestsPolicy and logEmptyRequests couldn't be seen in the haproxy.config file. Also for defaultHTTPVersion, there was just default value, there weren't valid values. I tried to update it with 2, the cluster was disconnected.
1. For httpEmptyRequestsPolicy: Ignore, " option http-ignore-probes" should be found in the haproxy.config
sh-5.1$ env | grep -w ROUTER_HTTP_IGNORE_PROBES
ROUTER_HTTP_IGNORE_PROBES=Ignore
sh-5.1$ cat haproxy.config | grep -i "http-ignore-probes"
sh-5.1$
2. For logEmptyRequests: Ignore, " option dontlognul" should be found in the haproxy.config
sh-5.1$ env | grep ROUTER_DONT_LOG_NULL
ROUTER_DONT_LOG_NULL=Ignore
sh-5.1$
sh-5.1$ cat haproxy.config | grep -i dontlognull
sh-5.1$
3. are the valid values available for the defaultHTTPVersion?
ingress:
# Determines default http version should be used for the ingress backends
# By default, using version 1.
defaultHTTPVersion: 1
ROUTER_HTTP_IGNORE_PROBES thanks @ShudiLi it appears that haproxy.config expects "true" in
ROUTER_HTTP_IGNORE_PROBESandROUTER_DONT_LOG_NULLENV, fixed.
@eslutsky: The following commands are available to trigger required jobs:
/test e2e-aws-footprint-and-performance/test e2e-aws-tests/test e2e-aws-tests-arm/test e2e-aws-tests-bootc/test e2e-aws-tests-bootc-arm/test e2e-aws-tests-bootc-periodic/test e2e-aws-tests-bootc-periodic-arm/test e2e-aws-tests-cache/test e2e-aws-tests-cache-arm/test e2e-aws-tests-periodic/test e2e-aws-tests-periodic-arm/test images/test ocp-full-conformance-optional-components-rhel-eus/test ocp-full-conformance-optional-components-rhel-eus-arm/test ocp-full-conformance-rhel-eus/test ocp-full-conformance-rhel-eus-arm/test ocp-full-conformance-serial-optional-components-rhel-eus/test ocp-full-conformance-serial-optional-components-rhel-eus-arm/test ocp-full-conformance-serial-rhel-eus/test ocp-full-conformance-serial-rhel-eus-arm/test test-rpm/test test-unit/test verify
The following commands are available to trigger optional jobs:
/test security/test test-rebase
Use /test all to run the following jobs that were automatically triggered:
pull-ci-openshift-microshift-main-e2e-aws-testspull-ci-openshift-microshift-main-e2e-aws-tests-armpull-ci-openshift-microshift-main-e2e-aws-tests-bootcpull-ci-openshift-microshift-main-e2e-aws-tests-bootc-armpull-ci-openshift-microshift-main-imagespull-ci-openshift-microshift-main-ocp-full-conformance-optional-components-rhel-euspull-ci-openshift-microshift-main-ocp-full-conformance-optional-components-rhel-eus-armpull-ci-openshift-microshift-main-ocp-full-conformance-rhel-euspull-ci-openshift-microshift-main-ocp-full-conformance-rhel-eus-armpull-ci-openshift-microshift-main-ocp-full-conformance-serial-optional-components-rhel-euspull-ci-openshift-microshift-main-ocp-full-conformance-serial-optional-components-rhel-eus-armpull-ci-openshift-microshift-main-ocp-full-conformance-serial-rhel-euspull-ci-openshift-microshift-main-ocp-full-conformance-serial-rhel-eus-armpull-ci-openshift-microshift-main-securitypull-ci-openshift-microshift-main-test-rebasepull-ci-openshift-microshift-main-test-rpmpull-ci-openshift-microshift-main-test-unitpull-ci-openshift-microshift-main-verify
In response to this:
/test ?
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.