k8s-bigip-ctlr
k8s-bigip-ctlr copied to clipboard
Published
20 hours ago •
F5Networks
F5Networks
Invalid iRule generation for HTTP/2 full proxy mode
Setup Details
CIS Version : 2.16.1
Build: f5networks/k8s-bigip-ctlr:2.16.1
BIGIP Version: BIG-IP 17.1.0.3 Build 0.75.4 Engineering Hotfix
AS3 Version: 3.x
Agent Mode: AS3
Orchestration: K8S
Orchestration Version: v1.28.7
Pool Mode: Nodeport
Additional Setup details: Cilium CNI
Description
When generating the irule for TLS virtual servers in the getTLSIRule function in routing.go, the generated iRule seems to be invalid and generates this log in BIG-IP.
TCL error: /k8s-dev/Shared/payroll_gateway_443_tls_irule <SERVER_CONNECTED> - can't read "sslprofile": no such variable while executing "if { not ($sslprofile equals "false") } { SSL::profile $reen }"
Here are the used configurations:
Virtual Server:
apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
annotations:
externaldns: enabled
meta.helm.sh/release-name: gateway
meta.helm.sh/release-namespace: payroll
objectset.rio.cattle.io/id: default-payroll-gitops-gateway
labels:
app.kubernetes.io/managed-by: Helm
f5cr: 'true'
objectset.rio.cattle.io/hash: a67b146037fca2e49ddc407b58b5821983ed7d48
name: gateway
namespace: payroll
spec:
host: payroll.dev.krd
httpMrfRoutingEnabled: true
ipamLabel: default
policyName: gateway
pools:
- path: /
service: traefik
serviceNamespace: traefik
servicePort: 443
tlsProfileName: payroll-tls
virtualServerName: payroll-gateway
TLS Profile:
apiVersion: cis.f5.com/v1
kind: TLSProfile
metadata:
annotations:
meta.helm.sh/release-name: gateway
meta.helm.sh/release-namespace: payroll
objectset.rio.cattle.io/id: default-payroll-gitops-gateway
labels:
app.kubernetes.io/managed-by: Helm
f5cr: 'true'
objectset.rio.cattle.io/hash: a67b146037fca2e49ddc407b58b5821983ed7d48
name: payroll-tls
namespace: payroll
spec:
hosts:
- payroll.dev.krd
tls:
clientSSL: payroll-tls-ssl
clientSSLParams:
profileReference: secret
renegotiationEnabled: false
reference: hybrid
serverSSL: /Common/serverssl
serverSSLParams:
profileReference: bigip
renegotiationEnabled: false
termination: reencrypt
Policy:
apiVersion: cis.f5.com/v1
kind: Policy
metadata:
annotations:
meta.helm.sh/release-name: gateway
meta.helm.sh/release-namespace: payroll
objectset.rio.cattle.io/id: default-payroll-gitops-gateway
labels:
app.kubernetes.io/managed-by: Helm
f5cr: 'true'
objectset.rio.cattle.io/hash: a67b146037fca2e49ddc407b58b5821983ed7d48
name: gateway
namespace: payroll
spec:
profiles:
http: /Common/http-k8s
http2:
client: /Common/http2
Generated iRule:
when CLIENT_ACCEPTED { TCP::collect }
proc select_ab_pool {path default_pool domainpath} {
set last_slash [string length $path]
set ab_class "/k8s-dev/Shared/payroll_gateway_443_ab_deployment_dg"
while {$last_slash >= 0} {
if {[class match $path equals $ab_class]} then {
break
}
set last_slash [string last "/" $path $last_slash]
incr last_slash -1
set path [string range $path 0 $last_slash]
}
if {$last_slash >= 0} {
set ab_rule [class match -value $path equals $ab_class]
if {$ab_rule != ""} then {
set weight_selection [expr {rand()}]
set service_rules [split $ab_rule ";"]
set active_pool ""
foreach service_rule $service_rules {
set fields [split $service_rule ","]
set pool_name [lindex $fields 0]
if { [active_members $pool_name] >= 1 } {
set active_pool $pool_name
}
set weight [expr {double([lindex $fields 1])}]
if {$weight_selection <= $weight} then {
#check if active pool members are available
if { [active_members $pool_name] >= 1 } {
return $pool_name
} else {
# select the any of pool with active members
if {$active_pool!= ""} then {
return $active_pool
}
}
}
}
}
# If we had a match, but all weights were 0 then
# retrun a 503 (Service Unavailable)
HTTP::respond 503
}
return $default_pool
}
when CLIENT_DATA {
# Byte 0 is the content type.
# Bytes 1-2 are the TLS version.
# Bytes 3-4 are the TLS payload length.
# Bytes 5-$tls_payload_len are the TLS payload.
binary scan [TCP::payload] cSS tls_content_type tls_version tls_payload_len
if { ! [ expr { [info exists tls_content_type] && [string is integer -strict $tls_content_type] } ] } { reject ; event disable all; return; }
if { ! [ expr { [info exists tls_version] && [string is integer -strict $tls_version] } ] } { reject ; event disable all; return; }
switch -exact $tls_version {
"769" -
"770" -
"771" {
# Content type of 22 indicates the TLS payload contains a handshake.
if { $tls_content_type == 22 } {
# Byte 5 (the first byte of the handshake) indicates the handshake
# record type, and a value of 1 signifies that the handshake record is
# a ClientHello.
binary scan [TCP::payload] @5c tls_handshake_record_type
if { ! [ expr { [info exists tls_handshake_record_type] && [string is integer -strict $tls_handshake_record_type] } ] } { reject ; event disable all; return; }
if { $tls_handshake_record_type == 1 } {
# Bytes 6-8 are the handshake length (which we ignore).
# Bytes 9-10 are the TLS version (which we ignore).
# Bytes 11-42 are random data (which we ignore).
# Byte 43 is the session ID length. Following this are three
# variable-length fields which we shall skip over.
set record_offset 43
# Skip the session ID.
binary scan [TCP::payload] @${record_offset}c tls_session_id_len
if { ! [ expr { [info exists tls_session_id_len] && [string is integer -strict $tls_session_id_len] } ] } { reject ; event disable all; return; }
incr record_offset [expr {1 + $tls_session_id_len}]
# Skip the cipher_suites field.
binary scan [TCP::payload] @${record_offset}S tls_cipher_suites_len
if { ! [ expr { [info exists tls_cipher_suites_len] && [string is integer -strict $tls_cipher_suites_len] } ] } { reject ; event disable all; return; }
incr record_offset [expr {2 + $tls_cipher_suites_len}]
# Skip the compression_methods field.
binary scan [TCP::payload] @${record_offset}c tls_compression_methods_len
if { ! [ expr { [info exists tls_compression_methods_len] && [string is integer -strict $tls_compression_methods_len] } ] } { reject ; event disable all; return; }
incr record_offset [expr {1 + $tls_compression_methods_len}]
# Get the number of extensions, and store the extensions.
binary scan [TCP::payload] @${record_offset}S tls_extensions_len
if { ! [ expr { [info exists tls_extensions_len] && [string is integer -strict $tls_extensions_len] } ] } { reject ; event disable all; return; }
incr record_offset 2
binary scan [TCP::payload] @${record_offset}a* tls_extensions
if { ! [info exists tls_extensions] } { reject ; event disable all; return; }
for { set extension_start 0 }
{ $tls_extensions_len - $extension_start == abs($tls_extensions_len - $extension_start) }
{ incr extension_start 4 } {
# Bytes 0-1 of the extension are the extension type.
# Bytes 2-3 of the extension are the extension length.
binary scan $tls_extensions @${extension_start}SS extension_type extension_len
if { ! [ expr { [info exists extension_type] && [string is integer -strict $extension_type] } ] } { reject ; event disable all; return; }
if { ! [ expr { [info exists extension_len] && [string is integer -strict $extension_len] } ] } { reject ; event disable all; return; }
# Extension type 00 is the ServerName extension.
if { $extension_type == "00" } {
# Bytes 4-5 of the extension are the SNI length (we ignore this).
# Byte 6 of the extension is the SNI type.
set sni_type_offset [expr {$extension_start + 6}]
binary scan $tls_extensions @${sni_type_offset}S sni_type
if { ! [ expr { [info exists sni_type] && [string is integer -strict $sni_type] } ] } { reject ; event disable all; return; }
# Type 0 is host_name.
if { $sni_type == "0" } {
# Bytes 7-8 of the extension are the SNI data (host_name)
# length.
set sni_len_offset [expr {$extension_start + 7}]
binary scan $tls_extensions @${sni_len_offset}S sni_len
if { ! [ expr { [info exists sni_len] && [string is integer -strict $sni_len] } ] } { reject ; event disable all; return; }
# Bytes 9-$sni_len are the SNI data (host_name).
set sni_start [expr {$extension_start + 9}]
binary scan $tls_extensions @${sni_start}A${sni_len} tls_servername
}
}
incr extension_start $extension_len
}
if { [info exists tls_servername] } {
set servername_lower [string tolower $tls_servername]
set domain_length [llength [split $servername_lower "."]]
set domain_wc [domain $servername_lower [expr {$domain_length - 1}] ]
# Set wc_host with the wildcard domain
set wc_host ".$domain_wc"
set passthru_class "/k8s-dev/Shared/payroll_gateway_443_ssl_passthrough_servername_dg"
if { [class exists $passthru_class] } {
# check if the passthrough data group has a record with the servername
set passthru_dg_key [class match $servername_lower equals $passthru_class]
set passthru_dg_wc_key [class match $wc_host equals $passthru_class]
if { $passthru_dg_key != 0 || $passthru_dg_wc_key != 0 } {
SSL::disable serverside
set dflt_pool_passthrough ""
# Disable Serverside SSL for Passthrough Class
set dflt_pool_passthrough [class match -value $servername_lower equals $passthru_class]
# If no match, try wildcard domain
if { $dflt_pool_passthrough == "" } {
if { [class match $wc_host equals $passthru_class] } {
set dflt_pool_passthrough [class match -value $wc_host equals $passthru_class]
}
}
if { not ($dflt_pool_passthrough equals "") } {
SSL::disable
HTTP::disable
}
set ab_class "/k8s-dev/Shared/payroll_gateway_443_ab_deployment_dg"
if { not [class exists $ab_class] } {
if { $dflt_pool_passthrough == "" } then {
log local0.debug "Failed to find pool for $servername_lower $"
} else {
pool $dflt_pool_passthrough
}
} else {
set selected_pool [call select_ab_pool $servername_lower $dflt_pool_passthrough ""]
if { $selected_pool == "" } then {
log local0.debug "Failed to find pool for $servername_lower"
} else {
pool $selected_pool
}
}
}
}
}
}
}
}
}
TCP::release
}
when CLIENTSSL_HANDSHAKE {
SSL::collect
}
when CLIENTSSL_DATA {
if { [llength [split [SSL::payload]]] < 1 }{
reject ; event disable all; return;
}
set sslpath [lindex [split [SSL::payload]] 1]
set domainpath $sslpath
set routepath ""
set wc_routepath ""
if { [info exists tls_servername] } {
set servername_lower [string tolower $tls_servername]
set domain_length [llength [split $servername_lower "."]]
set domain_wc [domain $servername_lower [expr {$domain_length - 1}] ]
set wc_host ".$domain_wc"
# Set routepath as combination of servername and url path
append routepath $servername_lower $sslpath
append wc_routepath $wc_host $sslpath
set routepath [string tolower $routepath]
set wc_routepath [string tolower $wc_routepath]
set sslpath $routepath
# Find the number of "/" in the routepath
set rc 0
foreach x [split $routepath {}] {
if {$x eq "/"} {
incr rc
}
}
# Disable serverside ssl and enable only for reencrypt routes
SSL::disable serverside
set reencrypt_class "/k8s-dev/Shared/payroll_gateway_443_ssl_reencrypt_servername_dg"
set edge_class "/k8s-dev/Shared/payroll_gateway_443_ssl_edge_servername_dg"
if { [class exists $reencrypt_class] || [class exists $edge_class] } {
# Compares the routepath with the entries in ssl_reencrypt_servername_dg and
# ssl_edge_servername_dg.
for {set i $rc} {$i >= 0} {incr i -1} {
if { [class exists $reencrypt_class] } {
set reen_pool [class match -value $routepath equals $reencrypt_class]
# Check for wildcard domain
if { $reen_pool equals "" } {
if { [class match $wc_routepath equals $reencrypt_class] } {
set reen_pool [class match -value $wc_routepath equals $reencrypt_class]
}
}
if { not ($reen_pool equals "") } {
set dflt_pool $reen_pool
SSL::enable serverside
}
}
if { [class exists $edge_class] } {
set edge_pool [class match -value $routepath equals $edge_class]
# Check for wildcard domain
if { $edge_pool equals "" } {
if { [class match $wc_routepath equals $edge_class] } {
set edge_pool [class match -value $wc_routepath equals $edge_class]
}
}
if { not ($edge_pool equals "") } {
set dflt_pool $edge_pool
}
}
if { not [info exists dflt_pool] } {
set routepath [
string range $routepath 0 [
expr {[string last "/" $routepath]-1}
]
]
set wc_routepath [
string range $wc_routepath 0 [
expr {[string last "/" $wc_routepath]-1}
]
]
}
else {
break
}
}
}
# handle the default pool for virtual server
set default_class "/k8s-dev/Shared/payroll_gateway_443_default_pool_servername_dg"
if { [class exists $default_class] } {
set dflt_pool [class match -value "defaultPool" equals $default_class]
}
# Handle requests sent to unknown hosts.
# For valid hosts, Send the request to respective pool.
if { not [info exists dflt_pool] } then {
# Allowing HTTP2 traffic to be handled by policies and closing the connection for HTTP/1.1 unknown hosts.
if { not ([SSL::payload] starts_with "PRI * HTTP/2.0") } {
reject ; event disable all;
log local0.debug "Failed to find pool for $servername_lower"
return;
}
} else {
pool $dflt_pool
}
set ab_class "/k8s-dev/Shared/payroll_gateway_443_ab_deployment_dg"
if { [class exists $ab_class] } {
set selected_pool [call select_ab_pool $servername_lower $dflt_pool $domainpath]
if { $selected_pool == "" } then {
log local0.debug "Unable to find pool for $servername_lower"
} else {
pool $selected_pool
}
}
}
SSL::release
}
when SERVER_CONNECTED {
log local0. "CUSTOM: SERVER_CONNECTED"
set reencryptssl_class "/k8s-dev/Shared/payroll_gateway_443_ssl_reencrypt_serverssl_dg"
set edgessl_class "/k8s-dev/Shared/payroll_gateway_443_ssl_edge_serverssl_dg"
if { [info exists sslpath] and [class exists $reencryptssl_class] } {
# Find the nearest child path which matches the reencrypt_class
for {set i $rc} {$i >= 0} {incr i -1} {
if { [class exists $reencryptssl_class] } {
set reen [class match -value $sslpath equals $reencryptssl_class]
# check for wildcard domain match
if { $reen equals "" } {
if { [class match $wc_routepath equals $reencryptssl_class] } {
set reen [class match -value $wc_routepath equals $reencryptssl_class]
}
}
if { not ($reen equals "") } {
set sslprofile $reen
}
}
if { [class exists $edgessl_class] } {
set edge [class match -value $sslpath equals $edgessl_class]
# check for wildcard domain match
if { $edge equals "" } {
if { [class match $wc_routepath equals $edgessl_class] } {
set edge [class match -value $wc_routepath equals $edgessl_class]
}
}
if { not ($edge equals "") } {
set sslprofile $edge
}
}
if { not [info exists sslprofile] } {
set sslpath [
string range $sslpath 0 [
expr {[string last "/" $sslpath]-1}
]
]
set wc_routepaath [
string range $wc_routepath 0 [
expr {[string last "/" $wc_routepath]-1}
]
]
}
else {
break
}
}
# Assign respective SSL profile based on ssl_reencrypt_serverssl_dg
if { not ($sslprofile equals "false") } {
SSL::profile $reen
}
}
}
In this configuration, when connecting to the VS we get an HTTP/2 stream error because the iRule is invalid.
By removing the last few lines of code from the iRule:
# Assign respective SSL profile based on ssl_reencrypt_serverssl_dg
# if { not ($sslprofile equals "false") } {
# SSL::profile $reen
# }
Now this works fine and HTTP/2 full proxy is working as expected
@shkarface Please share CIS configuration and error log, steps to reproduce this issue to automation_toolchain_pm [email protected]
@shkarface We are not able to reproduce this issue in local. Could you try with this build and confirm whether the error is still present
Build: cisbot/k8s-bigip-ctlr:browserUpdateIssue
Hello dear @vidyasagar-m , we have just updated to v2.17 and this is not yet fixed, all our virtual servers are responding with HTTP2 PROTOCOL ERROR
We have created ticket 00634117 on F5 support
Hi @shkarface ,
Please verify the fix with the following build and let us know the feedback.
Build: quay.io/f5networks/k8s-bigip-ctlr-devel:816503baf4e744f9228e51699bcfb966a6351eba
Hello @vklohiya,
Apologies for the late reply, I have tested your code and it still doesn't work. In fact, the new 2.17.1 version broke all VirtualServers with HostGroup assigned to them, I'm happy to jump on a call so we can look at this further.
I have a few environments experiencing this bug as well. Is there any workaround?
Hi @shkarface ,
Please verify the fix with the following build and let us know the feedback.
Build: quay.io/f5networks/k8s-bigip-ctlr-devel:816503baf4e744f9228e51699bcfb966a6351eba
I can confirm that this build fixes the issue
@walkingtub , Can you also confirm if following build fixes the issue?
Build: quay.io/f5networks/k8s-bigip-ctlr-devel:816503baf4e744f9228e51699bcfb966a6351eba
Can confirm this build works. I assume this will be in 2.18, do you know what the ETA for that is?
Resolved in CIS 2.18 - https://clouddocs.f5.com/containers/latest/reference/release-notes.html
