amazon-vpc-cni-k8s
amazon-vpc-cni-k8s copied to clipboard
aws
request NET_RAW capabilities in CNI manifests
What type of PR is this?
feature
Which issue does this PR fix: https://github.com/aws/amazon-vpc-cni-k8s/issues/2061
What does this PR do / Why do we need it: This PR request NET_RAW Capabilities in CNI manifests. Linux NET_RAW capabilities let us use RAW and PACKET sockets; bind to any address for transparent proxying which is quite powerful. Our application request NET_RAW capabilities that calls iptables, which opens up a SOCK_RAW netlink socket to configure netfilter settings. It's likely the VPC CNI will continue needing NET_RAW in the future and to be used by other applications.
If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:
Testing done on this change:
--- PASS: TestSetupHostNetworkIgnoringRpFilterUpdate (0.00s)
=== RUN TestSetupHostNetworkUpdateLocalRule
{"level":"info","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network_test.go:852","msg":"Setting up host network... "}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:280","msg":"Trying to find primary interface that has mac : "}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:280","msg":"Discovered interface: lo, mac: "}
{"level":"info","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:280","msg":"Discovered primary interface: lo"}
{"level":"info","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network_test.go:852","msg":"Skip updating RPF for primary interface: net/ipv4/conf/lo/rp_filter"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:307","msg":"Found the Link that uses mac address and its index is 0 (attempt 1/5)"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:383","msg":"Trying to find primary interface that has mac : "}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:383","msg":"Discovered interface: lo, mac: "}
{"level":"info","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:383","msg":"Discovered primary interface: lo"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:403","msg":"Total CIDRs to program - 0"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:403","msg":"Setup Host Network: iptables -N AWS-SNAT-CHAIN-0 -t nat"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:403","msg":"Setup Host Network: iptables -A POSTROUTING -m comment --comment \"AWS SNAT CHAIN\" -j AWS-SNAT-CHAIN-0"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:714","msg":"Setup Host Network: loading existing iptables nat rules with chain prefix AWS-SNAT-CHAIN"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:509","msg":"Setup Host Network: computing stale iptables rules for %s table with chain prefix %s"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:403","msg":"iptableRules: [nat/POSTROUTING rule first SNAT rules for non-VPC outbound traffic shouldExist false rule [-m comment --comment AWS SNAT CHAIN -j AWS-SNAT-CHAIN-0] nat/AWS-SNAT-CHAIN-0 rule last SNAT rule for non-VPC outbound traffic shouldExist false rule [! -o vlan+ -m comment --comment AWS, SNAT -m addrtype ! --dst-type LOCAL -j SNAT --to-source 10.10.10.20] mangle/PREROUTING rule connmark for primary ENI shouldExist true rule [-m comment --comment AWS, primary ENI -i lo -m addrtype --dst-type LOCAL --limit-iface-in -j CONNMARK --set-mark 0x80/0x80] mangle/PREROUTING rule connmark restore for primary ENI shouldExist true rule [-m comment --comment AWS, primary ENI -i eni+ -j CONNMARK --restore-mark --mask 0x80] mangle/PREROUTING rule connmark restore for primary ENI from vlan shouldExist true rule [-m comment --comment AWS, primary ENI -i vlan+ -j CONNMARK --restore-mark --mask 0x80]]"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : first SNAT rules for non-VPC outbound traffic"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule nat/POSTROUTING rule first SNAT rules for non-VPC outbound traffic shouldExist false rule [-m comment --comment AWS SNAT CHAIN -j AWS-SNAT-CHAIN-0] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : last SNAT rule for non-VPC outbound traffic"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule nat/AWS-SNAT-CHAIN-0 rule last SNAT rule for non-VPC outbound traffic shouldExist false rule [! -o vlan+ -m comment --comment AWS, SNAT -m addrtype ! --dst-type LOCAL -j SNAT --to-source 10.10.10.20] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : connmark for primary ENI"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule mangle/PREROUTING rule connmark for primary ENI shouldExist true rule [-m comment --comment AWS, primary ENI -i lo -m addrtype --dst-type LOCAL --limit-iface-in -j CONNMARK --set-mark 0x80/0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : connmark restore for primary ENI"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule mangle/PREROUTING rule connmark restore for primary ENI shouldExist true rule [-m comment --comment AWS, primary ENI -i eni+ -j CONNMARK --restore-mark --mask 0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : connmark restore for primary ENI from vlan"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule mangle/PREROUTING rule connmark restore for primary ENI from vlan shouldExist true rule [-m comment --comment AWS, primary ENI -i vlan+ -j CONNMARK --restore-mark --mask 0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:411","msg":"Total CIDRs to exempt from connmark rules - 0"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:411","msg":"Setup Host Network: iptables -N AWS-CONNMARK-CHAIN-0 -t nat"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:411","msg":"Setup Host Network: iptables -t nat -A PREROUTING -i eni+ -m comment --comment \"AWS, outbound connections\" -m state --state NEW -j AWS-CONNMARK-CHAIN-0"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:714","msg":"Setup Host Network: loading existing iptables nat rules with chain prefix AWS-CONNMARK-CHAIN"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:639","msg":"Setup Host Network: computing stale iptables rules for %s table with chain prefix %s"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:411","msg":"iptableRules: [nat/PREROUTING rule connmark rule for non-VPC outbound traffic shouldExist false rule [-i eni+ -m comment --comment AWS, outbound connections -m state --state NEW -j AWS-CONNMARK-CHAIN-0] nat/AWS-CONNMARK-CHAIN-0 rule connmark rule for external outbound traffic shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --set-xmark 0x80/0x80] nat/PREROUTING rule connmark to fwmark copy shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --restore-mark --mask 0x80] nat/PREROUTING rule connmark to fwmark copy shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --restore-mark --mask 0x80]]"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"execute iptable rule : connmark rule for non-VPC outbound traffic"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"rule nat/PREROUTING rule connmark rule for non-VPC outbound traffic shouldExist false rule [-i eni+ -m comment --comment AWS, outbound connections -m state --state NEW -j AWS-CONNMARK-CHAIN-0] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"execute iptable rule : connmark rule for external outbound traffic"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"rule nat/AWS-CONNMARK-CHAIN-0 rule connmark rule for external outbound traffic shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --set-xmark 0x80/0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"execute iptable rule : connmark to fwmark copy"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"rule nat/PREROUTING rule connmark to fwmark copy shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --restore-mark --mask 0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"execute iptable rule : connmark to fwmark copy"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"rule nat/PREROUTING rule connmark to fwmark copy shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --restore-mark --mask 0x80] exists false, err <nil>"}
--- PASS: TestSetupHostNetworkUpdateLocalRule (0.00s)
PASS
coverage: 70.8% of statements
ok github.com/aws/amazon-vpc-cni-k8s/pkg/networkutils 0.025s coverage: 70.8% of statements
? github.com/aws/amazon-vpc-cni-k8s/pkg/networkutils/mocks [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/nswrapper [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/nswrapper/mocks [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/procsyswrapper [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/procsyswrapper/mocks [no test files]
{"level":"info","ts":"2022-08-15T07:06:47.504Z","caller":"logger/logger.go:52","msg":"Constructed new logger instance"}
{"level":"info","ts":"2022-08-15T07:06:47.504Z","caller":"awssession/session.go:39","msg":"Initialized new logger as an existing instance was not found"}
=== RUN TestCloudWatchPublisherWithNoIMDS
{"level":"info","ts":"2022-08-15T07:06:47.504Z","caller":"publisher/publisher_test.go:42","msg":"Constructed new logger instance"}
{"level":"warn","ts":"2022-08-15T07:06:47.504Z","caller":"awssession/session.go:64","msg":"HTTP_TIMEOUT env is not set or set to less than 10 seconds, defaulting to httpTimeout to 10sec"}
{"level":"info","ts":"2022-08-15T07:06:47.504Z","caller":"publisher/publisher_test.go:49","msg":"Using REGION=us-west-2 and CLUSTER_ID=TEST_CLUSTER_ID"}
--- PASS: TestCloudWatchPublisherWithNoIMDS (0.00s)
=== RUN TestCloudWatchPublisherWithSingleDatum
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher_test.go:63","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
--- PASS: TestCloudWatchPublisherWithSingleDatum (0.00s)
=== RUN TestCloudWatchPublisherWithMultipleDatum
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher_test.go:86","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
--- PASS: TestCloudWatchPublisherWithMultipleDatum (0.00s)
=== RUN TestCloudWatchPublisherWithGreaterThanMaxDatapoints
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher_test.go:108","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
--- PASS: TestCloudWatchPublisherWithGreaterThanMaxDatapoints (0.00s)
=== RUN TestCloudWatchPublisherWithGreaterThanMaxDatapointsAndStop
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher_test.go:129","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.516Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.516Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.526Z","caller":"publisher/publisher.go:173","msg":"Missing data for publishing CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.536Z","caller":"publisher/publisher.go:173","msg":"Missing data for publishing CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.547Z","caller":"publisher/publisher.go:173","msg":"Missing data for publishing CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.556Z","caller":"publisher/publisher.go:173","msg":"Missing data for publishing CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.556Z","caller":"publisher/publisher_test.go:136","msg":"Stopping monitor loop for CloudWatch publisher"}
{"level":"info","ts":"2022-08-15T07:06:47.556Z","caller":"publisher/publisher.go:220","msg":"Stopping monitor loop for CloudWatch publisher"}
--- PASS: TestCloudWatchPublisherWithGreaterThanMaxDatapointsAndStop (0.10s)
=== RUN TestCloudWatchPublisherWithSingleDatumWithError
{"level":"info","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher_test.go:161","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
{"level":"warn","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher.go:173","msg":"Unable to publish CloudWatch metrics: test error"}
--- PASS: TestCloudWatchPublisherWithSingleDatumWithError (0.00s)
=== RUN TestGetCloudWatchMetricNamespace
--- PASS: TestGetCloudWatchMetricNamespace (0.00s)
=== RUN TestGetCloudWatchMetricDatumDimensions
--- PASS: TestGetCloudWatchMetricDatumDimensions (0.00s)
=== RUN TestGetCloudWatchMetricDatumDimensionsWithMissingClusterID
--- PASS: TestGetCloudWatchMetricDatumDimensionsWithMissingClusterID (0.00s)
=== RUN TestPublishWithNoData
{"level":"info","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher_test.go:209","msg":"Fetching CloudWatch dimensions"}
--- PASS: TestPublishWithNoData (0.00s)
=== RUN TestPushWithMissingData
{"level":"info","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher_test.go:217","msg":"Missing data for publishing CloudWatch metrics"}
--- PASS: TestPushWithMissingData (0.00s)
=== RUN TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 70.8% of statements
ok github.com/aws/amazon-vpc-cni-k8s/pkg/publisher 0.112s coverage: 70.8% of statements
? github.com/aws/amazon-vpc-cni-k8s/pkg/publisher/mock_publisher [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/rpcwrapper [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/rpcwrapper/mocks [no test files]
=== RUN TestBuildHostVethNamePrefix
=== RUN TestBuildHostVethNamePrefix/standard_mode_should_use_configured_vethNamePrefix
=== RUN TestBuildHostVethNamePrefix/strict_mode_should_use_vlan_vethNamePrefix
--- PASS: TestBuildHostVethNamePrefix (0.00s)
--- PASS: TestBuildHostVethNamePrefix/standard_mode_should_use_configured_vethNamePrefix (0.00s)
--- PASS: TestBuildHostVethNamePrefix/strict_mode_should_use_vlan_vethNamePrefix (0.00s)
=== RUN TestLoadEnforcingModeFromEnv
=== RUN TestLoadEnforcingModeFromEnv/use_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_set_to_strict
=== RUN TestLoadEnforcingModeFromEnv/use_standard_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_set_to_standard
=== RUN TestLoadEnforcingModeFromEnv/default_to_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_not_set
=== RUN TestLoadEnforcingModeFromEnv/default_to_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_incorrectly_configured
--- PASS: TestLoadEnforcingModeFromEnv (0.00s)
--- PASS: TestLoadEnforcingModeFromEnv/use_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_set_to_strict (0.00s)
--- PASS: TestLoadEnforcingModeFromEnv/use_standard_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_set_to_standard (0.00s)
--- PASS: TestLoadEnforcingModeFromEnv/default_to_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_not_set (0.00s)
--- PASS: TestLoadEnforcingModeFromEnv/default_to_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_incorrectly_configured (0.00s)
PASS
coverage: 88.9% of statements
ok github.com/aws/amazon-vpc-cni-k8s/pkg/sgpp 0.008s coverage: 88.9% of statements
? github.com/aws/amazon-vpc-cni-k8s/pkg/typeswrapper [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/typeswrapper/mocks [no test files]
=== RUN Test_FindInterfaceByName
=== RUN Test_FindInterfaceByName/found_the_CNI_interface_at_index_0
=== RUN Test_FindInterfaceByName/found_the_CNI_interface_at_index_1
=== RUN Test_FindInterfaceByName/didn't_found_CNI_interface
--- PASS: Test_FindInterfaceByName (0.00s)
--- PASS: Test_FindInterfaceByName/found_the_CNI_interface_at_index_0 (0.00s)
--- PASS: Test_FindInterfaceByName/found_the_CNI_interface_at_index_1 (0.00s)
--- PASS: Test_FindInterfaceByName/didn't_found_CNI_interface (0.00s)
=== RUN Test_FindIPConfigsByIfaceIndex
=== RUN Test_FindIPConfigsByIfaceIndex/single_matched_IPConfig
=== RUN Test_FindIPConfigsByIfaceIndex/multiple_matched_IPConfig
=== RUN Test_FindIPConfigsByIfaceIndex/none_matched_IPConfig
=== RUN Test_FindIPConfigsByIfaceIndex/interface_is_not_set
--- PASS: Test_FindIPConfigsByIfaceIndex (0.00s)
--- PASS: Test_FindIPConfigsByIfaceIndex/single_matched_IPConfig (0.00s)
--- PASS: Test_FindIPConfigsByIfaceIndex/multiple_matched_IPConfig (0.00s)
--- PASS: Test_FindIPConfigsByIfaceIndex/none_matched_IPConfig (0.00s)
--- PASS: Test_FindIPConfigsByIfaceIndex/interface_is_not_set (0.00s)
PASS
coverage: 100.0% of statements
ok github.com/aws/amazon-vpc-cni-k8s/pkg/utils/cniutils 0.024s coverage: 100.0% of statements
{"level":"info","ts":"2022-08-15T07:06:47.752Z","caller":"logger/logger.go:52","msg":"Constructed new logger instance"}
{"level":"info","ts":"2022-08-15T07:06:47.753Z","caller":"k8sapi/k8sutils.go:23","msg":"Initialized new logger as an existing instance was not found"}
=== RUN TestBroadcastEvents
{"level":"debug","ts":"2022-08-15T07:06:47.763Z","caller":"eventrecorder/eventrecorder_test.go:99","msg":"Broadcasting event on pod mockPodWithLabelAndSpec"}
--- PASS: TestBroadcastEvents (0.01s)
PASS
coverage: 34.6% of statements
ok github.com/aws/amazon-vpc-cni-k8s/pkg/utils/eventrecorder 0.035s coverage: 34.6% of statements
=== RUN TestEnvLogFilePath
--- PASS: TestEnvLogFilePath (0.00s)
=== RUN TestLoggerGetSameInstance
--- PASS: TestLoggerGetSameInstance (0.00s)
=== RUN TestLoggerNewAndGetSameInstance
--- PASS: TestLoggerNewAndGetSameInstance (0.00s)
=== RUN TestGetLogFileLocationReturnsDefaultPath
--- PASS: TestGetLogFileLocationReturnsDefaultPath (0.00s)
=== RUN TestLogLevelReturnsOverriddenLevel
--- PASS: TestLogLevelReturnsOverriddenLevel (0.00s)
=== RUN TestLogLevelReturnsDefaultLevelWhenEnvNotSet
--- PASS: TestLogLevelReturnsDefaultLevelWhenEnvNotSet (0.00s)
=== RUN TestLogLevelReturnsDefaultLevelWhenEnvSetToInvalidValue
--- PASS: TestLogLevelReturnsDefaultLevelWhenEnvSetToInvalidValue (0.00s)
=== RUN TestGetPluginLogFilePathEmpty
--- PASS: TestGetPluginLogFilePathEmpty (0.00s)
=== RUN TestGetPluginLogFilePathStdout
--- PASS: TestGetPluginLogFilePathStdout (0.00s)
=== RUN TestGetPluginLogFilePath
--- PASS: TestGetPluginLogFilePath (0.00s)
PASS
coverage: 62.5% of statements
ok github.com/aws/amazon-vpc-cni-k8s/pkg/utils/logger 0.008s coverage: 62.5% of statements
=== RUN TestSimpleBackoff
--- PASS: TestSimpleBackoff (0.00s)
=== RUN TestJitter
--- PASS: TestJitter (0.00s)
=== RUN TestRetryWithBackoff
=== RUN TestRetryWithBackoff/retries
=== RUN TestRetryWithBackoff/no_retries
--- PASS: TestRetryWithBackoff (0.00s)
--- PASS: TestRetryWithBackoff/retries (0.00s)
--- PASS: TestRetryWithBackoff/no_retries (0.00s)
=== RUN TestRetryWithBackoffCtx
=== RUN TestRetryWithBackoffCtx/retries
=== RUN TestRetryWithBackoffCtx/no_retries
=== RUN TestRetryWithBackoffCtx/cancel_context
--- PASS: TestRetryWithBackoffCtx (0.00s)
--- PASS: TestRetryWithBackoffCtx/retries (0.00s)
--- PASS: TestRetryWithBackoffCtx/no_retries (0.00s)
--- PASS: TestRetryWithBackoffCtx/cancel_context (0.00s)
=== RUN TestRetryNWithBackoff
=== RUN TestRetryNWithBackoff/count_exceeded
=== RUN TestRetryNWithBackoff/retry_succeeded
--- PASS: TestRetryNWithBackoff (0.00s)
--- PASS: TestRetryNWithBackoff/count_exceeded (0.00s)
--- PASS: TestRetryNWithBackoff/retry_succeeded (0.00s)
=== RUN TestRetryNWithBackoffCtx
=== RUN TestRetryNWithBackoffCtx/count_exceeded
=== RUN TestRetryNWithBackoffCtx/retry_succeeded
=== RUN TestRetryNWithBackoffCtx/cancel_context
--- PASS: TestRetryNWithBackoffCtx (0.00s)
--- PASS: TestRetryNWithBackoffCtx/count_exceeded (0.00s)
--- PASS: TestRetryNWithBackoffCtx/retry_succeeded (0.00s)
--- PASS: TestRetryNWithBackoffCtx/cancel_context (0.00s)
PASS
coverage: 100.0% of statements
ok github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry 0.004s coverage: 100.0% of statements
? github.com/aws/amazon-vpc-cni-k8s/pkg/utils/ttime [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/utils/ttime/mocks [no test files]
? github.com/aws/amazon-vpc-cni-k8s/pkg/version [no test files]
Automation added to e2e:
Will this PR introduce any new dependencies?:
No.
Will this break upgrades or downgrades. Has updating a running cluster been tested?: No.
Does this change require updates to the CNI daemonset config files to work?:
Does this PR introduce any user-facing change?:
AWS VPC CNI requests NET_RAW capabilities by default.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
CAP_NET_ADMIN
Perform various network-related operations:
* interface configuration;
* administration of IP firewall, masquerading, and
accounting;
* modify routing tables;
* bind to any address for transparent proxying;
* set type-of-service (TOS);
* clear driver statistics;
* set promiscuous mode;
* enabling multicasting;
* use [setsockopt(2)](https://man7.org/linux/man-pages/man2/setsockopt.2.html) to set the following socket options:
SO_DEBUG, SO_MARK, SO_PRIORITY (for a priority outside
the range 0 to 6), SO_RCVBUFFORCE, and SO_SNDBUFFORCE.
CAP_NET_RAW
* Use RAW and PACKET sockets;
* bind to any address for transparent proxying.
The only additional capability will be this - "Use RAW and PACKET sockets;".
Thanks for the PR, we will look into it.
Also instead of updating in the sample manifests since this would add additional capabilities for every user by default, will you be able to use helm and you can either update in https://github.com/aws/amazon-vpc-cni-k8s/blob/master/charts/aws-vpc-cni/values.yaml or use "set" to include the capability?
@jayanthvn, I was the one that found this issue while working on internal stuff so I wanted to chime in.
This won't change the default permissions for anyone. If CAP_NET is not provided either explicitly or as part of the default permissions, aws-vpc-cni will simply not work. We wanted to surface this to make sure the permission is explicitly requested so there's no issues using runtimes that provide more restrictive permissions by default 😄.
Edit - Forgot to say, I'm from the same team as @JingmingGuo
Also instead of updating in the sample manifests since this would add additional capabilities for every user by default, will you be able to use helm and you can either update in https://github.com/aws/amazon-vpc-cni-k8s/blob/master/charts/aws-vpc-cni/values.yaml or use "set" to include the capability?
Thanks for pointing! Yes we should add this additional capabilities for every user by default. I updated and I'm looking forward to the reply.
Also instead of updating in the sample manifests since this would add additional capabilities for every user by default, will you be able to use helm and you can either update in https://github.com/aws/amazon-vpc-cni-k8s/blob/master/charts/aws-vpc-cni/values.yaml or use "set" to include the capability?
Thanks for pointing! Yes we should add this additional capabilities for every user by default. I updated and I'm looking forward to the reply.
Sorry for the confusion, I meant not everyone would need NET_RAW capability. I feel this has to be added on need basis. Please let me know your thoughts.
Sorry for the confusion, I meant not everyone would need NET_RAW capability. I feel this has to be added on need basis. Please let me know your thoughts.
Thanks for the response @jayanthvn!
I did a little testing (v1.10.2 of the plugin) to see if I could get the plugin running with NET_RAW explicitly dropped. Here's my securityContext:
securityContext:
capabilities:
add:
- NET_ADMIN
drop:
- NET_RAW
When I set ENABLE_IPv4 set to "true" and ENABLE_IPv6 set to "false" I see the following logs from ipamd:
{"level":"error","ts":"2022-08-19T18:19:03.111Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to set up host network: host network setup: failed to add chain: running [/usr/sbin/iptables -t nat -N AWS-SNAT-CHAIN-0 --wait]: exit status 3: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}
I also tried the reversed configuration (enabling IPv6 and disabling IPv4) and got the following logs:
{"level":"error","ts":"2022-08-19T18:22:54.759Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to set up host network: failed to enable IPv6: setupVeth network: failed to setup route to block pod access via IPv4 address: failed adding v4 drop route: running [/usr/sbin/iptables -t filter -I FORWARD 1 -d 169.254.172.0/22 -m conntrack --ctstate NEW -m comment --comment Block Node Local Pod access via IPv4 -j REJECT --wait]: exit status 3: conntrack: Could not determine whether revision 1 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 1 is supported, assuming it is.\niptables v1.8.4 (legacy): can't initialize iptables table `filter': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}
In both cases the logs seem to indicate https://github.com/aws/amazon-vpc-cni-k8s/blob/v1.10.2/pkg/ipamd/ipamd.go#L457-L461 is causing the error. Just glancing at the code, I don't see any way to configure the plugin to avoid invoking iptables, which seems to require NET_RAW. Let me know if you can reproduce.
I didn't exhaustively test configuration, so I might be missing a case where iptables is not used.
Sorry for the confusion, I meant not everyone would need NET_RAW capability. I feel this has to be added on need basis. Please let me know your thoughts.
Thanks for the response @jayanthvn!
I did a little testing (v1.10.2 of the plugin) to see if I could get the plugin running with NET_RAW explicitly dropped. Here's my securityContext:
securityContext: capabilities: add: - NET_ADMIN drop: - NET_RAWWhen I set
ENABLE_IPv4set to"true"andENABLE_IPv6set to"false"I see the following logs from ipamd:{"level":"error","ts":"2022-08-19T18:19:03.111Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to set up host network: host network setup: failed to add chain: running [/usr/sbin/iptables -t nat -N AWS-SNAT-CHAIN-0 --wait]: exit status 3: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}I also tried the reversed configuration (enabling IPv6 and disabling IPv4) and got the following logs:
{"level":"error","ts":"2022-08-19T18:22:54.759Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to set up host network: failed to enable IPv6: setupVeth network: failed to setup route to block pod access via IPv4 address: failed adding v4 drop route: running [/usr/sbin/iptables -t filter -I FORWARD 1 -d 169.254.172.0/22 -m conntrack --ctstate NEW -m comment --comment Block Node Local Pod access via IPv4 -j REJECT --wait]: exit status 3: conntrack: Could not determine whether revision 1 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 1 is supported, assuming it is.\niptables v1.8.4 (legacy): can't initialize iptables table `filter': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}In both cases the logs seem to indicate https://github.com/aws/amazon-vpc-cni-k8s/blob/v1.10.2/pkg/ipamd/ipamd.go#L457-L461 is causing the error. Just glancing at the code, I don't see any way to configure the plugin to avoid invoking iptables, which seems to require NET_RAW. Let me know if you can reproduce.
I didn't exhaustively test configuration, so I might be missing a case where iptables is not used.
Yes the plugin will require iptables and there is no scenario where we won't use.
regarding your use case -> "Our application request NET_RAW capabilities that calls iptables, which opens up a SOCK_RAW netlink socket to configure netfilter setting" and NET_RAW adds additional capability of "* Use RAW and PACKET sockets;" for the existing NET_ADMIN which would not be required to be added by default for all applications.
If you are on K8S slack, we can sync up on a time slot and get on a call to further discuss about this.
Thanks for taking the time to set the discussion meeting with us today. @S-Chan explained the reason and the fact why everyone needs NET_RAW. I just put the summary here for easier tracking:
AWS VPC CNI k8s use coreos/go-iptables/iptables for networking in multiple places(e.g. https://github.com/aws/amazon-vpc-cni-k8s/blob/d43309bdfdb5034df86907944e682d78608ba165/pkg/networkutils/network.go#L40) which coreos/go-iptables/iptables trigger system iptables call: https://github.com/coreos/go-iptables/blob/d2b8608923d15b0800af7d9f4bb6dea90e03b7d5/iptables/iptables.go#L564-L570. In Amazon Linux AMI:
yum info iptables
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Installed Packages
Name : iptables
Arch : x86_64
Version : 1.8.4
Release : 10.amzn2.1.2
Size : 1.6 M
Repo : installed
Summary : Tools for managing Linux kernel packet filtering capabilities
URL : http://www.netfilter.org/
License : GPLv2 and Artistic 2.0 and ISC
Description : The iptables utility controls the network packet filtering code in the
: Linux kernel. If you need to set up firewalls and/or IP masquerading,
: you should install this package.
Netfilter iptables includes libiptc: https://git.netfilter.org/iptables/tree/include/iptables.h#n6 which uses RAW sockets: https://git.netfilter.org/iptables/tree/libiptc/libiptc.c#n1312.
So that all the users use AWS VPC CNI k8s will need the CAP_NET_RAW which is reasonable to set as default. Also prevent the failures if users didn't notice it.
Hi @JingmingGuo, thanks for the PR!
I was able to verify that NET_RAW capability is required in aws-node by experimenting with these values:
securityContext:
capabilities:
add:
- NET_ADMIN
- NET_RAW
drop:
- ALL
When I removed NET_RAW, ipamd fails to initialize with the following error(as already pointed out):
{"level":"error","ts":"2022-10-14T15:06:20.300Z","caller":"aws-k8s-agent/main.go:28","msg":"Initialization failure: ipamd init: failed to set up host network: host network setup: failed to add chain: running [/usr/sbin/iptables -t nat -N AWS-SNAT-CHAIN-0 --wait]: exit status 3: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}
Also verified that NET_RAW capability is included by default in docker & containerd runtimes, see:
kubectl exec -it -n kube-system aws-node-r476b -- capsh --print
Defaulted container "aws-node" out of: aws-node, aws-vpc-cni-init (init)
Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =
Current IAB: !cap_dac_read_search,!cap_linux_immutable,!cap_net_broadcast,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_lease,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read
Securebits: 00/0x0/1'b0 (no-new-privs=0)
secure-noroot: no (unlocked)
secure-no-suid-fixup: no (unlocked)
secure-keep-caps: no (unlocked)
secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=
Guessed mode: UNCERTAIN (0)
@renan-airbnb 's comment is valid:
We wanted to surface this to make sure the permission is explicitly requested so there's no issues using runtimes that provide more restrictive permissions by default 😄.
We will need to include NET_RAW to ensure that aws-node can work on runtimes that don't have it by default.
For example, cri-o does not include this by default https://github.com/cri-o/cri-o/pull/3119
@JingmingGuo can you please update the branch?
Sure I updated the branch.
@JingmingGuo thanks for updating, but you might still need to run make check-format as unit tests are failing here
@JingmingGuo thanks for updating, but you might still need to run
make check-formatas unit tests are failing here
Thanks for pointing! I run 'make format' and updated the PR. See the checks have passed now.