pulumi
constant diff from wafv2 with no changes
The following wafv2 config...
portal_frontend_whitelist_ipset = aws.wafv2.IpSet(
f'portal-frontend-whitelist-{environment_name}',
opts=ResourceOptions(provider=aws_provider),
addresses=portal_frontend_whitelist,
ip_address_version='IPV4',
scope='REGIONAL',
tags={
'ResourceGroup': aws_resource_group_name
}
)
portal_frontend_acl = aws.wafv2.WebAcl(
f'portal-frontend-{environment_name}',
opts=ResourceOptions(provider=aws_provider),
scope='REGIONAL',
default_action={
'Block': {}
},
rules=[
WebAclRuleArgs(
name='AWS-AWSManagedRulesCommonRuleSet',
priority=1,
statement=WebAclRuleStatementArgs(
managed_rule_group_statement=WebAclRuleStatementManagedRuleGroupStatementArgs(
vendor_name='AWS',
name='AWSManagedRulesCommonRuleSet',
)
),
override_action=WebAclRuleOverrideActionArgs(
none=WebAclRuleOverrideActionNoneArgs()
),
visibility_config=WebAclRuleVisibilityConfigArgs(
sampled_requests_enabled=True,
cloudwatch_metrics_enabled=True,
metric_name='AWS-AWSManagedRulesCommonRuleSet',
)
),
WebAclRuleArgs(
name='portal_frontend_whitelist_ipset',
priority=2,
statement=WebAclRuleStatementArgs(
ip_set_reference_statement=WebAclRuleStatementIpSetReferenceStatementArgs(
arn=portal_frontend_whitelist_ipset.arn,
)
),
action=WebAclRuleActionArgs(
allow=WebAclRuleActionAllowArgs()
),
visibility_config=WebAclRuleVisibilityConfigArgs(
sampled_requests_enabled=True,
cloudwatch_metrics_enabled=True,
metric_name='portal_frontend_whitelist_ipset',
)
),
],
visibility_config={
'cloudwatchMetricsEnabled': True,
'metric_name': 'portal-frontend',
'sampledRequestsEnabled': True,
},
tags={
'ResourceGroup': aws_resource_group_name
},
)
produces this diff every time I run pulumi up. The update succeeds, with no changes as far as I can see.
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:portal.test1::portal::pulumi:pulumi:Stack::portal-portal.test1]
> pulumi:pulumi:StackReference: (read)
[id=acme/xxx/xxx.devx]
[urn=urn:pulumi:portal.test1::portal::pulumi:pulumi:StackReference::acme/xxx/xxx.devx]
name: "acme/xxx/xxx.devx"
~ aws:wafv2/webAcl:WebAcl: (update)
[id=xxx]
[urn=urn:pulumi:portal.test1::portal::aws:wafv2/webAcl:WebAcl::portal-frontend-test1]
[provider=urn:pulumi:portal.test1::portal::pulumi:providers:aws::aws_provider::xxxxx]
~ rules: [
~ [0]: {
+ name : "AWS-AWSManagedRulesCommonRuleSet"
+ priority : 1
~ statement : {
~ managedRuleGroupStatement: {
+ name : "AWSManagedRulesCommonRuleSet"
+ vendorName: "AWS"
}
}
~ visibilityConfig: {
+ cloudwatchMetricsEnabled: true
+ metricName : "AWS-AWSManagedRulesCommonRuleSet"
+ sampledRequestsEnabled : true
}
}
~ [1]: {
+ name : "portal_frontend_whitelist_ipset"
+ priority : 2
~ statement : {
~ ipSetReferenceStatement: {
+ arn: "arn:aws:wafv2:ap-southeast-2:111111111:regional/ipset/portal-frontend-whitelist-test1-072a323/xxxxxxxxx"
}
}
~ visibilityConfig: {
+ cloudwatchMetricsEnabled: true
+ metricName : "portal_frontend_whitelist_ipset"
+ sampledRequestsEnabled : true
}
}
]
requirements.txt
Package Version
----------------- ---------
Arpeggio 1.10.1
attrs 20.3.0
certifi 2020.11.8
chardet 3.0.4
dill 0.3.3
grpcio 1.33.2
idna 2.10
parver 0.3.1
pip 20.2.3
protobuf 3.14.0
pulumi 2.23.2
pulumi-aws 3.35.0
pulumi-kubernetes 2.8.4
pulumi-postgresql 2.8.1
pulumi-random 3.1.1
PyYAML 5.3.1
requests 2.25.0
semver 2.13.0
setuptools 49.2.1
six 1.15.0
urllib3 1.26.2
Is anyone from Pulumi looking into this? We're running into the same issues with the TypeScript sdk (@pulumi/aws 4.10.0): webAcl's diff always renders as if creating all the rules from scratch, even when there are no changes to apply. #1098 identified the same problem (among others) nearly a year ago, and the lack of activity on both issues leaves me worried that this behavior will continue to fall through the cracks indefinitely.
I took a brief look and I can't figure out why this is happening. It seems to be some combination of: the way we check/diff old state versus new state, populate the inputs/outputs based on prior program runs, and/or, possibly, set diffing.
Here is some raw data. I think someone like @pgavlin, @mikhailshilkov, or @EvanBoyle will need to weigh in, however.
The rules property is the culprit. Running the program the first time yields these inputs and outputs for the rules.
Inputs:
"rules": [
{
"__defaults": [],
"name": "AWS-AWSManagedRulesCommonRuleSet",
"overrideAction": {
"__defaults": [],
"none": {
"__defaults": []
}
},
"priority": 1,
"statement": {
"__defaults": [],
"managedRuleGroupStatement": {
"__defaults": [],
"name": "AWSManagedRulesCommonRuleSet",
"vendorName": "AWS"
}
},
"visibilityConfig": {
"__defaults": [],
"cloudwatchMetricsEnabled": true,
"metricName": "AWS-AWSManagedRulesCommonRuleSet",
"sampledRequestsEnabled": true
}
},
{
"__defaults": [],
"action": {
"__defaults": [],
"allow": {
"__defaults": []
}
},
"name": "portal_frontend_whitelist_ipset",
"priority": 2,
"statement": {
"__defaults": [],
"ipSetReferenceStatement": {
"__defaults": [],
"arn": "arn:aws:wafv2:us-east-1:153052954103:regional/ipset/portal-frontend-whitelist-test-42db0ef/ecb38af0-0560-4f06-8e4c-2dc50072c66b"
}
},
"visibilityConfig": {
"__defaults": [],
"cloudwatchMetricsEnabled": true,
"metricName": "portal_frontend_whitelist_ipset",
"sampledRequestsEnabled": true
}
}
],
Outputs:
"rules": [
{
"action": null,
"name": "AWS-AWSManagedRulesCommonRuleSet",
"overrideAction": {
"count": null,
"none": null
},
"priority": 1,
"statement": {
"andStatement": null,
"byteMatchStatement": null,
"geoMatchStatement": null,
"ipSetReferenceStatement": null,
"managedRuleGroupStatement": {
"excludedRules": [],
"name": "AWSManagedRulesCommonRuleSet",
"scopeDownStatement": null,
"vendorName": "AWS"
},
"notStatement": null,
"orStatement": null,
"rateBasedStatement": null,
"regexPatternSetReferenceStatement": null,
"ruleGroupReferenceStatement": null,
"sizeConstraintStatement": null,
"sqliMatchStatement": null,
"xssMatchStatement": null
},
"visibilityConfig": {
"cloudwatchMetricsEnabled": true,
"metricName": "AWS-AWSManagedRulesCommonRuleSet",
"sampledRequestsEnabled": true
}
},
{
"action": {
"allow": null,
"block": null,
"count": null
},
"name": "portal_frontend_whitelist_ipset",
"overrideAction": null,
"priority": 2,
"statement": {
"andStatement": null,
"byteMatchStatement": null,
"geoMatchStatement": null,
"ipSetReferenceStatement": {
"arn": "arn:aws:wafv2:us-east-1:153052954103:regional/ipset/portal-frontend-whitelist-test-42db0ef/ecb38af0-0560-4f06-8e4c-2dc50072c66b",
"ipSetForwardedIpConfig": null
},
"managedRuleGroupStatement": null,
"notStatement": null,
"orStatement": null,
"rateBasedStatement": null,
"regexPatternSetReferenceStatement": null,
"ruleGroupReferenceStatement": null,
"sizeConstraintStatement": null,
"sqliMatchStatement": null,
"xssMatchStatement": null
},
"visibilityConfig": {
"cloudwatchMetricsEnabled": true,
"metricName": "portal_frontend_whitelist_ipset",
"sampledRequestsEnabled": true
}
}
],
Here is what we send over RPC during the Diff command, which perceives a difference:
Olds: 434 I0814 12:57:18.079817 12171 rpc.go:72] Marshaling property for RPC[Provider[aws, 0xc00146c2a0].Diff(urn:pulumi:dev::wafwaf::aws:wafv2/webAcl:WebAcl::portal-frontend-test,e94302a1-b7ee-4765-ba7b-35f6ce2c9a23).olds]: rules ={[{map[action:{<nil>} name:{AWS-AWSManagedRulesCommonRuleSet} overrideAction:{map[count:{<nil>} none:{<nil>}]} priority:{1} statement:{map[andStatement:{<nil>} byteMatchStatement:{<nil>} geoMatchStatement:{<nil>} ipSetRef erenceStatement:{<nil>} managedRuleGroupStatement:{map[excludedRules:{[]} name:{AWSManagedRulesCommonRuleSet} scopeDownStatement:{<nil>} vendorName:{AWS}]} notStatement:{<nil>} orStatement:{<nil>} rateBasedStatement:{<nil> } regexPatternSetReferenceStatement:{<nil>} ruleGroupReferenceStatement:{<nil>} sizeConstraintStatement:{<nil>} sqliMatchStatement:{<nil>} xssMatchStatement:{<nil>}]} visibilityConfig:{map[cloudwatchMetricsEnabled:{true} m etricName:{AWS-AWSManagedRulesCommonRuleSet} sampledRequestsEnabled:{true}]}]} {map[action:{map[allow:{<nil>} block:{<nil>} count:{<nil>}]} name:{portal_frontend_whitelist_ipset} overrideAction:{<nil>} priority:{2} stateme nt:{map[andStatement:{<nil>} byteMatchStatement:{<nil>} geoMatchStatement:{<nil>} ipSetReferenceStatement:{map[arn:{arn:aws:wafv2:us-east-1:153052954103:regional/ipset/portal-frontend-whitelist-test-42db0ef/ecb38af0-0560-4 f06-8e4c-2dc50072c66b} ipSetForwardedIpConfig:{<nil>}]} managedRuleGroupStatement:{<nil>} notStatement:{<nil>} orStatement:{<nil>} rateBasedStatement:{<nil>} regexPatternSetReferenceStatement:{<nil>} ruleGroupReferenceStat ement:{<nil>} sizeConstraintStatement:{<nil>} sqliMatchStatement:{<nil>} xssMatchStatement:{<nil>}]} visibilityConfig:{map[cloudwatchMetricsEnabled:{true} metricName:{portal_frontend_whitelist_ipset} sampledRequestsEnabled :{true}]}]}]}
News: 505 I0814 12:57:18.080419 12171 rpc.go:72] Marshaling property for RPC[Provider[aws, 0xc00146c2a0].Diff(urn:pulumi:dev::wafwaf::aws:wafv2/webAcl:WebAcl::portal-frontend-test,e94302a1-b7ee-4765-ba7b-35f6ce2c9a23).news]: rules ={[{map[__defaults:{[]} name:{AWS-AWSManagedRulesCommonRuleSet} overrideAction:{map[__defaults:{[]} none:{map[__defaults:{[]}]}]} priority:{1} statement:{map[__defaults:{[]} managedRuleGroupStatement:{map[__defaults:{[]} n ame:{AWSManagedRulesCommonRuleSet} vendorName:{AWS}]}]} visibilityConfig:{map[__defaults:{[]} cloudwatchMetricsEnabled:{true} metricName:{AWS-AWSManagedRulesCommonRuleSet} sampledRequestsEnabled:{true}]}]} {map[__defaults: {[]} action:{map[__defaults:{[]} allow:{map[__defaults:{[]}]}]} name:{portal_frontend_whitelist_ipset} priority:{2} statement:{map[__defaults:{[]} ipSetReferenceStatement:{map[__defaults:{[]} arn:{arn:aws:wafv2:us-east-1:1 53052954103:regional/ipset/portal-frontend-whitelist-test-42db0ef/ecb38af0-0560-4f06-8e4c-2dc50072c66b}]}]} visibilityConfig:{map[__defaults:{[]} cloudwatchMetricsEnabled:{true} metricName:{portal_frontend_whitelist_ipset} sampledRequestsEnabled:{true}]}]}]}
And here is the final determination resulting from that invocation of Diff, which decides there is a difference: 550 I0814 12:57:18.361747 12171 provider_plugin.go:677] Provider[aws, 0xc00146c2a0].Diff(urn:pulumi:dev::wafwaf::aws:wafv2/webAcl:WebAcl::portal-frontend-test,e94302a1-b7ee-4765-ba7b-35f6ce2c9a23) success: changes=2 #replace s=[] #stables=[name scope] delbefrepl=false, diffs=#[rules rules rules rules rules rules rules rules rules rules rules rules rules], detaileddiff=map[rules[0].name: rules[0].priority: rules[0].statement.managedRuleGroupSta tement.name: rules[0].statement.managedRuleGroupStatement.vendorName: rules[0].visibilityConfig.cloudwatchMetricsEnabled: rules[0].visibilityConfig.metricName: rules[0].visibilityConfig.sampledRequestsEnabled: rules[1].nam e: rules[1].priority: rules[1].statement.ipSetReferenceStatement.arn: rules[1].visibilityConfig.cloudwatchMetricsEnabled: rules[1].visibilityConfig.metricName: rules[1].visibilityConfig.sampledRequestsEnabled:] I0814 12:57:18.361834 12171 step_generator.go:652] Planner decided to update 'urn:pulumi:dev::wafwaf::aws:wafv2/webAcl:WebAcl::portal-frontend-test'
I was encountering this and switched to the AWS Native provider (just for wafv2, the rest of my stack is still using the AWS Classic provider), adjusted a few parameter names (different capitalization on a few things), and it doesn't have this issue.
I'm also running into this issue and also tried the AWS native provider for this resource. That one has it's own issues that I didn't manage to workaround, which lead me to try this classic version.
This is really a pain for us, running pulumi with the CI we have all this noise in the preview output each time.
I can confirm we're also seeing this internally at Pulumi. I tested 3 versions of pulumi-aws (ts) which all show the bug -- 4.14, 4.28, and 5.9.2.
This is really a pain for us, running pulumi with the CI we have all this noise in the preview output each time.
Sorry you're running into this. It's the kind of bug that degrades the usefulness of the CI integration, so it's top of mind for me. If we always show the user "hey check this out!" when in reality nothing is changing, it's less likely they'll notice or care when something actually does change.
@blampe I would like to help solving this issue, did you already know what could be? Is the state that is not saved correctly?
@blampe I would like to help solving this issue, did you already know what could be? Is the state that is not saved correctly?
I'm not sure where the problem is. Last week I confirmed pulumi preview --json showed identical newState and oldState for the waf resource, but then it also included this which seems to be why it's getting replaced?
"diffReasons": [
"rules",
"rules",
"rules",
"rules",
"rules",
"rules",
"rules",
"rules",
"rules",
"rules",
"rules",
"rules",
"rules"
],
"detailedDiff": {
"rules[0].name": {
"kind": "add",
"inputDiff": false
},
Verified that this is still an issue and also does not seem to occur with an equivalent TF template. Going to check into the Diff behavior in the bridge.
Verified that this is still an issue and also does not seem to occur with an equivalent TF template
Can you paste the TF config here for posterity?
@pgavlin Any commented out stuff is from me and @stack72 attempting to debug, but it should be pretty close to the original:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.0"
}
}
}
# Configure the AWS Provider
provider "aws" {
region = "us-east-1"
}
resource "aws_wafv2_web_acl" "web_acl" {
name = "josh-test"
default_action {
block {}
}
scope = "REGIONAL"
visibility_config {
cloudwatch_metrics_enabled = false
metric_name = "josh-test"
sampled_requests_enabled = true
}
rule {
name = "josh-test-aws-rules"
override_action {
count {}
}
priority = 1
statement {
managed_rule_group_statement {
vendor_name = "AWS"
name = "AWSManagedRulesCommonRuleSet"
# version = "Version_1.3"
}
}
visibility_config {
cloudwatch_metrics_enabled = false
metric_name = "josh-test-aws-rules"
sampled_requests_enabled = true
}
}
tags = {
"key" = "value"
}
}
And the equivalent Pulumi:
import * as aws from "@pulumi/aws";
new aws.wafv2.WebAcl("web-acl", {
defaultAction: {
block: {},
},
scope: "REGIONAL",
visibilityConfig: {
cloudwatchMetricsEnabled: false,
metricName: "josh-test",
sampledRequestsEnabled: true,
},
rules: [{
name: "josh-test-aws-rules",
overrideAction: {
count: {},
},
priority: 1,
statement: {
managedRuleGroupStatement: {
vendorName: "AWS",
name: "AWSManagedRulesCommonRuleSet",
// version: "Version_1.3",
},
},
visibilityConfig: {
cloudwatchMetricsEnabled: false,
metricName: "josh-test-aws-rules",
sampledRequestsEnabled: true,
}
}]
});
@stack72 and I took a look at this for several hours today. We checked the TF provider schema, nothing doing.
Here's some dumped olds and news, if it's any help:
Q LOG
[15:56:21 tfbridge/provider.go:768 github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Diff]
[33m0.001s[0m [1molds[0m=[36mresource.PropertyMap{
"arn": {
V: "arn:aws:wafv2:us-east-1:616138583583:regional/webacl/web-acl-1f4eae1/9d218587-551e-4529-95a6-32796d38a4fd",
},
"capacity": {
V: float64(700),
},
"customResponseBodies": {
V: []resource.PropertyValue{
},
},
"defaultAction": {
V: resource.PropertyMap{
},
},
"description": {},
"id": {
V: "9d218587-551e-4529-95a6-32796d38a4fd",
},
"lockToken": {
V: "61cac8fb-c9d1-41d4-bafe-63974353a5b7",
},
"name": {
V: "web-acl-1f4eae1",
},
"rules": {
V: []resource.PropertyValue{
{
V: resource.PropertyMap{
"name": {
V: "josh-test-aws-rules",
},
"overrideAction": {
V: resource.PropertyMap{
},
},
"priority": {
V: float64(1),
},
"ruleLabels": {
V: []resource.PropertyValue{
},
},
"statement": {
V: resource.PropertyMap{
"managedRuleGroupStatement": {
V: resource.PropertyMap{
"excludedRules": {
V: []resource.PropertyValue{
},
},
"name": {
V: "AWSManagedRulesCommonRuleSet",
},
"vendorName": {
V: "AWS",
},
"version": {},
},
},
},
},
"visibilityConfig": {
V: resource.PropertyMap{
"cloudwatchMetricsEnabled": {},
"metricName": {
V: "josh-test-aws-rules",
},
"sampledRequestsEnabled": {
V: bool(true),
},
},
},
},
},
},
},
"scope": {
V: "REGIONAL",
},
"tags": {
V: resource.PropertyMap{
},
},
"tagsAll": {
V: resource.PropertyMap{
},
},
"visibilityConfig": {
V: resource.PropertyMap{
"cloudwatchMetricsEnabled": {},
"metricName": {
V: "josh-test",
},
"sampledRequestsEnabled": {
V: bool(true),
},
},
},
}[0m
[1mnews[0m=[36mresource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
{
V: "name",
},
},
},
"defaultAction": {
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
"block": {
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
},
},
},
},
"name": {
V: "web-acl-1f4eae1",
},
"rules": {
V: []resource.PropertyValue{
{
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
"name": {
V: "josh-test-aws-rules",
},
"overrideAction": {
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
"count": {
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
},
},
},
},
"priority": {
V: float64(1),
},
"statement": {
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
"managedRuleGroupStatement": {
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
"name": {
V: "AWSManagedRulesCommonRuleSet",
},
"vendorName": {
V: "AWS",
},
},
},
},
},
"visibilityConfig": {
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
"cloudwatchMetricsEnabled": {},
"metricName": {
V: "josh-test-aws-rules",
},
"sampledRequestsEnabled": {
V: bool(true),
},
},
},
},
},
},
},
"scope": {
V: "REGIONAL",
},
"visibilityConfig": {
V: resource.PropertyMap{
"__defaults": {
V: []resource.PropertyValue{
},
},
"cloudwatchMetricsEnabled": {},
"metricName": {
V: "josh-test",
},
"sampledRequestsEnabled": {
V: bool(true),
},
},
},
}[0m
Per @stack72, it looks like the program itself may not be correctly translating to state, but the TF schema does not appear to have anything ususual as far as we can tell.
We've worked around this with:
// NOTE: while updating "rules", you must comment out "ignoreChanges" setting.
// see https://github.com/pulumi/pulumi-aws/issues/1423
new aws.wafv2.WebAcl("xxx", {
rules: [ ... ],
...
}, {
ignoreChanges: ["rules"],
});
This is a bad workaround. rules is the main content of the declaration, so developers need to be aware of the hack when making legit changes.
In playing around with a test stack, this seems to in part be related to empty actions in rules. For instance action: { block {} } or override_action: { count: {} }
For instance, this rule won't converge
{
action: {
block: {},
},
name: "rate-limit",
priority: 2,
statement: {
rateBasedStatement: {
limit: 30000,
aggregateKeyType: "IP",
},
},
visibilityConfig: {
cloudwatchMetricsEnabled: true,
metricName: "rate-limit",
sampledRequestsEnabled: true,
},
},
But changing block {} to block: { customResponse: { responseCode: 403 }}} will cause it stop emitting diffs.
Unfortunately some actions are designed to only ever be empty, (e.g. override_action: { count: {} }} ), so this is not a full solution for all rule types. That might give some hints as to where the problem lies though.
For my own purposes, I switched to the aws-native provider for this resource because it also has better behavior wrt the lockToken. Though, that version of the resource is not without it's own bugs.
