terraform-provider-sumologic
terraform-provider-sumologic copied to clipboard
SumoLogic
Unable to add new items to match list
Terraform Version
% terraform --version
Terraform v1.5.6
on darwin_arm64
+ provider registry.terraform.io/sumologic/sumologic v2.25.0
Affected Resource(s)
sumologic_cse_match_list
Terraform Configuration Files
resource "sumologic_cse_match_list" "match_list" {
name = "my-list"
description = "Testing Terraform management"
target_column = "SrcIp"
items {
description = "test"
value = "1.1.1.1"
}
}
However Adding a new item to the list causes a recreation of all items in a list, and this fails apply (it hangs and never completes). Additionally, as shown in the output below, the item ids are the same, I don't know why it wants to recreate them.
Debug Output
Terraform will perform the following actions:
# sumologic_cse_match_list.match_list will be updated in-place
~ resource "sumologic_cse_match_list" "match_list" {
id = "76"
name = "my-list"
# (7 unchanged attributes hidden)
- items {
- description = "test" -> null
- id = "5ebcb1d684fe3a9e8571470397e888c0e086aa137fa13f67927b39d0eca186106d792d6c697374" -> null
- value = "1.1.1.1" -> null
}
+ items {
+ description = "test"
+ id = "5ebcb1d684fe3a9e8571470397e888c0e086aa137fa13f67927b39d0eca186106d792d6c697374"
+ value = "1.1.1.1"
}
+ items {
+ description = "test2"
+ id = (known after apply)
+ value = "8.8.8.8"
}
}
[....20 minutes later]
sumologic_cse_match_list.match_list: Still modifying... [id=76, 20m0s elapsed]
╷
│ Error: error waiting for match list (76) to be updated: timeout while waiting for state to become '3' (last state: '1', timeout: 20m0s)
│
│ with sumologic_cse_match_list.match_list,
│ on cse_match_list.tf line 24, in resource "sumologic_cse_match_list" "match_list":
│ 24: resource "sumologic_cse_match_list" "match_list" {
│
Expected Behavior
I should be able to add new items to a match list in terraform.
Actual Behavior
Adding a new item forces a recreation of all items, which then fails.
Steps to Reproduce
- Create a match list
- Add some items to it
- Apply
- Add new items
- Try applying again
was not able to reproduce, if match-list and/or match-list-items are also being managed outside terraform there might be an issue. See https://github.com/SumoLogic/terraform-provider-sumologic/blob/master/website/docs/r/cse_match_list.html.markdown
@pmontiel-sumo no rep so auto close without reply?
I can continue to reproduce locally.
The match list is not being managed outside of terraform as documented in the issue.
I change my resource to
resource "sumologic_cse_match_list" "match_list" {
name = "my-list"
description = "Testing Terraform management"
target_column = "SrcIp"
items {
description = "test"
value = "1.1.1.1"
}
items {
description = "test2"
value = "8.8.8.8"
}
}
and get the plan that is in the issue description
going to ask access support to your org, in the meantime could you try with a new match list please?
as mentioned, asked for support access to your org, and was able to perform steps - you can see the test I ran https://upgrade.us2.sumologic.com/sec/content/match-lists/match-list/83
CREATION:
resource "sumologic_cse_match_list" "match_list" {
default_ttl = "10800"
description = "Match List Description"
name = "Terraform Test Match List 7e8401c5-9a9d-42d1-98f7-d40c7d555134"
target_column = "SrcIp"
items {
description = "Match List Item Description 0 95b419f3-0c4f-4cff-9236-0ccac6ed004e"
expiration = "2122-02-27T04:00:00"
value = "value 0 95b419f3-0c4f-4cff-9236-0ccac6ed004e"
}
}
PLAN:
CREATE: sumologic_cse_match_list.match_list
created: "" => "<computed>"
created_by: "" => "<computed>"
default_ttl: "" => "10800"
description: "" => "Match List Description"
id: "" => "<computed>"
items.#: "" => "1"
items.0.description: "" => "Match List Item Description 0 95b419f3-0c4f-4cff-9236-0ccac6ed004e"
items.0.expiration: "" => "2122-02-27T04:00:00"
items.0.id: "" => "<computed>"
items.0.value: "" => "value 0 95b419f3-0c4f-4cff-9236-0ccac6ed004e"
last_updated: "" => "<computed>"
last_updated_by: "" => "<computed>"
name: "" => "Terraform Test Match List 7e8401c5-9a9d-42d1-98f7-d40c7d555134"
target_column: "" => "SrcIp"
ADD NEW ITEM:
resource "sumologic_cse_match_list" "match_list" {
default_ttl = "3600"
description = "Updated Match List Description"
name = "Terraform Test Match List 7e8401c5-9a9d-42d1-98f7-d40c7d555134"
target_column = "SrcIp"
items {
description = "Updated Match List item Description 0 e95984ed-b9a3-45ab-a789-ccca3e52bab8"
expiration = "2122-02-27T05:00:00+00:00"
value = "updated value 0 e95984ed-b9a3-45ab-a789-ccca3e52bab8"
}
items {
description = "Updated Match List item Description 1 0d712e9e-d419-4a2c-8660-3ce7d6834d7d"
expiration = "2122-02-27T05:00:00+00:00"
value = "updated value 1 0d712e9e-d419-4a2c-8660-3ce7d6834d7d"
}
PLAN:
UPDATE: sumologic_cse_match_list.match_list
created: "2023-09-20T18:48:31.936704+00:00" => "2023-09-20T18:48:31.936704+00:00"
created_by: "[email protected]" => "[email protected]"
default_ttl: "10800" => "3600"
description: "Match List Description" => "Updated Match List Description"
id: "83" => "83"
items.#: "1" => "2"
items.0.description: "Match List Item Description 0 95b419f3-0c4f-4cff-9236-0ccac6ed004e" => "Updated Match List item Description 0 e95984ed-b9a3-45ab-a789-ccca3e52bab8"
items.0.expiration: "2122-02-27T04:00:00" => "2122-02-27T05:00:00+00:00"
items.0.id: "5ebcb1d684fe3a9e8571470397e888c070e8748ae8ab3d53862b1e34c9d97bfb5465727261666f726d2054657374204d61746368204c6973742037653834303163352d396139642d343264312d393866372d643430633764353535313334" => "<computed>"
items.0.value: "value 0 95b419f3-0c4f-4cff-9236-0ccac6ed004e" => "updated value 0 e95984ed-b9a3-45ab-a789-ccca3e52bab8"
items.1.description: "" => "Updated Match List item Description 1 0d712e9e-d419-4a2c-8660-3ce7d6834d7d"
items.1.expiration: "" => "2122-02-27T05:00:00+00:00"
items.1.id: "" => "<computed>"
items.1.value: "" => "updated value 1 0d712e9e-d419-4a2c-8660-3ce7d6834d7d"
last_updated: "2023-09-20T18:48:34.556633+00:00" => "2023-09-20T18:48:34.556633+00:00"
last_updated_by: "[email protected]" => "[email protected]"
name: "Terraform Test Match List 7e8401c5-9a9d-42d1-98f7-d40c7d555134" => "Terraform Test Match List 7e8401c5-9a9d-42d1-98f7-d40c7d555134"
target_column: "SrcIp" => "SrcIp"
Tested with a new match list
resource "sumologic_cse_match_list" "test2" {
name = "tf-provider-test"
description = "Testing"
target_column = "SrcIp"
items {
description = "test"
value = "1.1.1.1"
}
}
Plan:
Terraform will perform the following actions:
# sumologic_cse_match_list.test2 will be created
+ resource "sumologic_cse_match_list" "test2" {
+ created = (known after apply)
+ created_by = (known after apply)
+ description = "Testing"
+ id = (known after apply)
+ last_updated = (known after apply)
+ last_updated_by = (known after apply)
+ name = "tf-provider-test"
+ target_column = "SrcIp"
+ items {
+ description = "test"
+ id = (known after apply)
+ value = "1.1.1.1"
}
}
Plan: 1 to add, 0 to change, 0 to destroy.
Apply:
sumologic_cse_match_list.test2: Creating...
sumologic_cse_match_list.test2: Still creating... [10s elapsed]
sumologic_cse_match_list.test2: Creation complete after 16s [id=86]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
Add item
resource "sumologic_cse_match_list" "test2" {
name = "tf-provider-test"
description = "Testing"
target_column = "SrcIp"
items {
description = "test"
value = "1.1.1.1"
}
items {
description = "test2"
value = "8.8.8.8"
}
}
Plan:
Terraform will perform the following actions:
# sumologic_cse_match_list.test2 will be updated in-place
~ resource "sumologic_cse_match_list" "test2" {
id = "86"
name = "tf-provider-test"
# (7 unchanged attributes hidden)
- items {
- description = "test" -> null
- id = "5ebcb1d684fe3a9e8571470397e888c0e086aa137fa13f67927b39d0eca1861074662d70726f76696465722d74657374" -> null
- value = "1.1.1.1" -> null
}
+ items {
+ description = "test"
+ id = "5ebcb1d684fe3a9e8571470397e888c0e086aa137fa13f67927b39d0eca1861074662d70726f76696465722d74657374"
+ value = "1.1.1.1"
}
+ items {
+ description = "test2"
+ id = (known after apply)
+ value = "8.8.8.8"
}
}
Plan: 0 to add, 1 to change, 0 to destroy.
Apply fails. In debug logs I see
2023-09-20T15:00:10.387-0400 [DEBUG] provider.terraform-provider-sumologic_v2.25.0: 2023/09/20 15:00:10 [WARN] An error occurred while adding match list items to match list id: 86, err: {"data": null, "errors": [{"message": "Invalid values provided for parameter \"items\": ", "code": "BAD_REQUEST"}]}
log of an invalid plan
2023-09-20T15:00:07.552-0400 [WARN] Provider "registry.terraform.io/sumologic/sumologic" produced an invalid plan for sumologic_cse_match_list.test2, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .default_ttl: planned value cty.NumberIntVal(0) for a non-computed attribute
please bear with me, could you remove the empty line after "target_column" please, so:
resource "sumologic_cse_match_list" "test2" {
name = "tf-provider-test"
description = "Testing"
target_column = "SrcIp"
items {
description = "test"
value = "1.1.1.1"
}
items {
description = "test2"
value = "8.8.8.8"
}
}
Ok i've updated to
resource "sumologic_cse_match_list" "test2" {
name = "tf-provider-test"
description = "Testing"
target_column = "SrcIp"
items {
description = "test"
value = "1.1.1.1"
}
items {
description = "test2"
value = "8.8.8.8"
}
}
Plan shows item replacement still:
# sumologic_cse_match_list.test2 will be updated in-place
~ resource "sumologic_cse_match_list" "test2" {
id = "86"
name = "tf-provider-test"
# (7 unchanged attributes hidden)
- items {
- description = "test" -> null
- id = "5ebcb1d684fe3a9e8571470397e888c0e086aa137fa13f67927b39d0eca1861074662d70726f76696465722d74657374" -> null
- value = "1.1.1.1" -> null
}
+ items {
+ description = "test"
+ id = "5ebcb1d684fe3a9e8571470397e888c0e086aa137fa13f67927b39d0eca1861074662d70726f76696465722d74657374"
+ value = "1.1.1.1"
}
+ items {
+ description = "test2"
+ id = (known after apply)
+ value = "8.8.8.8"
}
}
Plan: 0 to add, 1 to change, 0 to destroy.
Apply fails on same error. And still producing invalid plan log
2023-09-20T15:18:29.069-0400 [WARN] ValidateProviderConfig from "provider[\"registry.terraform.io/sumologic/sumologic\"]" changed the config value, but that value is unused
2023-09-20T15:18:29.072-0400 [WARN] Provider "registry.terraform.io/sumologic/sumologic" produced an invalid plan for sumologic_cse_match_list.test2, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .default_ttl: planned value cty.NumberIntVal(0) for a non-computed attribute
ran a new test more similar to yours and think was able to replicate the BAD_REQUEST error, will come back to you shortly, thank you for your patience
sorry for the delay, @eambrosyupgrade as you already saw there is a code change coming out on next release, once 2.26.1 could you please give it a try?
@pmontiel-sumo New provider worked. I had to do 2 applies as first apply behaved similar as original issue. However plan for 2nd attemp did not show item ids nor removes and worked.
@eambrosyupgrade what do you mean by "2 applies"? I'm seeing similar behavior on 2.27 and can't figure this out.
OK did a little digging...
In resourceSumologicCSEMatchListCreate, existing state here for the id on new match list items is known after apply, so further on in the function when items is being populated, CSEMatchListItemPost instances are being initialized with no ID (empty string).
itemsData := d.Get("items").(*schema.Set).List()
var items []CSEMatchListItemPost
for _, data := range itemsData {
item := resourceToCSEMatchListItem([]interface{}{data})
items = append(items, item)
}
if len(items) > 0 {
err = c.CreateCSEMatchListItems(items, id)
if err != nil {
log.Printf("[WARN] An error occurred while adding match list items to match list id: %s, err: %v", id, err)
}
}
The provider is calling the AddItemsToMatchList endpoint (via SendCreateCSEMatchListItemsRequest) which doesn't take an id (and the ID property on CSEMatchListItemPost is specified with omitempty). So the initial apply works fine.
During an apply with an update via resourceSumologicCSEMatchListUpdate, is newItems being populated with CSEMatchListItemPost instances with an ID?
I also noticed the same line in my debug output that @eambrosyupgrade called out. Seems like the AddItemsToMatchList endpoint is being called with an id in the payload.
2023-10-04T19:22:27.274-0400 [DEBUG] provider.terraform-provider-sumologic_v2.27.0: 2023/10/04 19:22:27 [WARN] An error occurred while adding match list items to match list id: 40, err: {"data": null, "errors": [{"message": "Invalid values provided for parameter \"items\": In field \"id\": Unknown field. In field \"id\": Unknown field.", "code": "BAD_REQUEST"}]}
Does that sound plausible @pmontiel-sumo?
Re-opening this. Like @petergs mentioned this behavior persists.
1st apply always shows removals of existing items and fails. 2nd apply "succeeds" as it is adding new items. From what I've observeed, updating a match lists fails but all items are removed, re-running an apply adds the items back as now.
I've been able to consistently reproduce when adding new items. 2 applies function as work around for now, but running this in a pipeline and have to wait 20 mins for terraform to timeout and give up before kicking off a second job is not ideal.
I think the solution might be as simple as leaving item.ID uninitialized or set to an empty string in resourceToCSEMatchListItem.
But I'm not sure if it's a better idea to remove the ID property from CSEMatchListItemPost altogether. It's used for UpdateCSEMatchListItem which doesn't appear to be called anywhere.
I think this change would prevent posting an ID in AddItemsToMatchList:
func resourceToCSEMatchListItem(data interface{}) CSEMatchListItemPost {
itemsSlice := data.([]interface{})
item := CSEMatchListItemPost{}
if len(itemsSlice) > 0 {
itemObj := itemsSlice[0].(map[string]interface{})
item.ID = "" // leave as default zero value
item.Description = itemObj["description"].(string)
item.Active = true
item.Expiration = itemObj["expiration"].(string)
item.Value = itemObj["value"].(string)
}
return item
}
Also - here's a curl example to reproduce the error we're seeing in debug logs. You'll need to replace the values, MATCHLIST_ID, and ENDPOINT to relevant values for your environment.
#!/bin/bash
CREDS=$(echo -n "$SUMO_ACCESS_ID:$SUMO_ACCESS_KEY" | base64 )
MATCHLIST_ID=41
ENDPOINT="https://api.us2.sumologic.com/api/sec/v1/match-lists/$MATCHLIST_ID/items"
curl \
--header "Content-Type: application/json" \
--header "Authorization: Basic $CREDS" \
--request POST \
--data '{"items": [{"value": "8.8.8.8", "active": "true", "description": "test", "id": "0"}, {"value": "9.9.9.9", "active": "true", "description": "test", "id": "1"}]}' \
"$ENDPOINT"
Output:
{"data": null, "errors": [{"message": "Invalid values provided for parameter \"items\": In field \"id\": Unknown field. In field \"id\": Unknown field.", "code": "BAD_REQUEST"}]}