confluent cloud topic config values default not working

[mundey]

this behavior is weird and not sure if the reason is because of constant changes in confluent cloud or something else. we created with topics web/cli in the past and now intend to manage in this tool and the topics were created with defaults and no specific configs. So, intending to add the topics in this tool. here is a sample

settings:
  ccloud:
    enabled: true
  topics:
    defaults:
      # seems to be required for kafka, gives error if removed. but it's not required/shown for confluent cloud cli
      replication: 1
    blacklist:
      prefixed:
        - _confluent
topics:
  temp.test.hello-kafka:
    partitions: 3
# omitting writing some topic names in issue   

this topic above was created I think with web ui and then some other created with cli 2 days ago. one we created with cli 2 days ago does not show any change in plan but this one above shows change in config:

14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka exists, it will not be created.
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] cleanup.policy
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] max.compaction.lag.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] max.message.bytes
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] min.compaction.lag.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] message.timestamp.type
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] min.insync.replicas
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] segment.bytes
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] retention.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] segment.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] message.timestamp.difference.max.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] retention.bytes
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] delete.retention.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka exists, it will not be created.
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] cleanup.policy
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] max.compaction.lag.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] max.message.bytes
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] min.compaction.lag.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] message.timestamp.type
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] min.insync.replicas
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] segment.bytes
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] retention.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] segment.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] message.timestamp.difference.max.ms
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] retention.bytes
14:05:11.117 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] delete.retention.ms
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update

The following actions will be performed:
~ [TOPIC] temp.test.hello-kafka
	~ configs:
		- cleanup.policy
		- max.compaction.lag.ms
		- max.message.bytes
		- min.compaction.lag.ms
		- message.timestamp.type
		- min.insync.replicas
		- segment.bytes
		- retention.ms
		- segment.ms
		- message.timestamp.difference.max.ms
		- retention.bytes
		- delete.retention.ms


~ [TOPIC] temp.stage.hello-kafka
	~ configs:
		- cleanup.policy
		- max.compaction.lag.ms
		- max.message.bytes
		- min.compaction.lag.ms
		- message.timestamp.type
		- min.insync.replicas
		- segment.bytes
		- retention.ms
		- segment.ms
		- message.timestamp.difference.max.ms
		- retention.bytes
		- delete.retention.ms

when I execute apply, got errors:

14:08:58.853 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka exists, it will not be created.
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] cleanup.policy
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] max.compaction.lag.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] max.message.bytes
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] min.compaction.lag.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] message.timestamp.type
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] min.insync.replicas
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] segment.bytes
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] retention.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] segment.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] message.timestamp.difference.max.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] retention.bytes
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.test.hello-kafka | [REMOVE] delete.retention.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka exists, it will not be created.
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] cleanup.policy
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] max.compaction.lag.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] max.message.bytes
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] min.compaction.lag.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] message.timestamp.type
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] min.insync.replicas
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] segment.bytes
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] retention.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] segment.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] message.timestamp.difference.max.ms
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] retention.bytes
14:08:58.854 [main] INFO com.devshawn.kafka.gitops.manager.PlanManager - [PLAN] Topic temp.stage.hello-kafka | [REMOVE] delete.retention.ms
Applying: [UPDATE]

~ [TOPIC]  temp.test.hello-kafka
	~ configs:
		- max.compaction.lag.ms
		- max.message.bytes
		- min.compaction.lag.ms
		- message.timestamp.type
		- min.insync.replicas
		- segment.bytes
		- retention.ms
		- segment.ms
		- message.timestamp.difference.max.ms
		- retention.bytes
		- delete.retention.ms


[ERROR] Error thrown when attempting to update a Kafka topic config:
org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=TOPIC, name='redacted_ temp.test.hello-kafka'): null

[ERROR] An error has occurred during the apply process.
[ERROR] The apply process has stopped in place. There is no rollback.
[ERROR] Fix the error, re-create a plan, and apply the new plan to continue.

I have no choice but to fill up these values from cloud.

It would be nice to have:

• if the topic config properties are not specified in state.yaml, should not try to update. that way defaults will work even if these are changed by provider
• another nice feature would be to have the values under defaults just like replication.

[devshawn]

I ran into this with Confluent Cloud as well. They set defaults from the UI and they appear to be user-set from Kafka's API, so kafka-gitops sees them as custom values.

I agree, default config values would be a good feature.