Kafka advanced config defaults: set default/max values for advanced config settings

[dweinshenker]

What

• Sets defaults for advanced config settings related to Kafka
• Sets maximum values for various params, based on the size of int64

[loosla]

:wave: Hello, thanks for setting defaults for Kafka :pray:

• Just curious if these defaults were released and accessible through API or they are planned to be released soon?

When I create a Kafka topic without advanced config parameters with curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" -d '{"name":"customer-events", "partition_count":3, "replication_factor": 3, "config": {}}' "https://api.digitalocean.com/v2/databases/90b1549f-f9a4-4927-9ece-30303ebab89a/topics"

I'm getting

"config": {
      "cleanup_policy": "delete",
      "compression_type": "producer",
      "delete_retention_ms": 86400000,
      "file_delete_delay_ms": 60000,
      "flush_messages": 9223372036854776000,
      "flush_ms": 9223372036854776000,
      "index_interval_bytes": 4096,
      "max_compaction_lag_ms": 9223372036854776000,
      "max_message_bytes": 1048588,
      "message_down_conversion_enable": true,
      "message_format_version": "3.0-IV1",
      "message_timestamp_difference_max_ms": 9223372036854776000,
      "message_timestamp_type": "create_time",
      "min_cleanable_dirty_ratio": 0.5,
      "min_compaction_lag_ms": 0,
      "min_insync_replicas": 1,
      "preallocate": false,
      "retention_bytes": -1,
      "retention_ms": 604800000,
      "segment_bytes": 1073741824,
      "segment_index_bytes": 10485760,
      "segment_jitter_ms": 0,
      "segment_ms": 604800000
    }

• Can it cause potential problems for out customers for example when we change default for compression_type from "producer" to "uncompressed"?

Many thanks :pray:

[dweinshenker]

👋 Hello, thanks for setting defaults for Kafka 🙏

• Just curious if these defaults were released and accessible through API or they are planned to be released soon?

When I create a Kafka topic without advanced config parameters with curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" -d '{"name":"customer-events", "partition_count":3, "replication_factor": 3, "config": {}}' "https://api.digitalocean.com/v2/databases/90b1549f-f9a4-4927-9ece-30303ebab89a/topics"

I'm getting

"config": {
      "cleanup_policy": "delete",
      "compression_type": "producer",
      "delete_retention_ms": 86400000,
      "file_delete_delay_ms": 60000,
      "flush_messages": 9223372036854776000,
      "flush_ms": 9223372036854776000,
      "index_interval_bytes": 4096,
      "max_compaction_lag_ms": 9223372036854776000,
      "max_message_bytes": 1048588,
      "message_down_conversion_enable": true,
      "message_format_version": "3.0-IV1",
      "message_timestamp_difference_max_ms": 9223372036854776000,
      "message_timestamp_type": "create_time",
      "min_cleanable_dirty_ratio": 0.5,
      "min_compaction_lag_ms": 0,
      "min_insync_replicas": 1,
      "preallocate": false,
      "retention_bytes": -1,
      "retention_ms": 604800000,
      "segment_bytes": 1073741824,
      "segment_index_bytes": 10485760,
      "segment_jitter_ms": 0,
      "segment_ms": 604800000
    }

• Can it cause potential problems for out customers for example when we change default for compression_type from "producer" to "uncompressed"?

Many thanks 🙏

Hi @loosla - you're correct on the default for compression_type, it's a mistake in the PR. I have correct that. I was originally looking at producer compression config that we do not currently support.

[loosla]

Couple more things that you may want to update :pray:

1. Would it be possible to update segment_bytes too? :pray: Now its default in the documentation is 209715200, however API sets it to 1073741824 by default.

2. Some fields are missing:

segment_index_bytes is missing in the documentation, however it is set to 10485760 by default.

segment_bytes is missing in the documentation, however it is set to 1073741824 by default.

message_timestamp_difference_max_ms is missing in the documentation, however it is set to 9223372036854776000 by default.

message_format_version is missing in the documentation, however it is set to 3.0-IV1 by default.

Thank you.