kafka-configurator
kafka-configurator copied to clipboard
Update circe(-yaml) when max aliases is configurable
We use yaml anchors and aliases to reduce duplication for some of our topic configurations. Following #51 this is now broken in a project that is creating lots of topics due to an update in snakeyaml that sets a maximum number of allowed aliases. We see the error below:
ERROR com.sky.kafka.configurator.Main$ - Number of aliases for non-scalar nodes exceeds the specified max=50
Example configuration:
test-topic: &DEFAULT
partitions: 9
replication: 3
config:
cleanup.policy: compact
topic-one: *DEFAULT
topic-two: *DEFAULT
topic-three: *DEFAULT
# Repeats > 50 times
This is configurable in snakeyaml but circe-yaml (which translates between the snakeyaml AST and circe's JSON AST) does not currently give you this option - see circe/circe-yaml#136.
For now we should probably revert the circe update and wait for the linked issue to be resolved.
0.14.0 includes #53 which reverted the change in circe version and so means there is no limit on the maximum number of aliases again.
I think we can leave this issue open to track being able to update to a future version of circe-yaml (when configuring the maximum should be possible).