Cannot set the `kafka.schemaRegistry.urls` with environment variable (KAFKA_SCHEMAREGISTRY_URLS)

[joseph-isaacs]

Hey,

Firstly love the tool.

I using v1.4.0 of kowl. I have managed to open and use kowl with schema registry set in the config file.

kafka:
  brokers:
    - localhost:9092
  schemaRegistry:
    enabled: true
    urls: [http://localhost:8081]

I am now trying to use the environment variables to configure the schema registry (kafka.schemaRegistry.urls: [...]) so that would be KAFKA_SCHEMAREGISTRY_URLS correct? export KAFKA_SCHEMAREGISTRY_URLS="http://localhost:8081". I have managed to use the environment variable KAFKA_BROKERS to set the broker (removing the brokers from the config file).

Any help would be appreciated.

Thanks Joe

[weeco]

Hi @Curdled , yes you'd need to export the env variables KAFKA_SCHEMAREGISTRY_URLS and also enable schema registry via KAFKA_SCHEMAREGISTRY_ENABLED=TRUE

Are you facing an issue with that? I'm unsure what problem you are facing

[joseph-isaacs]

Hey thanks for getting back to me to fast!! @weeco.

When I have the config .yaml file as below and the env variables KAFKA_SCHEMAREGISTRY_URLS=http... and KAFKA_SCHEMAREGISTRY_ENABLED=TRUE

kafka:
  brokers:
    - localhost:9092
  schemaRegistry:
    enabled: true
  protobuf:
    enabled: true
    schemaRegistry:
      enabled: true
      refreshInterval: 1m

I get the response

$ ./kowl -config.filepath kowl.yaml
{"level":"fatal","msg":"failed to validate config","error":"failed to validate Kafka config: schema registry is enabled but no URL is configured"}

however removing the 3rd and 4th lines of the yaml corrects the problem.

It seems like having kafka.schemaRegistry.enabled without .urls set (in the yaml) and urls set through env variable ..._URLS doesn't work. Which is where the confusion occurred.

Thanks, this does solve my problem. Joe

[weeco]

Hey Joe, ah that makes sense. I agree that this might be a bit unexpected, I'd expect a strictly additive config (if multiple options are used) as well. I think a very common use case is that secrets are set via env variables while the rest of the config may be in YAML.

We are using Koanf for the configuration management (see: https://github.com/cloudhut/kowl/blob/2991089d0a3e7271b395b14263502477a826f6ce/backend/pkg/api/config.go#L103-L145), and I might need to take a look at the UnmarshalConf, maybe it is configurable how multiple configurations from different sources shall be merged.

[weeco]

This should be fixed already in the latest release