scio
scio copied to clipboard
spotify
Consider using another configuration mechanism other than properties
It seems like scio uses System Properties for some configurations.
Motivated by reasons listed here https://medium.com/@hussachai/configurations-are-you-doing-it-wrong-e4ea2672a4a9 , I would like to ask whether we should consider supporting configuration via another mechanism.
Unless I am missing something, none of the cons of this article affect Scio since https://github.com/spotify/scio/pull/1404 was merged.
Ah, interesting. Thanks for the clarification @jto !
In some cases, it would also be good to have these configurations defined in a file. Do you know a way to do that?
Maybe it would be good to have a short section about configuration (with some pointers/references) to clarify basics of scio properties.
FYI internally in Spotify we have been hacking on pipeline-config project. It uses HOCON configuration files to configure Beam's PipelineOptions.
If we find that project valuable, we will consider in the future to open source it. Still just want to provide an update to this issue and check if there is more interest on this.