incubator-heron
incubator-heron copied to clipboard
add command line arg to CheckpointManager to take overridden config
Currently, there is no command line argument that CheckpointManager supports to take in the heron-conf/override.yaml
, which holds all the configuration property that user specifies, for example, via -D
property argument to the Heron API server deployment.
This means that users who deploy Heron on Kubernetes will be locked to org.apache.heron.statefulstorage.localfs.LocalFileSystemStorage
as the storage backend of any stateful topology. But, local filesystem is ephemeral in Docker container, therefore it defeats the purpose of a stateful topology, since all the states get lost when the topology gets re-submitted and its pods get recreated.
User can work around this issue by creating their own Heron docker image and override the configuration file, but that is static approach, which won't work well because certain configuration varies from deployment to deployment, and is unknown until deploy time, such as the DistributedLog namespace including zookeeper service/url.
So, the better solution is to make CheckpointManager aware of the override.yaml
via CLI argument, and Executor can specify it when it starts the CheckpointManager process in each Heron Container Pod.