cloudflow icon indicating copy to clipboard operation
cloudflow copied to clipboard

Checkpointing configurations are not taking effect for flink streamlet

Open JustinPihony opened this issue 3 years ago • 0 comments

The below checkpointing configurations are not taking effect when tried to configure enabling the Rocksdb backend with Azure blob storage .

cloudflow.runtimes.flink.config {
  flink.state.backend = filesystem
  flink.state.checkpoints.dir = "wasbs://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>"
  flink.fs.azure.account.key.<your-azure-acount>.blob.core.windows.net = "<blob-access-key>"
}

The current workaround is to do it via code:

getContext().env().setStateBackend(new EmbeddedRocksDBStateBackend(true));
getContext().env().getCheckpointConfig().setCheckpointStorage("wasbs://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>");
getContext().env().getCheckpointConfig().setCheckpointStorage(new FileSystemCheckpointStorage(file:///tmp/checkpoint-dir));

JustinPihony avatar Oct 05 '22 16:10 JustinPihony