Checkpointing to s3 using minio doesn't work

[lkokhreidze]

Hello @alpinegizmo

I've tried to use your example to have checkpointing in s3 using minio. I'm using docker compose setup and what I've noticed that checkpointing doesn't really and I see following in job manager logs:

2020-06-04 19:22:19,450 INFO  com.amazonaws.latency                                         - ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://minio-service:9000], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 16156D972F7BCCCA; S3 Extended Request ID: null), S3 Extended Request ID: null], RequestType=[GetObjectMetadataRequest], AWSRequestID=[16156D972F7BCCCA], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[3.172], HttpClientSendRequestTime=[0.113], HttpRequestTime=[2.379], RequestSigningTime=[0.294], CredentialsRequestTime=[0.003, 0.001], HttpClientReceiveResponseTime=[0.675],
2020-06-04 19:22:19,454 INFO  com.amazonaws.latency                                         - ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://minio-service:9000], RequestType=[ListObjectsRequest], AWSRequestID=[16156D972FA9FC2F], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.077], ClientExecuteTime=[3.384], HttpClientSendRequestTime=[0.104], HttpRequestTime=[1.511], RequestSigningTime=[0.289], CredentialsRequestTime=[0.001, 0.001], HttpClientReceiveResponseTime=[0.967],
2020-06-04 19:22:22,419 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 33 @ 1591298542418 for job 44a454e5e0a555403eb80ef36e2dc375.
2020-06-04 19:22:32,420 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Checkpoint 33 of job 44a454e5e0a555403eb80ef36e2dc375 expired before completing.

Btw, in order to pick up the flink-conf.yaml file I had to add following to docker-compose:

volumes:
      - ./flink/flink-conf.yaml:/opt/flink/conf/flink-conf.yaml

Any pointers on the error above?

Thanks in advance