ProgressDeadlineExceeded cannot be extended

[syzmekk]

Pods are getting killed after 100 minutes and marked as Failed progressing. We have some big boys in our environment. Our image is about ~5GB and needs around 90-110 minutes to be up and running (1st deployment). After approximately 100 mins our pod(s) is/are getting deleted without a reason, even though everything inside was going well.

Version

oc v3.11.0+0cbc58b kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO openshift v3.11.69 kubernetes v1.11.0+d4cacc0

Steps To Reproduce

1. Prepare DeploymentConfig with a ~5GB image
2. Set all timeouts to 7200 (2hrs)
3. Deploy it

Current Result

Example from one of the timeouts:

 conditions:
    - lastTransitionTime: '2019-09-27T05:33:28Z'
      lastUpdateTime: '2019-09-27T05:33:28Z'
      message: Deployment config does not have minimum availability.
      status: 'False'
      type: Available
    - lastTransitionTime: '2019-09-27T07:30:32Z'
      lastUpdateTime: '2019-09-27T07:30:32Z'
      message: replication controller "app-wls-1" has failed progressing
      reason: ProgressDeadlineExceeded
      status: 'False'
      type: Progressing

We have been also trying to patch the DC and include it in YAML DC file, but it seems that Openshift is ignoring this spec in YAML (is it available only for deployments?). Patch command returns following output:

$ oc patch dc app --patch='{"spec":{"progressDeadlineSeconds":7200}}'
deploymentconfig.apps.openshift.io/app not patched

Expected Result

Successful deployment without exceeding any deadline. ;)

Additional Information

$ oc get all -o yaml -n szymon-sandbox >> namespace.yml

namespace.yml

$ oc describe rc/app-wls-1

rc.yml

Please, kindly advise. :) Feel free to ask me for any additional info or missing details. Thanks!

[syzmekk]

Another pod randomly deleted after about an hour:

  Type    Reason            Age   From                    Message
  ----    ------            ----  ----                    -------
  Normal  SuccessfulCreate  1h    replication-controller  Created pod: app-wls-10-mtlpz
  Normal  SuccessfulDelete  13m   replication-controller  Deleted pod: app-wls-10-mtlpz

I have been forwarding the logs to a local file and the last entry is:

rpc error: code = Unknown desc = Error: No such container: df6088d60dd12b4d2ff69108b350a66487db07541d1ab45712a06e8ea5e42956

I belive it's not related with the application itself :)

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[syzmekk]

/remove-lifecycle stale /lifecycle frozen