workflow icon indicating copy to clipboard operation
workflow copied to clipboard

Secret Env Not available, container couldn't start

Open Cryptophobia opened this issue 7 years ago • 3 comments

From @pixeleet on February 14, 2017 9:13


25m       15h       9         api-cmd-4281948355-qecb1   Pod       spec.containers{api-cmd}   Normal    Pulling      {kubelet ip-10-0-10-170.eu-central-1.compute.internal}   pulling image "bannerwise/core:production-735dd428f31808e52bd422e6e54f5d78bddde289"
25m       15h       9         api-cmd-4281948355-qecb1   Pod       spec.containers{api-cmd}   Normal    Pulled       {kubelet ip-10-0-10-170.eu-central-1.compute.internal}   Successfully pulled image "bannerwise/core:production-735dd428f31808e52bd422e6e54f5d78bddde289"
25m       31m       8         api-cmd-4281948355-qecb1   Pod                                  Warning   FailedSync   {kubelet ip-10-0-10-170.eu-central-1.compute.internal}   Error syncing pod, skipping: failed to "StartContainer" for "api-cmd" with RunContainerError: "GenerateRunContainerOptions: secrets \"api-v115-env\" not found"

What happens is, one of my containers died, a new machine which hasn’t pulled the image yet tried to pull but it somehow didn’t have the latest env secret and thus the container could not start, eventually this resulted in all api containers being not available, still investigating why but if you know something please let me know :stuck_out_tongue: It’s an annoying little thing happening every now and then and I till can’t put my finger on it

Kube version

Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:38:31Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Deis version

v2.4.0

Cloud Provider: AWS 4.7.3-coreos-r2

Copied from original issue: deis/workflow#736

Cryptophobia avatar Mar 20 '18 20:03 Cryptophobia

From @bacongobbler on March 28, 2017 19:58

@pixeleet can you please try upgrading to something more recent like v2.13.0? There have been plenty of significant changes/bugfixes in the last 11 months and we haven't seen this in recent versions. I'm inclined to think this is long past, and I'd try upgrading first.

Cryptophobia avatar Mar 20 '18 20:03 Cryptophobia

From @bacongobbler on March 28, 2017 19:59

Sorry this ticket fell through the cracks, by the way. We do appreciate the bug report!

Cryptophobia avatar Mar 20 '18 20:03 Cryptophobia

From @pixeleet on March 28, 2017 20:2

Sure, already put upgrading to the latest and greatest deis on the roadmap. It's rarely occuring anyway. On Tue, 28 Mar 2017 at 21:59, Matthew Fisher [email protected] wrote:

Sorry this ticket fell through the cracks, by the way. We do appreciate the bug report!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/deis/workflow/issues/736#issuecomment-289886394, or mute the thread https://github.com/notifications/unsubscribe-auth/AFgmZVbKSWOogw8xNTbhEHutTIJdI5vzks5rqWaMgaJpZM4MAOfX .

Cryptophobia avatar Mar 20 '18 20:03 Cryptophobia