berglas icon indicating copy to clipboard operation
berglas copied to clipboard

CrashLoopBackOff when setting command in my deployments

Open mikejoseph-ah opened this issue 2 years ago • 2 comments

Context

I am trying to use berglas webhook with gke, kubernetes setup worked and tested with sample.yaml, it fetched the secrets, so now I have few helm charts that doesn't use command spec but I added one that shouldn't do anything than trigger berglas ["/bin/sh"]

my _deployment.tpl

{{/*
Default Template for Deployment. All Sub-Charts under this Chart can include the below template.
*/}}
{{- define "helm-adh.deploymenttemplate" }}
{{- $PROJECT_ID := .Values.global.projectId -}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "helm-adh.name" . }}
  labels:
    {{- include "helm-adh.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "helm-adh.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "helm-adh.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.global.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      serviceAccountName: {{ .Values.global.serviceAccount }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "gcr.io/adh-artifactory/{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command: {{ .Values.entryPoint }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          # livenessProbe:
          #   httpGet:
          #     path: /
          #     port: http
          # readinessProbe:
          #   httpGet:
          #     path: /
          #     port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          {{- if .Values.configMap.enabled }}
          envFrom:
            - configMapRef:
                name: {{ include "helm-adh.name" . }}-configmap
          {{- end }}
          env:
            {{- range .Values.secrets }}
            - name: {{ .name }}
              value: sm://{{ $PROJECT_ID }}/{{ .name }}
            {{- end }}
      {{- with .Values.global.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.global.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.global.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
{{- end }}

values.yaml of microservice that doesn't work

replicaCount: 1

nameOverride: "calls-be"
fullnameOverride: ""

entryPoint: ["/bin/sh"]

image:
  repository: calls-be
  pullPolicy: Always
  # Overrides the image tag whose default is the chart appVersion.
  tag: "charts-latest"

service:
  type: ClusterIP
  port: 80
  targetPort: http

resources: {}
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

serviceAccount:
  # Specifies whether a service account should be created
  create: false

# insert here envrionment variables
configMap:
  enabled: true
  data:
    REDIS_HOST: "redie"

secrets: 
  - name: EMR_PASSWORD
  - name: TWILIO_ACCOUNT_SID
  - name: TWILIO_API_KEY_SID
  - name: TWILIO_API_KEY_SECRET
  - name: TWILIO_AUTH_TOKEN

podAnnotations: {}

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

by saying "this doesn't work" I mean that berglas does initiate and fetches the secrets, but the service stuck on CrashLoopBackOff and doesn't run with this settings, the Dockerfile of that container is nginx without ENTRYPOINT

Edit: the namespace that the deployment is on, does have the needed service account

mikejoseph-ah avatar Nov 17 '22 09:11 mikejoseph-ah

berglas exec will run the container command, so if the command is /bin/sh, it will resolve secrets and then start a shell. Without logs, it's very difficult to diagnose.

sethvargo avatar Nov 17 '22 14:11 sethvargo

what can I set the command: for it to resolve and keep docker containers as is? is it not overwriting container's ENTRYPOINT if not included?

Edit: I also don't have any logs, it just crashes

justmike1 avatar Nov 17 '22 15:11 justmike1