enhancement request: Native support for Kubernetes livenessProbe, readinessProbe

[jdinsel-xealth]

Is there an existing issue for this?

• [X] I have searched the existing issues

Enhancement description

In Kubernetes, livenessProbe and readinessProbes determine their result based on the HTTP.Status of 200. Thus, when using localstack, the endpoints /_localstack/health and /_localstack/ready will signal that the service is ready even when the startup scripts are still running.

To reproduce, add a lengthy /etc/localstack/init/ready.d/setup.sh script (or a simple one with a sleep 60) to a k8s deployment of localstack. Deploy the localstack container into a cluster and watch the status report. It will be reported as running and ready as soon as the first readiness probe is executed. As mentioned, this is because the HTTP request returned a 200 (Ok) status. It does also return "completed": false, as part of the body, but kubernetes is not able to parse that information.

A partial deployment is included below.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: localstack
  name: localstack
spec:
  template:
    metadata:
      labels:
        app: localstack
    spec:
      containers:
      - name: localstack
        image: localstack/localstack
        ports:
        - name: localstack
          containerPort: 4566
          protocol: TCP
        livenessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
          httpGet:
            path: /_localstack/health
            port: localstack
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
          httpGet:
            path: /_localstack/init/ready
            port: localstack
        volumeMounts:
        - mountPath: /etc/localstack/init/ready.d/01-setup.sh
          name: localstack-ready-scripts
          subPath: 01-setup.sh

🧑‍💻 Implementation

Please consider adding an additional ready query that returns 200 when the check on /_localstack/ready would return "completed": true. It should return a 200 when ready and 503 if the init scripts are still running.

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[thrau]

Hey @jdinsel-xealth, thanks for reporting! Since #10942, this should actually in principle work correctly now. We're first running the ready scripts before we set the runtime to ready: https://github.com/localstack/localstack/blob/44d2462963375a9196686aa2f9d298001b2cf91d/localstack-core/localstack/runtime/runtime.py#L120-L122

Compared to what we did previously, which was the other way around: https://github.com/localstack/localstack/blob/44d2462963375a9196686aa2f9d298001b2cf91d/localstack-core/localstack/services/infra.py#L307-L312

That said, the health endpoint currently doesn't care about runtime readiness at all at the moment, so we'd have to properly expose the ready state somewhere.

[localstack-bot]

Hello 👋! It looks like this issue hasn’t been active in longer than five months. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.