microservices-demo icon indicating copy to clipboard operation
microservices-demo copied to clipboard

Base resource requests on actual usage

Open NimJay opened this issue 1 year ago • 8 comments

Background

  • This fixes https://github.com/GoogleCloudPlatform/microservices-demo/issues/591.
  • I looked at a deployment of Online Boutique from 4 days ago. I noted the max (actually used) vCPU and memory values of each microservice — within the full 4 days. (The deployment included the loadgenerator.) Olivier did a similar analysis in 2022.
  • For each value, I rounded up to the next 10m (for CPU) and the next 10MiB for memory — to add a buffer.

Change Summary

  • I've updated the resource.requests.cpu and resource.requests.memory values of each default microservices's Deployment.

Testing Procedure

  • We just have to make sure the staging URL works fine. :)

NimJay avatar May 10 '24 23:05 NimJay

🚲 PR staged at http://34.42.127.234

github-actions[bot] avatar May 10 '24 23:05 github-actions[bot]

🚲 PR staged at http://34.42.127.234

github-actions[bot] avatar May 11 '24 00:05 github-actions[bot]

I have tested the new Kubernetes manifests on the regular channel and rapid channel (see delete-me-test-resources-new and delete-me-test-resources-new-rapid below). The new manifests work. Screenshot 2024-05-13 at 11 32 42 AM

However, I believe, on GKE autopilot, this only shaved off 0.25 vCPUs because the Managed Service for Prometheus is enabled by default on GKE autopilot and requires a lot (relative to Online Boutique) of resources.

Screenshot 2024-05-13 at 11 34 52 AM Screenshot 2024-05-13 at 11 36 30 AM

NimJay avatar May 13 '24 15:05 NimJay

🚲 PR staged at http://34.42.127.234

github-actions[bot] avatar May 14 '24 14:05 github-actions[bot]

🚲 PR staged at http://34.42.127.234

github-actions[bot] avatar Jun 02 '24 17:06 github-actions[bot]

FYI: I just deployed this in my own GKE Autopilot cluster, and I'm getting these values, k top pods -n onlineboutique-development:

NAME                                    CPU(cores)   MEMORY(bytes)
adservice-bc9667f7c-cdbrd               254m         128Mi
cartservice-69754cfcc9-pvf88            7m           89Mi
checkoutservice-856c5f7f68-cwnnn        84m          49Mi
currencyservice-66cc88ff55-5mk9l        11m          74Mi
emailservice-7d69bb6774-cvx2b           14m          81Mi
frontend-7fcf78c9bd-nxrz8               4m           50Mi
loadgenerator-76f787d859-nthg6          12m          142Mi
paymentservice-7495f4f8d9-b94rh         4m           66Mi
productcatalogservice-5dffd678c-8b7nn   5m           51Mi
recommendationservice-ff8fcbf9d-cgbjz   6m           91Mi
redis-575569978c-2z5fh                  6m           40Mi
shippingservice-7b6764896f-9zhd5        5m           47Mi

In addition to the comment on the recommendationservice, I'm getting these restarts because of memory limits too restrictive:

adservice-bc9667f7c-cdbrd               1/2     CrashLoopBackOff   115 (72s ago)   5h39m
checkoutservice-856c5f7f68-cwnnn        2/2     Running            1 (7m50s ago)   5h39m
currencyservice-66cc88ff55-5mk9l        2/2     Running            5 (62m ago)     5h39m```

Should we have:

  • For adservice: 150Mi for memory limits? and 150m for cpu limits?
  • For checkoutservice: 80Mi for memory limits?
  • For currencyservice: 90Mi for memory limits?

mathieu-benoit avatar Jun 03 '24 18:06 mathieu-benoit

🚲 PR staged at http://34.42.127.234

github-actions[bot] avatar Jun 04 '24 14:06 github-actions[bot]

@NimJay do you remember where this PR was left off?

bourgeoisor avatar Sep 10 '24 20:09 bourgeoisor