layotto icon indicating copy to clipboard operation
layotto copied to clipboard

bookinfo部署问题

Open LXPWing opened this issue 2 years ago • 15 comments

环境

preview server k8s yaml: https://github.com/LXPWing/preview-server-k8s-yml

线上实验室配置文件: https://github.com/LXPWing/mosn-tutorial/tree/master/layotto/layotto-with-bookinfo

bookinfo k8s yaml: https://github.com/LXPWing/layotto/tree/main/deploy/k8s/bookinfo

线上实验室演示:https://killercoda.com/lixingpeng/course/layotto/layotto-with-bookinfo

问题

preview server与bookinfo都部署在default namespace下, preview server yml中设置replicas: 1但实际部署中会超出1个pod且报错,同时preview server pod中的镜像数量会比yml中设置的多1个。bookinfo 的一些pod也会出现超出设置数量且报错。

image

尝试解决

preview server和bookinfo的namespace分离,但是preview serve一直处于ContainerCreating

image

LXPWing avatar Aug 16 '22 02:08 LXPWing

Try:

kubectl describe pods pod_name -n layotto

Then we saw the error events: image

For more details, see https://serverfault.com/questions/728727/kubernetes-stuck-on-containercreating

seeflood avatar Aug 26 '22 02:08 seeflood

It's ok now. image

I fixed it by adding a namespace for the configmap: image

But we have new problems now:

image

I tried k describe pods productpage-v1-6564fc549f-w6zl7: image

seeflood avatar Aug 26 '22 02:08 seeflood

@LXPWing You can try:

  • run k8s with more nodes
  • increase the resource limit

看着是资源不足,可以换成更多节点的k8s试试,以及调高对容器占用资源的限制 我先去开会了……

seeflood avatar Aug 26 '22 03:08 seeflood

Looks like we can't run K8s with more than 2 nodes in killercoda :( see https://killercoda.com/creators image

You can try to solve the problem by raising the limit on resources. If it doesn't solve the problem, we have to deploy the preview-server and the storage systems in other environment instead of in the lab. For example, we only deploy the preview-server and redis on alibaba-cloud and aws

seeflood avatar Aug 26 '22 09:08 seeflood

Currently using "kubernetes-kubeadm-2nodes" store can be deployed in the lab but preview cannot be deployed

LXPWing avatar Aug 26 '22 09:08 LXPWing

May need to use other cloud servers

LXPWing avatar Aug 26 '22 09:08 LXPWing

@LXPWing OK. Let me deploy it on alibaba-cloud first.

seeflood avatar Aug 26 '22 11:08 seeflood

@LXPWing I'm thinking about deploying the latest binary automatically in our CI/CD. Are u interested in doing some research on the solution? This is not necessary, it doesn't matter if we can't implement it

seeflood avatar Sep 06 '22 11:09 seeflood

@seeflood I don't understand what this “the latest binary automatically in our CI/CD.”

LXPWing avatar Sep 06 '22 11:09 LXPWing

I mean: Currently, after a new PR get merged into the main branch, our github workflow will compile the code into a binary, and then upload the docker images and binary files to github. For example, you can see the uploaded binary files in https://github.com/mosn/layotto/actions/runs/2924861106 image

That's good. I'm thinking about making it better. For example, after uploading the images and binary files, the tools can notify our preview server, and the preview server will download the new docker image (or binary file) and then deploy it. I guess there are already tools for it ( they call it "continuous deployment", a.k.a CD)

seeflood avatar Sep 06 '22 12:09 seeflood

Will jenkins be used?

LXPWing avatar Sep 06 '22 12:09 LXPWing

@LXPWing We don't have Jenkins yet :( It's better that we achieve this goal using github action. It's just an idea. As the first step, we only need to do some research on "whether there are existing tools to do it". If not, or if the tools are too complex to use, we give up :)

seeflood avatar Sep 06 '22 12:09 seeflood

“the tools can notify our preview server”github action may not be completed.stop containers -> delete image -> pull new image -> start Containers may jenkins can do it.

LXPWing avatar Sep 06 '22 12:09 LXPWing

@LXPWing Deploying jenkins sounds complex 😢 I hope there's some "agent" on the server, which:

  • listen to the webhook events
  • download image
  • remove the existing container
  • start a new container

If there isn't a simple solution, let's give up and do something more important...

seeflood avatar Sep 06 '22 12:09 seeflood

https://github.com/layotto/java-sdk/pull/35 @LXPWing I added startup & stop scripts for preview service. But we still need several things to deploy preview service, including:

  • correct config.json file to use redis as state store and local file system as file component.
  • initialize the data in the storage system image

seeflood avatar Sep 09 '22 03:09 seeflood

This issue has been automatically marked as stale because it has not had recent activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue or help wanted) or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Oct 09 '22 03:10 github-actions[bot]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue or help wanted. Thank you for your contributions.

github-actions[bot] avatar Oct 16 '22 04:10 github-actions[bot]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue or help wanted. Thank you for your contributions.

github-actions[bot] avatar Oct 24 '22 04:10 github-actions[bot]