The future of this project/Wishlist

[luxas]

I want this to be a kind of tracking issue for things the community (you) want from this project. Note: I do not promise to make all features one may wish for/propose, but I would like to know what you think.

Also, feel free to comment here if you are using this project and it is working, that will probably give more fuel to this project :)

I'm working on merging this functionality into mainline Kubernetes, and for that feedback would be greatly appreciated, so the Kubernetes guys know which features count. Here is the ARM tracking issue: https://github.com/kubernetes/kubernetes/issues/17981

[nsteinmetz]

Hi,

Not sure it's doable/feasible as I'm not completely familar with k8s yet:

• Could we access the pods not only via the master to avoid the SPOF ? I saw some presentation of Openshift and it seemd we could expose API or proxy on several nodes
• A UI could be interesting too - I know it's in your roadmap
• Monitoring topic (at master / node / pods level)
• ...

My 2 first cents for this project for 2016 ;-)

[luxas]

Haha, your typo of Openshift was a good one. Well, I guess you mean this doc. That would be great to have here is a lite-variant which I am going to try to implement in future versions. I wanted it for v0.6.0, but I gave up with it. It just didn't work.

The UI is probably coming soon

Monitoring, I will investigate if the Heapster addon is worth running on Pis, I think it's quite resource-heavy. It isn't worth it if it just outputs that it takes all resources ;) However, it would be nice to have!

[nsteinmetz]

Oops!

Yes, I mean routers ; I didn't remember the name. service-loadbalancer seems indeed an alternative. HA Proxy seems to be everywhere now ; would have thought about something lighter like a basic nginx (don't know much HA Proxy but seems overkill for my needs)

For the monitoring, there is indeed the resource issue you mention.

[luxas]

@nsteinmetz Check out v0.6.3! Released today. It's most bugfixes and such, but it may be useful.

[lohmander]

Regarding the monitoring, my PR (https://github.com/kubernetes/dashboard/pull/232) to the dashboard might be a good option since it just uses the cAdvisor instance already running on each node to poll usage metrics. Just sayin...

[luxas]

Works fine for me if you rebase your PR to the released beta version when it's released. So we'll have the "official" beta and your features on top of it

[luxas]

FYI, there's an official ARM etcd image for Kubernetes now :) kubernetes/kubernetes#19216 was pushed some days ago Wanna try? docker pull gcr.io/google_containers/etcd-arm:2.2.1

Soon, I think kubernetes/kubernetes#19769 will be merged, and then we'll get official binaries and some images for every release >= 1.2.0-alpha.7

So although there hasn't been so much commits on this project lately, ARM support is moving fast forward :+1:

RancherOS has announced they are working on ARM support: rancher/os#735 When it's working (probably a month away), this project will include it

[ibuildthecloud]

Just FYI. What we are working on at Rancher is first RancherOS, then Rancher (our full container platform), and finally k8s. In the end we want Rancher running Kubernetes on RancherOS on ARM (specifically ARM64 servers, but rpis will work too). We expect this to take at least another 2-3 months. But I'm watching this project closely.

[luxas]

@ibuildthecloud Directly when we have RancherOS support for ARM, we'll be able to run Kubernetes on ARM and ARM64: kubernetes/kubernetes#19769 (I hope we're gonna merge this before v1.2) and kubernetes/kubernetes#17981 Of course, it will take some time to build up a stable rootfs, but it should be possible to run Kubernetes on ARM in docker from the first boot of RancherOS on ARM

BTW, I made a PR some time ago to enhance Kubernetes setup on RancherOS: kubernetes/kubernetes#19109 and kubernetes/kubernetes#19193. It was so frustrating to not be able to download kubectl from the release page for controlling Kubernetes on RancherOS. I'm also making the k8s-in-docker method better (e.g. now serviceAccounts are working)

[luxas]

Well, v0.6.5 is here!

And things have evolved on the mainline side too. kubernetes/kubernetes#19769 is merged, and that means that there's no need to build k8s binaries no more... I will probably do it for some future releases though

We will probably have linux/arm64 (and linux/ppc64le) binaries available in v1.3.0-alpha.1, because in v1.3 they'll use go1.5 that has support for those new platforms.

On the dashboard side, I will use official ARM builds only (gcr.io/google_containers/kubernetes-dashboard-arm:version). Sorry @lohmander, but that's the best way.

RancherOS is running on my Pi 2, and I'll work with @ibuildthecloud to get things working OOTB

I'll start working on v0.7.0 of this project now, which probably will include many, many changes. Some technical things I have in mind:

• Kubernetes 1.2.0
• go1.5.3 or go1.6, registry v2.3.0, etcd v2.3.0 or v2.2.5
• flannel VXLAN backend, try to switch to iptables proxying again (hopefully fixed in v1.2)
• build flannel statically => minimize image size
• get rid of --containerized => more like the "native" kubelet
• experimental heapster running alongside dashboard
• see if RancherOS on ARM is ready for this project

But a non-technical thing would be to include some kind of catalog with prebuilt apps that are ready-to-run. E.g. only type kube-config run-app gogs or something.

Right now I have two apps running on my local cluster, gogs image updated 24.1 and owncloud image updated 27.1 Test them out if you want, and suggest more apps that we may run! @nsteinmetz @w0mbat @saturnism @larmog @kyletravis @lavvy @sokoow @gyoho @TimCook1

[lavvy]

Thumbs up @Lucas . One question : Could the app catalog have a gui fronted wit a git repo backend. I think it will make so much sense, it may have to integrate with dashboard, though I don't know how you are planning it On Feb 15, 2016 10:05 PM, "Lucas Käldström" [email protected] wrote:

Well, v0.6.5 is here!

And things have evolved on the mainline side too. kubernetes/kubernetes#19769 https://github.com/kubernetes/kubernetes/pull/19769 is merged, and that means that there's no need to build k8s binaries no more... I will probably do it for some future releases though

We will probably have linux/arm64 (and linux/ppc64le) binaries available in v1.3.0-alpha.1, because in v1.3 they'll use go1.5 that has support for those new platforms.

On the dashboard side, I will use official ARM builds only ( gcr.io/google_containers/kubernetes-dashboard-arm:version). Sorry @lohmander https://github.com/lohmander, but that's the best way.

RancherOS is running on my Pi 2, and I'll work with @ibuildthecloud https://github.com/ibuildthecloud to get things working OOTB

I'll start working on v0.7.0 of this project now, which probably will include many, many changes. Some technical things I have in mind:

• Kubernetes 1.2.0
• go1.5.3 or go1.6, registry v2.3.0, etcd v2.3.0 or v2.2.5
• flannel VXLAN backend, try to switch to iptables proxying again (hopefully fixed in v1.2)
• build flannel statically => minimize image size
• get rid of --containerized => more like the "native" kubelet
• experimental heapster running alongside dashboard
• see if RancherOS on ARM is ready for this project

But a non-technical thing would be to include some kind of catalog with prebuilt apps that are ready-to-run. E.g. only type kube-config run-app gogs or something.

Right now I have two apps running on my local cluster, gogs image updated 24.1 https://hub.docker.com/r/luxas/gogs/ and owncloud image updated 27.1 https://hub.docker.com/r/luxas/owncloud/ Test them out if you want, and suggest more apps that we may run! @nsteinmetz https://github.com/nsteinmetz @w0mbat https://github.com/w0mbat @saturnism https://github.com/saturnism @larmog https://github.com/larmog @kyletravis https://github.com/kyletravis @lavvy https://github.com/lavvy @sokoow https://github.com/sokoow @gyoho https://github.com/gyoho @TimCook1 https://github.com/TimCook1

— Reply to this email directly or view it on GitHub https://github.com/luxas/kubernetes-on-arm/issues/40#issuecomment-184393255 .

[larmog]

Great Job Lucas! I installed v0.6.5 yesterday on to v0.6.3. and hypriotos. The Docker upgrade took some time on each node, which made the docker service start timeout.

I’ve tried the new dashboard with heapster (larmog/heapster-armhf image) but something is broken in cadvisor and docker subcontainers. I found some issue reports from people running debian (on x86_64) with the same problem. I guess we have to wait for 1.2 release.

Have you had a look at https://helm.sh/ package manager for Kubernetes?

I’m running Gogs, Drone and mysql on a 6 node mixed Pi cluster using NFS volumes. I’m not happy with the NFS setup so I will try to add some nodes and setup glusterfs.

I really would like to try out RancherOS if I can find the time.

I’ll try to help if there’s anything I can do. I’m more of a dev guy than ops, but I’m learning :)

/Regards

15 feb. 2016 kl. 22:05 skrev Lucas Käldström [email protected]:

Well, v0.6.5 is here!

And things have evolved on the mainline side too. kubernetes/kubernetes#19769 https://github.com/kubernetes/kubernetes/pull/19769 is merged, and that means that there's no need to build k8s binaries no more... I will probably do it for some future releases though

We will probably have linux/arm64 (and linux/ppc64le) binaries available in v1.3.0-alpha.1, because in v1.3 they'll use go1.5 that has support for those new platforms.

On the dashboard side, I will use official ARM builds only (gcr.io/google_containers/kubernetes-dashboard-arm:version). Sorry @lohmander https://github.com/lohmander, but that's the best way.

RancherOS is running on my Pi 2, and I'll work with @ibuildthecloud https://github.com/ibuildthecloud to get things working OOTB

I'll start working on v0.7.0 of this project now, which probably will include many, many changes. Some technical things I have in mind:

Kubernetes 1.2.0 go1.5.3 or go1.6, registry v2.3.0, etcd v2.3.0 or v2.2.5 flannel VXLAN backend, try to switch to iptables proxying again (hopefully fixed in v1.2) build flannel statically => minimize image size get rid of --containerized => more like the "native" kubelet experimental heapster running alongside dashboard see if RancherOS on ARM is ready for this project But a non-technical thing would be to include some kind of catalog with prebuilt apps that are ready-to-run. E.g. only type kube-config run-app gogs or something.

Right now I have two apps running on my local cluster, gogs image updated 24.1 https://hub.docker.com/r/luxas/gogs/ and owncloud image updated 27.1 https://hub.docker.com/r/luxas/owncloud/ Test them out if you want, and suggest more apps that we may run! @nsteinmetz https://github.com/nsteinmetz @w0mbat https://github.com/w0mbat @saturnism https://github.com/saturnism @larmog https://github.com/larmog @kyletravis https://github.com/kyletravis @lavvy https://github.com/lavvy @sokoow https://github.com/sokoow @gyoho https://github.com/gyoho @TimCook1 https://github.com/TimCook1 — Reply to this email directly or view it on GitHub https://github.com/luxas/kubernetes-on-arm/issues/40#issuecomment-184393255.

[luxas]

@lavvy No it won't have an UI. It's just some .yaml template files.

@larmog Yeah, the delay between docker-1.9 to docker-1.10 is expected. As you may see, I've uploaded heapster to dev and I'm testing it now, so it integrates with dashboard. Maybe we'll have to wait for v1.2 and a newer dashboard release.

Yeah, I've heard/read about helm, but haven't used it. It's an interesting project (but only for x86_64 :smile:)

I have also thought about glusterfs, but haven't had time to convert it into ARM yet. Probably v0.7.5-ish if I do it for this project.

RancherOS was a lot of hacking, and had so many rough edges that I'm waiting for rancher/os#760 before I really start to think about it for this project.

I've read your blog posts, and they're very nice. Haven't tried drone yet. Interesting. Keep up the good work :)

[lavvy]

+1 billion for glusterfs in this project @Lucas :) On Feb 19, 2016 6:37 AM, "Lucas Käldström" [email protected] wrote:

@lavvy https://github.com/lavvy No it won't have an UI. It's just some .yaml template files.

@larmog https://github.com/larmog Yeah, the delay between docker-1.9 to docker-1.10 is expected. As you may see, I've uploaded heapster to dev and I'm testing it now, so it integrates with dashboard. Maybe we'll have to wait for v1.2 and a newer dashboard release.

Yeah, I've heard/read about helm, but haven't used it. It's an interesting project (but only for x86_64 [image: :smile:])

I have also thought about glusterfs, but haven't had time to convert it into ARM yet. Probably v0.7.5-ish if I do it for this project.

RancherOS was a lot of hacking, and had so many rough edges that I'm waiting for rancher/os#760 https://github.com/rancher/os/issues/760 before I really start to think about it for this project.

I've read your blog posts, and they're very nice. Haven't tried drone yet. Interesting. Keep up the good work :)

— Reply to this email directly or view it on GitHub https://github.com/luxas/kubernetes-on-arm/issues/40#issuecomment-186064402 .

[larmog]

Hi, +1 for "get rid of --containerized => more like the "native" kubelet"

@lavvy I'm now running glusterfs 3.5.2-2 instead of NFS and it just works: http://bit.ly/1XJRAJO

[lavvy]

Wow, real good. I wish @Lucas can support this upstream. It appears so much technical for us noobs. I believe persistent storage is so necessary especially for private cloud infrastructure like k8s on arm. On Feb 24, 2016 4:18 PM, "Lars Mogren" [email protected] wrote:

Hi, +1 for "get rid of --containerized => more like the "native" kubelet"

@lavvy https://github.com/lavvy I'm now running glusterfs 3.5.2-2 instead of NFS and it just works: http://bit.ly/1XJRAJO

— Reply to this email directly or view it on GitHub https://github.com/luxas/kubernetes-on-arm/issues/40#issuecomment-188302668 .

[luxas]

@larmog Thanks for your work again! I read your post yesterday, and it was amazing! I'm gonna investigate it more, but now I have a rough idea what it looks like thanks to your work.

+1 for "get rid of --containerized => more like the "native" kubelet"

I'm gonna do this when upstream supports it. There are known issues for the time being.

However, all this takes time. Now I'm busy with many things, building v0.7.0 and validating v1.2.0 for ARM, testing everything on RancherOS and porting all this to upstream Kubernetes at the same time, so it will take some time before glusterfs is running smoothly OOTB from this project. But anyway, just follow @larmog's guide and you're done in no time at all! :smile:

@lavvy Please ping me with @luxas. @Lucas is a completely other person which I don't know, and he probably don't want to get these conversations to his email.

[lucas]

Thanks @luxas, much appreciated :)

[lavvy]

Oh am sorry, thanks for hint. Noted On Feb 24, 2016 10:17 PM, "Lucas" [email protected] wrote:

Thanks @luxas https://github.com/luxas, much appreciated :)

— Reply to this email directly or view it on GitHub https://github.com/luxas/kubernetes-on-arm/issues/40#issuecomment-188456849 .

[larmog]

@luxas is there a reason why the flannel config uses udp as Backend instead of host-gw? I've switched to host-gw. See: http://machinezone.github.io/research/networking-solutions-for-kubernetes/

[luxas]

No there isn't a specific reason, it was just the default. I planned to include vxlan, but I'll switch to host-gw as the default thanks to your performance measuring link.

[larmog]

Sorry @luxas my fault, I was too quick when I said that I got it working. I checked the logs this morning and flannel still uses udp. I'll try to put some more work into it, and maybe create an issue to track it.

[luxas]

Progress! Now I've got Kubernetes v1.2.0-alpha.8 running on my Pis. I'm using the official binaries I've built in kubernetes/kubernetes#19769. Also heapster is somehow working.

I'm using hostgw as @larmog suggested, and flanneld is built statically, resulting in 72MB smaller kubernetesonarm/flannel image. Yay!

[luxas]

+1 for "get rid of --containerized => more like the "native" kubelet"

@larmog (and others) Please test this:

$ docker run -d --net=host kubernetesonarm/etcd
$ docker run \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:rw \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:shared \
    --volume=/var/run:/var/run:rw \
    --net=host \
    --pid=host \
    --privileged=true \
    -d \
    kubernetesonarm/hyperkube \
    /hyperkube kubelet \
        --pod_infra_container_image=kubernetesonarm/pause \
        --hostname-override="127.0.0.1" \
        --address="0.0.0.0" \
        --api-servers=http://localhost:8080 \
        --config=/etc/kubernetes/manifests-multi \
        --cluster-dns=10.0.0.10 \
        --cluster-domain=cluster.local \
        --allow-privileged=true --v=2

This spins up a single node cluster (do kube-config disable && kube-config delete-data first). Of course, you may hack the /usr/lib/systemd/system/k8s-master.service too.

Test mounting a secret inside a pod, using downward-api and such things. If these test passes, I'll switch to this more "native" approach in v0.7.0. It requires docker-1.10+, but since v0.6.5 all machines should run docker-1.10

[luxas]

Yeah, now it works! Got native kubelet support with: aefa2bca0a743edaf60c30f946ce2d231b4247bc

[larmog]

@luxas great, I had problem mounting secrets with your first suggested solution.

I'll test if:

-v /var/lib/kubelet:/var/lib/kubelet:rw \
 + -v /var/lib/kubelet:/var/lib/kubelet:shared \

makes a change

[larmog]

I ran some tests and it looks great. Seems that a bunch of errors from when unmounting and removing pods is gone. I saw that it's still the 0.6.2 image, so I'll restart all nodes in "native" mode. I'll even try to enable heapster in dashboard and see how it looks. Awesome job @luxas :D

[luxas]

Yes, I haven't published newer images to Docker Hub than v0.6.2, cause not much has changed. But soon, when v1.2 is released, we'll have brand new images!

[DorianGray]

Hey all, joining the party late. Can we start documenting each thing that needs to get done as a separate issue? I'd love to help with implementation and testing.

I'm planning on getting this running on odroid c2's personally, also have rasp pi 2's for testing.

[luxas]

And there we go! v0.7.0 released. See the release notes Please test it and give feedback :smile: