cloud.weave.works is down + missing page in weave documentation

[jpetazzo]

Hi!

I noticed that cloud.weave.works was down. (According to https://downforeveryoneorjustme.com/cloud.weave.works?proto=https, it's not just me, at least at the moment.)

This is (or at least, was) mentioned as the method of choice to install Weave on Kubernetes clusters:

kubever=$(kubectl version | base64 | tr -d '\n')
kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=$kubever

It worked a few days ago (at least last Monday 26-SEP-2022 according to my logs) but doesn't work today (Monday 03-OCT-2022). I thought - maybe this was decommissioned, and I should update my deployment scripts? But when I go to the Weave documentation (https://www.weave.works/docs/net/latest/kubernetes/), it says "The recommended way of using Weave with Kubernetes is via the new Kubernetes Addon". But the link to "Kubernetes Addon" redirects to the Weave Net overview (https://www.weave.works/docs/net/latest/overview/).

I'm not sure what to do next; I'll fetch a cached copy of the Weave YAML somewhere but I thought it might help others to report this and get it fixed ♥

[clacsinaRX-M]

Encountered the same issue this morning.

I was able to install weave on both a 1.24 and 1.25 cluster using the 1.11 yaml from the release page: https://github.com/weaveworks/weave/releases/tag/v2.8.1

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s-1.11.yaml

That seems like an older version so I'm unsure if that's going to continue being supported, but so far things are working as expected.

[prassurohith]

Thanks @clacsinaRX-M

You saved me its working for me

[jtgorny]

Oof https://www.weave.works/blog/weave-cloud-end-of-service

[fujitatomoya]

Is there any official and up-to-date one instead of https://github.com/weaveworks/weave/issues/3960#issuecomment-1265751492?

[zenhighzer]

Oh no, I also wondered why cloud.weave.works is down.

But if I access "https://www.weave.works/docs/net/latest/kubernetes/kube-addon" I land on the Overview-Page. So actually I cant find any Page showing how to install the kubernetes addon

[monadic]

Hi, we are looking into a workaround if we can provide, or some alternatives. Watch this space.

[lizwarner-weave]

Just confirming that the workaround here https://github.com/weaveworks/weave/issues/3960#issuecomment-1265751492 should work. We are updating our documentation and will comment again on this issue in a bit

[seanmcconkey]

While we're waiting for the fix, do you not think it prudent to get someone in the organisation to update https://www.weave.works/docs/net/latest/kubernetes/kube-addon/ with the fix and a link to the EOS announcement.

[lizwarner-weave]

Updates:

1. We have updated https://www.weave.works/docs/net/latest/kubernetes/kube-addon/ so users are no longer directed to Weave Cloud to obtain Weave Net
2. Please also see https://github.com/weaveworks/weave/issues/3948 if you’re interested in continuing to be involved with Weave Net as a community member
3. Please note our end-of-life page which covers both Weave Cloud and Weave Net

Thanks for your feedback and patience

[kingdonb]

JFYI, this is the top result when searching for "weave net one-liner" so at least 50% of people will probably click this first: https://www.weave.works/blog/weave-net-kubernetes-integration/

I was unable to find this blog page in any of our git repos, or I'd be updating it myself right now, but it still contains the reference to cloud.weave.works – thanks Weaveworks team for handling this issue so quickly already 🙇

[dholbach]

@AhmedAtefWW can you help?

[jpetazzo]

Thanks for updating the documentation, and confirming that the YAML on the GitHub release pages is now the canonical source; that's very much appreciated! ♥

One little remark: the YAML that was just released there a few days ago (weave-daemonset-k8s.yaml) uses the :latest image tag. Personally, I would rather use a pinned version, like in the other manifests in that release page. Pinned versions avoid issues due to unexpected upgrades; they ensure consistent versions across nodes; and they're easier to troubleshoot (because kubectl describe on the DaemonSet or Pod will show you exactly what you're running).

For folks wondering about the differences between the -1.8, -1.9, -1.11 YAML manifests:

• the 1.8 manifest uses a DaemonSet apiVersion extensions/v1beta1
• the 1.9 manifest switches to DaemonSet apps/v1
• the 1.11 manifest adds priorityClassName: system-node-critical to the Weave Pods

There was no change in DaemonSet API or behavior since then, so the 1.11 manifest (in fact, the 1.9 manifest!) will work just fine on Kubernetes 1.25.

Thanks again!

[lucj]

@lizwarner-weave is there a dedicated URL to install weave scope as well ? The installation instructions still refers to cloud.weave.works (https://www.weave.works/docs/scope/latest/installing/#k8s)

[enekofb]

@lizwarner-weave is there a dedicated URL to install weave scope as well ? The installation instructions still refers to cloud.weave.works (https://www.weave.works/docs/scope/latest/installing/#k8s)

@lucj looking into this

[enekofb]

@lucj see the update on scope docs

let me know any feedback at the back of it

[fireflycons]

@monadic Any chance of creating a helm chart? Would be useful to be able to configure things like IPALLOC_RANGE

Would you accept a PR for one?

[monadic]

Sure please email me about this Alexis at Weave dot works

[jpetazzo]

One little remark: the YAML that was just released there a few days ago (weave-daemonset-k8s.yaml) uses the :latest image tag. Personally, I would rather use a pinned version, like in the other manifests in that release page. Pinned versions avoid issues due to unexpected upgrades; they ensure consistent versions across nodes; and they're easier to troubleshoot (because kubectl describe on the DaemonSet or Pod will show you exactly what you're running).

... And indeed, using the :latest version brought some unexpected problems; it looks like the :latest tag isn't properly multiarch:

ubuntu@node1:~$ uname -a
Linux node1 5.15.0-1021-oracle #27-Ubuntu SMP Fri Oct 14 20:04:20 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@node1:~$ sudo docker run --entrypoint sh -ti  weaveworks/weave-kube:latest -c "apk -q add file && file weaver"
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
weaver: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=1a29ea0e5d7e33dcee87512f32068e17dae6f78c, stripped
ubuntu@node1:~$ sudo docker run --entrypoint sh -ti  weaveworks/weave-kube:2.8.1 -c "apk -q add file && file weaver"
weaver: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=892358108af8a6f0d0c5427a34943ebfe471b84c, stripped

I'll open a separate issue to track this.

[kallisti5]

I just ran into this. how lame of weave :cry:

[kingdonb]

I'm pretty sure you can still:

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

as described on: https://www.weave.works/docs/net/latest/kubernetes/kube-addon/

I use Weave Net, but I don't e2e test it daily. If there's an issue that it can't be installed, please cc me as it will be a problem for me sooner or later. But from what I've seen, they've still been taking care of the installation blockers as they come in.

🙏 🙇 📿

[jpetazzo]

Hi @kingdonb ! 👋🏻

I'm not sure which issue @kallisti5 is talking about; but perhaps it's about the multiarch problem described just above (and tracked in #3974) - if you kubectl apply the default YAML on an ARM cluster, it won't work because that YAML is referencing :latest, which isn't multiarch.

Cheers!

[kallisti5]

I'm not sure which issue @kallisti5 is talking about;

Just the shutdown of cloud.weave.works. It was a really nice service to use.

[kallisti5]

It also looks like the last weave release was in 2021. That doesn't bode well for its future...

https://github.com/weaveworks/weave/releases

[kingdonb]

The shutdown of Weave Cloud shouldn't really have had anything to do with Weave Net, but it unfortunately did impact

There are people interested in rekindling Weave Net here:

• https://github.com/weaveworks/weave/issues/3948

It's going to have to come from volunteer effort though, as it always has! There have been hotfixes to keep the one-liner instructions working, and there has been upstream issues that breaking changes impacted Weave Net, that got sorted out ultimately, and there are people who are enthusiastic about keeping Weave Net going, but it's definitely been on life support!

[fujitatomoya]

i would like to revise https://github.com/weaveworks/weave/issues/3960#issuecomment-1396556847, anyone working to address this issue to support multi-arch image?

[kingdonb]

I would like to see multi-arch get resolved, and I can likely help test it if someone else is driving the effort. 🙏

[jpetazzo]

Hi @kingdonb, all that is needed for multi-arch support is to update the released YAML so that it references tag :2.8.1 instead of :latest (see #3974 for details). I don't know how to update Github releases (it looks like they were done manually, at least the last YAML) otherwise I'd offer a PR. Let me know if that's not clear, I'm happy to provide better details!

[kingdonb]

Those are pretty concise and to the point, thank you for speaking up about it!

I'll see if I can find out who has the keys to push those tags around today. ❤️

[kingdonb]

OK, so this is really bad. I think this means that new clusters on Weave Net that just follow the released instructions cannot have Arm64 nodes without some modifications. This is really bad because, the arm64 support exists, it is there, it's just obscured by some mistake that occurred during triage of the cloud.weave.works website itself sunsetting, with Weave Cloud.

And it can technically be fixed with a single change, but there is no commitment from us at Weaveworks to continue the maintenance of Weave Net distro, as has been echoed from the start of this issue. And there has been a loose formation of a new team, and then the holidays. I'm just repeating the events so far, for anyone that is just joining the conversation.

I think the instructions that are hosted at Weaveworks that "need to be updated" now are actually on the Weaveworks blog or corporate brand site. I don't believe any keys have been handed over to anyone.

So if a new team is being formed that's composed of people who are not at Weaveworks, they're going to need somewhere else to post the instructions that explain how to get at the latest release. I'm not sure if that team has a name. Does anyone know the answer yet?

So is there a GitHub org for New-Weave-Net org yet?

Or has this conversation already been covered somewhere and I need to catch up with the recording? I think the team needs a name first, (and not to put the cart before the horse, but then there is the possibility that more than one competing team springs up... I'd be happy to work with one point of contact or multiple but I'd prefer a single team with at least one committed point of contact that I can name.)

I'm doing the best to facilitate this next step right now without being antagonistic to anyone, please nobody read this note as passive-aggressive or anything else, as I genuinely don't know if there is a new maintainer yet... but I want to help! 🌷

[kingdonb]

And for anyone that's aware of the precarious maintenance status of Weave Net right now, it should not be a problem to make a one-line change to the YAML in order to get access to multi-architecture for their cluster. It is risky because there has been no maintainer, I wouldn't recommend it for production.

So I think the new release team needs to create a website where they will host the instructions for Weave Net going forward, (and I will be glad and volunteer to help with this.)

Is there such a team where the maintainers can be enumerated now, or is it public?

I would like for this to be easy, and for things to just go on working for everyone, but I also want to avoid that someone winds up with Weave Net in production, and a CVE is not addressed, and then it's then Weaveworks who is indirectly responsible (in spite of the license which says we cannot be responsible), because of no maintenance and a poorly managed handover that resulted in incorrect signaling about the current maintenance status.

I'm trying to say, I think the bad news is we cannot "just put it back" from latest even though it's clearly in error. The doc may need to remain in error, and we keep it as incentive for folks to move on to the new maintainership... once it is extant!

I'm happy to be overruled by anyone that has the keys and can actually fix it now, but I'm actually not in a position to make the change requested myself, and I'm doing my best to telegraph the disposition of engineering resources in the company who are in position to trace that down, but have been asked to focus on other stuff.

So as respectfully as possible, I think right now the blocker is: there is no new website from the new maintainer team.