scope
scope copied to clipboard
weaveworks
weave scope doesn`t work behind a proxy
Hi,
I execute weave scope behind our proxy on docker.
The UI is hanging with word in the page "Trying to reconnect".
I think it because below log error:
test@ubuntu:~/kube/weavescope$ dclo 6055a77b8b3a
time="2018-06-27T06:05:58Z" level=info msg="publishing to: 127.0.0.1:4040"
--weave=false option.
--weave=false option.
--weave=false option.
--weave=false option.
--weave=false option.
--weave=false option.
--weave=false option.
I have add proxy and no_proxy ENV as below in the docker container: bash-4.3# env NO_PROXY=localhost,127.0.0.1 http_proxy=153.88.253.150:8080 https_proxy=153.88.253.150:8080 HTTPS_PROXY=153.88.253.150:8080 no_proxy=localhost,127.0.0.1 HTTP_PROXY=153.88.253.150:8080
Get information busyboxs wget cant support no_proxy env. Do you think in weavescope docker container should package other edition of wget?
Many thanks.
Does some one face this problem too? Or just me usage weave scope in a lab behind firewall and proxy?
I have the same issue. Is there a way to run weavescope behind a firewall which does not permit access outside the network.
What proxies are you guys using? The proxy needs to support websockets.
Also, it would be helpful to see the console logs of the browser.
when i use burpsuite , it happen again , and burp suite log : attempting to auto select ssl parameters for {ip}
I have the same issue, in Live mode it always showing "Trying to reconnect..." message, in Pause works ok. I'm running it behind ELB and nginx-ingress with websockets enabled, thus following directives are set in weave vhost: proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade;
weavescope is 1.11.0 version k8s is: Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:26:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
I can see the following output in firefox console: Firefox can’t establish a connection to the server at wss://DOMAIN/api/topology/kube-controllers/ws?snapshot=hide&storage=hide&pseudo=hide&namespace=NAMESPACE&t=5s. app-8bbac71a57f56a6fd775.js:17:119776 Firefox can’t establish a connection to the server at wss://DOMAIN/api/topology/kube-controllers/ws?snapshot=hide&storage=hide&pseudo=hide&namespace=NAMESPACE&t=5s. app-8bbac71a57f56a6fd775.js:17:119776 The connection to wss://DOMAIN/api/topology/kube-controllers/ws?snapshot=hide&storage=hide&pseudo=hide&namespace=NAMESPACE&t=5s was interrupted while the page was loading. app-8bbac71a57f56a6fd775.js:17:119776
and the logs of weave scope frontend are:
--weave=false option.
--weave=false option.
I'm running kops with weave net and actually this command "weave ps" failing with the following output: bash-4.4# weave ps /usr/bin/weave: line 249: docker: not found ERROR: Unable to parse docker version
@iershovnsk try adding nginx.org/websocket-services: 'service-name' annotation to your ingress file weavescope uses websockets in order to provide real time data. https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket
Thanks @CodeAholic .
Maybe what is needed is clear instructions in the docs how to get Scope working with a proxy. If someone could make a PR for that, that'd be great.
I want to deploy scope under the subpath of ingress like that:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: weave
name: scope-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- myhost.com
secretName: myhost.com
rules:
- host: myhost.com
http:
paths:
- path: /scope
backend:
serviceName: weave-scope-app
servicePort: 80
But it didn't work properly.
I thought specify the -app.externalUI to /scope in weave-scope-app can work, but the log told me -app.externalUI was a boolean flag.
What can I do then?
@JayceCao if by not working properly you mean:
- the app is loading but you see it constantly trying to connect then that is caused by 400 bad request on the web-socket, you can inspect it in your navigator developer tool/network. [SOLUTION]: Add this line:
annotations:
nginx.org/websocket-services: "weave-scope-app"
If this is not your case, provide more details.
@CodeAholic When I deploy scope with default official deployment and create a nginx ingress like I said above, then I opened the web page with myhost.com/scope, I got error message in the console:
vendors-0e09fc049edd1048be73.js:1 Failed to load resource: the server responded with a status of 404 ()
app-69f341e7438c0844544f.js:1 Failed to load resource: the server responded with a status of 404 ()
vendors-0e09fc049edd1048be73.js:1 Failed to load resource: the server responded with a status of 404 ()
favicon.ico:1 Failed to load resource: the server responded with a status of 404 ()
the corresponding requests in the network tab are:
Request URL: https://myhost.com/app-69f341e7438c0844544f.js?754df3ea8a568a4a1ee5
Request Method: GET
Status Code: 404
Remote Address: 192.168.3.61:443
Referrer Policy: no-referrer-when-downgrade
Request URL: https://cloud.metalcao.com/vendors-0e09fc049edd1048be73.js?754df3ea8a568a4a1ee5
Request Method: GET
Status Code: 404
Remote Address: 192.168.3.61:443
Referrer Policy: no-referrer-when-downgrade
In my opinion, the -app.externalUI is to specify the subURL, so I specify --app.externalUI in deployment file like this:
spec:
replicas: 1
revisionHistoryLimit: 2
template:
metadata:
labels:
name: weave-scope-app
app: weave-scope
weave-cloud-component: scope
weave-scope-component: app
spec:
containers:
- name: app
args:
- '--mode=app'
- '--app.externalUI=/scope' ## added line
command:
- /home/weave/scope
env: []
image: 'docker.io/weaveworks/scope:1.11.4'
imagePullPolicy: IfNotPresent
ports:
- containerPort: 4040
protocol: TCP
But it turned out a boolean flag.
I don't know if you've configured kibana in ELK. There is a paragraph in kibana's config file:
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
server.basePath: "/kibana"
I don't know if scope has a option like this.
@JayceCao can you locate weave-scope-app svc when you kubectl get svc -n 'name space where u deployed weave scope '
@CodeAholic
the output of kubectl -n weave get services:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
weave-scope-app ClusterIP 10.101.208.117 <none> 80/TCP 3h41m
the outout of kubectl -n weave describe service weave-scope-app:
Name: weave-scope-app
Namespace: weave
Labels: app=weave-scope
name=weave-scope-app
weave-cloud-component=scope
weave-scope-component=app
Annotations: cloud.weave.works/launcher-info:
{
"original-request": {
"url": "/k8s/v1.10/scope.yaml?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxNSIsIEdpdFZlcnNpb246InYxLjE1...
"date": "Thu Aug 08 2019 07:22:07 GMT+0000 (UTC)"
},
"email-address": "[email protected]"
}
Selector: app=weave-scope,name=weave-scope-app,weave-cloud-component=scope,weave-scope-component=app
Type: ClusterIP
IP: 10.101.208.117
Port: app 80/TCP
TargetPort: 4040/TCP
Endpoints: 10.244.0.157:4040
Session Affinity: None
Events: <none>
the output of kubectl -n weave describe ingresses scope-ingress:
Name: scope-ingress
Namespace: weave
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
myhost.com terminates myhost.com
Rules:
Host Path Backends
---- ---- --------
myhost.com
/scope weave-scope-app:80 (10.244.0.157:4040)
Annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
Events: <none>
the output of kubectl -n weave describe pods weave-scope-app-5fdbbc5668-tpst7:
Name: weave-scope-app-5fdbbc5668-tpst7
Namespace: weave
Priority: 0
Node: metalcao/192.168.3.61
Start Time: Thu, 08 Aug 2019 17:32:44 +0800
Labels: app=weave-scope
name=weave-scope-app
pod-template-hash=5fdbbc5668
weave-cloud-component=scope
weave-scope-component=app
Annotations: <none>
Status: Running
IP: 10.244.0.157
Controlled By: ReplicaSet/weave-scope-app-5fdbbc5668
Containers:
app:
Container ID: docker://58f151d505617c253ec12ed8d190f2c6651b92d74b6041f66a69d82bff76e4f9
Image: docker.io/weaveworks/scope:1.11.4
Image ID: docker-pullable://weaveworks/scope@sha256:8c44d6761e197cc5c901d8b017d3afc528d3437aa057d5f2dee7fab39a9bfb3f
Port: 4040/TCP
Host Port: 0/TCP
Command:
/home/weave/scope
Args:
--mode=app
State: Running
Started: Thu, 08 Aug 2019 17:32:45 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-gn8fc (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-gn8fc:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-gn8fc
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
BTW, the scope can be accessed properly through:
kubectl port-forward -n weave "$(kubectl get -n weave pod --selector=weave-scope-component=app -o jsonpath='{.items..metadata.name}')" --address 0.0.0.0 4040
It is long time ago, I create this issue. I think the problem is the wget package in the docker image doesn't support no-proxy well. So I tried to build a docker image with update wget version by myself. Eventually I give up and begin to use Prometheus for monitoring. I think weave scope is still very good, and provide information for network ingress... Not sure the last version makes the statuation better... I don't have time to verify it now, if someone has successful useage with proxy setup, please inform me.
I have similar issue with Traefik-ingress and HAProxy reverse proxy. It's hanging and showing nothing. Anyone has a suggestion how to solve this? HAProxy and Traefik should support websockets by default as far as I know.
Update: It wasn't issue with Traefik/HAProxy, it's weave scope wasn't working on Firefox. I switched to Chrome and everything was fine.
I ran in the same issue using nginx ingress controller 0.26.1 on AWS ELB - solved with the hint from here https://github.com/kubernetes/ingress-nginx/issues/3746#issuecomment-554239384 - remove "aws-load-balancer-proxy-protocol from annotations", works on both Chrome and Firefox.
@JayceCao
I want to deploy scope under the subpath of ingress like that: ... What can I do then?
I've a similar setup. The trick was just to append a trailing slash when browsing to the webapp:
https://my-domain.corp/scope/
Ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$2
labels:
app: weave-scope
name: weave-scope-app
weave-cloud-component: scope
weave-scope-component: app
name: weave-scope
namespace: weave
spec:
rules:
- host: my-domain.corp
http:
paths:
- backend:
serviceName: weave-scope-app
servicePort: app
path: /scope(/|$)(.*)
tls:
- hosts:
- my-domain.corp
secretName: my-domain-ingress-tls
status:
loadBalancer:
ingress:
- {}
I'm using traefik as the ingress controller for a cluster and I cant set a specific path to weavescope, like /scope, as it gives 404 not found errors, one solution to it is to just use a subdomain, example: scope.kube.test/
Now when I can actually see the weavescope site, I now get a Nothing to Show page
I lost 12h on this problem.
At the end, the issue was my adblock. I disabled it and worked! Dont forget to test your page in at least 2 browsers and pay attention to the console.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.org/websocket-services: weave-scope-app
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/configuration-snippet: |
rewrite ^(/map)$ /map/ permanent;
name: ingress-weavescope
spec:
rules:
- http:
paths:
- path: /map(/|$)(.*)
backend:
serviceName: weave-scope-app
servicePort: 80
@CodeAholic When I deploy scope with default official deployment and create a nginx ingress like I said above, then I opened the web page with
myhost.com/scope, I got error message in the console:vendors-0e09fc049edd1048be73.js:1 Failed to load resource: the server responded with a status of 404 () app-69f341e7438c0844544f.js:1 Failed to load resource: the server responded with a status of 404 () vendors-0e09fc049edd1048be73.js:1 Failed to load resource: the server responded with a status of 404 () favicon.ico:1 Failed to load resource: the server responded with a status of 404 ()the corresponding requests in the network tab are:
Request URL: https://myhost.com/app-69f341e7438c0844544f.js?754df3ea8a568a4a1ee5 Request Method: GET Status Code: 404 Remote Address: 192.168.3.61:443 Referrer Policy: no-referrer-when-downgradeRequest URL: https://cloud.metalcao.com/vendors-0e09fc049edd1048be73.js?754df3ea8a568a4a1ee5 Request Method: GET Status Code: 404 Remote Address: 192.168.3.61:443 Referrer Policy: no-referrer-when-downgradeIn my opinion, the
-app.externalUIis to specify the subURL, so I specify--app.externalUIin deployment file like this:spec: replicas: 1 revisionHistoryLimit: 2 template: metadata: labels: name: weave-scope-app app: weave-scope weave-cloud-component: scope weave-scope-component: app spec: containers: - name: app args: - '--mode=app' - '--app.externalUI=/scope' ## added line command: - /home/weave/scope env: [] image: 'docker.io/weaveworks/scope:1.11.4' imagePullPolicy: IfNotPresent ports: - containerPort: 4040 protocol: TCPBut it turned out a boolean flag.
I don't know if you've configured kibana in ELK. There is a paragraph in kibana's config file:
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. server.basePath: "/kibana"I don't know if scope has a option like this.
setting it to "true" doesn't work
I have a similar issue when exposing weave-scope via nginx-ingress
$ kubectl describe ingress -n weave weave-manual-ingress
Name: weave-manual-ingress
Namespace: weave
Address: internal-a06c2d71bd90c4b43b7344c38b8757c3-1682546038.eu-west-1.elb.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
weave.int.affinionservices.com
weave-scope-app:80 (100.120.15.197:4040)
Annotations: nginx.org/websocket-services: weave-scope-app
route53mapper: enabled
Any fix?
