thingsboard
thingsboard copied to clipboard
thingsboard
[Bug] Widget Error: Cannot read properties of undefined (reading 'ɵcmp')
Describe the bug Thingsboard works with no issue on Firefox.
On Chrome-based browsers (Chrome, Edge, ...), we often have the following error :
Cannot read properties of undefined (reading 'ɵcmp')
Clearing Chrome's cache makes the dashboard work again for a little while, but the error appears again, sometimes after only one refresh.
Your Server Environment
- own setup
- local
- 3.6.1 PE
- Debian Bullseye
Your Client Environment Desktop (please complete the following information):
- OS: Windows (untested with others)
- Browser: chrome, edge
- Version: Chrome 119.0.6045.161 (Official build, 64 bit)
To Reproduce Steps to reproduce the behavior: Create a dashboard with any kind of widget Refresh the page a few times
Expected behavior This widget error should not appear.
Sadly none found yet.
We will attempts to update the host OS this week in case it's due to a bad library.
This is similar to https://github.com/angular/angular/issues/36279 which seems to basically have solved itself.
We tried to upgrade the OS today, to no avail. Error is still consistent.
If this helps at all, here's the console output in chrome when the error occurs first after emptying the cache.
7502.0f89b887c1b0b1ac.js:1 TypeError: Cannot read properties of undefined (reading 'ɵcmp')
at _i (vendor.85d856bbf6e1608e.js:1:902618)
at Rg.createComponent (vendor.85d856bbf6e1608e.js:1:1021712)
at ee.configureDynamicWidgetComponent (7502.0f89b887c1b0b1ac.js:1:6877708)
at createDefaultSubscription.subscribe.subscriptionInited [as next] (7502.0f89b887c1b0b1ac.js:1:6876554)
at ve.next (vendor.85d856bbf6e1608e.js:1:450282)
at I._next (vendor.85d856bbf6e1608e.js:1:449952)
at I.next (vendor.85d856bbf6e1608e.js:1:449638)
at vendor.85d856bbf6e1608e.js:1:447568
at m (vendor.85d856bbf6e1608e.js:1:504463)
at m.next (vendor.85d856bbf6e1608e.js:1:447407)
And here's the affected line in "runtime.xxxxxx.js"
(Line 1, column 3618)
Thank you
@Webbeh @Backdraft007 @netbc Hi. Please describe your deployment in as much detail as possible. How exactly did you build your environment (OS, service/container/cluster, db type, message queue service, loadbalancer, etc) Also, do you use some third-party certificate for HTTPS? Is this problem reproduced using IP instead of Domain?
We are running version 3.6.2PE monolithic on docker using Kafka and PostgreSQL. We do not have loadbalancer in front and using Letsencrypt for TB TLS certificates (we have server.ssl.enabled) Cannot be reproduced when connecting directly to IP instead Domain
@Webbeh @Backdraft007 @netbc Hi. Please describe your deployment in as much detail as possible. How exactly did you build your environment (OS, service/container/cluster, db type, message queue service, loadbalancer, etc) Also, do you use some third-party certificate for HTTPS? Is this problem reproduced using IP instead of Domain?
Debian Bullseye using the .deb packages for both Thingsboard and Tb-web-report. We recently updated to 3.6.2 and still have the issue. Certs are provided by Digicert and the instance is public.
We're using postgres + cassandra for data retention. No loadbalancer. ~Using NGINX as reverse proxy.~ EDIT 2024.02.16 : we actually were not using NGINx on this machine, but the Thingsboard's SSL_ENABLED configuration. Sorry for the misunderstanding.
We are using another instance (PE as well) with k8s (using helm charts) and that one doesn't have this issue on our end. Same SSL certificates are used. I believe it uses an NGINX ingress as well.
I am having this same problem My backend database is postgres, version is 3.6.2 CE. I can repeat it by going into tenant admin and editing a dashboard - in this case all I did was added a widget title. When I go back, I get this exact error on every widget (even the ones I did not update).
In the browser, I get this an error message that might mean something to soembody:
3125.034b16032b37b4ce.js:1 TypeError: Cannot read properties of undefined (reading 'ɵcmp')
at yi (vendor.f85194cfffd980b7.js:1:881487)
at Rb.createComponent (vendor.f85194cfffd980b7.js:1:1000588)
at mn.configureDynamicWidgetComponent (3125.034b16032b37b4ce.js:1:6572937)
at createDefaultSubscription.subscribe.subscriptionInited [as next] (3125.034b16032b37b4ce.js:1:6571783)
at me.next (vendor.f85194cfffd980b7.js:1:429578)
at x._next (vendor.f85194cfffd980b7.js:1:429248)
at x.next (vendor.f85194cfffd980b7.js:1:428934)
at vendor.f85194cfffd980b7.js:1:426881
at m (vendor.f85194cfffd980b7.js:1:483509)
at m.next (vendor.f85194cfffd980b7.js:1:426720)
The only other thing I noticed is at about the same time I got this. It may be unrelated.
2024-02-07 14:33:00,342 [https-jsse-nio-0.0.0.0-443-exec-10] INFO o.a.coyote.http2.Http2UpgradeHandler - Connection [25], Stream [605] Closed due to error
Note: further occurrences of HTTP/2 stream errors will be logged at DEBUG level.
org.apache.coyote.http2.StreamException: The client attempted to use more than [100] active streams
at org.apache.coyote.http2.Http2UpgradeHandler.headersEnd(Http2UpgradeHandler.java:1723)
at org.apache.coyote.http2.Http2AsyncUpgradeHandler.headersEnd(Http2AsyncUpgradeHandler.java:41)
at org.apache.coyote.http2.Http2Parser.onHeadersComplete(Http2Parser.java:622)
at org.apache.coyote.http2.Http2Parser.readHeadersFrame(Http2Parser.java:284)
at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:254)
at org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:167)
at org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1122)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1704)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
Actually, let me take that back. It may be related after all. I noticed if I keep hitting refresh the dashboard might load. Other times I get this error. Watching the network tab I might see a clue here:
From time to time a compiled javascript component fails to load. There is no HTTP response code, just 0 bytes - typically I have seen this kind of response when the web server gets the request and just immediately closes the connection with no response at all (not even a 500).
If I keep hitting refresh, one out of every 8 or so times it will load:
Notice the one failed file finally worked. It's not always that file. It's acting as if the server is getting overloaded and just dumping connections.
Is this tomcat? What is maxConcurrentStreams set to?
One last thing. I confirmed it's definitely a problem with Chrome based browsers but on Firefox it seems to work. I think there must be some difference in how Chrome handles http/2 and either Firefox handles it differently or just isn't using it.
@Webbeh @Backdraft007 @netbc Hi. Please describe your deployment in as much detail as possible. How exactly did you build your environment (OS, service/container/cluster, db type, message queue service, loadbalancer, etc) Also, do you use some third-party certificate for HTTPS? Is this problem reproduced using IP instead of Domain?
These are all good background questions but in my case I don't think it's the issue ... but regardless, here are some data points:
- I am running this on docker, but the network mode is 'host' so docker isn't getting in the way - listen() in the container is just listen() on the host
- I'm using a letsencrypt certificate, but running without https at all gave me the same results
- No load balancer
I am pretty confident this is an HTTP/2 issue. There are some significant differences in how Chrome handles HTTP/2 versus other browser platforms, especially in the areas of coalescing and prioritization. There are also some differences in things lke frame size, idle timeout, and initial window size.
Increasing maxConcurrentStreams may be pushing the bump under the carpet. What would you increase it to? And at what point does that become a problem, too? Maybe what we need is a way to set maxConcurrentStreams, and also a way to disable http/2 entirely. There are some aspects of http/2 that still give people heartburn - for example the whole concept of Server Push seemed like a good idea, but both Chrome and Firefox are considering deprecating it due to questionable efficiency and low utilization.
Along the lines of the question about load balancing, one likely workaround here would be to put an nginx or apache in front of thingsboard, set it up as a proxy, and just disable http/2. It might make load/refresh time a little worse, but I'm pretty sure http/2 is the problem here.
Hello, any updates on this? It seems to us that the issue is related to the internal SSL processing when using configuration SSL_ENABLED=true . We have this issue only if HTTPS is terminated in the Thingsboard. When we connect using IP instead of domain or terminate HTTPS in nginx we do not experience this issue.
@Webbeh @Backdraft007 @netbc Hi. Please describe your deployment in as much detail as possible. How exactly did you build your environment (OS, service/container/cluster, db type, message queue service, loadbalancer, etc) Also, do you use some third-party certificate for HTTPS? Is this problem reproduced using IP instead of Domain?
Debian Bullseye using the .deb packages for both Thingsboard and Tb-web-report. We recently updated to 3.6.2 and still have the issue. Certs are provided by Digicert and the instance is public.
We're using postgres + cassandra for data retention. No loadbalancer. Using NGINX as reverse proxy.
We are using another instance (PE as well) with k8s (using helm charts) and that one doesn't have this issue on our end. Same SSL certificates are used. I believe it uses an NGINX ingress as well.
I actually have to revise what I said after @netbc 's latest comment. The instance we have that is showing this issue does NOT have an NGINX reverse proxy (I mean one is installed but we do not use it for Thingsboard).
Please discard my previous observation. We are indeed using the SSL termination in Thingsboard itself, and NOT one behind a reverse proxy. I confirm that we do not have this issue with the k8s instances which are terminated through an NGINX ingress, confirming what @netbc is saying on our end.
This issue still persists in Thingsboard version 3.6.4.
When HTTP2 is disabled in the Thingsboard configuration, the issue no longer occurs, confirming what @wz2b mentioned. Using the IP address instead of the domain also prevents the issue from occurring, as noted by @netbc. Another way to resolve the issue is by disabling browser caching through the network recorder in Chrome Developer Tools.
Our Thingsboard is running on Ubuntu 20.04 as a service without a proxy, using the SSL_ENABLED configuration with a PEM type certificate. Currently to avoid this issue we have HTTP2 disabled.
While trying to refresh my SSL certificates as they had expired, I put the wrong command in a few times and got blocked from making any more requests for an hour, in this time when I went to thingsboard I got the same error where every widget wasn't loading with that error: [Widget Error: Cannot read properties of undefined (reading 'ɵcmp')]
I downloaded firefox after seeing this thread and it worked fine on there. I think the difference is that chrome uses HTTP2 whereas firefox doesn't. Instead of disabling it on the thingsboard.yml file, I altered my HAproxy configuration as follows -
Changing this line in haproxy.cfg:
frontend http-in
bind *:80 alpn h2,http/1.1
To this line:
frontend http-in
bind *:80
Disabling the use of HTTP/2 solved the problem for me. I could then use it on chrome. I'm hoping once I can refresh my SSL certificates I can get HTTP/2 working again
