Unable to Normally Add Packages From Eclipse Marketplace

[LeoNerdoG]

When user is trying to add multiple packages from Eclipse Marketplace (I tried to install with drag-and-drop), Kura starts reporting an error in /var/kura/kura.log (see bellow). After some time packages start to appear (or not - i tried several times and sometimes they appeared, sometimes not).

Testflow:

1. Login to Kura as admin/admin
2. Go to Packages, start drag-and-dropping packages from Eclipse Marketplace (only the ones for Eclipse Kura 4.x.y)
3. observe the situation after "org.eclipse.kura.driver.ibeacon" is installed - Kura does not install another package and no error is shown It looks like the packages are stuck in some kind buffer?

Error log:

2020-06-10T08:31:51,340 [qtp27429980-2005] INFO o.e.k.w.s.s.WiresBlinkServlet - unregistered 2020-06-10T08:31:51,689 [qtp27429980-2006] INFO o.e.k.w.s.s.WiresBlinkServlet - Session ended: 1591770718033 2020-06-10T08:31:52,570 [AbstractNtpClockSyncProvider:schedule] WARN o.e.k.l.c.JavaNtpClockSyncProvider - Error while synchronizing System Clock with NTP host 0.pool.ntp.org. Please verify network connectivity ... 2020-06-10T08:31:52,570 [AbstractNtpClockSyncProvider:schedule] INFO o.e.k.l.c.AbstractNtpClockSyncProvider - Try to sync clock (59) 2020-06-10T08:32:00,606 [qtp27429980-2006] INFO o.e.k.w.s.GwtPackageServiceImpl - Installing deployment package from Eclipse Marketplace, URL http://download.eclipse.org/kura/releases/4.1.0/org.eclipse.kura.example.ibeacon.scanner_1.0.300.dp... 2020-06-10T08:32:00,611 [qtp27429980-2006] WARN o.e.k.w.s.GwtPackageServiceImpl - failed to start package install from Eclipse Marketplace org.eclipse.kura.web.shared.GwtKuraException: INTERNAL_ERROR at org.eclipse.kura.web.server.util.ServiceLocator.applyToServiceOptionally(ServiceLocator.java:124) at org.eclipse.kura.web.server.GwtPackageServiceImpl.installPackageFromMarketplace(GwtPackageServiceImpl.java:263) at jdk.internal.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373) at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at org.eclipse.kura.web.server.OsgiRemoteServiceServlet.service(OsgiRemoteServiceServlet.java:39) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.equinox.http.servlet.internal.HttpServiceRuntimeImpl$LegacyServlet.service(HttpServiceRuntimeImpl.java:1223) at org.eclipse.equinox.http.servlet.internal.registration.EndpointRegistration.service(EndpointRegistration.java:148) at org.eclipse.equinox.http.servlet.internal.servlet.ResponseStateHandler.processRequest(ResponseStateHandler.java:62) at org.eclipse.equinox.http.servlet.internal.context.DispatchTargets.doDispatch(DispatchTargets.java:131) at org.eclipse.equinox.http.servlet.internal.servlet.ProxyServlet.service(ProxyServlet.java:74) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(HttpServerManager.java:284) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:876) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) at java.base/java.lang.Thread.run(Thread.java:844) Caused by: java.lang.Exception: Element already exists at org.eclipse.kura.deployment.agent.impl.DeploymentAgent.installDeploymentPackageAsync(DeploymentAgent.java:262) at org.eclipse.kura.web.server.GwtPackageServiceImpl.lambda$1(GwtPackageServiceImpl.java:274) at org.eclipse.kura.web.server.util.ServiceLocator.applyToServiceOptionally(ServiceLocator.java:119) ... 53 more 2020-06-10T08:32:02,582 [AbstractNtpClockSyncProvider:schedule] WARN o.e.k.l.c.JavaNtpClockSyncProvider - Error while synchronizing System Clock with NTP host 0.pool.ntp.org. Please verify network connectivity ... 2020-06-10T08:32:02,582 [AbstractNtpClockSyncProvider:schedule] INFO o.e.k.l.c.AbstractNtpClockSyncProvider - Try to sync clock (60) 2020-06-10T08:32:12,602 [AbstractNtpClockSyncProvider:schedule] WARN o.e.k.l.c.JavaNtpClockSyncProvider - Error while synchronizing System Clock with NTP host 0.pool.ntp.org. Please verify network connectivity ... 2020-06-10T08:32:12,602 [AbstractNtpClockSyncProvider:schedule] INFO o.e.k.l.c.AbstractNtpClockSyncProvider - Try to sync clock (61)

Expected behavior If user tries to install package one after another, Kura should be able to handle this or throw an error if gets stuck somewhere.

Screenshot:

Target Environment (please complete the following information): Board: Raspberry Pi 3 OS version: Linux raspberrypi 4.19.75-v7+ # 1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux Kura version: 4.2.0_SNAPSHOT (Link to the specific Kura version -from May 20th 2020

[salvatore-coppola]

I was not able to reproduce the bug, i'have tried to install different packes from Eclipse Marketplace and everything went fine. Target Environment: Docker eclipse/kura 5.0.0-RC3-alpine-x86_64

[MMaiero]

@mstankovic Can you have a check on this?

[vincemulhollon]

I can verify this problem exists in docker eclipse/kura:latest and does not exist in docker eclipse/kura 5.0.0-RC3-alpine-x86_64

Attempts at installing using docker image latest fail.

Testing kura 5.0.0-RC3-alpine-x86_64 will initially fail as the SSL certificate in the docker expired back on Sep 5, 2021, 7:15:31 PM

However, one can visit the marketplace.eclipse.org, then click on the padlock, then keep clicking until show certificate, then details tab, then "copy to a file" button, then save the latest cert somewhere as a Base-64 encoded X.509 which is a .CER file, then windows will not allow you to look at .CER files, so rename the file to change the extension to .txt, then open the file in a text editor.

From there its a rather simple process to start up a kura 5.0.0-RC3-alpine-x86_64 in docker, log in as admin, go to Security, click on add, select certificate in the dropdown instead of the default private/public pair, then "next", then leave the key store as SSLKeystore, then provide a useful alias for the new key such as "unexpired eclipse" then copy and paste the entire certificate downloaded from the marketplace, then "apply".

From there its the usual process of going to the "Packages" then click and drag the "Eclipse Kura Heater Demo for Eclipse Kura 4/5" into Kura and in a couple seconds I have a working heater demo... cool...

So the problem starts somewhere between docker images kura 5.0.0-RC3-alpine-x86_64 and latest

[MMaiero]

@mstankovic Can you please have a check?

[vincemulhollon]

From a clean docker install of tag 5.1.0-M1-alpine-x86_64 which was pushed to hub.docker.com 8 days ago, if I try to install the Eclipse Kura Heater Demo from the eclipse marketplace, the package install fails, then using the Kura web interface if I check Device then System Logs I see:

2022-03-09T18:39:41,307 [priority: ERROR] Message: o.e.k.d.a.i.DeploymentAgent - Exception installing package at URL https://download.eclipse.org/kura/releases/5.0.0/org.eclipse.kura.demo.heater_1.0.600.dp Stacktrace: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

using older docker version tags there are other ways to access the logs ranging from downloading a large zipfile to opening a shell inside the docker container and looking at the raw files, but its always roughly the same error message, cert path problems.

If I check Security, I see the ssl-eclipse-marketplace cert shipped with 5.1.0-M1 expires in about three weeks on Apr 4 2022

If I download the current cert for marketplace using a web browser (chrome, although it doesn't matter), and upload the current SSL cert into Security with the name "ssl-eclipse-marketplace-new" I see the cert currently being used by the marketplace has an expiration date of Feb 5 2023

After installing the current marketplace cert I attempt installing Eclipse Kura Heater Demo from the eclipse marketplace and it works perfectly.

The first two lines of the good cert that works look like this:

-----BEGIN CERTIFICATE----- MIIHxzCCBq+gAwIBAgIQCHzxjmc9BShOegdIJ7tKlTANBgkqhkiG9w0BAQsFADBP MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE

And its the *.eclipse.org cert issued by DigiCert valid from 1/4/2022 to 2/5/2023

I don't know how to pull the cert data from the docker container, or how to examine a cert in the Kura web interface if that is even possible; but I DO know how to pull a cert in Kapua: "Devices", click the device, in the lower tab select "Keystore" click the ssl-eclipse-marketplace SSLKeystore, click "i Details" button (which is a weird name) and the SSL cert that ships in docker containers and does NOT work with marketplace.eclipse.org has the following first two lines aka this is the bad cert:

-----BEGIN CERTIFICATE----- MIIH7DCCBtSgAwIBAgIQC508hpoYxqByTXTyEIgBMDANBgkqhkiG9w0BAQsFADBw MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3

I think what's required to get marketplace access working again is ship new docker containers with the "MIIHxzC" cert instead of the "MIIH7DC" cert. On a fresh docker install, changing that cert is all I needed to do to get marketplace working.

I tried the above process with a docker container tag "latest" which is actually some months old, older than 5.1.0-M1-alpine-x86_64 yet newer than 5.0.0-RC3-alpine-x86_64 and could not get "latest" tag to work despite reporting the same cert path error, which seems weird to me, but maybe is a mistake in my testing procedure or bad luck or something. I note that docker tag "latest" points to 5.0.1-alpine-x86_64 so these are all alpine-x86_64 based images so I suspect it is NOT an OS related issue.

Or rephrasing the above, there seems to be a second marketplace problem I cannot figure out but did reproduce (edited: twice , on two separate days...), possibly unrelated to the SSL problem, with 5.0.1 that does not seem to exist in 5.1.0 nor in 5.0.0-RC3, both of which work great if the SSL cert for marketplace is updated to the current cert.

Have a great day!

[MMaiero]

This should be fixed now on develop as we have added the new Eclipse marketplace SSL certificate