Not able to visit the Admin User Interface, ota-user-profile: System error

[adlingepa]

I am trying to setup the ostree meta-update yocto build environment and OTA CE server on same VM server machine. I am able to setup the OTA CE server using make start with docker as minikube driver. But while trying to access the Admin user interface using http://app.ota.local url, this url is redirected to http://app.ota.local/login page automatically and gives error as "ota-user-profile: System error"

I've check below things while troubleshooting this, All microservices are running,

prashanta@hiuyoctovmnew:/u02/ats_repo_poky/build/tmp$ kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
app-76cccf7cb4-fzw6h                      1/1     Running   0          4d
campaigner-76cf7cc696-bkjrp               1/1     Running   895        4d
campaigner-daemon-6c99974f96-p8sb2        1/1     Running   0          4d
device-registry-68fbc6d858-f499g          1/1     Running   0          4d
device-registry-daemon-6c4b5989cc-9j8l7   1/1     Running   0          4d
director-57d4758bc5-msjxl                 1/1     Running   0          4d
director-daemon-5fdd469fd6-hlm9h          1/1     Running   1          4d
gateway-deployment-6567f4d664-k5q7c       1/1     Running   0          4d
kafka-0                                   1/1     Running   0          4d
mysql-0                                   1/1     Running   0          4d
treehub-0                                 1/1     Running   3          4d
tuf-keyserver-68c654d575-5dkd4            1/1     Running   0          4d
tuf-keyserver-daemon-5d6c475575-z8fmn     1/1     Running   0          4d
tuf-reposerver-684b7b8f7c-4glk7           1/1     Running   0          4d
web-events-7f8645678-4p7m8                1/1     Running   3          4d
zookeeper-0                               1/1     Running   0          4d

tried to accessing via http tool,

prashanta@hiuyoctovmnew:/u02/ats_repo_poky/build/tmp$ http http://app.ota.local
HTTP/1.1 303 See Other
Connection: keep-alive
Content-Length: 0
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval'  https://cdn.auth0.com https://cdn.eu.auth0.com https://  https://js-agent.newrelic.com/ https://bam.nr-data.net/ https://tags.tiqcdn.com 'self' 'unsafe-inline' https://wego.here.com/services/iplookup/ ; style-src 'unsafe-inline' 'self' maxcdn.bootstrapcdn.com https://tags.tiqcdn.com; font-src 'self' data: cdn.auth0.com maxcdn.bootstrapcdn.com; img-src 'self' data: cdn.auth0.com https://*.gravatar.com https://*.githubusercontent.com https://*.googleusercontent.com https://*.wp.com https://sheredeutschlandgmbh.d1.sc.omtrdc.net; connect-src 'self' ws://*.ota.local:80 https://bam.nr-data.net/  https://
Date: Mon, 31 Aug 2020 11:22:14 GMT
Location: /login
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Server: nginx/1.13.9
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: master-only
X-XSS-Protection: 1; mode=block
x-b3-traceid: 0038c31551b4e50d

Below the browser screenshot,

While checking the logs of application pod, I found some exceptions, which might help you to guide us what going wrong,

prashanta@hiuyoctovmnew:~$ kubectl logs app-76cccf7cb4-fzw6h
I|2020-08-27 10:33:03,508|akka.event.slf4j.Slf4jLogger|Slf4jLogger started
W|2020-08-27 10:33:03,920|akka.util.ManifestInfo|Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.5.30] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries. Possibly conflicting versions [2.5.30, 2.5.26] in libraries [akka-protobuf:2.5.30, akka-actor:2.5.30, akka-slf4j:2.5.26, akka-stream:2.5.30]
I|2020-08-27 10:33:05,248|play.api.Play|Application started (Prod)
I|2020-08-27 10:33:05,505|play.core.server.NettyServer|Listening for HTTP on /0.0.0.0:9001
{"at":"2020-08-27T10:33:33.666Z","level":"INFO","logger":"c.a.l.h.logging.RequestLoggingActor","msg":{"http_stime":"275","http_content_ln":"15","http_query":"''","http_method":"GET","http_path":"/health","http_service_name":"ota-plus-web","traceid":"68db55c14c20cedf","http_status":"200"}}
W|2020-08-27 10:33:34,275|z.r.AsyncReporter$BoundedAsyncReporter|Spans were dropped due to exceptions. All subsequent errors will be logged at FINE level.
W|2020-08-27 10:33:34,277|z.r.AsyncReporter$BoundedAsyncReporter|Dropped 1 spans due to ConnectException(Failed to connect to localhost/127.0.0.1:9411)
java.net.ConnectException: Failed to connect to localhost/127.0.0.1:9411
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
	at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
...
I|2020-08-27 13:44:16,800|o.a.kafka.common.utils.AppInfoParser|Kafka version: 2.4.1
I|2020-08-27 13:44:16,800|o.a.kafka.common.utils.AppInfoParser|Kafka commitId: c57222ae8cd7866b
I|2020-08-27 13:44:16,800|o.a.kafka.common.utils.AppInfoParser|Kafka startTimeMs: 1598535856724
W|2020-08-27 13:44:29,373|o.apache.kafka.clients.NetworkClient|[Producer clientId=producer-1] Error while fetching metadata with correlation id 2 : {UserLogin-ce=LEADER_NOT_AVAILABLE}
I|2020-08-27 13:44:29,388|org.apache.kafka.clients.Metadata|[Producer clientId=producer-1] Cluster ID: pxGRY3FmSKyOZb0HW1kGQA
W|2020-08-27 13:44:29,455|o.apache.kafka.clients.NetworkClient|[Producer clientId=producer-1] Error while fetching metadata with correlation id 5 : {UserLogin-ce=LEADER_NOT_AVAILABLE}
W|2020-08-27 13:44:29,744|o.apache.kafka.clients.NetworkClient|[Producer clientId=producer-1] Error while fetching metadata with correlation id 6 : {UserLogin-ce=LEADER_NOT_AVAILABLE}
W|2020-08-27 13:44:30,918|o.apache.kafka.clients.NetworkClient|[Producer clientId=producer-1] Error while fetching metadata with correlation id 7 : {UserLogin-ce=LEADER_NOT_AVAILABLE}
W|2020-08-27 13:44:31,151|o.apache.kafka.clients.NetworkClient|[Producer clientId=producer-1] Error while fetching metadata with correlation id 8 : {UserLogin-ce=LEADER_NOT_AVAILABLE}
W|2020-08-27 13:44:31,255|o.apache.kafka.clients.NetworkClient|[Producer clientId=producer-1] Error while fetching metadata with correlation id 9 : {UserLogin-ce=LEADER_NOT_AVAILABLE}
W|2020-08-27 13:44:31,357|o.apache.kafka.clients.NetworkClient|[Producer clientId=producer-1] Error while fetching metadata with correlation id 10 : {UserLogin-ce=LEADER_NOT_AVAILABLE}
I|2020-08-27 13:44:31,695|c.a.PlayMessageBusPublisher|published UserLogin - default
E|2020-08-27 13:44:32,197|p.c.server.netty.PlayRequestHandler|Cannot invoke the action
com.advancedtelematic.api.Errors$RemoteApiIOError: java.net.UnknownHostException: ota-user-profile: System error
Caused by: java.net.UnknownHostException: ota-user-profile: System error
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
...

Here are the complete logs, ota-user-profile_system_error.zip

Please help me in troubleshooting this issue.

Thanks , Prashant

[ddunlap2005]

@adlingepa About 8 months ago, commit https://github.com/advancedtelematic/ota-plus-server/commit/b03ac42d3537007aa2f3cba637c278e575cc78e0 was introduced. This commit adds a call to userProfileApi within the NoLoginAction. OTA Community Edition does not use the ota-user-profile that I can see; Ota-plus-server does not even define the USER_PROFILE_HOST/PORT environment variables.

I was able to bring my Ota-community-edition server up earlier without seeing this error when I connected to app.ota.local. I corrected a few more issues with the environment and now I'm seeing this error. If I figure out why I got past it, I'll let you know.

[houcros]

ota-plus-server always had a reference to ota-user-profile, see the config. The thing is that, somewhere along the way, ota-plus-server became more dependent on ota-user-profile, to a point where it won't work without it. But ota-user-profile is not part of the CE, hence the problems.

I guess one could implement a new NoLoginAction to do the same that the old one was doing, but I'm afraid that would only uncover further issues of the same nature.

[msatpathy26]

This issue still persists and keeps coming back every now and then. Not able to open an issue in ota-plus-server repo as well.

[karymei]

Is there any plan to fix this issue? Thanks.

[houcros]

Sorry, but I don't think so. To the best of my knowledge, this repo is not actively maintained.

[wzq619]

I got the same issue，do anyone have the solution?thanks!

[simao]

I believe a patch like the following would allow the webapp to be used without user-profile, which is a closed source component:

https://gist.github.com/simao/3b347729c568c8929422fce57efc222a

However, like houcros said this project is not actively maintained, further development will continue at https://github.com/uptane/ota-community-edition Though a web frontend is not supported and the ota ce solution is api based only.

[lpcysmi]

I believe a patch like the following would allow the webapp to be used without user-profile, which is a closed source component:

https://gist.github.com/simao/3b347729c568c8929422fce57efc222a

However, like houcros said this project is not actively maintained, further development will continue at https://github.com/uptane/ota-community-edition Though a web frontend is not supported and the ota ce solution is api based only.

Where do I apply this patch?

I have been trying to use https://github.com/uptane/ota-community-edition and always end up at this error.

[simao]

You'll need to apply that patch on the webapp (https://github.com/advancedtelematic/ota-plus-server) build a webpp container, and make ota-community edition use that container.

[jinlingyi]

Are there any guide to resolve this issue now?

You'll need to apply that patch on the webapp (https://github.com/advancedtelematic/ota-plus-server) build a webpp container, and make ota-community edition use that container.

How to use this as web app of OTA CE?

[simao]

This project is no longer under active development, you can use https://github.com/uptane/ota-community-edition which doesn't use a web app, or you can apply that patch to webapp yourself.

Again, this is not supported, but you can just

git checkout <webapp>
git apply user-profile-disable.patch
sbt docker:publishLocal

Use that image instead of the one included in this repo.

If you are just trying ota-ce and see how it works you could try https://github.com/simao/ota-lith there is a webapp branch with a web ui.

[jinlingyi]

This project is no longer under active development, you can use https://github.com/uptane/ota-community-edition which doesn't use a web app, or you can apply that patch to webapp yourself.

Again, this is not supported, but you can just

git checkout <webapp>
git apply user-profile-disable.patch
sbt docker:publishLocal

Use that image instead of the one included in this repo.

If you are just trying ota-ce and see how it works you could try https://github.com/simao/ota-lith there is a webapp branch with a web ui.

Thank you for your reply! But the https://github.com/uptane/ota-community-edition has the same issue. Will it be updated? I will try your patch for this issue. And I need a web UI so I will try https://github.com/simao/ota-lith also.

[simao]

See my comments on those projects.

[jinlingyi]

See my comments on those projects.

Thank you very much! Now I can run this branch too. And the web UI can find device with qemu. But the web UI can't create update and campaigner, and the ota-cli can't control this project. This feature can only be used with API?

[simao]

in which project are you having problems with ota-cli?

in which projet are you having problems with the web ui creating updates? And what exactly is the error?

in which project are you having trouble using ota-cli? ota-cli should be able to be used with both. See https://github.com/simao/ota-lith/blob/master/docs/updates-ota-cli.md

[jinlingyi]

The web UI of advancedtelematic/ota-community-edition (also the local docker image of advancedtelematic/ota-plus-server) can't create update, the button of 'Create update' is missing. And I can't set ota-cli with advancedtelematic/ota-community-edition because when I use 'ota init' the servers seems can't be reach of 'http://campaigner.ota.local, etc.', even if I set it with Ingress.

[jinlingyi]

Now I can use the web UI to create update. I change the UIFeature to all open in the ota-plus-server. Thank you for your reply!

[Jeredriq]

@jinlingyi Can you please provide the road you take step by step?