Fresh installation of Artifactory OSS 7.12.6 doesn't start, missing jffe service

[wl2776]

I have done everything according to manuals from https://jfrog.com/open-source/

My Ubuntu version is 20.04. Since there is no "focal" in https://releases.jfrog.io/artifactory/artifactory-debs/, I've added "bionic":

$ cat /etc/apt/sources.list.d/artifactory.list
deb https://releases.jfrog.io/artifactory/artifactory-debs  bionic main

Then I've installed jfrog-artifactory-oss version 7.12.6 and tried launching the service. It has launched, but browser, connected to ports :8081 or :8082 of localhost, shows that 3 services don't start:

Attached is the relevant part of logs:

art.log

[amithins]

Can you please compress and upload the content of /opt/jfrog/artifactory/var/log

[wl2776]

Here it is artifactory-oss-log.tar.gz

[Suphax]

Having the same issue on Ubuntu 20.04, on a fresh intall

$ cat /etc/apt/sources.list.d/jfrog.list
deb https://jfrog.bintray.com/artifactory-debs bionic main

artifactory-oss-log.tar.gz

Edit: I it running by purging Artifactory, installing an older version, and starting with a different command

DON'T DO THIS IF YOU HAVE ANY CONFIG YOU WISH TO KEEP

sudo apt purge jfrog-artifactory-purge
sudo rm -rf /opt/jfrog/artifactory
sudo rm -rf /var/opt/jfrog
sudo apt update
sudo apt install jfrog-artifactory-oss=7.12.5
sudo service artifactory start

Wait for a little then try to open the page, all should be good.

[amithins]

@Suphax how long did you wait for services to come up in 7.12.6 ? In a VM i tried, it took me around 28 seconds for all the services to come up.

[Suphax]

@Suphax how long did you wait for services to come up in 7.12.6 ? In a VM i tried, it took me around 28 seconds for all the services to come up.

Roughly 30 minutes

[Iril]

We are also seeing this issue on a Redhat 7 New Installation:

2021-02-16T23:49:15.747Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 502 code 2021-02-16T23:49:16.752Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 502 code 2021-02-16T23:49:17.757Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 502 code

and the status shows a Service is Healthy missing jffe error as well. Thanks Martin

[RRadziejewski]

2021-03-26T12:53:45.078Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 404 code

[saas2813]

Can anyone tell me what this JFFE service is? I have an Artifactory that suddenly stoped working and the web-page reports,: "Missing services: jffe" For some reason it got back online after restarting it but I would really like to know what it was.

[grierellis]

I found that libvirtd was running. It wasn't libvirtd, itself, that prevented artifactory from coming up. It was the fact that the virbr0 interface was up. It doesn't seem to be a problem bringing this interface back up after artifactory comes up.

[jordaniac89]

I'm also having this issue. These three services do not start up. Seeing lots of "connection refused" errors from the different services as well as a java stacktrace when trying to initialize Home

2022-05-03T16:26:40.382Z [jfrt ] [ERROR] [ad8423ca882374a8] [tifactoryHomeConfigListener:55] [Catalina-utility-1  ] - Failed initializing Home. Caught exception: 
java.lang.IllegalStateException: Failed to initialize file watcher.
	at org.jfrog.config.watch.FileWatcher.create(FileWatcher.java:65)
	at org.jfrog.config.ConfigurationManagerImpl.<init>(ConfigurationManagerImpl.java:82)
	at org.jfrog.config.ConfigurationManagerImpl.create(ConfigurationManagerImpl.java:63)
	at org.artifactory.lifecycle.webapp.servlet.BasicConfigurationManager.<init>(BasicConfigurationManager.java:84)
	at org.artifactory.lifecycle.webapp.servlet.ArtifactoryHomeConfigListener.getBasicConfigManagers(ArtifactoryHomeConfigListener.java:78)
	at org.artifactory.lifecycle.webapp.servlet.ArtifactoryHomeConfigListener.contextInitialized(ArtifactoryHomeConfigListener.java:52)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690)
	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Function not implemented
	at java.base/sun.nio.fs.LinuxWatchService.<init>(LinuxWatchService.java:64)
	at java.base/sun.nio.fs.LinuxFileSystem.newWatchService(LinuxFileSystem.java:47)
	at org.jfrog.config.watch.JdkWatchService.create(JdkWatchService.java:24)
	at org.jfrog.config.watch.WatchServiceAdapter.<init>(WatchServiceAdapter.java:21)
	at org.jfrog.config.watch.JdkWatchService.<init>(JdkWatchService.java:19)
	at org.jfrog.config.watch.FileWatcher.create(FileWatcher.java:63)
	... 20 common frames omitted
2022-05-03T16:26:40.420Z [jfrt ] [ERROR] [ad8423ca882374a8] [actoryContextConfigListener:90] [Catalina-utility-1  ] - Failed initializing Artifactory context: Artifactory home not initialized.

Eventually, after many failed pings, the services all shut down.

[bryaend]

I've been getting stuck on this for the better part of the last day now. What I think is happening is the services are starting up before the jfrt service is fully bootstrapped. The other services seem to be smart enough (or have timeouts high enough) that they'll continue retrying until they're able to connect to the router at localhost:8046, however, the jffe (JFrog Front End) service appears to be dying out and not restarting itself, thus causing the issue. This explains why a restart fixes it.

artifactory      | 2022-09-14T14:54:26.787Z [jffe ] [INFO ] [                ] [                              ] [main                ] - pinging artifactory, attempt number 180
artifactory      | 2022-09-14T14:54:26.816Z [jffe ] [INFO ] [                ] [                              ] [main                ] - pinging artifactory attempt number 180 failed with code : ECONNREFUSED
artifactory      | 2022-09-14T14:54:27.834Z [jffe ] [ERROR] [                ] [                              ] [main                ] - Error starting application - Error: Failed pinging artifactory for 180connect ECONNREFUSED 127.0.0.1:8046
artifactory      | 2022-09-14T14:54:27.947Z [jffe ] [INFO ] [                ] [                              ] [main                ] - exit detected
artifactory      | 2022-09-14T14:54:27.948Z [jffe ] [INFO ] [                ] [                              ] [main                ] - doing clean up
artifactory      | 2022-09-14T14:54:27.949Z [jffe ] [INFO ] [                ] [                              ] [main                ] - unregistering from router
artifactory      | 2022-09-14T14:54:27.950Z [jffe ] [INFO ] [                ] [                              ] [main                ] - client is already unregistered from router
artifactory      | 2022-09-14T14:54:27.951Z [jffe ] [INFO ] [                ] [                              ] [main                ] - exit code : 0
artifactory      | 2022-09-14T14:54:27.951Z [jffe ] [INFO ] [                ] [                              ] [main                ] - closing recurring jobs
artifactory      | 2022-09-14T14:54:27.952Z [jffe ] [INFO ] [                ] [                              ] [main                ] - closing recurring jobs

The jfac service, for example, appears to keep trying to connect well past 180 seconds (3 minutes), unlike the jffe service:

artifactory      | 2022-09-14T14:56:58.186Z [jfac ] [WARN ] [24718c04392e7d4b] [o.j.c.ExecutionUtils:177      ] [jf-common-pool-1    ] - Retry 400 Elapsed 3.46 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again
artifactory      | 2022-09-14T14:57:03.200Z [jfac ] [WARN ] [24718c04392e7d4b] [o.j.c.ExecutionUtils:177      ] [jf-common-pool-1    ] - Retry 410 Elapsed 3.55 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again

I'll see if I can gather enough evidence to open a bug with JFrog for this. I had to do the same workaround and start with 7.12.6 and then change up the version which is not how any of this should work.

[eldada]

This is an old issue with an old version. I recommend you use the latest version of the Artifactory image as there are many improvements there since 7.12. Latest version is 7.41.12.

[bryaend]

This is an old issue with an old version. I recommend you use the latest version of the Artifactory image as there are many improvements there since 7.12. Latest version is 7.41.12.

Please re-read :) The only way to get this to work is to start with 7.12.6 and then upgrade after it is bootstrapped.

[bryaend]

logs.txt FWIW this exact scenario is still happening with 7.41.12 on a fresh install. Logs attached. This was run per the documentation at https://www.jfrog.com/confluence/display/JFROG/Installing+Artifactory#InstallingArtifactory-DockerInstallation

[eldada]

Thx. I'll have someone in the team review this again with the newer version.

[mhvelplund]

I ran into this issue while trying to bootstrap the configuration of my server. I installed the latest version (7.41.12?) on an Amazon Linux 2 instance in AWS and it started fine.

Then I added users and repositories, and copied the config- and security descriptors from the admin page to /var/opt/jfrog/artifactory/etc/artifactory as artifactory.config.import.xml and security.import.xml respectively. Finally I added a artifactory.lic with the license and a binarystore.xml.

After rebooting, I hade the problem described above. By tinkering with the configuration over multiple resets, I found out that the jffe service would fail to start if any user in the security import file had encoded <userProperty> elements.

Funnily enough, the admin user can have an unencoded user property (the one it has after the first boot with JSON), but after completing the setup, the security descriptor in the export contains some sort of encoded or encrypted value, and that will cause the server to fail on start-up, hanging forever while waiting for the jffe service.

I wasn't able to find anything in the log files about invalid configurations though, and it was purely (protracted) trial and error that led me to this.

[bryaend]

I ran into this issue while trying to bootstrap the configuration of my server. I installed the latest version (7.41.12?) on an Amazon Linux 2 instance in AWS and it started fine.

Then I added users and repositories, and copied the config- and security descriptors from the admin page to /var/opt/jfrog/artifactory/etc/artifactory as artifactory.config.import.xml and security.import.xml respectively. Finally I added a artifactory.lic with the license and a binarystore.xml.

After rebooting, I hade the problem described above. By tinkering with the configuration over multiple resets, I found out that the jffe service would fail to start if any user in the security import file had encoded <userProperty> elements.

Funnily enough, the admin user can have an unencoded user property (the one it has after the first boot with JSON), but after completing the setup, the security descriptor in the export contains some sort of encoded or encrypted value, and that will cause the server to fail on start-up, hanging forever while waiting for the jffe service.

I wasn't able to find anything in the log files about invalid configurations though, and it was purely (protracted) trial and error that led me to this.

I actually found something similar. I got a case open with JFrog and worked with them live to troubleshoot. They had me use their config.sh script that is available from the Downloads page for Docker Compose and it worked first try. When comparing the output of their script (which I don't like for various unrelated reasons), I found that the config it bootstrapped was far slimmer than their sample configs. I essentially took that config, stripped out the stuff I didn't want (again, for various unrelated reasons), and it works every time, now. I didn't dig enough to prove exactly what part(s) of the default sample config are to blame, but, it sounds like a similar issue to what you found. Whatever is happening, there's not enough logging to find what the error is before services start dying. This would be a good challenge for somebody to meticulously add/remove each portion of the config until they find what is causing the issues and get a bug open with JFrog.

[mhvelplund]

They had me use their config.sh script that is available from the Downloads page for Docker Compose and it worked first try.

Maybe add a deep link to help those who come here later?

[bryaend]

They had me use their config.sh script that is available from the Downloads page for Docker Compose and it worked first try.

Maybe add a deep link to help those who come here later?

It's the same as the docker-compose download link on the Download page but this looks like it should be a fairly good permalink.

https://releases.jfrog.io/artifactory/artifactory-pro/org/artifactory/pro/docker/jfrog-artifactory-pro/[RELEASE]/jfrog-artifactory-pro-[RELEASE]-compose.tar.gz

[romain-cros]

I had the same issue that was due to another service using the artifactory default service port 8081. To check if a process is listening on the port, the following command can be used.

netstat -tunapl | grep 8081

I fixed the issue by defining an alternative port in the system.yaml file.

artifactory:
    port: <another port>

[taxi333]

I am having the same issue on a fresh install

[dotkas]

I had the same issue, only to discover I was running on a M1 mac with the DOCKER_DEFAULT_PLATFORM set to amd64, causing it to run emulated. Not sure why, but running it natively on arm64 fixed it for me.