Dynomite Manager deployment on EC2

[netgurru]

Guys I am trying to deploy dynomite manager on tomcat on EC2 and enabled AWS binding and disabled local one. But still getting this error in logs

21-Feb-2021 14:32:13.650 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/opt/apache-tomcat-9.0.41/webapps/dynomite-manager.war] 21-Feb-2021 14:32:16.939 INFO [Catalina-utility-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. log4j:WARN No appenders could be found for logger (com.netflix.governator.LifecycleManager). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 21-Feb-2021 14:32:29.525 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file 21-Feb-2021 14:32:29.531 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [/dynomite-manager] startup failed due to previous errors 21-Feb-2021 14:32:29.537 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.538 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.539 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.539 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.540 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.546 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.546 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.547 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.548 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.548 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_Worker-10] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543) 21-Feb-2021 14:32:29.549 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [Timer-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) 21-Feb-2021 14:32:29.550 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [pool-56-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) 21-Feb-2021 14:32:29.551 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [pool-55-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) 21-Feb-2021 14:32:29.551 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [pool-54-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) 21-Feb-2021 14:32:29.552 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dynomite-manager] appears to have started a thread named [DefaultQuartzScheduler_QuartzSchedulerThread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:244) 21-Feb-2021 14:32:29.554 INFO [Catalina-utility-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/opt/apache-tomcat-9.0.41/webapps/dynomite-manager.war] has finished in [15,903] ms Can you let me know what more steps are required or what other config needs to be updated? I am using openjdk version "1.8.0_272" and Apache Tomcat/9.0.41.

[ipapapa]

This sounds like an issue on how the .war is unpacked in Tomcat. I have not seen that in the past, @smukil can probably provide some context.

[smukil]

@netgurru We've not noticed this error before. This seems like it might be specific to your setting. I did some googling around and I found this which might help: https://stackoverflow.com/questions/40204124/how-to-fix-jars-that-were-scanned-but-no-tlds-were-found-in-them-in-tomcat-9

[netgurru]

But @smukil I have checked all options. Don't understand why application is crashing. I have followed the documentation, although it is very limited. Are there ant config steps to run on AWS EC2 single node or on local ? @ipapapa I have also followed the doc to build the war.

./gradlew build

as per this url https://github.com/Netflix/dynomite-manager instructions. I do not see any specific configurations for it?

[ipapapa]

cc @diegopacheco who had built dynomite-manager external to the Netflix tomcat ecosystem.

[diegopacheco]

@netgurru

These resources might help you out

https://github.com/Netflix/dynomite-manager/wiki/Step-by-step-installation

http://diego-pacheco.blogspot.com/2017/11/running-dynomite-on-aws-with-docker-in.html http://diego-pacheco.blogspot.com/2016/06/running-netflix-dynomite-and-dynomite.html http://diego-pacheco.blogspot.com/2015/11/netflix-dynomitedyno-cluster-for-redis.html http://diego-pacheco.blogspot.com/2018/04/dynomite-remediation.html

[netgurru]

Thanx @diegopacheco let me check this one and get back. Also can you tell me max versions to be used/supported for tomcat, cassandra and java etc?