Ant-Media-Server icon indicating copy to clipboard operation
Ant-Media-Server copied to clipboard

RTMP stream not showing on instances launched using AWS marketplace image (with & without GPU)

Open yashtandon113 opened this issue 3 years ago • 5 comments

Short description

After launching the Ant media server using AWS marketplace image the RTMP stream is connected but not showing on dashboard but after restarting the server it started working and reflecting over dashboard.

Environment

  • Ant Media Server version: v2.4.2.1

Steps to reproduce

  1. Launch AMS through AWS marketplace image
  2. Publish RTMP stream

Expected behavior

Stream should show on dashboard panel.

Actual behavior

Stream is connected and publishing fine as well but not showing on dashboard but after restarting the server it works.

Logs

undefinedionContext - Refreshing Root WebApplicationContext: startup date [Sun Mar 06 19:13:18 UTC 2022]; parent: org.springframework.context.support.FileSystemXmlApplicationContext@3e906375 2022-03-06 19:13:18,264 [Loader:/LiveApp] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from ServletContext resource [/WEB-INF/red5-web.xml] 2022-03-06 19:13:18,405 [Loader:/LiveApp] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring 2022-03-06 19:13:18,640 [Loader:/LiveApp] INFO org.red5.server.Context - Setting parent bean factory as core 2022-03-06 19:13:19,918 [Loader:/LiveApp] INFO i.a.datastore.db.DataStoreFactory - Used Datastore:mapdb db name:liveapp 2022-03-06 19:13:19,933 [Loader:/LiveApp] INFO org.red5.server.scope.WebScope - Set server [Server@4fa7daafmap['/root' -> 'default']] 2022-03-06 19:13:19,934 [Loader:/LiveApp] INFO org.red5.server.Server - Add mapping global: default host: context: LiveApp 2022-03-06 19:13:19,935 [Loader:/LiveApp] INFO i.a.AntMediaApplicationAdapter - Delete the ".closed" file in LiveApp 2022-03-06 19:13:19,940 [Loader:/LiveApp] INFO i.a.enterprise.webrtc.WebRTCAdaptor - setting excessive bandwidth algorithm enabled:false 2022-03-06 19:13:19,940 [Loader:/LiveApp] INFO i.a.AntMediaApplicationAdapter - LiveApp started 2022-03-06 19:13:19,948 [vert.x-eventloop-thread-1] INFO i.a.AntMediaApplicationAdapter - Stream source size: 0 2022-03-06 19:13:19,949 [vert.x-eventloop-thread-1] INFO i.a.s.StreamFetcherManager - StreamFetcherSchedule job name 6 2022-03-06 19:13:19,974 [Loader:/WebRTCAppEE] INFO o.s.w.c.s.XmlWebApplicationContext - Refreshing Root WebApplicationContext: startup date [Sun Mar 06 19:13:19 UTC 2022]; parent: org.springframework.context.support.FileSystemXmlApplicationContext@3e906375 2022-03-06 19:13:19,975 [Loader:/WebRTCAppEE] INFO o.s.b.f.xml.XmlBeanDefinitionReader - Loading XML bean definitions from ServletContext resource [/WEB-INF/red5-web.xml] 2022-03-06 19:13:20,128 [Loader:/WebRTCAppEE] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring 2022-03-06 19:13:20,202 [Loader:/WebRTCAppEE] INFO org.red5.server.Context - Setting parent bean factory as core 2022-03-06 19:13:20,445 [Loader:/WebRTCAppEE] INFO i.a.datastore.db.DataStoreFactory - Used Datastore:mapdb db name:webrtcappee 2022-03-06 19:13:20,450 [Loader:/WebRTCAppEE] INFO org.red5.server.scope.WebScope - Set server [Server@4fa7daafmap['/root' -> 'default', '/LiveApp' -> 'default']] 2022-03-06 19:13:20,450 [Loader:/WebRTCAppEE] INFO org.red5.server.Server - Add mapping global: default host: context: WebRTCAppEE 2022-03-06 19:13:20,451 [Loader:/WebRTCAppEE] INFO i.a.AntMediaApplicationAdapter - Delete the ".closed" file in WebRTCAppEE 2022-03-06 19:13:20,452 [Loader:/WebRTCAppEE] INFO i.a.enterprise.webrtc.WebRTCAdaptor - setting excessive bandwidth algorithm enabled:false 2022-03-06 19:13:20,452 [Loader:/WebRTCAppEE] INFO i.a.AntMediaApplicationAdapter - WebRTCAppEE started 2022-03-06 19:13:20,462 [vert.x-eventloop-thread-1] INFO i.a.AntMediaApplicationAdapter - Stream source size: 0 2022-03-06 19:13:20,462 [vert.x-eventloop-thread-1] INFO i.a.s.StreamFetcherManager - StreamFetcherSchedule job name 15 2022-03-06 19:14:12,259 [pool-4-thread-1] INFO org.red5.server.tomcat.WarDeployer - Deploy wars with app start 2022-03-06 19:18:09,844 [http-nio2-0.0.0.0-5080-exec-1] INFO io.antmedia.rest.RestServiceBase - Version Name 2.4.2.1 Version Type Enterprise Edition 2022-03-06 19:18:11,075 [http-nio2-0.0.0.0-5080-exec-10] INFO io.antmedia.statistic.GPUUtils - cuda initialized 2022-03-06 19:18:12,819 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - System cpu load:6 process cpu load:16 available memory: 15032716 KB used memory(RSS): 489372 KB 2022-03-06 19:18:12,820 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - Vertx worker queue size:0 WebRTCVertx worker queue size:0 2022-03-06 19:18:38,108 [http-nio2-0.0.0.0-5080-exec-6] WARN i.a.AntMediaApplicationAdapter - app settings bean updated for WebRTCAppEE 2022-03-06 19:19:00,673 [RTMPConnectionExecutor-1] INFO i.a.e.webrtc.WebRTCApplication - W3C x-category:session x-event:connect c-ip:103.87.56.50 c-client-id:0 2022-03-06 19:19:00,791 [RTMPConnectionExecutor-1] INFO i.a.s.AcceptOnlyStreamsWithWebhook - AcceptOnlyStreamsWithWebhook is not activated for stream testing webhook url: 2022-03-06 19:19:00,792 [RTMPConnectionExecutor-1] INFO i.a.s.AcceptOnlyStreamsInDataStore - AcceptOnlyStreamsInDataStore is not activated. Accepting all streams testing 2022-03-06 19:19:00,808 [RTMPConnectionExecutor-1] INFO org.red5.server.scope.BasicScope - ClientBroadcastStream already exists: org.red5.server.stream.ClientBroadcastStream@5d38f8fa new: org.red5.server.stream.ClientBroadcastStream@5d38f8fa 2022-03-06 19:19:00,809 [RTMPConnectionExecutor-1] INFO o.r.s.stream.ClientBroadcastStream - Stream start: testing 2022-03-06 19:19:00,813 [RTMPConnectionExecutor-1] INFO i.a.e.webrtc.WebRTCApplication - W3C x-category:stream x-event:publish c-ip:103.87.56.50 x-sname:8b73ad92-509a-4823-9d70-995885438f25 x-name:testing 2022-03-06 19:19:02,790 [DeadlockGuardScheduler-1] WARN o.r.s.n.r.ReceivedMessageTaskQueue - Interrupting unfinished active task [sessionId: ODAUQPMBSBQVV; packetNumber: 5; processing: true] 2022-03-06 19:19:02,800 [RTMPConnectionExecutor-2] INFO o.red5.server.stream.AbstractStream - metadata read data type -->>>> core map 2022-03-06 19:19:03,597 [RTMPConnectionExecutor-1] ERROR o.r.server.service.ServiceInvoker - Error executing call: Service: null Method: publish Num Params: 2 0: testing 1: live java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:189) at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:213) at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:309) at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:115) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:72) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.UnsatisfiedLinkError: java.nio.channels.FileLockInterruptionException at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1739) at org.bytedeco.javacpp.Loader.load(Loader.java:1328) at org.bytedeco.javacpp.Loader.load(Loader.java:1132) at org.bytedeco.ffmpeg.global.avcodec.(avcodec.java:18) at io.antmedia.plugin.PacketFeeder.(PacketFeeder.java:27) at io.antmedia.muxer.MuxAdaptor.init(MuxAdaptor.java:354) at io.antmedia.enterprise.adaptive.EncoderAdaptor.init(EncoderAdaptor.java:183) at io.antmedia.muxer.MuxAdaptor.init(MuxAdaptor.java:308) at org.red5.server.stream.ClientBroadcastStream.startPublishing(ClientBroadcastStream.java:984) at org.red5.server.stream.StreamService.publish(StreamService.java:721) ... 16 common frames omitted Caused by: java.nio.channels.FileLockInterruptionException: null at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1138) at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1063) at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:647) at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1582) ... 25 common frames omitted 2022-03-06 19:19:03,598 [RTMPConnectionExecutor-1] WARN o.r.s.net.rtmp.RTMPMinaConnection - Interrupted while waiting for write lock. State: connected java.lang.InterruptedException: null at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1367) at java.base/java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:415) at org.red5.server.net.rtmp.RTMPMinaConnection.write(RTMPMinaConnection.java:342) at org.red5.server.net.rtmp.Channel.write(Channel.java:139) at org.red5.server.net.rtmp.Channel.sendStatus(Channel.java:194) at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:314) at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:115) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:72) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2022-03-06 19:19:03,599 [RTMPConnectionExecutor-1] INFO o.r.s.net.rtmp.RTMPMinaConnection - Session id: ODAUQPMBSBQVV in queue size: 0 pending msgs: 1 last ping/pong: 0 2022-03-06 19:19:03,599 [RTMPConnectionExecutor-1] INFO o.r.s.net.rtmp.RTMPMinaConnection - Available permits - decoder: 1 encoder: 1

yashtandon113 avatar Mar 06 '22 19:03 yashtandon113

java.lang.reflect.InvocationTargetException: null at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:189) at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:213) at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:309) at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:115) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:72) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.UnsatisfiedLinkError: java.nio.channels.FileLockInterruptionException at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1739) at org.bytedeco.javacpp.Loader.load(Loader.java:1328) at org.bytedeco.javacpp.Loader.load(Loader.java:1132) at org.bytedeco.ffmpeg.global.avcodec.(avcodec.java:18) at io.antmedia.plugin.PacketFeeder.(PacketFeeder.java:27) at io.antmedia.muxer.MuxAdaptor.init(MuxAdaptor.java:354) at io.antmedia.enterprise.adaptive.EncoderAdaptor.init(EncoderAdaptor.java:183) at io.antmedia.muxer.MuxAdaptor.init(MuxAdaptor.java:308) at org.red5.server.stream.ClientBroadcastStream.startPublishing(ClientBroadcastStream.java:984) at org.red5.server.stream.StreamService.publish(StreamService.java:721) ... 16 common frames omitted Caused by: java.nio.channels.FileLockInterruptionException: null at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1138) at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1063) at org.bytedeco.javacpp.Loader.cacheResource(Loader.java:647) at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1582) ... 25 common frames omitted 2022-03-07 12:12:57,983 [RTMPConnectionExecutor-1] WARN o.r.s.net.rtmp.RTMPMinaConnection - Interrupted while waiting for write lock. State: connected java.lang.InterruptedException: null at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1367) at java.base/java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:415) at org.red5.server.net.rtmp.RTMPMinaConnection.write(RTMPMinaConnection.java:342) at org.red5.server.net.rtmp.Channel.write(Channel.java:139) at org.red5.server.net.rtmp.Channel.sendStatus(Channel.java:194) at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:314) at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:115) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:72) at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

burak-58 avatar Mar 07 '22 12:03 burak-58

Probably related or the same as #3400

justinh24 avatar Apr 29 '22 19:04 justinh24

It still continues on v2.4.3. @burak-58 FYI. ams.log

SelimEmre avatar Aug 01 '22 21:08 SelimEmre

Any estimates on when this will be fixed? I am still running old versions because this bug prevents our automated system from working.

justinh24 avatar Sep 02 '22 20:09 justinh24

Hi @justinh24 ,

Do you have scenario that reproduces this problem? We don't have the scenario to reproduce it every time.

I think it will be easier for us if we can reproduce the problem.

mekya avatar Sep 05 '22 10:09 mekya

It's possible to reproduce even with AMS 2.5.1. I use Hashicorp Packer to create the AMI from Ubuntu 20.04 LTS and in the Packer script I call a bash script to install AMS. When I afterwards launch an EC2 from the generated AMI, the issue does occur. Restarting the antmedia service on the box solves the problem.

kboek avatar Jan 09 '23 12:01 kboek

FYI please

The solution is recommended to @kboek on the community post below. https://community.antmedia.io/topic/1549-invocationtargetexception-and-when-launching-from-custom-ami/#comment-5711

yashtandon113 avatar Jan 09 '23 14:01 yashtandon113

@yashtandon113 Link is broken

gorghino avatar Feb 29 '24 04:02 gorghino