Ant-Media-Server icon indicating copy to clipboard operation
Ant-Media-Server copied to clipboard

Published 20 hours ago • verified publisher icon ant-media

Server hangs after start for a while.

Open MoonTTMM opened this issue 3 years ago • 4 comments

It works fine when just started. But after a while, it stops working, cannot load any liveStreams anymore. And below is the error. It seems the recording causes that.


2022-09-14 18:38:27,978 [vertx-blocked-thread-checker] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-worker-thread-48,5,main] has been blocked for 1406778 ms, time limit is 10000 ms
io.vertx.core.VertxException: Thread blocked
	at io.antmedia.datastore.db.MapDBStore.lambda$new$0(MapDBStore.java:115)
	at io.antmedia.datastore.db.MapDBStore$$Lambda$469/0x00000007c06b6c40.handle(Unknown Source)
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
	at io.vertx.core.impl.ContextImpl$$Lambda$466/0x00000007c06b6040.handle(Unknown Source)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
	at io.vertx.core.impl.ContextImpl$$Lambda$462/0x00000007c06b2c40.run(Unknown Source)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at [email protected]/java.lang.Thread.run(Thread.java:829)
2022-09-14 18:38:27,978 [vertx-blocked-thread-checker] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-worker-thread-58,5,main] has been blocked for 1161809 ms, time limit is 10000 ms
io.vertx.core.VertxException: Thread blocked
	at io.antmedia.datastore.db.MapDBStore.get(MapDBStore.java:187)
	at io.antmedia.AntMediaApplicationAdapter.muxingFinished(AntMediaApplicationAdapter.java:527)
	at io.antmedia.muxer.RecordMuxer.lambda$writeTrailer$0(RecordMuxer.java:149)
	at io.antmedia.muxer.RecordMuxer$$Lambda$501/0x00000007c0766840.handle(Unknown Source)
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
	at io.vertx.core.impl.ContextImpl$$Lambda$466/0x00000007c06b6040.handle(Unknown Source)
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
	at io.vertx.core.impl.ContextImpl$$Lambda$462/0x00000007c06b2c40.run(Unknown Source)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at io.vertx.core.impl.TaskQueue$$Lambda$228/0x00000007c0376440.run(Unknown Source)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at  [email protected]/java.lang.Thread.run(Thread.java:829)

MoonTTMM avatar Sep 14 '22 11:09 MoonTTMM

Hi @MoonTTMM Thanks for creating the issue. For better understanding, kindly send the complete log files. The log files are located in the /usr/local/antmedia/log folder.

yashtandon113 avatar Sep 14 '22 14:09 yashtandon113

According to the log, server hangs at 2022-09-15 04:11 ant-media-server.1.log.zip

MoonTTMM avatar Sep 15 '22 02:09 MoonTTMM

@MoonTTMM Could you please check the status of disk space when server starts misbehaving?

yashtandon113 avatar Sep 19 '22 14:09 yashtandon113

There is plenty of disk space.

MoonTTMM avatar Sep 22 '22 10:09 MoonTTMM

@MoonTTMM Yes, there is a plenty of disk space but it appears that there was other disk-related problem as well after we looked at your server logs. Have you run across this problem again? Have you mounted any additional external disk for the recordings?

yashtandon113 avatar Sep 26 '22 10:09 yashtandon113

@yashtandon113 I still have this problem. I have mounted additional external disk for recording.

MoonTTMM avatar Sep 27 '22 01:09 MoonTTMM

@MoonTTMM Is it possible for you to share your Ant media server credentials (both Dashboard and SSH) so that we can investigate further? You can share it via email at [email protected].

yashtandon113 avatar Oct 04 '22 14:10 yashtandon113

@yashtandon113 I have setup in a LAN，not able to share.

MoonTTMM avatar Oct 10 '22 05:10 MoonTTMM

@MoonTTMM No worries, we can also arrange a meeting to look into this issue together. Please let me know when you are available for the meeting this week. Please share your email address as well, or you can send an email directly to me.

yashtandon113 avatar Oct 10 '22 06:10 yashtandon113

@yashtandon113 I am in China, and all available from 19:00-22:00 UTC+8 except Monday email: [email protected] Let's arrange a meeting.

MoonTTMM avatar Oct 29 '22 09:10 MoonTTMM

@MoonTTMM I have scheduled a meeting for Thursday at 19:00 China time, and an invitation has been sent to your email address. See you at the meeting.

yashtandon113 avatar Nov 01 '22 12:11 yashtandon113

@yashtandon113 very sorry, I am too busy dealing with an accident and miss the meeting, could we arrange another? sorry again.

MoonTTMM avatar Nov 04 '22 07:11 MoonTTMM

some new clues, if antmedia service started, even 'ls' command in the streams folder is very slow，and has some print like, "ls: cannot visit '3087497205821119872383410000.ts': no such file or directory�". If I stop antmedia, 'ls' works fine.

MoonTTMM avatar Nov 04 '22 08:11 MoonTTMM

@MoonTTMM No worries, I can schedule another meeting for tomorrow at 20:00 pm or Thursday at 19:00 pm China time. Please let me know if that works for you.

yashtandon113 avatar Nov 08 '22 13:11 yashtandon113

@yashtandon113 Could we arrange for next Tuesday at 19:00 pm China time?

MoonTTMM avatar Nov 12 '22 09:11 MoonTTMM

@MoonTTMM I have scheduled a meeting for Tuesday (tomorrow) at 19:00 China time, and an invitation has been sent to your email address.

yashtandon113 avatar Nov 14 '22 11:11 yashtandon113

I have add the MP4 Muxing script. But the muxing seems never finish and the mp4.tmp_extension became large. When would the muxing finish and trigger the script?

-rw-r--r-- 1 antmedia antmedia 3.7G 11月 18 04:56 746980858495546707148579-2022-11-17_10-21-44.864.mp4.tmp_extension -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 74698085849554670714857921224.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 74698085849554670714857921225.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 74698085849554670714857921226.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 74698085849554670714857921227.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 74698085849554670714857921228.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 74698085849554670714857921229.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 746980858495546707148579.m3u8 -rw-r--r-- 1 antmedia antmedia 3.2G 11月 18 04:56 779280695186162428552037-2022-11-17_10-21-44.870.mp4.tmp_extension -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 77928069518616242855203721224.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 77928069518616242855203721225.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 77928069518616242855203721226.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 77928069518616242855203721227.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 77928069518616242855203721228.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 77928069518616242855203721229.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 779280695186162428552037.m3u8 -rw-r--r-- 1 antmedia antmedia 3.3G 11月 18 04:56 800408253755660603030520-2022-11-17_10-21-44.796.mp4.tmp_extension -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 80040825375566060303052021224.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 80040825375566060303052021225.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 80040825375566060303052021226.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 80040825375566060303052021227.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 80040825375566060303052021228.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 80040825375566060303052021229.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 800408253755660603030520.m3u8 -rw-r--r-- 1 antmedia antmedia 3.2G 11月 18 04:56 809212786698286597875347-2022-11-17_10-21-44.886.mp4.tmp_extension -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 80921278669828659787534721224.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 80921278669828659787534721225.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 80921278669828659787534721226.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 80921278669828659787534721227.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 80921278669828659787534721228.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 80921278669828659787534721229.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 809212786698286597875347.m3u8 -rw-r--r-- 1 antmedia antmedia 2.2G 11月 18 04:56 871690017092925781971837-2022-11-17_10-21-44.869.mp4.tmp_extension -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 87169001709292578197183721224.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 87169001709292578197183721225.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 87169001709292578197183721226.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 87169001709292578197183721227.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 87169001709292578197183721228.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 87169001709292578197183721229.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 871690017092925781971837.m3u8 -rw-r--r-- 1 antmedia antmedia 3.9G 11月 18 04:56 908781290605994701960032-2022-11-17_10-23-53.820.mp4.tmp_extension -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 90878129060599470196003221224.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 90878129060599470196003221225.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 90878129060599470196003221226.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 90878129060599470196003221227.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 90878129060599470196003221228.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 90878129060599470196003221229.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 908781290605994701960032.m3u8 -rw-r--r-- 1 antmedia antmedia 1.6G 11月 18 04:56 912236832573731522144770-2022-11-17_10-21-44.859.mp4.tmp_extension -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 91223683257373152214477021224.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:56 91223683257373152214477021225.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 91223683257373152214477021226.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 91223683257373152214477021227.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 91223683257373152214477021228.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 91223683257373152214477021229.ts -rw-r--r-- 1 antmedia antmedia 0 11月 18 09:57 912236832573731522144770.m3u8

MoonTTMM avatar Nov 18 '22 02:11 MoonTTMM

Only finalize record to make it mp4 when stop recording?

MoonTTMM avatar Nov 19 '22 09:11 MoonTTMM

@MoonTTMM Yes, you are right. User defined script triggers just after the recording is stopped.

yashtandon113 avatar Nov 21 '22 08:11 yashtandon113

@yashtandon113 So this would not solve my problem. the script would not trigger when keep recording until the disk is full.

MoonTTMM avatar Nov 22 '22 04:11 MoonTTMM

There are two problems:

  1. the script would not trigger until I stop recording manually, so the disk could still get full;
  2. I run the script alone successfully, but when triggered by antmedia, it fails and log shows completing with return value 1, but no more details.

MoonTTMM avatar Nov 28 '22 07:11 MoonTTMM

Hi @MoonTTMM As we continue to follow up on issue #4638, I am closing this one. Regarding the script issue, my colleague will get back to you.

yashtandon113 avatar Dec 19 '22 07:12 yashtandon113