bmdtools
bmdtools copied to clipboard
lu-zero
Memory usage issue
I am trying to use bmdcapture to pipe to an instance of ffmpeg segmented streaming for 24/7 recording but I keep running into the same issue, running out of memory. The server it is running on has 6GB of ram but after about 2 and a half days the ram fills up completely and bmdcapture dies then the ram is cleared. I have confirmed that piping bmdcapture to /dev/null still fills up the ram.
Are there any suggestions?
ffmpeg isn't supported anymore, please use avconv.
make sure you are processing frames in real-time, the capture just piles up frames if they aren't processed fast enough.
I must still have something incorrect... its memory usage still vastly surpassed the 1GB mark.
This is the command that I am using to capture my live stream: /usr/bin/bmdcapture -C 0 -m 12 -A 2 -V 4 -F nut -f pipe:1 2> /data/bmdCard0.txt | avconv -y -i - -vf "yadif" -c:v "mpeg2video" -q:v 2 -s 640x360 -c:a "mp2" -pix_fmt "yuv420p" -map 0 -loglevel fatal -f segment -vol 24576 -segment_time 60 /dev/shm/card0/out%03d.mpg &> /data/captureCard0.txt
Right now the recordings are being saved as one minute clips on ram disk for later processing. For testing I have them immediately deleted with a cron every minute until I can get this memory issue under control.
Am I leaving something out?
I have tried many different iterations of the command that I posted the other day but I still cannot get this to stop the continuous growth.
What do you mean "processing frames in real-time"?
Unfortunately I can't provide a solution either -- I can just leave a "mee too" here and share some experience. I had the same problem: memory usage just grew over time. During testing, I also tried sending the data to /dev/null which did work for me. So my conclusion was that the encoder was too slow (for whatever reason -- it wasn't maxing out the CPUs). If sending to /dev/null exhibits the same problem for you, it might either be a different issue or a performance problem as well (but even without an encoder).
If the problem itself (encoding is delayed somewhere) cannot be solved, maybe it would be useful to drop frames in bmdcapture. I made some attempts to do so, but failed to drop video and audio frames properly, which led to problems in the encoder (waiting for the related audio frame which would never be sent because it was dropped...).
I have no idea how easy or hard it would be to implement that.
I don't know which libav version you are using, but it might be possible, that some dev version leaked memory (didn't check), so using a release version might be a good idea.
(In my setup I just had to do live streaming for a specific time frame; just upping the bmdcapture memory limit worked for me because of that)
On 23/05/12 12:06, Christian Hoffmann wrote:
Unfortunately I can't provide a solution either -- I can just leave a "mee too" here and share some experience. I had the same problem: memory usage just grew over time. During testing, I also tried sending the data to /dev/null which did work for me. So my conclusion was that the encoder was too slow (for whatever reason -- it wasn't maxing out the CPUs). If sending to /dev/null exhibits the same problem for you, it might either be a different issue or a performance problem as well (but even without an encoder).
If the problem itself (encoding is delayed somewhere) cannot be solved, maybe it would be useful to drop frames in bmdcapture. I made some attempts to do so, but failed to drop video and audio frames properly, which led to problems in the encoder (waiting for the related audio frame which would never be sent because it was dropped...).
I have no idea how easy or hard it would be to implement that.
I'll try to implement something more graceful but IF the problem is slow processing there isn't much you can do that won't spoil the result.
I don't know which libav version you are using, but it might be possible, that some dev version leaked memory (didn't check), so using a release version might be a good idea.
libav is testrun on valgrind in our regression checking system, so it is unlikely.
(In my setup I just had to do live streaming for a specific time frame; just upping the bmdcapture memory limit worked for me because of that)
lu
Luca Barbato Gentoo/linux http://dev.gentoo.org/~lu_zero
The cards being used are the DeckLink SDI and the DeckLink Duo. I am using the following versions: libav-0.8.2-2.fc16.x86_64 bmdcapture (latest from github)
I don't think that the machine is having a problem processing frames because it is a server class box with a quad core and 6GB of ram and 1TB of HD space.
top - 09:18:13 up 2 days, 17:48, 3 users, load average: 0.05, 0.06, 0.05
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.7%sy, 0.0%ni, 98.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6103252k total, 2177248k used, 3926004k free, 15240k buffers
Swap: 0k total, 0k used, 0k free, 133288k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17688 root 20 0 1892m 1.6g 3244 S 17.0 27.4 210:20.79 bmdcapture
This is bmdcapture running for about 20 hours with the following command:
/usr/bin/bmdcapture -C 0 -m 12 -A 2 -V 4 -F nut - -f pipe:1 > /dev/null
It appears that the memory issue is still there.
On 24/05/12 07:21, thenexxuz wrote:
top - 09:18:13 up 2 days, 17:48, 3 users, load average: 0.05, 0.06, 0.05 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 0.7%us, 0.7%sy, 0.0%ni, 98.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 6103252k total, 2177248k used, 3926004k free, 15240k buffers Swap: 0k total, 0k used, 0k free, 133288k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17688 root 20 0 1892m 1.6g 3244 S 17.0 27.4 210:20.79 bmdcaptureThis is bmdcapture running for about 20 hours with the following command: /usr/bin/bmdcapture -C 0 -m 12 -A 2 -V 4 -F nut - -f pipe:1 > /dev/null
It appears that the memory issue is still there.
I'd need a valgrind run, looks like there is a memory leak.
lu
Luca Barbato Gentoo/linux http://dev.gentoo.org/~lu_zero
I used the following command:
valgrind --tool=memcheck --leak-check=full --show-reachable=yes --track-origins=yes -v /usr/bin/bmdcapture -C 0 -m 12 -A 2 -V 4 -F nut - -f pipe:1 > /dev/null
The output is listed here http://pastebin.com/u1kCbAH8
I have the same problem with latest bmdcapture, libav 9_beta1 and BlackMagic SDK961. Still not exist solution for this problem ?
Coworkers have a workaround... cron restarts the capture process after about 9 to 13 hours. We have safeguards in place in case if it wants to restart during a scheduled time then it waits... we do lose about 3 minutes for the restart.
I'm trying my best to track possible memory leaks and plug what valgrind reports. I also integrated in capture support in libav (check my local bmd branch and libbmd) to compare two different implementations of the same concept.
I'm running some tests as this week to compare the result.
Could you please report me exactly what you are doing when the problem appears?
if you just write to /dev/null what happens?
I have the same issue. I compiled bmdtool with libavformat from avconv and ffmpeg. It is the same, the memory usage increases continuously. It is then impossible to use bmdtool for 24/7.
After 2 hours, 31488 root 20 0 425M 285M 2672 S 2.0 28.8 2:30.87 ./bmdcapture -m 11 -A 2 -V 4 -F nut -f pipe:1 The memory usage was 120M at startup and is 285M now.
ffmpeg is not supported. If you are capturing use my bmd branch from libav tree.
On Fri, Apr 19, 2013 at 11:29 AM, ralawa [email protected] wrote:
I have the same issue. I compiled bmdtool with libavformat from avconv and ffmpeg. It is the same, the memory usage increases continuously. It is then impossible to use bmdtool for 24/7.
After 2 hours, 31488 root 20 0 425M 285M 2672 S 2.0 28.8 2:30.87 ./bmdcapture -m 11 -A 2 -V 4 -F nut -f pipe:1 The memory usage was 120M at startup and is 285M now.
— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-16643973 .
https://github.com/lu-zero/libav/tree/bmd
On Fri, Apr 19, 2013 at 2:03 PM, ralawa [email protected] wrote:
Where can we found your "bmd branch from libav tree"?
Thanks.
— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-16649402 .
nut is the problem, for me works avi instead nut without memory leaking
2013/4/19 Luca Barbato [email protected]
https://github.com/lu-zero/libav/tree/bmd
On Fri, Apr 19, 2013 at 2:03 PM, ralawa [email protected] wrote:
Where can we found your "bmd branch from libav tree"?
Thanks.
— Reply to this email directly or view it on GitHub< https://github.com/lu-zero/bmdtools/issues/5#issuecomment-16649402> .
— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-16652133 .
On Fri, Apr 19, 2013 at 3:21 PM, digitalx [email protected] wrote:
nut is the problem, for me works avi instead nut without memory leaking
I'll triplecheck that, thanks for the pointer.
Yes, bmdcapture works fine with avi. I have now an audio/video desync but I currently don't now if it comes from bmdcapture or something else.
On Tue, Apr 23, 2013 at 3:17 PM, ralawa [email protected] wrote:
Yes, bmdcapture works fine with avi. I have now an audio/video desync but I currently don't now if it comes from bmdcapture or something else.
Avi is horrible, I'll try to fix nut later if the problem is there... may I ask some volunteers for testing some variants soon?
I can confirm ralawas test results . With nut memory usage is increasing. With avi memory usage is low and stable but audio-/videosync issue starts. I would also test new variants.
Could you please use massif (part of the valgrind tool collection) and provide the ms_print results?
Once it starts filling up On Apr 28, 2013 5:21 PM, "felixdreinulldrei" [email protected] wrote:
I've started valgrind. How many hours should I run the tool?
— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-17136694 .
here is the link to my result. https://www.dropbox.com/s/yst4gcs2nkgbeuk/massif.out.1534
and here the result of a 12 hour capture process. https://www.dropbox.com/s/j8dqmqljzycc0xb/massif.out.1785
Hi guys - i seem to have the same issue with memory usage.
Using latest Ubuntu 12.04 LTS with latest libav compiled from git and latest Blackmagic driver+sdk (9.7.1) with Blackmagic Intensity Pro.
Memory usage is growing - after about 28hrs process bmdcapture dies with Out of memory and the following syslog message.
http://pastebin.com/pEEW72b5
Machine has 4GB ram and its an i5 processor
Hi,
If you use "asf" instead of "nut" or "avi" it works fine without a/v sync after 12hrs now.
-- Now 48hrs
Found out why using NUT causes the problem, I'll fix that problem soon on Libav. (on why I never noticed with valgrind memcheck, well, it is not a memory leak but a different usage assumption so the whole file index is being built in memory and never freed till the file isn't closed)
Is there any workaround to avoid problem with memory. I got same problem on bmdplay.. a memory is full within 24+ hours
try to use the pipe mode in my nut branch.
On Tue, May 21, 2013 at 8:00 AM, AmnoNBB [email protected] wrote:
I tried.. :) in case bmdplay it's does not work
— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-18190686 .
Is that example of pipe mode? avconv ....... -f nut - | bmdplay -m 2 -f pipe:0 or maybe i don't understand something..
I crafted an experimental mode in nut so it doesn't keep an index and called that mode pipe (since you should use it only in that case) see the documentation in my libav nut branch.
On Wed, May 22, 2013 at 11:43 PM, AmnoNBB [email protected] wrote:
Is that example of pipe mode? avconv ....... -f nut - | bmdplay -m 2 -f pipe:0 or maybe i don't understand something..
— Reply to this email directly or view it on GitHubhttps://github.com/lu-zero/bmdtools/issues/5#issuecomment-18310003 .
I have compiled from experimental branch but, how exactly to enter this format? i try: avconv ....... -f nut -nut_flags pipe - | bmdplay -m 2 -f pipe:0 and it gives me:
Unrecognized option 'nut_flags'. Error splitting the argument list:
how to format the line?
thanks
any idea why this happens sometimes? used command: bmdcapture -m 2 -V 1 -A 1 -F nut -f pipe:1|cat > /dev/null . it happens right after pressing enter. sometimes it does not happen.
[nut @ 0xf52260] pts < dts in stream 0 [nut @ 0xf52260] pts < dts in stream 1 [nut @ 0xf52260] pts < dts in stream 0 [nut @ 0xf52260] pts < dts in stream 1 [nut @ 0xf52260] pts < dts in stream 0 [nut @ 0xf52260] pts < dts in stream 1 [nut @ 0xf52260] pts < dts in stream 0 [nut @ 0xf52260] pts < dts in stream 1 [nut @ 0xf52260] pts < dts in stream 0
or
[asf @ 0x25ac260] pts < dts in stream 0 [asf @ 0x25ac260] pts < dts in stream 1 [asf @ 0x25ac260] pts < dts in stream 0 [asf @ 0x25ac260] pts < dts in stream 1 [asf @ 0x25ac260] pts < dts in stream 0 [asf @ 0x25ac260] pts < dts in stream 1
is there a way to make bmdcapture process kill itself and restart if any error like this occurs?
On Sat, May 25, 2013 at 4:03 PM, felixdreinulldrei <[email protected]
wrote:
is there a way to make bmdcapture process kill itself and restart if any error like this occurs?
I guess I should either open an irc channel or some other media, the bug tracker isn't the best place for this.
lu
Please somebody can explane how to use experimental pipe mode in libav nut branch
I have tried your experimental avconv libav nut branch and successfully compiled it. But the problem with nut still persists.
machine kills process due to "oom-killer" as memory is exhausted completely.
please provide the exact command line and make sure the problem isn't due memory usage.
I'm experiencing this as well and I've tried using both the bmd and nut branches with the same result. Here's the command I'm running:
bmdcapture -C 1 -m 9 -V 4 -M 4 -F nut -f pipe:1 | avconv -y -i - -acodec aac -b:a 128k -strict experimental -vcodec libx264 -filter:v yadif -r 30 -b:v 900k -preset superfast -threads 4 -bt 2500 -qcomp 0.7 -f flv rtmp://myserver.com/
Hi
I think there is some problem with memory handling. I have DeckLink's Quatro card with three inputs enabled and after a while inputs randomly will give me an error:
root@ODE:/DVB# bmdcapture -m 2 -C 2 -A 2 -V 4 -F asf -f pipe:1 Failed to enable video input. Is another application using the card? Segmentation fault (core dumped)
I'm using this card 24/7 and start and stop automatic capture hundreds of times per a day. When I get this error, only way to make input to work again is reboot.
So my question is, that how I can avoid this and/or is there something that can do with orginal code to fix this?
ps. I'm using latest ubuntu server/kernel, latest bmdtools and latest drivers from declink.
Br, Jiii
On Jul 25, 2013 8:29 AM, "jii2" [email protected] wrote:
Hi
I think there is some problem with memory handling. I have DeckLink's Quatro card with three inputs enabled and after a while inputs randomly will give me an error:
root@ODE:/DVB# bmdcapture -m 2 -C 2 -A 2 -V 4 -F asf -f pipe:1 Failed to enable video input. Is another application using the card? Segmentation fault (core dumped)
A backtrace might be nice even if I doubt I can fix the problem.
I'm using this card 24/7 and start and stop automatic capture hundreds of times per a day. When I get this error, only way to make input to work again is reboot.
So my question is, that how I can avoid this and/or is there something that can do with orginal code to fix this?
Ask blackmagic for driver support. If you do not have zombie bmdcapture...
I got 17140 Segmentation fault | ~/bmdplay -C 1 -b 0 -f pipe:0 when tried experimental pipe with bmd branch NUT without flags work but still with memory problem
any new experiences with memory leak problem? maybe some solution meanwhile? using the latest drivers Desktop Video 10.2.3, I am still experiencing increasing memory.
Hi,
this is problem only with NUT. When you use different container, everything is OK. For me work very good stupid .avi container.
Majo
2014-10-28 19:17 GMT+01:00 felix303 [email protected]:
any new experiences with memory leak problem? maybe some solution meanwhile? using the latest drivers Desktop Video 10.2.3, I am still experiencing increasing memory.
— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-60804607.
unfortunately problem still exists. no matter which container. with asf or avi it is just increasing slower. forgot to mention, bmdplay is the one which makes the memory increase.
Its not true. I am using this app for 24h recording system. Whole system has 2GB RAM and run-time 247 days. System continuously record and playback one stream on same machine with Decklink Duo card, everything is passing through AVI containers.
2014-11-01 23:08 GMT+01:00 felix303 [email protected]:
unfortunately problem still exists. no matter which container. with asf or avi it is just increasing slower. forgot to mention, bmdplay is the one which makes the memory increase.
— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-61385560.
There is no problem at all with bmdcapture unless you are using "nut" as mentioned before. You can use avi or asf and test this but with avi you can get A/V sync issues.
I added support to avoptions to use the index-less nut to avoid this problem. I guess I should update the usage info to take that in account. Sorry for the belate response.
On Mon, Nov 3, 2014 at 11:11 AM, CoMMyz [email protected] wrote:
There is no problem at all with bmdcapture unless you are using "nut" as mentioned before. You can use avi or asf and test this but with avi you can get A/V sync issues.
— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-61458496.
@digitalx , in my case I have this problem specifically with bmdplay. I have installed the latest version of bmdtools three days ago. libav was installed from https://github.com/lu-zero/libav. everything builds fine and works. Using Ubuntu 12.04 . I'm using the input from an Intensity Card and output to SDI Monitor Card. commandline is: bmdcapture -C 0 -m 10 -A 1 -V 3 -F nut -f pipe:1 | bmdplay -m 10 -b 2000 -C 1 -f pipe:0 or bmdcapture -C 0 -m 10 -A 1 -V 3 -F avi -f pipe:1 | bmdplay -m 10 -b 2000 -C 1 -f pipe:0 . I have also tried asf which was the one with the slowest increasing memory issue. I wanted to use this system 24/7 without interruption. Maybe it's a little detail I'm missing. Currently I'm using the latest Decklink Driver. Which one are you using?
Hi Lu-zero,
we have the same RAM Problem by 24hour capture Loop
cmdline:
/opt/DecklinkCapture/004_BmdCapture -C 0 -m 13 -A 2 -V 3 -F nut -f pipe:1 |
sudo -u vdr /opt/DecklinkCapture/005_avconv -re -y -i - -c:v mpeg2video -b:v 12000k -r 50 -pix_fmt yuv420p -g:v 5 -q:v 5 -qmin:v 5 -c:a mp2 -r:a 48000 -b:a 384k -g 15 -loglevel debug -aspect 16:9 -f mpegts output.ts
The process starts with low RAM PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 14595 test 20 0 121m 16m 2680 S 59,2 0,4 3:12.49 005_avconv 14592 root 20 0 323m 125m 3392 S 13,3 3,2 0:36.09 004_BmdCapture
Start at 00:00 Uhr, at ~17:00Uhr the bmdcapture process used ~60% and avconv the same. So sporadic at 18:00 hours the bmdcapture process is killing (see not more in the process list). After restart them, it works fine for the next hours.
What we can do... ? Have you a idea ? We use the lastes build of libav-11 (avconv) and the last bmdcapture....
Greets...
Hello lu-zero,
Could you update this thread once you updated the usage info with the index-less nut option? Currently dealing with this same issue of memory leak with nut, and av sync issues with avi and asf. Looking forward to testing that out, as I have been trying to work around this issue for awhile now.
Much appreciated!
The issue is that nut is a raw format and takes up a lot of space. If your input is something like ffmpeg, it will decode the file very quickly and fill up your pipe or buffer space (RAM). bmdplay is a simple encoder which isn't using much CPU and designed to play back in realtime. The result is a backlog of data in the buffer since the decoder is running really fast.
For ffmpeg you can fix this by using the -re flag to do realtime encoding. This prevents buffer fill-up by only encoding at real time.
Im have tried both ffmpeg and avconv using the -re option, and it still leaks memory. What I am running is a Decklink Quad >> bmdcapture >> avconv >> RTMP out. With nut, I get a memory leak, and with avi or asf I get an AV Sync issue.
Sorry, I was referring to bmdplay. For bmdcapture I had to eliminate the -re option and set a sizeable buffer (4 gig?). What are your options?
On Thu, Nov 6, 2014 at 4:24 PM, Klindg [email protected] wrote:
Im have tried both ffmpeg and avconv using the -re option, and it still leaks memory. What I am running is a Decklink Quad >> bmdcapture >> avconv
RTMP out. With nut, I get a memory leak, and with avi or asf I get an AV Sync issue.
— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-62053834.
Hey Rokclimb15... Currently this is what I am running...
./bmdcapture -C 3 -m 9 -V 4 -A 2 -F nut -f pipe:1 | avconv -y -re -r 29.97 -i - -codec:v libx264 -preset veryfast -bf 0 -qmax 24 -g 60 -b:video 4000k -aspect 16:9 -s 1280x720 -codec:a libfdk_aac -b:audio 128k -f flv rtmp://localhost/live/live
Your suggesting...
./bmdcapture -C 3 -m 9 -V 4 -A 2 -F nut -f pipe:1 | avconv -y -r 29.97 -i - -codec:v libx264 -preset veryfast -bf 0 -qmax 24 -g 60 -b:video 4000k -aspect 16:9 -s 1280x720 -codec:a libfdk_aac -b:audio 128k -buffersize 4000000000 -f flv rtmp://localhost/live/live
??
This will need to run 24/7.
Thanks
Yes, drop the -re since it can underflow the buffer and crash. I don't do the exact thing you're doing (I'm doing ffmpeg with HLS segmented output), but I don't set buffersize explicitly on avconv/ffmpeg. I'm also using a named pipe instead of a numbered one (mkfifo). Also, mine runs for a long time sometimes (hours), but never 24/7. How many does in a row do you run it? It is possible that it slowly leaks memory and over enough time, crashes.
On Thu, Nov 6, 2014 at 4:36 PM, Klindg [email protected] wrote:
Hey Rokclimb15... Currently this is what I am running...
./bmdcapture -C 3 -m 9 -V 4 -A 2 -F nut -f pipe:1 | avconv -y -re -r 29.97 -i - -codec:v libx264 -preset veryfast -bf 0 -qmax 24 -g 60 -b:video 4000k -aspect 16:9 -s 1280x720 -codec:a libfdk_aac -b:audio 128k -f flv rtmp://localhost/live/live
Your suggesting...
./bmdcapture -C 3 -m 9 -V 4 -A 2 -F nut -f pipe:1 | avconv -y -r 29.97 -i
- -codec:v libx264 -preset veryfast -bf 0 -qmax 24 -g 60 -b:video 4000k -aspect 16:9 -s 1280x720 -codec:a libfdk_aac -b:audio 128k -buffersize 4000000000 -f flv rtmp://localhost/live/live
??
This will need to run 24/7.
Thanks
— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-62055596.
You can compile latest ffmpeg master with decklink this patch http://git.videolan.org/?p=ffmpeg.git;a=commit;h=bac6cfcb3a1d13c04702e3bd6e54a7515a4c46b9 add deckling input i build auto record system that take SDI input signal and work already 3 month 24/7. One problem with this patch you can't choose video/audio input if you got SDI/Analog.
Rokclimb15,
Up and running minus the -re and with a bufsize set. Thanks for the input, Ill see if this works for me.
Amnonbb,
I have a couple of test boxes here to work with, and did read something about that elsewhere. Thanks for the link, Ill get that spun up on one of my other test servers.
Appreciate it guys, Ill update with my results.
Hi Rokclimb15,
Unfortunately, this caused a memory leak and an AVSync issue.
Amnonbb,
I have not had a chance to try ffmpeg with decklink yet. I was trying to compile ffmpeg with decklink support, via https://trac.ffmpeg.org/wiki/CompilationGuide/Centos, and also https://www.ffmpeg.org/ffmpeg-devices.html section 3.21, but am uncertain as to what I need to add with --extra-cflags and --extra-ldflags. cflags should point to decklinkSDK include directory I believe, but Im unsure as to ldflags... I may be over thinking/going down the wrong path here, and I dont want to fill up this thread with non BMDtools related questions, but would it be possible to private chat about this, or continue this via Email? Very interested as you say you have a working solution thats running 24/7.
Thanks!
Pass to bmdcapture -o strict=experimental:syncpoints=none if you hadn't do that yet. It should solve your problem with NUT completely.
Meanwhile I'm adding encoding support in bmdcapture so you might not need to pass to avconv for the simplest usage patterns =)
Hi Lu-zero,
Sounds good on the encoding support. Im in the middle of reconfiguring the server I am testing with. Ill let you know how -o strict=experimental:syncpoints=none goes once I get that done.
Thanks!
Hi Lu-zero,
Im unable to get this to work. Not sure if I missed something, but here is what I am getting...
./bmdcapture -C 3 -m 9 -I 1 -F nut -o strict=experimental:syncpoints=none -f test.dat and ./bmdcapture -C 3 -m 9 -A 2 -V 4 -F nut -o strict=experimental:syncpoints=none -f test.dat
Just returns the -h output.
using the -I option returns /bmdcapture: invalid option -- 'I'
I can still pull the SDI feed without the -o option via... ./bmdcapture -C 3 -m 9 -A 2 -V 4 -F nut -f test.dat
I think I may be missing something here. I pulled down the latest bmdtools via... https://github.com/lu-zero/bmdtools.git
Of course, trying to pipe it to avconv gives the pipe:: Invalid data found when processing input error.
Thanks!
are you sure On 12 Nov 2014 22:16, "Klindg" [email protected] wrote:
Hi Lu-zero,
Im unable to get this to work. Not sure if I missed something, but here is what I am getting...
./bmdcapture -C 3 -m 9 -I 1 -F nut -o strict=experimental:syncpoints=none -f test.dat
and
./bmdcapture -C 3 -m 9 -A 2 -V 4 -F nut -o strict=experimental:syncpoints=none -f test.dat
Just returns the -h output.
using the -I option returns /bmdcapture: invalid option -- 'I'
I forgot a break;. now should work fine.
Thanks for the quick response Lu-zero!
All set now, up and running. Will update the thread with my results.
Again, Appreciate the quick response.
Now it should work as expected. please tell me if it does not. Sadly I have a good deal of different work task to take care of so I couldn't test bmd in a while =/
Hi lu-zero, I am using your fix strict=experimental:syncpoints=none. In a single pipe this fix completly the memory leak. But now I am trying to tee the pipe, see:
bmdcapture -M 10G -m 11 -A 2 -F nut -o strict=experimental:syncpoints=none -f pipe:1 \
| tee >(record_ffmpeg) | streaming_ffmpeg
Only the first pipe is being filled (the tee pipe), the second pipe seems to not receive any data.
PS: without the syncpoints option the pipe works normally.
FFmpeg is not supported to begin with, the line reported seems to missing a - but I guess is just a typo while reporting.
I hope to have a little time to check if it happens with avconv. Sadly I hadn't had time to push some updates to libbmd and bmdtools I was planning to do this December due some emergencies =/
I guess I should start populating the wiki with examples and scripts on how to do something like that.
I am using the latest avconv from git (not ffmpeg, it is just a naming convention in my bash functions, sry).
just reading it again:
tee is getting is stdout captured twice and that won't work. in order to do something like that you should do:
mkfifo /path/to/recording_pipe
mkfifo /path/to/streaming_pipe
recordme recording_pipe &
streamme streaming_pipe &
bmdcapture -M 10G -m 11 -A 2 -F nut -o strict=experimental:syncpoints=none -f pipe:1 | tee recording_pipe > streaming_pipe
rm pipes
I tested it locally and works as expected.
Hi lu-zero
I'm using bmdcapture to make multiple SDI stream recordings of uncompressed audio/video and I'm seeing a memory leak. When the tool exits it would appear that a copy of the full recording exists in cache. When I run the tool a subsequent time (with the same parameters) then the memory is freed when avio_open() is being called. That said, in the event that the filename parameter does change in a subsequent execution of the tool then the cached file is not overwritten and yet more cache gets eaten up by the tool.
Has anyone else noticed this issue? Is there a possible solution to this memory leak?
--thanks
Linux uses a cache to speed up the io http://www.tldp.org/LDP/sag/html/buffer-cache.html probably that's what you are experiencing.
On Wed, Jan 28, 2015 at 10:19 PM, IDCmbray [email protected] wrote:
Hi lu-zero
I'm using bmdcapture to make multiple SDI stream recordings of uncompressed audio/video and I'm seeing a memory leak. When the tool exits it would appear that a copy of the full recording exists in cache. When I run the tool a subsequent time (with the same parameters) then the memory is freed when avio_open() is being called. That said, in the event that the filename parameter does change in a subsequent execution of the tool then the cached file is not overwritten and yet more cache gets eaten up by the tool.
Has anyone else noticed this issue? Is there a possible solution to this memory leak?
--thanks
— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-71918267.
OK, I was afraid that was the case.
Because of the way my scripts work my I end up with a full 16GB of RAM being used up as cache which causes problems in the processing that must be done on the SDI recordings. I added a hack into my script where I explicitly clear the cache (echo 3 > /proc/sys/vm/drop_caches) between recordings thus freeing up the memory required for the processing but I was hoping for a "cleaner" solution to this problem.
--thanks for the reply :)
You could use a swapfile as big as twice your ram or tune the the vm to make it less impacting. I might get you a patch for avio to improve the situation letting you to pass O_DIRECT and skip the cache totally, but I can't do that today =/
On Thu, Jan 29, 2015 at 7:54 PM, IDCmbray [email protected] wrote:
OK, I was afraid that was the case.
Because of the way my scripts work my I end up with a full 16GB of RAM being used up as cache which causes problems in the processing that must be done on the SDI recordings. I added a hack into my script where I explicitly clear the cache (echo 3 > /proc/sys/vm/drop_caches) between recordings thus freeing up the memory required for the processing but I was hoping for a "cleaner" solution to this problem.
--thanks for the reply :)
— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-72081858.
Hm, why should specific options (for O_DIRECT or something) or special tuning be necessary at all? Filling the free RAM with buffer caches should not have any impact on performance or RAM accessible to other processes.
@IDCmbray, what kind of processing problems are you seeing? If some tool bases its mode of operation on the "cached" memory value, this sounds wrong to me. If it just tries to allocate memory, Linux should flush these buffer caches as-needed without you having to worry about or am I missing something?
@lu-zero That would be awesome and greatly appreciated! Thanks!
@hoffie When re-reading my previous post I realized it may be a little (and by that I mean a lot) misleading. The system I have implemented is for testing a hardware encoder. Basically I'm using a Decklink Duo to output a test pattern over one SDI port of the Decklink, passing that through a hardware encoder who's output is then looped back to the Decklink where I make the recording. The issue I'm seeing is that when the RAM gets eaten up by cache the bmdplay tool begins to fail by either not playing audio and/or playing choppy video (presumably because the pts are not being properly processed). I believe the playback becomes choppy because the playback is always the first process to start on the machine it is therefore always the oldest data and therefore first to be overwritten when new memory is required. The bad playback resulting from the overfilled RAM is what affects my processing. It should be noted that on a typical run through a test sequence I would be obtaining upwards or 1000 recordings of raw SDI each of at least 30 seconds... hence 16 GB of RAM is quickly filled.
With the latest version of bmdcapture and the recommended -o option, there seems no memory leak to be anymore. I have used nut container with no problem so far. Bmdplay is still eating up memory until crash. Tried over days. Maybe it's a little problem with the buffer implementation?
Could you please paste how are you using it?
On Mon, Feb 23, 2015 at 11:43 AM, felixdreinulldrei < [email protected]> wrote:
With the latest version of bmdcapture and the recommended -o option, there seems no memory leak to be anymore. I have used nut container with no problem so far. Bmdplay is still eating up memory until crash. Tried over days. Maybe it's a little problem with the buffer implementation?
— Reply to this email directly or view it on GitHub https://github.com/lu-zero/bmdtools/issues/5#issuecomment-75521048.
for longtime testing i'm using this line with two blackmagic studio2 cards: bmdcapture -C 0 -m 8 -A 2 -V 3 -o strict=experimental:syncpoints=none -F nut -f pipe:1 | bmdplay -C 1 -b 5000 -m 8 -f pipe: