FFmpegFrameGrabber - How to set devicePath or deviceFile on Linux

[jkody]

Hi

I am found this source which shows how to read video from cam and audio from microphone https://github.com/bytedeco/javacv/blob/master/samples/WebcamAndMicrophoneCapture.java#L120 ; Every time I run the example I get this exception :

-----begin----- VIDEOIO ERROR: V4L: index 1 is not correct! Exception in thread "main" org.bytedeco.javacv.FrameGrabber$Exception: cvCreateCameraCapture() Error: Could not create camera capture. at org.bytedeco.javacv.OpenCVFrameGrabber.start(OpenCVFrameGrabber.java:179) at my.atest1.WebcamAndMicrophoneCapture.main(WebcamAndMicrophoneCapture.java:64) Java Result: 1 -----end-----

...so define index as constant is not working and there should be a dynamic way (a filter)?

The thing is I watching this source https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameGrabber.java#L79 and I can see there as arg should be devicePath or deviceFile for example; So to get stream from web cam or microphone I need to get their device paths or files but I don't know how? So my question is... how to get(?filter) the FFmpegFrameGrabber required devices info in the standard way? And if there are related tutorials please share me the link(s)

Thanks

p.s.

• OS : Linux Arch x64
• jdk 1.7
• javacv-1.0

[saudet]

Yes, we need to set the index to the desired one. OpenCV doesn't offer a way to list devices, but it looks like recent versions of FFmpeg do, so it would be something to try and add to FFmpegFrameGrabber. Would you be interested in working on that yourself? It would be a welcome contribution :)

The paths we can use for video4linux2, directshow, and avfoundation are listed here: https://www.ffmpeg.org/ffmpeg-devices.html

[jkody]

@saudet Thank you for your reply

I'd love to do some contribution but I am not sure where to start at :) I am just testing some javacv examples right now; Correct me if I am wrong but seems like the ffmpeg is written on C; So Is there a Java wrapper for the lib? I mean wrapper do you recommend?

p.s. I have Linux x64 platform right now

[saudet]

Great! Let me know if have encounter any issues that are preventing you from making progress, and send a PR once you have something working. Thanks!

Yes, FFmpeg is written in C, and the wrappers are here: https://github.com/bytedeco/javacpp-presets/tree/master/ffmpeg

[jkody]

I am using Linux x64 right now; As a test I tried to get audio from speakers or headphones like a :

FFmpegFrameGrabber grabbera = FFmpegFrameGrabber.createDefault("hw:0,0"); ... graberra.setFormat("alsa");

but, as a result, I get sound from microphone only :( I have duplex jack which may work with headphones and microphone; So what am I doing wrong? I mean how to get audio from speakers/headphones? I am watching the ffmpeg docs http://ffmpeg.org/ffmpeg.html#Video-and-Audio-grabbing right now but cannot see any related example :S

So give me a piece of advice please

[saudet]

I don't see "microphone", "speakers", or "headphones" written anywhere in FFmpeg's documentation. Where did you see that?

[jkody]

See these audio interaction args "ffmpeg -f alsa -ac 1 -i hw:1 -f video4linux2 -i /dev/video0 /tmp/out.mpg" here http://ffmpeg.org/ffmpeg.html#Video-and-Audio-grabbing I just tried to adapt the example for using FFmpegFrameGrabber and changed device path according to my /proc/asound/... data... But still I am not sure how to set correct device path in case of speakers or headphones? Seems like all I tried (according to aplay -L) lead to microphone or does nothing :P

------aplay -L output---- null Discard all samples (playback) or generate zero samples (capture) pulse PulseAudio Sound Server default Default ALSA Output (currently PulseAudio Sound Server) sysdefault:CARD=PCH HDA Intel PCH, ALC270 Analog Default Audio Device front:CARD=PCH,DEV=0 HDA Intel PCH, ALC270 Analog Front speakers -------------etc... -------------

So I wanted to ask you how to get audio from speakers with FFmpegFrameGrabber or there should be another grabber or device path or...?

EDIT: Concerning the output... I just found this docs thread https://www.ffmpeg.org/ffmpeg-devices.html#Output-Devices and this https://www.ffmpeg.org/ffmpeg-devices.html#pulse-1 ; OK the $pactl list sinks outputs :

-----pactl list sinks outputs : ------- Sink #4 State: RUNNING Name: alsa_output.pci... ... Definition Audio Controller" device.form_factor = "internal" device.string = "front:0" ... ------------------etc...--------------

...but still I cannot grab any output audio from headphones (in this particular case) with code like a :

FFmpegFrameGrabber grabberra = FFmpegFrameGrabber.createDefault("front:0"); graberra.setFormat("alsa");

So please give me a tip here

[saudet]

I don't know, you'll need to figure out the correct device path. Maybe ask the ALSA guys?

[jkody]

What exactly mean args "-i" and "-f" in javacv wrapper in case of FFmpegFrameGrabber or FFmpegFrameRecorder? Is there an args parser table or similar? Please give me a tip

[saudet]

"-i" is the input filename or something, and the argument to the constructor of FFmpegFrameGrabber. "-f" is the format, which we can use with setFormat().

[saudet]

It's not well defined and there's no one-to-one mapping. It's something you could work on if you would to? It would make a great contribution. :) I guess a Wiki page on that would be most appropriate. Everyone should have write access to it, so feel free to create one. Thanks!

[jkody]

@saudet

OpenCV doesn't offer a way to list devices, but it looks like recent versions of FFmpeg do...

what older versions of FFmpeg list devices? Do you have the versions links?

[saudet]

According to the doc (https://ffmpeg.org/doxygen/3.0/avdevice_8h.html) they are functions like avdevice_list_devices()...

[jkody]

@saudet OK; Seems like wrapper https://github.com/bytedeco/javacpp-presets/tree/master/ffmpeg has ability to get devices list as well

int avdevice_list_devices = avdevice.avdevice_list_devices(afc, adil);

...so I want to test the way of getting devices list with java wrapper but I cannot find the exact algorythm example of methods, consts, fields optimal invoking; so maybe there are some similar examples in C lang which demo that or still there is an example I needed to analyze in javacpp presets collection? And concerning javacpp presets native C wrapping is getting devices list working with linux? Please guide me

[jkody]

I don't know, you'll need to figure out the correct device path. Maybe ask the ALSA guys

Yeah, but I wanted to ask does the ffmpeg java wrapper can read speakers stream at all? I mean do I go right direction?

[saudet]

I think FFmpeg can capture audio, but we have Java Sound anyway, so why not use that?

[jkody]

I think FFmpeg can capture audio, but we have Java Sound anyway, so why not use that?

Thank you for your soon reply.

I know. But seems like Java Sound has problems with extracting speakers stream out. Java Sound API just reads microphone or(and) write to speakers for example (the more or less standard usage way) :P

I am asking about ffmpeg java wrapper how far it supports original ffmpeg args? I tried just device path but what about terminal input like a :

$ffmpeg -f pulse -i ...

I mean ffmpeg args using ffmpeg java wrapper

[saudet]

setFormat() is used to set the format, so have you tried that?

[jkody]

setFormat() is used to set the format, so have you tried that?

No, I haven't; OK. I'll try it but it has less docs, as I can see that... Are there some tutorials about that?

[saudet]

There are samples, the wiki, the API docs... Contributions are welcome though! It's not possible for me to do everything on my own. Let me know if you encounter any problems and I'll help. Thanks

[saudet]

FFmpeg has some support for listing devices: https://trac.ffmpeg.org/wiki/Capture/Webcam

We just need someone to make a contribution :)

[alanwhite]

Been researching this, the FFmpeg code seems to log it's list of devices, rather than libavdevice return them somehow, in response to ffmpeg -f 'avfoundation' -list_devices true -i '', the following code achieves the same (except for some reason it logs it twice)

import org.bytedeco.ffmpeg.avformat.AVInputFormat;
import org.bytedeco.ffmpeg.global.avdevice;
import org.bytedeco.javacv.FFmpegFrameGrabber;

public class Cameras {
	public static void main(String[] args) {
		avdevice.avdevice_register_all();
		AVInputFormat inp = avdevice.av_input_video_device_next(null);
		while ( inp != null ) {
			String format = inp.name().getString();

			FFmpegFrameGrabber lister = new FFmpegFrameGrabber(""); 
			lister.setFormat(format); 
			lister.setOption("list_devices", "true"); 
			try { 
				lister.start(); 
			} catch (Exception e) { 
				// cannot open "dummy": ignore exception 
			} 

			inp.close();
			inp = avdevice.av_input_video_device_next(inp);
		}
	}
}

Capturing stdout and parsing it seems quite clunky, in my apps so far I open 3 tiny video streams with devices 0,1 and 2 and ask the user to pick which camera to use but that's also very clunky.

There must be a way somewhere in here to provide a extract of all the devices FFmpeg knows about and their capabilities to provide a reasonable ux. Despite research and lots of trial and error I'm coming up with nothing, is anyone else actively working how to list cameras and their capabilities plse?

[xiaomengxin123]

so if I just wanna to use FFmpegFrameGrabber to open local camera, it's inconvenient？

[bthai1234]

Not sure how relevant this is now to this older issue, but ill comment it here in case anyone else stumbles upon this. i was able to get some input device names in windows using the maven javacpp ffmpeg-platform version 5.1.2-1.5.8 with the following code:

avdevice.avdevice_register_all();
AVInputFormat fmt = avdevice.av_input_video_device_next( null );
ArrayList<String> list = new ArrayList<>();
while ( fmt != null )
{
  AVDeviceInfoList deviceInfo = new AVDeviceInfoList();
  AVDictionary options = new AVDictionary();
  avutil.av_dict_set( options, "list_devices", "true", 0 );
  int size = avdevice.avdevice_list_input_sources( fmt,"video=dummy", options, deviceInfo  );

  if( !deviceInfo.isNull() )
  {
    for ( int i = 0; i < size; i++ )
    {
      list.add( deviceInfo.devices( i ).device_description().getString() );
    }
  }

  fmt.close();
  fmt = avdevice.av_input_video_device_next( fmt );
}
System.out.println( list );

I only tested this on windows, not sure how it holds up on linux or mac. This also doesn't seem to work in older versions of ffmpeg-platform, im not sure why. If anyone knows how to get it working in older versions that would be nice.