Implement AVOptions support

[kmsquire]

AVOptions is meant to be a version-independent method for getting and setting options in ffmpeg/libav structs. It works kind of like a dictionary.

This would make solutions like #26 much nicer, in that they could properly call library functions instead of command line tools.

[kmsquire]

@kmsquire. I have been running and testing through the AVOptions in ffmpeg and there is obviously quite a few. However, since audio is not a priority right now and as far as I understand it is not functional yet with VideoIO, I would prefer to concentrate primarily on the video options for now.

Of the "generic" AVOptions, I found AVFormatContext in libavformat_h.jl (below), and AVCodecContext in libavcodec_h.jl. However, I am a little confused about the organization of these options (and their naming) in the VideoIO files relative to the original c/h files in libavformat, libavdevice and libavcodec. Could you explain how you organized these in your files? I need to find all the private options (containers, devices and codecs) as well. Thanks for your help.

In libavformat_h.jl immutable AVFormatContext

av_class::Ptr{AVClass} iformat::Ptr{AVInputFormat} oformat::Ptr{AVOutputFormat} priv_data::Ptr{Void} . . . end

In libavcodec_h.jl immutable AVCodecContext av_class::Ptr{AVClass} bit_rate::Cint bit_rate_tolerance::Cint flags::Cint . . . end

@maxruby, sorry not to get back to you sooner. I moved this discussion to #27 because I thought the discussion might get long for #38.

The files you pointed out were auto generated by Clang.jl, using the script in the util directory. The README file there gives some details. One thing that's missing from there is related to your question.

When Clang.jl wraps a bunch of header files, such as those for each library in ffmpeg, it

1. Converts all structs to types
2. Converts #defines to either type aliases or constants
3. Wraps function definitions

The wrapped functions end up in a file with the same base name as the header file. Everything else (types, type aliases, and constants) are collected together and put in one big file (e.g., libavcodec_h.jl).

For ffmpeg, there's a larger overall structure which is described in detail in various README files.

For AVOption, I would suggest focusing first on getting something simple working with one of the types/structs that supports the AVOption interface, such as one of those you mentioned. I would focus on supporting the string interface first, since (I think) all options can be set using strings (ffmpeg does the conversion).

It's also possible that the structs currently in use aren't initialized properly to support AVOptions, so that might need to change.

Hope this helps. Let me know what might not be clear. Cheers!

[maxruby]

This definitely helps! The overall structure of ffmpeg is nicely described in README - I got that far. The part I did not understand is to do with calling all the private (rather than generic) options from AVOptions since they are all dependent on the container, device or codec. Although I am still trying to work it out, I think that you might be right that the structs are not currently initalized in a way that they can be used directly to support AVOptions. I am hoping to clear this up first before moving to the next level. . .

[maxruby]

Having gone through your files, its now much more clear what I need to do to access the parameters in the immutable types AVCodecContext and AVForrmatConext. I see you created e.g., the AVFrame() function in LIBAVCODEC.jl. to call all the parameters (types) from AVFrame (defined in libavcodec_h.jl). I have a couple of related questions:

1.LIBAVCODEC.jl in AVCodecs v53 and v54 contains the AVFrame() function, but there isn´t one in v55. Is there a reason for this? Are those the versions you were working with? The latest ffmpeg release of libavcodec (as of today) has changed to v56 and the rest of them too. libavutil 54. 7.100 libavcodec 56. 1.100 libavformat 56. 4.101 libavdevice 56. 0.100 libavfilter 5. 1.100 libavresample 2. 1. 0 libswscale 3. 0.100 libswresample 1. 1.100 libpostproc 53. 0.100

Do you think we should take this into account when completing the wrappers? How much change is there between recent versions?

2.Since I plan to add a function AVCodecContext() similar tot AVFrame() to access the parameters from the AVCodecContext type, I wonder whether it is really is useful to support multiple version of the libraries (v52, v53, v55). I assume that support for v55/v56 should be sufficient to do the job?

Cheers, Max

[kmsquire]

Addressing your second question first: it would certainly be easier to support only the latest version of ffmpeg/libav. However, the main reason I went the more complicated route is that I run Linux, and

1. Different Linux distributions distribute different versions of the libraries (e.g., Debian/Ubuntu only have libav in their repositories, and only a particular version).
2. Some users do not have admin rights on the machines they use, and are stuck with whatever version of the libraries are installed on their machines.

In particular, some users who have expressed interest in using this package (e.g., Lucas Beyer) have some of these issues.

This affects Mac and Windows users less, since we have greater control over what's installed there.

In general, the libraries haven't changed much since the first version I mentioned, so this hasn't been too much of a problem so far. But, e.g., AVFrame moved from libavcodec to libavutil in later versions of libav, so the AVFrame() function you found in LIBAVCODEC.jl for versions v53 and v54 appears instead in AVUtil/v53/LIBAVUTIL.jl. (One downside of the current organization is that it isn't obvious which library versions go with each other.)

We should wrap the latest ffmpeg versions. Would you mind creating an issue for it?

[maxruby]

Now it makes sense! Sure, I will create a new issue entitled "Wrapping latest versions of ffmpeg".

[lucasb-eyer]

Especially Ubuntu 12.04 LTS will still be used for quite some time in and around my office, and they only ship v53. That's why I'd like to keep support for that version as long as the effort for it stays reasonable.

Edit: if you ever need testing for something with these old versions, just ping me.

[maxruby]

OK. I will work on support for as many libraries as I can. Personally I would prefer to support the latest ones and those essential for Linux systems like v53.

[kmsquire]

Hopefully, the files under ffmpeg and libav are mostly create once--at least the auto-generated parts. For the others, some rearrangement/consolidation is probably possible, and may be worthwhile, e.g., for files like src/ffmpeg/AVCodecs/src/AVCodecs.jl. That still means duplication for some of the other files, though.

[maxruby]

@kmsquire. I have been testing several ideas about how to create access to the AVOptions in ffmpeh/libav within avio.jl and possibly some in LIBCODEC.jl, LIBAVUTIL.jl depending on what is appropriate. One is simple with minimal interface, the other might involve better documentation info, e.g., show() to the user, so that we can have better control to choose which parameters can be changed in e.g., AVInput, AVFrame, AVCodecContext, AVFrameContext, etc. I think this is actually something even FFMPEG/LIBAV could improve as it is challenging for a beginner especially to remember all the options available, even if they are available at the shell prompt.

I am finding it rather challenging to keep track of the many types and parameters from which AVCodecContext and AVFormatContext inherit their types. For example, in your avio.jl, I noticed your comment to create av_opt_set as follows:

     #av_opt_set(avin.apFormatContext[1], "probesize", "100000000", 0)
     #av_opt_set(avin.apFormatContext[1], "analyzeduration", "1000000", 0)

I got that avin is an AVInput type, with apFormatContext, a vector of pointers to AVFormatContext that contains "probesize" a AVRational type. I assume your idea here was to set probesize (Uint32) to "100000000".

Here are my questions:

1. Do you think it is worth trying to have as many of these options loaded with VideoIO or should AVOptions be considered a sort of separate module within VideoIO?
2. Is there a way in Julia to easily list all the parameters/types and functions declared within each AVFrame, AVFormatContext, etc - an inheritance tree for types and functions?

Thanks for your help.

[kmsquire]

Hi Max, I really appreciate you looking into this, and I think I haven't been that clear what I had in mind.

1. I would focus first on what the interface might look like. I haven't written this anywhere, but I had in mind something like supporting

   avin.apFormatContext[1][:probesize] = 100000000
   avin.apFormatContext[1][:analyzeduration] = 1000000
   

   This should be straightforward, and would only involve creating appropriate getindex and setindex! functions. Let me know if you need/want more details.

2. Initially I think access to this would be mostly for internal use. So instead of trying to support all possible options of all possible types, I would suggest focusing on one problem that needs to be solved using AVOptions:

   • For https://github.com/kmsquire/VideoIO.jl/issues/25#issuecomment-53836419, the iformat field of AVFormatContext needs to be set (see https://github.com/kmsquire/VideoIO.jl/issues/25#issuecomment-53674736 for some sample code) (Windows and Mac).
   • For #31, we need to use a AVDeviceCapabilitiesQuery struct (see https://www.ffmpeg.org/doxygen/trunk/structAVDeviceCapabilitiesQuery.html), which needs to be set up using AVOptions

1. Do you think it is worth trying to have as many of these options loaded with VideoIO or should AVOptions be considered a sort of separate module within VideoIO?

At this point, I would just include any needed functions in src/util.jl.

1. Is there a way in Julia to easily list all the parameters/types and functions declared within each AVFrame, AVFormatContext, etc - an inheritance tree for types and functions?

Again, I wouldn't want to support everything right now, because we don't use most of the ffmpeg/libav API. That said:

julia> using VideoIO

julia> collect(zip(names(AVCodecs.AVFrame), AVCodecs.AVFrame.types))
42-element Array{(Symbol,DataType),1}:
 (:data,Array_8_Ptr)                 
 (:linesize,Array_8_Cint)                           
 ⋮                                   
 (:sample_rate,Int32)                
 (:channel_layout,Uint64)            

(AVFrame doesn't support AVOptions, because the first member isn't an AVClass type.)

[maxruby]

That is a relief. Being a perfectionist-type, I thought that I should pursue a general support of as many AVOptions as possible. . . But I also agree that it is much better to start with a basic support for the immediately essential functions at least internally. That said, my main motivation is to access the filtering/encoding options from ffmpeg to do live video streaming for now (and hoping eventually to link this with an OpenCV wrapper) . I noticed that some people are using OpenCV with ffmpeg and it would be fantastic to transfer this combination to Julia for live streaming/computer vision. http://siddhantahuja.wordpress.com/2011/08/15/ros-opencv-read-a-video-file-using-ffmpeg/

[kmsquire]

:+1

[maxruby]

Hi Kevin, Thanks again for your patience. I apologize for asking so many questions.

Q1. Is it possible to use the av_setfield function (defined in util.jl) to set iformat, probesize and max_analyze_duration (the same way it was used to set pb -> apAVIOContext[1] in line 193 of avio.jl)?

iformat::Ptr{AVInputFormat} -> avin.apAVInputFormat[1]

    av_setfield(avin.apFormatContext[1], :iformat, avin.apAVInputFormat[1]) ? (Q2 below)

probesize::Uint32 -> 100000000

    av_setfield(avin.apFormatContext[1], :probesize, 100000000)

max_analyze_duration::Int32 -> 1000000

   av_setfield(avin.apFormatContext[1], :max_analyze_duration, 1000000)

Q2. Should iformat = avin.apAVInputFormat[1]? If so should apAVInputFormat be added to the type declaration?

   type AVInput{I} 
      . . .   
       apAVInputFormat::Vector{Ptr{AVInputFormat}} 
    end

[kmsquire]

Hi Max, No worries! I've been finding it difficult to make time to actually work on this stuff, and it really will be easier in the long run if someone other than just me understands it. So I'm quite happy to help you in that direction.

Q1. Is it possible to use the av_setfield function (defined in util.jl) to set iformat, probesize and max_analyze_duration (the same way it was used to set pb -> apAVIOContext[1] in line 193 of avio.jl)?

Short answer: only as a short-term solution. av_setfield itself is slightly sketchy. Useful, but sketchy.

Long answer: from version to version of ffmpeg, the structs supporting the AVOption interface can change (even for minor versions), and the fields set with the AVOption interface may disappear, even if the option itself doesn't (see, e.g., here).

If we implement things right, eventually we wouldn't even need definitions for structs that support AVOptions--we could just create, initialize, and destroy the objects with library calls, set values in them with the AVOptions API, and pass them around as opaque pointers. Getting rid of larger immutable type definitions in Julia should also allow compilation of VideoIO to be faster.

That said, it might be nice just to get something working for now, and if we can fix #25 and #31 with av_setfield, we could just do it and worry about AVOptions later. ;-)

Q2. Should iformat = avin.apAVInputFormat[1]? If so should apAVInputFormat be added to the type declaration?

I don't really know. For #25, we're wanting to get the list of video devices, which means we don't know the format. I only know that it needs to be set by looking at the source.

[maxruby]

I am happy to implement properly the library AVOptions API in VideoIO. After reading carefully the ffmpeg h files (e.g., avformat.h), it is more clear what these different options mean. I suggest the following plan:

1. I add code with av_setfield or something similar in src/util.jl to fix #25 and #31 (and for any other requests that might come up soon). For #25, I think apAVInputFormat[1] should contain all the options (including formats) which is passed to :iformat in AVFormatContext, but I will check again if this correct and how to use it later to actually set the format.
2. In the meantime, I will work on the AVOptions API with direct library calls as you suggest. Given as you say that the structs and fields in ffmpeg may disappear even with minor bug fixes, it makes a lot of sense to talk directly to the library API rather than creating immutable types in Julia and passing them around. I was actually wondering whether so many immutable declarations could be avoided. . . Cheers, Max

[kmsquire]

Sounds like a plan!

[maxruby]

@kmsquire. By the way, I think we should include in the VideoIO.jl file: using Images
using ImageView
I understand that it may slow down the first time you import VideoIO, but it seems at least right now essential to play videos and do anything with the images captured. Anything against this?

[kmsquire]

The current situation was actually requested, to reduce startup times for those who don't need or don't want to use those packages (e.g., someone doing batch processing or using GLPlot).

I haven't followed all of the module related changes recently, however, so there may be a middle ground, where, e.g., we can pass parameters when loading a module. But I don't know if that functionality was merged yet.

Cheers, Kevin

On Tuesday, September 30, 2014, Maximiliano Suster [email protected] wrote:

@kmsquire https://github.com/kmsquire. By the way, I think we should include in the VideoIO.jl file: using Images

using ImageView

I understand that it may slow down the first time you import VideoIO, but it seems at least right now essential to play videos and do anything with the images captured. Anything against this?

— Reply to this email directly or view it on GitHub https://github.com/kmsquire/VideoIO.jl/issues/27#issuecomment-57301593.

[timholy]

Even I can understand why you might want to support running without Images and ImageView :smile:.

That said, the best solution is to make loading fast, and then make use of these two packages. I'm keeping a close eye on (and mildly helping out with) https://github.com/vtjnash/Speed.jl.

[maxruby]

@kmsquire. In addition to the av_opt_set and av_opt_get AVOptions we can also use the AVDictionary structure and set key,value pairs by calling av_opt_set_dict() on either AVFormatContext or AVCodecContext. One particular case where this is crucial is when we don´t know the input file format, i.e., before the file is actually opened. I found the AVDictionaryEntry declaration in livabutil_h.jl, but the declaration of AVDictionary fields in dict.c is not present in VideoIO. I would like to try to use AVDictionary (and it should not be difficult to use) - so we can have a a more global solution.

I can not find a declaration of AVDictionary like this in the AVUtil module, is there a reason for this? Sorry for my ignorance.

   type AVDictionary               # struct in dict.c
      count::Int32
      elems:Ptr{AVDictionaryEntry}
   end

Cheers, Max

[kmsquire]

Hi Max, making AVDictionary accessible sounds like a good plan. The reason that the type doesn't exist is precisely because it's defined in dict.c, and not in any of the header files. In the header files, it's just available as typedef struct AVDictionary AVDictionary;, which means its implementation is not part of the external api, and it's meant to be used as an opaque structure or pointer. So, all operations on AVDictionaries (creation, adding and removing keys/values, etc.) should be done using the public APIs.

Actually, I think AVDictionary, not AVOption is what I was thinking of when I proposed the dictionary-like interface above. So I'm glad you found and proposed to wrap it. ;-)

[maxruby]

Thanks for the explanation. Yes, I have used AVDictionary in VideoIO for the simple case above, used av_dict_set() and passed the AVDictionary to avformat_open_input. The AVOptions API also relies on key, value pairs like AVDictionary but has more functionality (e.g., we can use av_opt_find to search all the formats, codecs, etc in the AVFormatContext or AVCodecContext structures). And we can also use AVDictionary as input to av_opt_set_dict. . .

[kmsquire]

Hi Max, I'm on vacation this week and have limited internet access. I'll take a look when I get the chance, but it possibly won't be until Sunday or Monday.

On Wednesday, October 8, 2014, Maximiliano Suster [email protected] wrote:

@kmsquire https://github.com/kmsquire. I have completed a draft version of the AVOptions API wrapper for VideoIO.jl which should contain most of the functionality needed to support AVOptions (I still need to do testing and correct/refactor code). I tried to follow as closely as possible the original ffmpeg library documentation.

Issues #25 https://github.com/kmsquire/VideoIO.jl/issues/25 and #31 https://github.com/kmsquire/VideoIO.jl/issues/31 should be addressed by these new AVOptions functions. Currently, they are packaged in a single file named avoptions.jl.

However, I would like to make sure that getting/setting options with AVOptions functions is integrated in such a way that it works smoothly with the current flow in avio.jl. In other words, it only makes sense to get/set devices before we call opencamera -> AVInput(), and setting formats (AVFormatConext) and codecs (AVCodecContext) of the input frames should be done when calling open_avinput(avin, source, input_format)

For example, avformat_open_input can take AVDictionary as user_options = Dict{String,String}()

— Reply to this email directly or view it on GitHub https://github.com/kmsquire/VideoIO.jl/issues/27#issuecomment-58338314.

[maxruby]

No problem, have a nice holiday!

I have finished the AVOptions wrapper and I will be testing it over the next couple of days. It has support for all essential functions with AVDictionary, AVDeviceCapabilitiesQuery. It is all contained in a new file called avoptions.jl. Hopefully I will have polished it further by Monday when you get a chance to look into it. Meanwhile, can you please suggest what is the best way for me to test the new module together with the VideoIO.jl module? In other words, how should I update the Package just for testing/debugging?

Thank you. Max

[kmsquire]

I think this is what you're asking; if not, let me know.

What you'll want to do is create an test/avoptions.jl file, with simple tests that exercise the new functionality. Then you'll want to modify test/runtests.jl to include this file.

You'll probably want to move using Base.Test from test/avio.jl to test/runtests.jl, and you should use the test macros in test/avoptions.jl. See test/avio.jl (or the test directory in most other packages) for examples.

When you submit the PR, automated (Travis) testing runs test/runtests.jl and indicates whether or not the tests pass. This file is also run by @IainNZ's automated testing framework to see how well packages are tracking changes in mainline Julia. (VideoIO is currently broken in this regard because we haven't yet updated to the most recent ffmpeg libraries.)

Thanks again for taking this on. My intention is to add you as a collaborator in this repo after reviewing the PR.

[maxruby]

Thanks for the info. I will test thoroughly the new functions with test/avoptions.jl and get back to you as soon as I see the tests pass. . .

[maxruby]

@kmsquire. I am sorry for the delay - I wished to have this done by today, but I also had to spend time on a C++/OpenCV project. I will try to get back to you latest end of the week.

Is it possible at all to run my modified version of the VideoIO package this way? julia > Pkg.rm("VideoIO") -> include("avoptions.jl") in my modified VideoIO.jl julia > "include("path/to/VideoIO.jl") julia > using VideoIO

Then test any new functions in the REPL?

Max

[maxruby]

It seems it should be possible to rapidly test changes by adding files/modifying my ~/.julia/VideoIO folder and then: reload("VideoIO")
Followed by require("../src/VideoIO.jl") include("../test/avoptions.jl")

Or do you have an alternative suggestion for rapid testing Pkg module scripts without committing changes/PR to github?

[kmsquire]

No worries, Max.

Regarding testing, you can run package tests with Pkg.test("VideoIO"). I'm not sure how this works if you change things. reload("VideoIO") between tests might work, but I haven't tested it.

In general, reload will work unless using VideoIO was called--in that case, you'll probably need to restart the REPL.

workspace() is supposed to help here by giving you a new Main module, but it's noisy when you the reload something, and I've found it often doesn't work as I need it to. But you might try it as well.

Sorry not to be able to give a canonical answer here!

[maxruby]

I have two questions now:

1. I may have read about this issue already somewhere before, but Pkg.test("VideoIO") gives errors even before I load my modified VideoIO package, e.g.,

julia> Pkg.test("VideoIO") .... downloaded several files .... Testing file reading... Testing annie_oakley.ogg... ERROR: test failed: img == first_frame in error at error.jl:21 in default_handler at test.jl:19 in do_test at test.jl:39 in anonymous at no file:37 in include at ./boot.jl:245 in include_from_node1 at ./loading.jl:128 in include at ./boot.jl:245 in include_from_node1 at loading.jl:128 in process_options at ./client.jl:285 in _start at ./client.jl:354 in _start at /Users/maximilianosuster/julia/usr/lib/julia/sys.dylib
while loading /Users/maximilianosuster/.julia/v0.3/VideoIO/test/avio.jl, in expression starting on line 19 while loading /Users/maximilianosuster/.julia/v0.3/VideoIO/test/runtests.jl, in expression starting on line 1

failed process: Process(/Users/maximilianosuster/julia/usr/bin/julia /Users/maximilianosuster/.julia/v0.3/VideoIO/test/runtests.jl, ProcessExited(1)) [1]

However, if I simply use the example in your tutorial, VideoIO.playvideo("anne_oakley"), this works fine and I can see the video in the display window.

The fatal "ERROR: test failed: img == first_frame" refers to the macro @test @test img == first_frame

I suppose this is to do with the swapping of .ogg to .png extension (which you do elegantly with swaptext)? which is indicated in the file. I would like to know if this is the only problem with the current version of Pkg.test("VideoIO") before I go on using this approach.

2. How can I load my fork of your VideoIO repo (now containing avoptions.jl) into the Julia environment and use it instead of the VideoIO package in the ./julia directory?