audio function does not work

[pangguoming]

hey @vshcryabets, did you find that audio function does not work? did you find a way to fix it? if you did it please let me know. thanks.,

[danman81]

Have you gotten this fixed, @pangguoming? If you take a look near the bottom of this thread:

https://github.com/youtube/yt-watchme/issues/16

You can see what might be wrong with the audio is a depricated call to avcodec_encode_audio in avcodec.h, which should be calling avcodec_encode_audio2 instead, although this will require a re-write of those calls and it doesn't look trivial. I've tried compiling ffmpeg on my own, as well as using precompiled libraries provided by @garyjamessilva at the above thread. Hope this helps u!

[Lightartist]

Hey everybody, is someone able to solve the audio problem? If it is complicated or takes too much time, we will pay for your service! Thanks in advance for your message!

[pangguoming]

@danman81 thanks a lot for your suggestion, i will try to fix it now.

if i have any progress , i will tell you @Lightartist

[danman81]

thanks, @pangguoming! let us know how it goes!

[danman81]

hey, @pangguoming have you made any progress? let me know if you run into errors so i can help you debug!

[pangguoming]

@danman81 sorry, i am occupied in other important work, i did not fix the audio function yet. i will continue to try to fix it in my free time

[danman81]

Thank you for the update, @pangguoming ! please let us know if you make more progress! and feel free to post any errors you run into!

[GavinCaptain]

@pangguoming , @danman81 : I got the audio part worked! The points are as follows:

1.

In the file "ffmpeg-jni.c", in the function 'Java_xxx_Ffmpeg_init'. Check this code: audio_codec_ctx->channels = 2;

If your device only support mono, you need to set this to 1. Also, in the file "AudioFrameGrabber.java", in the method "recordThread()",

Check this code: int channelConfiguration = AudioFormat.CHANNEL_CONFIGURATION_STEREO; you need to set it to AudioFormat.CHANNEL_CONFIGURATION_MONO

2.

In the file "ffmpeg-jni.c", in the function 'Java_xxx_Ffmpeg_init'. Check this code: audio_codec_ctx->sample_fmt = AV_SAMPLE_FMT_FLT; This is the format which ffmpeg's native aac supports. However,after checking the file AudioFrameGrabber.java, I found that it sent out 'AudioFormat.ENCODING_PCM_16BIT' data .

So, I changed the code to: audio_codec_ctx->sample_fmt = AV_SAMPLE_FMT_S16; However, this audio format needs libfdk_aac when compiling ffmpeg to *.a So I added the argument "--enable-libfdk_aac" to build.sh. By the way, here's my build.sh when I compiled ffmpeg2.3.6. I think it may be useful. (referenced from http://magiclen.org/):

NDK=$HOME/ffmpeg2.3.6/android-ndk-r10d SYSROOT=$NDK/platforms/android-19/arch-arm/ TOOLCHAIN=$NDK/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64

PREFIX=""

function build_one { PREFIX="$HOME/ffmpeg_build" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig/"

./configure \
--enable-static \
--prefix="$HOME/ffmpeg_build" \
--enable-avresample \
--enable-libfdk_aac \
--enable-gpl \
--enable-nonfree \
--enable-libx264 \
--disable-doc \
--disable-ffplay \
--disable-ffprobe \
--disable-ffserver \
--disable-avdevice \
--disable-doc \
--disable-symver \
--cross-prefix=$TOOLCHAIN/bin/arm-linux-androideabi- \
--target-os=linux \
--arch=arm \
--enable-cross-compile \
--sysroot=$SYSROOT \
--extra-cflags="-I$HOME/ffmpeg_build/include" \
--extra-ldflags="-L$HOME/ffmpeg_build/lib" \
$ADDITIONAL_CONFIGURE_FLAG

make clean make -j4 make install }

#-----For neon------ CPU=arm_neon

ADDI_CFLAGS="-marm -march=armv7-a -mfpu=neon -mfloat-abi=softfp -mvectorize-with-neon-quad"

ADDI_LDFLAGS="-Wl,--fix-cortex-a8"

build_one

sudo mv $PREFIX/bin/ffmpeg $PREFIX/bin/libffmpeg.so

#-----For general----- #CPU=arm

#ADDI_CFLAGS="-marm"

#ADDI_LDFLAGS=""

#build_one

#sudo mv $PREFIX/bin/ffmpeg $PREFIX/bin/libffmpeg.so

#--------------------- sudo chmod 777 -R $(pwd)

3.

In the file "ffmpeg-jni.c", in the function 'Java_xxx_Ffmpeg_init'. Add this code:

if (fmt_context->oformat->flags & AVFMT_GLOBALHEADER){
                audio_codec_ctx->flags |= CODEC_FLAG_GLOBAL_HEADER;
}

before the line: open_res = avcodec_open2(audio_codec_ctx, audio_codec, &opts);

Or you may get this error (in your log file): Malformed AAC bitstream detected: use audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)

4.

In the file "ffmpeg-jni.c", in the function 'Java_xxx_Ffmpeg_init'. Replace this code: av_dict_set(&opts, "strict", "experimental", 0);

to this code: av_dict_set(&opts, "-bsf:a", "aac_adtstoasc", 0);

However, I am not sure if this step is a MUST. I did it and step 3 as the same time to fix the "Malformed AAC bitstream detected" error. So, maybe you could try taking this step off and see if it still works if you're interested.

[danman81]

@gavincaptain I love you from the bottom of my heart! I will try this tonite and let u know how it goes!

[pangguoming]

@gavincaptain i love you deeply, did you get married?

[pangguoming]

@gavincaptain chould you please share your whole project ?

[danman81]

@GavinCaptain just wanted to update everyone on my progress. when I try your optional suggestion #4, I get the following error:

The encoder 'aac' is experimental but experimental codecs are not enabled, add '-strict -2' if you want to use it.

But this was resolved when I added "av_dict_set(&opts, "strict", "experimental", 0);" back in along with "av_dict_set(&opts, "-bsf:a", "aac_adtstoasc", 0);" instead of replacing.

Also, when I try suggestion #2, with or without #4, I get the following error during runtime:

Specified sample format s16 is invalid or not supported

i'm running this on windows, so I did not add "--enable-libfdk_aac" to build.sh. but I can recompile libffmpeg.so without problems. i also have libfdk-aac.a built. anyways, when I google the error, looks like someone was able to resolve it by downgrading ffmpeg:

https://bbs.archlinux.org/viewtopic.php?pid=1245854#p1245854

I think i'm using ffmpeg-2.5.3, so maybe that is the problem? maybe i'll try downgrading to ffmpeg2.3.6 to see if that helps.

Either way, this is much farther than I've gotten, so thanks so much again!!

[GavinCaptain]

@danman81 :

1.

Yes, the level of ffmpeg DOES matter. I also failed when I used higher level ffmpeg. (but I forgot the error message...^^U)

2.

"Specified sample format s16 is invalid or not supported" -> Yes, if you don't add '--enable-libfdk_aac', you will encounter this problem.

I think the ffmpeg module is a "big machine", and all the arguments is to tell the machine which 'component' should be used in each part during the compiling process. Each component may support different audio format.

So, it's ok for compliing with no error, if you do not use ''--enable-libfdk_aac'. Because you will get the default ffmpeg's audio format support with that "*.a" So, if you add '--enable-libfdk_aac' and want to set: audio_codec_ctx->sample_fmt = AV_SAMPLE_FMT_FLT; You will get some error message like "not supported..", too.

3.

Don't know if this helps, my whole process is as follows:

1. compile x264, aac module to get .a in Ubuntu
2. compile lame module to get .a in Ubuntu (but I don't think this will be used, because I did not add '--enable-libmp3lame' argument in step 3)
3. compile ffmpeg module to get .a in Ubuntu
4. copy all the .a to the "watchme" project folder
5. Use Eclipse to compile whole project(including the libffmpeg.so) on Window7 [Ps.] I use ndk-r10d both on Windows and Ubuntu

Hope you could make audio work!

[GavinCaptain]

@pangguoming : Sorry, I cannot share the whole project. Because this is my company's project, not mine. I am afraid of the copyright issue if I do so. But I will keep answering the problem as possible as I could.

[pangguoming]

@GavinCaptain it does not matter. i will try to do it as you descripted thanks a lot

[sybercoda]

I'm having issues getting this app to work as well. I just don't understand how so many changes are required to get it to work when it was just uploaded to github 5 months ago.

[garyjamessilva]

Well you will understand what is required to get this working if you start with only what Google posted to Github originally, they didn't post any of the ffmpeg & associated libraries (or various other required parts), just a note saying -

'Include cross-platform compiled streaming libraries. Either libffmpeg.so under src/main/jniLibs/armeabi, or another streaming library with modifying VideoStreamingInterface'

Originally Google did not include the JNI code either, but did subsequently after it was asked for.

In one comment someone said this was like being given a car with no engine, I thought that was about right as you had to find, assemble (& fix) all those parts with only few mostly out of date instructions.

On Wed, Apr 22, 2015 at 11:40 PM, sybercoda [email protected] wrote:

I'm having issues getting this app to work as well. I just don't understand how so many changes are required to get it to work when it was just uploaded to github 5 months ago.

— Reply to this email directly or view it on GitHub https://github.com/youtube/yt-watchme/issues/23#issuecomment-95181081.

[sybercoda]

@garyjamessilva I compiled ffmpeg for android, rmtpdump, openssl after jumping through hoops. Im now trying to compile the app with the libraries via Android Studio but I'm getting errors so I think I'm going to try eclipse instead. I hope things are smoother porting this to iOS but that is probably a whole new can of worms. FYI, look Into open source streaming solutions for drones as a viable substitute for this project. Thanks again for your reply.

[pangguoming]

i am doing it with windows7 , there is a lot of questions. i had better to do it with ubuntu

[sybercoda]

@pangguoming I tried it with windows/android studio at first with no luck as well. Then tried with Ubuntu and was able to compile, much easier! Now I'm going to use eclipse on ubuntu because Android Studio is becoming troublesome. I think Eclipse on Ubuntu is your best bet. Command line compile as much as possible. I still may have to go back and recompile again if audio isn't working, per the above procedure.

[pangguoming]

@sybercoda thanks a lot for your suggestion

[pangguoming]

@GavinCaptain could you please share your libffmpeg.so file ? so that i don't have to compile it with windows .

[GavinCaptain]

@pangguoming : Here you are. Try it~ https://onedrive.live.com/?cid=eaf2ffc47d5c37b0&id=EAF2FFC47D5C37B0%21720&ithint=file,so&authkey=!AEfiV8h2FkG92Xs

[GavinCaptain]

By the way, I found that compiling ffmpeg to .a on Windows is not so easy. Lots of error happened..., and I did not succeed. However, compiling ffmpeg to .a on Ubuntu is much easier.

[pangguoming]

@GavinCaptain i want to use ubuntu, but my project is created with windows,so.... you konw.

with your libffmpeg.so file , google returned 403 error: stream is inactive. and ffmpeg-ini init function return false. Could not open file /data/data/com.google.android.apps.watchme/files/outputStreamFile.flv

[GavinCaptain]

@pangguoming : Oh, you need to change the "ffmpeg-jni.c" to meet your own need. It seems that only share libFFmpeg.so does not work, because it includes all my customized settings.

However, I am not so sure what do you mean "my project is created with windows". In my case, I import the project 'yt-watchme' with Eclipse on Window7, too. I only compile ffmpeg on Ubuntu, for getting all *.a which the project 'yt-watchme' needs. Then I finish all other works with Eclipse on Window7. (Of course, you need to install Cygwin and NDK on Windows first)

[sybercoda]

Is there any particular reason or benefit to using ffmpeg for android for live streaming to YouTube vs using the MediaCodec class ? It is to my understanding that MediaCodec offers hardware encoding and is much faster.

[pangguoming]

@GavinCaptain yes , sir, you are right , i mean i need install JDK, eclipse, ADT NDK softer ware etc to an ubuntu computer. i am so lazy.

[sybercoda]

@pangguoming You will also need libssl library on Ubuntu or you will receive an error

[GavinCaptain]

@sybercoda : yes, I am trying to use MediaCodec encoder now, and got stuck.... In short, Android only provides video/audio encoder and muxer for mp4. However, if we want to live stream to Youtube, we need to produce .flv stream. So,I'm trying to use ffmpeg for producing .flv with MediaCodec encoder. But no luck......