Update of periscope streaming API

[DmT021]

Hi, As I can see, you create a broadcast with /liveorigin path in the rtmp URL. When I create a broadcast i get vidmanlive application in response of createBroadcast function. Do you get the same?

[DmT021]

Oh, I got it. I used to specify User Agent as "Periscope/2699 (iPhone; iOS 8.1.2; Scale/2.00)" within a request and periscope returns vidmanlive. If I do not specify anything periscope returns liveorigin.

[Pmmlabs]

Thanks for info, but if I specify user agent and get url to vidman application, ffmpeg crashes with error RTMP_ReadPacket, failed to read RTMP packet header. If user agent not specified, liveorigin application returns, and streaming runs normally, besides broadcast can't be viewed on smartphone, only in browser. Do you get it working with vidmanlive ? Both on smartphone and in browser?

[DmT021]

Nope, can't fight vidmanlive. But I found something very intresting in the RTMP client of periscope for android. I let it crash for my sample broadcast and checked out the crash report. If I'm correct it fails with NullPointerException (getting array length of null) of something called trackinfo of onMetaData object.

    public final boolean m10316(aun o_aun) {
        aun o_aun2 = o_aun;
        if (o_aun.bMN == 18) {
            o_aun2 = o_aun;
            Object[] objArr = o_aun.bMO;
            if (((String) objArr[0]).equals("onMetaData") && this.boH == null) {
                C0467 c0467;
                C1778 c1778;
                Map map = (Map) objArr[1];
                afy o_afy = this;
                Object[] objArr2 = (Object[]) r9.get("trackinfo");
                for (int i = 0; i < objArr2.length; i++) { // crashes here
                    Map map2 = (Map) objArr2[i];
                    String str = (String) map2.get("type");
                    if (str.equals("video")) {
                        Double d = (Double) r9.get("height");
                        o_afy.boH = new agb(i, (String) map2.get("sprop-parameter-sets"), ((Double) r9.get("width")).intValue(), d.intValue(), o_afy.boI);
                        o_afy.boH.m7692(o_afy.boF);
                    } else if (str.equals("audio")) {
                        o_afy.boG = new afx(i, (String) map2.get("config"), ((Double) r9.get("audiochannels")).intValue(), ((Double) r9.get("audiosamplerate")).intValue());
                        o_afy.boG.m7692(o_afy.boF);
                    }
                }
                if (o_afy.boH == null) {
                    c0467 = null;
                    C1942.log("Stream with no video encountered: " + o_afy.boC);
                } else {
                    c0467 = new C2102(o_afy, o_afy.boH, 1, 5000, o_afy.boD.jf, o_afy.boD, 50);
                }
                if (o_afy.boG == null) {
                    c1778 = null;
                    C1942.m9770(new Exception("Stream with no audio encountered: " + o_afy.boC));
                } else {
                    c1778 = new C1778(o_afy.boG);
                }
                C0467[] c0467Arr = new C0467[5];
                c0467Arr[0] = c0467;
                c0467Arr[1] = c1778;
                o_afy.boE.m7649(null, null, c0467Arr);
            }

As I can understand it sends/recieves some additional metadata with streams' parameters. I'm not an expert in RTMP protocol, so I can't explain what exactly type of RTPM message it sends, but I know that it encode it as string with RTMPMessage prefix and hexadeciamal representation of data.

[Pmmlabs]

Metadata can be posted using -metadata option of FFmpeg. Without it, error in logcat is

java.lang.NullPointerException: Attempt to get length of null array

And with -metadata trackinfo="<someinfo>" option, error is

java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Object[]

Instead <someinfo> I tried JSON-encoded object, but no luck. Seems to be trackinfo must be some complex structure in Metadata section, as here. And I have not found, how to do it with ffmpeg.

[Pmmlabs]

Sample onMetaData object (from logs.txt in pingBroadcast request):

{
  frameHeight=568,
  trackinfo=[
    {
      sampledescription=[{
                    sampletype = "H264";
                }],
      language="eng",
      description = "{H264CodecConfigInfo: codec:H264, profile:Main, level:2.1, frameSize:320x568, displaySize:320x568, crop: l:0 r:0 t:0 b:0}"; // iPhone: l:0 r:0 t:0 b:8
      type="video",
      profile-level-id="640015",   // iPhone: 4d0015
      timescale=90000,
      sprop-parameter-sets="Z2QAFazSBQEn5YBtChNQAA==,aM4G4sA=" // iPhone: "Z00AFatgoCT8qA==,KO4GcMA="
    },
    {
      config="1208",
      sampledescription=[{
                    sampletype = "mpeg4-generic";
                }],
      description = "{AACFrame: codec:AAC, channels:1, frequency:44100, samplesPerFrame:1024, objectType:LC}";
      language="eng",
      type="audio",
      timescale=90000
    }
  ],
  audiocodecid="mp4a",
  frameWidth=320,
  rtpsessioninfo={
    name="Live stream from Periscope",
    connectiondata="In IP4 0.0.0.0",
    protocolversion=0,
    timing="0 0"
  },
  audiosamplerate=44100,
  height=568,
  width=320,
  displayWidth=320,
  videocodecid="avc1",
  displayHeight=568,
  audiochannels=1
}

[DmT021]

Yep, I checked out ffmpeg sources https://github.com/FFmpeg/FFmpeg/blob/a0174f67298ba9494c146183dd360e637b03db64/libavformat/flvenc.c#L205 There is no way to pass anything but string. Probably there is another streaming software that can do it, but I didn't find. I going to check out RTMP libs with streaming feature.

[Pmmlabs]

There is something called SDP for posting metadata. Here is example for rtpsessioninfo object, I think for trackinfo must be similarly. UPD: no, it will not help.

[Pmmlabs]

Found something: rtmp_conn ffmpeg option. Allow to write arbitrary AMF connection parameters, even objects.

[DmT021]

rtmp_conn - Append arbitrary AMF data to the Connect message As I can understand the rtmp flow is following: handshake send connect send createStream send publish send onMetadata

[Pmmlabs]

I tried to patch ffmpeg, but one parameter, sprop-parameter-sets, somehow doesn't seen in Periscope (i.e. null). I don't know, why.

[Pmmlabs]

I edited the patch and got it working. Unfortunately, on the screen I see gray porridge instead of video. Audio working great.

[Pmmlabs]

Got video working! But only on Android. iPhone still crashes. Patch fixed.

[MichaelZaporozhets]

@Pmmlabs Do you have any insight into why it's crashing on iphone? I tried passing through meta data but it still crashes on iphone for some reason..

[DmT021]

It's much harder for me to debug iOS version of periscope. I'm sure both versions works with the same metadata, but actually iOS and android decode h264/avc differently. So it's possible that iOS simply can't decode current config of h264 stream. Did you tried different options for profile (baseline, main, high) and profile-level (2.0, 2.1, 4.0, 4.1)?

[MichaelZaporozhets]

@DmT021 I'm trying a bunch of option combinations now. Will update on results

[MichaelZaporozhets]

@DmT021 I'm on a jail broken iPhone too- hoping to find a way to look at the video buffer

[Pmmlabs]

I tried Main 2.1, because it was in log.txt from pingBroadcast. Yes, maybe need to try other options. I have no iPhone, therefore testing will be long.

[MichaelZaporozhets]

@Pmmlabs perhaps this will help

[MichaelZaporozhets]

Here is the crash log for periscope on the iphone:

Mon Mar 14 00:58:08 2016: Periscope (com.bountylabs.periscope): Error response Error Domain=PubNub Channel is nil Code=0 "(null)"
Mon Mar 14 00:58:08 2016: Periscope (SecLogging):  SecTrustEvaluate  [leaf AnchorTrusted ValidLeaf ValidRoot]
Mon Mar 14 00:58:22 2016: Periscope (com.bountylabs.periscope): -[__NSCFConstantString objCType]: unrecognized selector sent to instance 0x5346c8
Mon Mar 14 00:58:22 2016: Periscope (com.bountylabs.periscope): *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFConstantString objCType]: unrecognized selector sent to instance 0x5346c8'
*** First throw call stack:
(0x24e4b86b 0x3654adff 0x24e51035 0x24e4ec8f 0x24d7e2b8 0x24d669e7 0x14f849 0x14f157 0x14ebbd 0x1c01ab 0x1c0fb7 0x1bbecd 0x36c4ced7 0x36c4cec3 0x36c51729 0x24e0e595 0x24e0ca8f 0x24d5f1e9 0x24d5efdd 0x2e003af9 0x28fc418d 0x97ec7 0x36c75873)
Mon Mar 14 00:58:23 2016: ReportCrash (Crash Reporter): Formulating report for corpse[6258] Periscope
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Notice: Injecting: com.bountylabs.periscope [Periscope] (1240.10)
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Notice: Loading: /Library/MobileSubstrate/DynamicLibraries/SSLKillSwitch2.dylib
Mon Mar 14 00:58:24 2016: Periscope (com.bountylabs.periscope): === SSL Kill Switch 2: Preference set to 1.
Mon Mar 14 00:58:24 2016: Periscope (com.bountylabs.periscope): === SSL Kill Switch 2: Subtrate hook enabled.
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Notice: Loading: /Library/MobileSubstrate/DynamicLibraries/SpeedIntensifier.dylib
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector slowAnimations
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector slowDownFactor
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector slowAnimations
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector slowDownFactor
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector backlightFadeDuration
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector setIsEditing:
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector _beginEditing
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector _stopEditing
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Notice: Loading: /Library/MobileSubstrate/DynamicLibraries/XMScreenShotEx.dylib
Mon Mar 14 00:58:24 2016: Periscope (com.apple.coregraphics): CGContextSaveGState: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Mon Mar 14 00:58:24 2016: Periscope (com.apple.coregraphics): CGContextTranslateCTM: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Mon Mar 14 00:58:24 2016: Periscope (com.apple.coregraphics): CGContextRestoreGState: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Mon Mar 14 00:58:25 2016: Periscope (com.bountylabs.periscope): [Crashlytics] Version 3.6.0 (99)
Mon Mar 14 00:58:25 2016: Periscope (com.bountylabs.periscope): <WARNING> -[Twitter session] will soon be deprecated. It is recommended that users use -[TWTRSessionStore session] or -[TWTRSessionStore sessionForUserID:] if they are managing multiple users
Mon Mar 14 00:58:25 2016: Periscope (com.apple.console): MS:Warning: message not found [UITextField becomeFirstResponder:]
Mon Mar 14 00:58:25 2016: Periscope (com.apple.console): MS:Warning: message not found [UITextField becomeFirstResponder:]
Mon Mar 14 00:58:25 2016: Periscope (com.bountylabs.periscope): [Crashlytics:Crash:Reports] Submitting async /var/mobile/Containers/Data/Application/45D70B75-4330-4129-9AFE-1A95BCF796C7/Library/Caches/com.crashlytics.data/com.bountylabs.periscope/v3/prepared/6CD26C40-3C7F-4FE3-8C26-87475D544C7D.multipartmime
Mon Mar 14 00:58:26 2016: Periscope (com.bountylabs.periscope): Unbalanced calls to begin/end appearance transitions for <SENavigationController: 0x15398800>.
Mon Mar 14 00:58:26 2016: Periscope (SecLogging):  SecOSStatusWith error:[-50] Error Domain=NSOSStatusErrorDomain Code=-50 "query missing class name" UserInfo={NSDescription=query missing class name}
Mon Mar 14 00:58:28 2016: Periscope (com.bountylabs.periscope): Error response Error Domain=NSURLErrorDomain Code=-1002 "unsupported URL" UserInfo={NSErrorFailingURLStringKey=, NSErrorFailingURLKey=, NSLocalizedDescription=unsupported URL, NSUnderlyingError=0x146dd5a0 {Error Domain=kCFErrorDomainCFNetwork Code=-1002 "(null)"}}
Mon Mar 14 00:58:30 2016: Periscope (com.bountylabs.periscope): Error response Error Domain=PubNub Channel is nil Code=0 "(null)"
Mon Mar 14 00:58:31 2016: Periscope (SecLogging):  SecTrustEvaluate  [leaf AnchorTrusted ValidLeaf ValidRoot]
Mon Mar 14 00:58:31 2016: Periscope (com.apple.lsuseractivity): Caching encoded userInfo to use until we are marked dirty again (UAUserActivity.m #1567)
Mon Mar 14 00:58:31 2016: Periscope (user): LaunchServices: disconnect event received for service com.apple.lsd.modifydb
Mon Mar 14 00:58:36 2016: Periscope (com.bountylabs.periscope): -[__NSCFConstantString objCType]: unrecognized selector sent to instance 0x4f96c8
Mon Mar 14 00:58:36 2016: Periscope (com.bountylabs.periscope): *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFConstantString objCType]: unrecognized selector sent to instance 0x4f96c8'
*** First throw call stack:
(0x24e4b86b 0x3654adff 0x24e51035 0x24e4ec8f 0x24d7e2b8 0x24d669e7 0x114849 0x114157 0x113bbd 0x1851ab 0x185fb7 0x180ecd 0x36c4ced7 0x36c4cec3 0x36c51729 0x24e0e595 0x24e0ca8f 0x24d5f1e9 0x24d5efdd 0x2e003af9 0x28fc418d 0x5cec7 0x36c75873)
Mon Mar 14 00:58:36 2016: ReportCrash (Crash Reporter): Formulating report for corpse[6276] Periscope

[Pmmlabs]

@MichaelZaporozhets from 3-rd party libraries, only libx264 is needed. What error do you get while compiling?

[MichaelZaporozhets]

@Pmmlabs I overcame the error; It was a whitespace issue. That being said, how do we pass meta data with the patch?

[Pmmlabs]

@MichaelZaporozhets nohow, it's hardcoded for a while.

[Pmmlabs]

@MichaelZaporozhets thanks for the crashlog, NSInvalidArgumentException means that periscope expected some type, but got string. I think, the reason is not in video encoding, but in metadata.

[MichaelZaporozhets]

@Pmmlabs I assume it's looking for a pubnub channel to be passed with the stream somehow?

[Pmmlabs]

@MichaelZaporozhets android app don't passes pubnub channel with the stream. but streams, created with android, can be viewed on iPhone. Therefore, that is not the reason.

[MichaelZaporozhets]

I assumed that Error response Error Domain=PubNub Channel is nil Code=0 "(null)" referred to the metadata- no?

[MichaelZaporozhets]

@Pmmlabs Would it help at all if I gave you the .app file, you could probably decompile some with Hopper

[Pmmlabs]

@MichaelZaporozhets I don't know, how to decompile iphone apps and is it possible. I am zero in iphone development.

[lepikhinb]

@MichaelZaporozhets have you ever tried to mess up with Hopper?

[MichaelZaporozhets]

@Pmmlabs yeah, i'm the same

@lepikhinb nope, but know it's quite popular for his sort of thing