SharpRTSP not working with Janus Gateway

[REPRESSION]

SharpRTSP's CameraExample does not work with Janus. Every other stream i've tested works on Janus. Is there anything non standard done in SharpRTSP with RTSP streams ? It does work with VLC, live555 RTSP Proxy, KMP, webrtc-streamer (this one transcode from H264 to VP8, Janus does not). I'm struggling with this since a few days, I thought the problem came from UDP (as Janus only support UDP).

Janus' Github: https://github.com/meetecho/janus-gateway

webrtc-streamer's Github: https://github.com/mpromonet/webrtc-streamer

[RogerHardiman]

I see janus uses libcurl for RTSP. I've never tested that, but doing some Google searches shows me libcurl had issues with Digest Authentication.

Try with an empty username and password

Also try with a change from .Digest to .Basic in this code

		if (String.IsNullOrEmpty(username) == false
		    && String.IsNullOrEmpty(password) == false) {
		    String realm = "SharpRTSPServer";
		    auth = new Authentication(username,password,realm,Authentication.Type.Digest);
		} else {
			auth = null;
		}

[REPRESSION]

Thank you for your quick answer. I am not using any authentication (password ans username are set to null, this is the only change I made to CameraExample)

[RogerHardiman]

Can you get some Verbose logs from Janus to show me what it is doing, or how far through it gets through the OPTIONS, DESCRIBE, SETUP, PLAY sequence.

[REPRESSION]

I will post Janus' logs later (no access to them right now), but here are the logs from the RtspCameraExample:

Connection from 192.168.1.223:57478
1 RTSP clients connected. 0 RTSP clients in PLAY mode
1 RTSP clients connected. 0 RTSP clients in PLAY mode
RTSP message received Rtsp.Messages.RtspRequestDescribe
Request for rtsp://192.168.1.129:8554/
RTSP message received Rtsp.Messages.RtspRequestSetup
1 RTSP clients connected. 0 RTSP clients in PLAY mode
RTSP message received Rtsp.Messages.RtspRequestPlay
1 RTSP clients connected. 1 RTSP clients in PLAY mode
Sending video session 1 UDP Timestamp(ms)=7215. RTP timestamp=649350. Sequence=1
1 RTSP clients connected. 1 RTSP clients in PLAY mode
Sending video session 1 UDP Timestamp(ms)=7241. RTP timestamp=651690. Sequence=2
1 RTSP clients connected. 1 RTSP clients in PLAY mode
Sending video session 1 UDP Timestamp(ms)=7291. RTP timestamp=656190. Sequence=3
1 RTSP clients connected. 1 RTSP clients in PLAY mode
Sending video session 1 UDP Timestamp(ms)=7320. RTP timestamp=658800. Sequence=4
1 RTSP clients connected. 1 RTSP clients in PLAY mode
Sending video session 1 UDP Timestamp(ms)=7369. RTP timestamp=663210. Sequence=5
1 RTSP clients connected. 1 RTSP clients in PLAY mode
Sending video session 1 UDP Timestamp(ms)=7400. RTP timestamp=666000. Sequence=6
1 RTSP clients connected. 1 RTSP clients in PLAY mode

[RogerHardiman]

Thanks for the logs. This tells me RtspCameraClient is sending the video OK. It has gone from DESCRIBE to SETUP to PLAY and it has sent 6 chunks of H264 video in UDP mode.

So we need to check Janus to see what the problem is

[REPRESSION]

Yes, that was my first thought too, but the stream coming from RtspCameraExample is the only one not working in Janus (any IP Camera will work). So my guess right now is that there is something not totally standard in the stream coming from RtspCameraExample (maybe SDP?) and makes Janus failing the decoding.

[RogerHardiman]

There are a few possible places it can go wrong a) The SD has some hard coded values for the SDP that may be wrong (eg "profile-level-id=42A01E")

b) Maybe it does not like the very basic H264 encoder that is being used. It is a very simple encoder with a very large bitstream

c) Or maybe it does not like that fact that I am sending the H264 as one large RTP packet and letting the OS fragment a large RTP packet. You can change that. Look at this code

        // The H264 Payload could be sent as one large RTP packet (assuming the receiver can handle it)
        // or as a Fragmented Data, split over several RTP packets with the same Timestamp.
        bool fragmenting = false;

and change 'false' to 'true'

It is supposed to be 'true'. I'd left it 'false' when doing some tests.

[REPRESSION]

a) I tried with other profile-level-ids, still no result. (4d002a which seems to be the one used in the RTSP stream coming from the IPCamera)

b) Originally I was feeding h264 frames coming from an IPCamera, but I am now trying with the original RtspCameraExample to be sure the problem isnt in my modifications. The RTSP stream from the same camera works in Janus.

c) I just tried with this code, still no image in Janus:

        // The H264 Payload could be sent as one large RTP packet (assuming the receiver can handle it)
        // or as a Fragmented Data, split over several RTP packets with the same Timestamp.
        bool fragmenting = true;
        //int packetMTU = 65000; 
	int packetMTU = 1400;

[REPRESSION]

Here are some logs of Janus, those lines keep repeating so I only pasted a few occurrences of them:

https://pastebin.com/C00UqcSW

[Rakiah]

Spoke directly to @lminiero which is the creator & maintainer of Janus Gateway and redirected to this issue, maybe he can see something that we cannot about this problem

[REPRESSION]

Here is an image resuming working and not working cases:

[REPRESSION]

I enabled webrtc logging on chrome here are the logs:

https://pastebin.com/TJQ3Urfz

seems like this is the problem

[22404:20404:1228/140614.833:WARNING:h264_sps_pps_tracker.cc(85)] No PPS with id << 0 received

test made with RtspCameraExample

sdp generated by RtspCameraExample:

v=0
o=user 123 0 IN IP4 0.0.0.0
s=SharpRTSP Test Camera
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
a=control:trackID=0
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=42A01E; sprop-parameter-sets=Z0IACvhgif+AAIAAiABB6wAM3+YEEA==,aM44gA==;

[RogerHardiman]

Hi I found and fixed two bugs in the H264 encoding in SimpleH264 Encoder (one in the SPS, one in the video NALs Slice Header). These prevented the RTSP stream playing properly on BenSoft SecuitySpy CCTV camera viewer on the Mac and with VLC on the Mac. It may not be the cause of the Janus issue, but it is worth mentioning. The changes are in the Git repository.

[RogerHardiman]

I have a question. You said you are using the raw H264 frames from an IP camera. Can you tell me how you are getting the raw frames?

eg are you using some RTSP client and looking at the received UDP packets? If you are, then these packets are not simple NALs. They will be NALs wrapped with a RTP header as per the H264 over RTP RFC standard

The best way to check your NALs are correct is to write each one to a .264 file, with a 0x00 0x00 0x00 0x01 4 byte header. Then see if you can play with ffplay myvideo.264

[REPRESSION]

Hi,

I used 2 ways to get those raw frames: by using RtspClient to get the frames from the RTSP Stream of the IP camera and by requesting them from a VMS.

I just tried with the updated SimpleH264Encoder, same result.

In Janus' source code, these is a function that detects keyframes, it never detects keyframes when the stream is coming from SharpRTSP.

	if(fragment == 5 ||
			((fragment == 28 || fragment == 29) && (nal == 5 || nal == 7) && start_bit == 128)) {
		JANUS_LOG(LOG_HUGE, "Got an H264 key frame\n");
		return TRUE;

This is the error in Chrome's webrtc debug:

[8920:12576:0102/154613.367:WARNING:generic_decoder.cc(236)] Failed to decode frame with timestamp 4265197366, error code: -1

I just resend the nals I get from RtspClient.Received_NALs using RtspServer.video_source_ReceivedYUVFrame.

I suspect the problem to be linked to SPS/PPS, NAL HEADER ?

Sadly I dont understand everything below:

               int rtp_version = 2;
                int rtp_padding = 0;
                int rtp_extension = 0;
                int rtp_csrc_count = 0;
                int rtp_marker = (end_bit == 1 ? 1 : 0); // Marker set to 1 on last packet
                int rtp_payload_type = 96;

                RTPPacketUtil.WriteHeader(rtp_packet, rtp_version, rtp_padding, rtp_extension, rtp_csrc_count, rtp_marker, rtp_payload_type);

				UInt32 empty_sequence_id = 0;
				RTPPacketUtil.WriteSequenceNumber(rtp_packet, empty_sequence_id);

                RTPPacketUtil.WriteTS(rtp_packet, rtp_timestamp);

                UInt32 empty_ssrc = 0;
                RTPPacketUtil.WriteSSRC(rtp_packet, empty_ssrc);

                // Now append the Fragmentation Header (with Start and End marker) and part of the raw_nal
                byte f_bit = 0;
                byte nri = (byte)((first_byte >> 5) & 0x03); // Part of the 1st byte of the Raw NAL (NAL Reference ID)
                byte type = 28; // FU-A Fragmentation

                rtp_packet[12] = (byte)((f_bit << 7) + (nri << 5) + type);
                rtp_packet[13] = (byte)((start_bit << 7) + (end_bit << 6) + (0 << 5) + (first_byte & 0x1F));

Here is my sdp code, I hardcoded it using values i got from RtspClient.Received_SPS_PPS (I tried a lot of stuff on this, like changing profile-level-id, removing sprop-parameter-sets)

sdp.Append("a=fmtp:96 packetization-mode=1; profile-level-id=4d002a; sprop-parameter-sets=J00AKpY1APAET8s3BQYFQAAA+kAAOpgm+oA=,KO4EYg==;");

[RogerHardiman]

Let's focus on this code in Janus if(fragment == 5 || ((fragment == 28 || fragment == 29) && (nal == 5 || nal == 7) && start_bit == 128)) { JANUS_LOG(LOG_HUGE, "Got an H264 key frame\n"); return TRUE;

I've built Janus on a Linux box (raspberry pi) but I don't know how to configure it properly. I did edit a config file in /opt/janus/etc and uncommented an example RTSP stream and changed the settings. So Janus does go and fetch the video. But it does not even call the is_keyframe() function.

So I need to do more - eg does it only call that function when you have a client. Opening http://ip:8088 or :8188 does not give me any sort of web page

[Rakiah]

Hello Roger, Thanks a lot for the effort you put in, i'll drop you everything required to make Janus work in a minute

[Rakiah]

To make it work correctly you should: 1 - Follow janus instruction for building it 2 - you should be able to run Janus with /opt/janus/bin/janus 3 - start the web server hosting the .js/html file cd /YOUR_JANUS_SOURCES/html php -S 0.0.0.0:8082 4 - replace all of the files in the html folder with the following ones: html.zip 5 - open up chrome browser and hit this url http://192.168.1.223:8082/streamingtest.html but replace the ip address with yours 6 - open up the chrome developer console and type in the console generateStream("rtsp://192.168.1.223/20", 0); but replace the rtsp link with the one generated by SharpRTSP.

you should see errors in the chrome console if all levels logging are enabled, and if you try with the direct rtsp stream from the camera it should work, you can also run Janus like this /opt/janus/bin/janus --debug-level=7 so that you can see the logs at max level in janus, when launching a stream that is working you will see a "Got a keyframe" message while trying with one that doesn't work you will see nothing,

please do not hesitate if you have any question

[RogerHardiman]

Ok. I've got there. I installed Apache and copied in the html files. I went to the /streamingtest/html page and used the Chrome Console to run generateStream (never used the Chrome Console before - I don't write apps that run inside the browser) I know have it printing the error that there are no keyframes

[REPRESSION]

To get more precise debugging from chrome, you need to start it with those parameters:

./chrome --enable-logging --vmodule=*/webrtc/*=1

this will output the logs into C:\Users\USER\AppData\Local\Google\Chrome\User Data

[Rakiah]

good, if you wanna try with a stream that is working, please run this command:

generateStream("rtsp://184.72.239.149/vod/mp4:BigBuckBunny_175k.mov", 0);

you should see an endless trailer video with a big bunny

[RogerHardiman]

big buck bunny works

[Rakiah]

nice, you should have a setup that is kind of equivalent to ours

[RogerHardiman]

The keyframe detection code is fine for me. It detects RTP packets with raw NALs (so spots fragment type 5 which means an IDR I-frame) and it spots NALs wrapped in a Fragmented Packet (so spots fragment type 28)

So no idea why you don't see the Keyframe message, unless you are not passing in NALs correctly.

As for Chrome, you are right, it does not show SharpRTSP, even with my recent changes.

I'll try and take a look, but am on customer sites the rest of this week and most of next week

[RogerHardiman]

Ok. I have a possible suggestion There are two ways you can send the SPS and PPS. a) in the SDP b) in the RTP video stream, mixed in with the video NALs

RtspCameraExample only sends the SPS and PPS in the SDP So maybe Janus only looks for the SPS and PPS embedded in the RTP video stream. Or Maybe Janus is not parsing the SDP that I generate (I see there was a Janus Issue in 2017 about Janus parsing the SDP)

I would suggest the first step is to get the SPS and PPS into the RTP stream.

[RogerHardiman]

Another thing you will need to implement in RtspCameraExample is to handle the case where there are multiple NALs forming a video frame. Most H264 encoders will return an Array of NALs for a frame of video. (Note that SimplyH264encoder and TinyH264Encoder do not do this) There may be multiple NALs forming a single video frame because the H264 encoder is set to emit small NALs. Or maybe one of the NALs are video and some of them are the SPS and PPS NALs)

Anyway the main thing is that a RTSP Server needs to be passed an Array of NALs and then send thm properly in a way that the RTSP Client can tell that all these NALs are from the same Array of NALs. It does this by using the same RTP sequence number on all the individual NALs and uses the M-Bit.

But RtspCameraExample only handles the case where there is a a single NAL for the whole frame of video. So in the real world it will not work very well as most H264 encoders return lots of small NALs to make a video frame, and it means you have no way to inject the SPS NAL and the PPS NAL.

So RtspCameraExample needs to cope with an Array of NALs.

You'll be able to read the RtspClientExample to see how it receives RTP packets and forms the Array of NALs. But at the moment I don't have time to implement it in the RtspCameraServer as paid-for projects take priority.

[RogerHardiman]

Ok. quick update. I did a quick change to SharpRTSP Server to send multiple NALs and got it to send the SPS and PPS before the I-Frame. Janus now tells me it is sending Key Frames all the time (good news). On Chrome it now displays a Green Rendering rectangle where the video would go (ie where BigBuckBunny was rendered) and all the PPS errors have gone from the logs.

But, it still does not actually show any video.

Maybe it is waiting on RTCP Sender Reports or something else

Anyway at least I can see that the next step is to send the SPS and PPS before the KeyFrame to resolve some issues.

[lminiero]

Still on my holidays break, so apologies if I'll be brief. In the Janus Streaming plugin (the one with RTSP support) we just relay media, so there's no transcoding: what we receive from the camera is what we send to the viewers. As such, when the browser fails to decode/render an H.264 stream, it's usually one of two things: broken video, or the SDP advertises an H.264 profile the browser doesn't support. The latter can be fixed by overriding the fmtp property when configuring the streaming mountpoint in Janus, while for the former there's nothing we can do since we don't touch the media.

The keyframe method you see is not used much in that plugin, or at all, so it might be useful for debugging, but you shouldn't rely on that much: the logs from Chrome are much more indicative. I see you made some changes where you now send a new SPS/PPS before every keyframe: I think that's the right thing to do, and what I believe most encoders do. I guess you'll want to send the SPS/PPS in the same packet that contains the keyframe NAL, as if an endpoints waits for a keyframe before doing anything and the SPS/PPS precedes that, the SPS/PPS nal may be missed (I seem to remember this being an issue in Janus too the past, but I'm not sure).

As to the RTCP Sender Reports, we send them on a regular basis when sending media.

[RogerHardiman]

Hi @lminiero Thanks for the extra info. Never used Janus before so background is very welcome. You are right most IP cameras include the SPS and PPS in-band. I think Axis never used to include it inband a few years back and the viewer relied on the SPS/PPS in the SDP. (I recall fixing my own bugs where I relied in it being in-band and then not being able to view Axis cameras until I used an SDP parser and did the base64 decode.

That little keyframe detection logic was good - always handy as a log entry to help tell what a H264 stream is doing.

SharpRTSP's Camera Example does not generate any Sender Reports. So did not know if that caused Janus problems if you were waiting on the SR to match the RTP timestamp to Wall Clock time.

Anyway will dig on next week when I'm back home

[REPRESSION]

Hi, thanks again for your help.

If I understood everything correctly, I need to send NALU containing SPS/PPS before each keyframe, Do I need to change this part of the code ? Or will also handle NALU containing SPS/PPS ?

               int rtp_version = 2;
                int rtp_padding = 0;
                int rtp_extension = 0;
                int rtp_csrc_count = 0;
                int rtp_marker = (end_bit == 1 ? 1 : 0); // Marker set to 1 on last packet
                int rtp_payload_type = 96;

                RTPPacketUtil.WriteHeader(rtp_packet, rtp_version, rtp_padding, rtp_extension, rtp_csrc_count, rtp_marker, rtp_payload_type);

				UInt32 empty_sequence_id = 0;
				RTPPacketUtil.WriteSequenceNumber(rtp_packet, empty_sequence_id);

                RTPPacketUtil.WriteTS(rtp_packet, rtp_timestamp);

                UInt32 empty_ssrc = 0;
                RTPPacketUtil.WriteSSRC(rtp_packet, empty_ssrc);

                // Now append the Fragmentation Header (with Start and End marker) and part of the raw_nal
                byte f_bit = 0;
                byte nri = (byte)((first_byte >> 5) & 0x03); // Part of the 1st byte of the Raw NAL (NAL Reference ID)
                byte type = 28; // FU-A Fragmentation

                rtp_packet[12] = (byte)((f_bit << 7) + (nri << 5) + type);
                rtp_packet[13] = (byte)((start_bit << 7) + (end_bit << 6) + (0 << 5) + (first_byte & 0x1F));

Could you push (on a branch?) your changes that lead you to a green screen on janus ?