How to modify encode and decode video process?

[alexivaner]

Hi, after tracing this library, it seems that encoding and decoding use the class from "libwebrtc.jar" library. If I would to modify how we decode and encode the video, we need to modify and build our own libwebrtc.jar? is it mean that we need to modify this ? Could you add more clues and direction for this case? Thank you so much for helps.

[saghul]

What are you trying to accomplish exactly?

[alexivaner]

I would to do preprocessing like encrypt the frame after it being encoded and decrypt before it being decoded. I believe this is really a scratch process. Is it possible?

[saghul]

There is already an API for this, but only on the native side. See setEncryptor on the RtpSender and setDecryptor on the RtpReceiver.

[alexivaner]

There is already an API for this, but only on the native side. See setEncryptor on the RtpSender and setDecryptor on the RtpReceiver.

I already see in RTPSender.java and inside it has setFrameEncryptor that I guest then finally connect to fake_frame_encryptor.cc. So if I am correct, we could add our own method for encryption in "fake_frame_encryptor.cc", is that correct? Also, I wonder why this RTPSender seems never called anywhere even inside PeerConnectionObserver.java. Is this an API that we usually use by ourselves only if it is needed? Thank you for the clue and help as well.

[saghul]

Senders will be available when we release unified plan support.

We are also working on E2EE which would bring in a default encryptor / decryptor using SFrame. No timeline for that as of yet though.

[alexivaner]

Senders will be available when we release unified plan support.

We are also working on E2EE which would bring in a default encryptor / decryptor using SFrame. No timeline for that as of yet though.

I see, thank you for confirming. So, does RTPSender currently just preparation for unified plan support? Does react-native-webrtc still only supports plan B for now? Do you have any estimation for unified plan support itself? Is it possible for me to use this RTPSender now? Sorry, I have so many questions, and thanks again.

[saghul]

Senders were initially part of unified plan IIrC but they ended up being available on plan B too. We never exposed them in this plugin.

Currently this plugin only implements plan B. Unified plan is coming, see the open PRs.

[alexivaner]

Is it mean that for now, we could not use RTPSenders yet? If I would add a bit of preprocessing after encoding and decoding in Jitsi Android, where I should start? I also found e2ee-worker.js inside lib-jitsi-meet, should we start from here? Thank you

[saghul]

Right now you cannot use them.

Even when we add them, the encryptor and decrypt or are just thin wrappers over C++ objects, our current plan is to embed an SFrame entity for this purpose since it's not easy to build otherwise.

If what you want is to modify the video as comes from the camera to add things such as effects, you can start looking at the VideoProcessor interface which you can attach to a VideoSource: https://github.com/jitsi/webrtc/blob/df80aa37cc45f98c6be28740bc5c8a68f5c36483/sdk/android/api/org/webrtc/VideoSource.java#L128

[alexivaner]

HI, I would like to make sure about this diagram:

Is it correct that currently Android also works in the same way as this? Or does this diagram only works for browser on PC?

If Android does use It, I would like to make sure, does android also use this code to send the encoded frame? rtp_sender_video.cc

bool RTPSenderVideo::SendEncodedImage(
    int payload_type,
    absl::optional<VideoCodecType> codec_type,
    uint32_t rtp_timestamp,
    const EncodedImage& encoded_image,
    RTPVideoHeader video_header,
    absl::optional<int64_t> expected_retransmission_time_ms) {
  if (frame_transformer_delegate_) {
    // The frame will be sent async once transformed.
    return frame_transformer_delegate_->TransformFrame(
        payload_type, codec_type, rtp_timestamp, encoded_image, video_header,
        expected_retransmission_time_ms);
  }
  return SendVideo(payload_type, codec_type, rtp_timestamp,
                   encoded_image.capture_time_ms_, encoded_image, video_header,
                   expected_retransmission_time_ms);
}

Currently, I plan to add some preprocessing here after encoding like only doing simple encryption. For now, I will not care how we will exchange the code for decryption. Thank you for your help and confirmation.

[saghul]

Android it's a wrapper over the C++ layer so yeah, it works that way, with the addition that you have a mechanism to modify the encoded frames in the Java layer too.

[alexivaner]

Could you give me the clue on how to modify encoded frames in the Java layer? because I am more familiar with java than c++. Thanks again

[saghul]

It's what I mentioned earlier: https://github.com/react-native-webrtc/react-native-webrtc/issues/1165#issuecomment-1178736979

[alexivaner]

It's what I mentioned earlier: #1165 (comment)

Since I could not use RTPSender in the java wrapper, will it work if I modify rtp_sender_video.cc? Also do you have any clue how to see RTC_LOG from C++ if we debug in android? For example the code below:

    if (first_frame) {
      if (i == 0) {
        RTC_LOG(LS_INFO)
            << "Sent first RTP packet of the first video frame (pre-pacer)";
      }
      if (i == num_packets - 1) {
        RTC_LOG(LS_INFO)
            << "Sent last RTP packet of the first video frame (pre-pacer)";
      }
    }

Thank you and sorry for having many question

[alexivaner]

Hi @saghul , I tried to build webrtc by my own after trying to implement fake frame encryptor but I always get error like this: error: undefined symbol:webrtc::FakeFrameEncryptor::FakeFrameEncryptor(unsigned char,unsigned char)

Sorry I am really new to C++ and I could not understand where should I put this FakeFrameEncryptor

I already do this in PeerConnection.cc

#include "api/test/fake_frame_decryptor.h"
#include "api/test/fake_frame_encryptor.h"
#include "api/crypto/frame_decryptor_interface.h"
#include "api/crypto/frame_encryptor_interface.h"

Below is how I attached the fake encryptor:

if (kind == MediaStreamTrackInterface::kAudioKind) {
auto audio_sender = AudioRtpSender::Create(
worker_thread(), rtc::CreateRandomUuid(), stats_.get(), rtp_manager());
audio_sender->SetMediaChannel(rtp_manager()->voice_media_channel());
new_sender = RtpSenderProxyWithInternal<RtpSenderInternal>::Create(
signaling_thread(), audio_sender);
rtp_manager()->GetAudioTransceiver()->internal()->AddSender(new_sender);
} else if (kind == MediaStreamTrackInterface::kVideoKind) {
// Initiate frame encryptor
rtc::scoped_refptr<FrameEncryptorInterface> fake_frame_encryptor(
new FakeFrameEncryptor());
auto video_sender = VideoRtpSender::Create(
worker_thread(), rtc::CreateRandomUuid(), rtp_manager());
video_sender->SetMediaChannel(rtp_manager()->video_media_channel());

// Set frame encryptor to the sender.
video_sender->SetFrameEncryptor(fake_frame_encryptor);

new_sender = RtpSenderProxyWithInternal<RtpSenderInternal>::Create(
signaling_thread(), video_sender);
rtp_manager()->GetVideoTransceiver()->internal()->AddSender(new_sender);
} else {
RTC_LOG(LS_ERROR) << "CreateSender called with invalid kind: " << kind;
return nullptr;
}
new_sender->internal()->set_stream_ids(stream_ids);

return new_sender;
}

Thank you so much, I really appreciate your time and help

[tmoldovan8x8]

Hi @alexivaner, I am working at the moment on the e2ee for mobile, I've added the encryptors and decryptors, maybe you can find some inspiration in this. https://github.com/tmoldovan8x8/webrtc/tree/new-m100-e2ee

This is still work in progress, so a lot of drafty things.

[alexivaner]

Hi @alexivaner, I am working at the moment on the e2ee for mobile, I've added the encryptors and decryptors, maybe you can find some inspiration in this. https://github.com/tmoldovan8x8/webrtc/tree/new-m100-e2ee

This is still work in progress, so a lot of drafty things.

Wow, this is a really great help. I am really frustrated doing research about this. Can't thank you enough, hope it could become an inspiration for others too.

[alexivaner]

Hi @tmoldovan8x8 may I know what builder tools you use to build this webrtc repository? I tried to build by myself using tools from this repository . I am using v1.100.1 since I know that you are using M100.

The command for the building is shown below:

python buid-webrtc.py --sync --android ~/src
python build-webrtc.py --build --android --debug ~/src

I always got errors like shown below:

In file included from ../../sdk/android/src/jni/pc/gcm_frame_encryptor.cc:2:
gen/sdk/android/generated_peerconnection_jni/GCMFrameEncryptor_jni.h:49:13: error: function 'webrtc::jni::JNI_GCMFrameEncryptor_SetKey' has internal linkage but is not defined [-Werror,-Wundefined-internal]
static void JNI_GCMFrameEncryptor_SetKey(JNIEnv* env, const base::android::JavaParamRef<jintArray>&
^
gen/sdk/android/generated_peerconnection_jni/GCMFrameEncryptor_jni.h:56:10: note: used here
return JNI_GCMFrameEncryptor_SetKey(env, base::android::JavaParamRef<jintArray>(env, key));
^
1 error generated.
[2137/3902] CXX obj/sdk/android/peerconnection_jni/rtc_certificate.o
ninja: build stopped: subcommand failed.

I also try to create a branch and just add simple XOR Encryption and build, then I also get the same error as below:

In file included from ../../sdk/android/src/jni/pc/octon_frame_encryptor.cc:3:
gen/sdk/android/generated_peerconnection_jni/OctonFrameEncryptor_jni.h:41:14: error: function 'webrtc::jni::JNI_OctonFrameEncryptor_GetOctonrameEncryptor' has internal linkage but is not defined [-Werror,-Wundefined-internal]
static jlong JNI_OctonFrameEncryptor_GetOctonrameEncryptor(JNIEnv* env);
             ^
gen/sdk/android/generated_peerconnection_jni/OctonFrameEncryptor_jni.h:46:10: note: used here
  return JNI_OctonFrameEncryptor_GetOctonrameEncryptor(env);
         ^
1 error generated.
[3502/4071] CXX obj/sdk/android/peerconnection_jni/stats_observer.o
ninja: build stopped: subcommand failed.

My guess that this is maybe the problem with the build tools, do we need to modify something in builder tools in order to make building this JNI wrapper successful? Thank you so much, really appreciate your time and help.

Hi @alexivaner, I am working at the moment on the e2ee for mobile, I've added the encryptors and decryptors, maybe you can find some inspiration in this. https://github.com/tmoldovan8x8/webrtc/tree/new-m100-e2ee

This is still work in progress, so a lot of drafty things.

[tmoldovan8x8]

Hi @alexivaner, I am building the library in a Docker with debian:stable-slim, but do not think this is the problem. In the branch I've sent you I was trying out some stuff and indeed I missed something, sorry about that. On this branch it should work fine https://github.com/tmoldovan8x8/webrtc/tree/august-m100-e2ee. This is for the error in the GCMFrameEncryptor. For the error in your own encryptor it is very likely that the JNI functions are not properly defined so the build tool is not able to generate the glue code properly.

[alexivaner]

Hi @alexivaner, I am building the library in a Docker with debian:stable-slim, but do not think this is the problem. In the branch I've sent you I was trying out some stuff and indeed I missed something, sorry about that. On this branch it should work fine https://github.com/tmoldovan8x8/webrtc/tree/august-m100-e2ee. This is for the error in the GCMFrameEncryptor. For the error in your own encryptor it is very likely that the JNI functions are not properly defined so the build tool is not able to generate the glue code properly.

Thank you so much, now I could build using august-m100-e2ee already, but how did you test this E2EE function and confirm whether it encrypt our video successfully? I tried to inject this new lib jingle_peerconnection.so.jar and libwebrtc.jar in official jitsi app then install to both of android phone and it will force close when someone joined. After seen the code, do we start to do addEncryption() when we switch the camera?

[tmoldovan8x8]

We do not turn it on at the moment, so the fact that it crashes must be due to some other issue. The decryption can be tested quite easy since in https://github.com/react-native-webrtc/react-native-webrtc/blob/master/android/src/main/java/com/oney/WebRTCModule/PeerConnectionObserver.java we have access to the RTPReceivers. The encryption it is a bit trickier, since the migration to the Unified Plan is not completed yet.

[alexivaner]

Yeah, I see in PeerConnectionObserver that we never use RTPSenders directly. But do you think is it possible to use encryption in plan B? What is the best way to set this frame encryptor in plan B? Should we set from react-native-webrtc or inside webrtc itself? And how to turn on the encryption?

We do not turn it on at the moment, so the fact that it crashes must be due to some other issue. The decryption can be tested quite easy since in https://github.com/react-native-webrtc/react-native-webrtc/blob/master/android/src/main/java/com/oney/WebRTCModule/PeerConnectionObserver.java we have access to the RTPReceivers. The encryption it is a bit trickier, since the migration to the Unified Plan is not completed yet.

[alexivaner]

Hi @tmoldovan8x8, thanks for your help, seems I successfully encrypt both video and audio using XOR. I add the frameEncryptor to the corresponding sender directly in rtp_transmission_manager.cc, using this code:

  rtc::scoped_refptr<webrtc::XORFrameEncryptor> xor_encryptor(
      new webrtc::XORFrameEncryptor());
  sender->internal()->SetFrameEncryptor(xor_encryptor);

For audio it successfully encrypts and decrypts I guess, if I did not attach the decryptor via PeerConnectionObserver.java, I will hear only some noise, then if I attach the decryptor, I clearly can hear my voice. The new problem happened for video, the video was only shown as black video whether I attach the decryptor or not, and also it showed error like below, seems that the decoder failed to decode:

W/vp8_header_parser.cc: (line 172): Failed to get QP, invalid length: 49727
W/video_receive_stream2.cc: (line 756): Failed to extract QP from VP8 video frame
W/generic_decoder.cc: (line 296): Failed to decode frame with timestamp 4036674814, error code: -1

What I’m confused about is why this happened, seems payload size already changed. Isn’t it that when the data payload arrives at the receiver the payload size will decrypt and back to the original data and size ? And why this only happens on video? Do you ever face these issues or have some insight regarding this?

For now, I still tested using this repository, if I deploy in Jitsi, it still forces close if someone joins, maybe I need the same client that already uses my modified library. Thanks for all your patience and helps.

We do not turn it on at the moment, so the fact that it crashes must be due to some other issue. The decryption can be tested quite easy since in https://github.com/react-native-webrtc/react-native-webrtc/blob/master/android/src/main/java/com/oney/WebRTCModule/PeerConnectionObserver.java we have access to the RTPReceivers. The encryption it is a bit trickier, since the migration to the Unified Plan is not completed yet.

[tmoldovan8x8]

Hi @alexivaner, for e2ee we leave the first bytes of the frame unencrypted in order for the bridge to detect keyframes properly https://github.com/jitsi/lib-jitsi-meet/blob/master/modules/e2ee/Context.js For VP9 it is a bit more complicated to detect what bytes need to be left unencrypted, we do not have the functionality yet.

[alexivaner]

Ah, I see so this is the reason why you add this in your gcm_frame_encryptor right?

switch (media_type) {
  case cricket::MEDIA_TYPE_AUDIO:
    unencrypted_bytes = 1;
    break;
  case cricket::MEDIA_TYPE_VIDEO:
    unencrypted_bytes = 10;
    break;
  case cricket::MEDIA_TYPE_DATA:
    break;
  case cricket::MEDIA_TYPE_UNSUPPORTED:
    break;
}

It also according to this explanation as well:

// We copy the first bytes of the VP8 payload unencrypted.
// For keyframes this is 10 bytes, for non-keyframes (delta) 3. See
//   https://tools.ietf.org/html/rfc6386#section-9.1
// This allows the bridge to continue detecting keyframes (only one byte needed in the JVB)
// and is also a bit easier for the VP8 decoder (i.e. it generates funny garbage pictures
// instead of being unable to decode).
// This is a bit for show and we might want to reduce to 1 unconditionally in the final version.

Thank you so much, I will try it

[alexivaner]

Thank you so much for your help, I already successfully encrypt the video and audio but still get some problems in decrypting. Decrypting audio seems fine, it could back to the original voice, but the video still looks like in the encrypted stage. Another problem is, that it will only work when I deploy this compiled library in the example program I mentioned before, but in Jitsi, it still forces closed. Is there anything / additional steps in Jitsi that we should do to make this compiled library work? Thank you so much for all of your nice help so far. Hope these open issues could help others as well to contribute in webrtc, react-native-webrtc, and also Jitsi.

Here I show you the image after I attach the decryption as well (seems the video not decrypting really well, but it was fine for audio):

Looks like maybe I still did something wrong here, here is my encryption code

int OctonFrameEncryptor::Encrypt(cricket::MediaType media_type,
                                 uint32_t ssrc,
                                 rtc::ArrayView<const uint8_t> additional_data,
                                 rtc::ArrayView<const uint8_t> frame,
                                 rtc::ArrayView<uint8_t> encrypted_frame,
                                 size_t* bytes_written) {
  if (fail_encryption_) {
    RTC_LOG(LS_INFO) << "Ivan, Encryption failed";
    return static_cast<int>(FakeEncryptionStatus::FORCED_FAILURE);
  }

  uint8_t unencrypted_bytes = 1;
  switch (media_type) {
    case cricket::MEDIA_TYPE_AUDIO:
      unencrypted_bytes = 1;
      break;
    case cricket::MEDIA_TYPE_VIDEO:
      unencrypted_bytes = 10;
      break;
    case cricket::MEDIA_TYPE_DATA:
      break;
    case cricket::MEDIA_TYPE_UNSUPPORTED:
      break;
  }

  std::vector<uint8_t> frame_header;
  for (size_t i = 0; i < unencrypted_bytes; i++) {
    encrypted_frame[i] = frame[i];
    frame_header.push_back(encrypted_frame[i]);
  }
  for (size_t i = unencrypted_bytes; i < frame.size(); i++) {
    encrypted_frame[i] = frame[i] ^ fake_key_;
  }

  encrypted_frame[frame.size()] = postfix_byte_;
  *bytes_written = encrypted_frame.size();
  RTC_LOG(LS_INFO) << "Ivan, Encryption successs" << frame.size();

  return static_cast<int>(FakeEncryptionStatus::OK);
}

and here is my decryption code:

OctonFrameDecryptor::Result OctonFrameDecryptor::Decrypt(
    cricket::MediaType media_type,
    const std::vector<uint32_t>& csrcs,
    rtc::ArrayView<const uint8_t> additional_data,
    rtc::ArrayView<const uint8_t> encrypted_frame,
    rtc::ArrayView<uint8_t> frame) {
  if (fail_decryption_) {
    RTC_LOG(LS_INFO) << "Ivan, Decryption Failed";
    return Result(Status::kFailedToDecrypt, 0);
  }

  uint8_t unencrypted_bytes = 1;
  switch (media_type) {
    case cricket::MEDIA_TYPE_AUDIO:
      unencrypted_bytes = 1;
      break;
    case cricket::MEDIA_TYPE_VIDEO:
      unencrypted_bytes = 10;
      break;
    case cricket::MEDIA_TYPE_DATA:
      break;
    case cricket::MEDIA_TYPE_UNSUPPORTED:
      break;
  }

  std::vector<uint8_t> frame_header;

  // RTC_CHECK_EQ(frame.size() + 1, encrypted_frame.size());
  for (size_t i = 0; i < unencrypted_bytes; i++) {
    frame[i] = encrypted_frame[i];
    frame_header.push_back(encrypted_frame[i]);
  }
  for (size_t i = unencrypted_bytes; i < frame.size(); i++) {
    frame[i] = encrypted_frame[i] ^ fake_key_;
  }

  if (encrypted_frame[frame.size()] != expected_postfix_byte_) {
    RTC_LOG(LS_INFO) << "Ivan, Decryption Failed the dimension is not the same";
    return Result(Status::kFailedToDecrypt, 0);
  }

  RTC_LOG(LS_INFO) << "Ivan, Decryption is successfull and pass everything"
                   << frame.size();
  return Result(Status::kOk, frame.size());
}

UPDATE :: I rebuild the library and this time I did not use --debug mode and it works in Jitsi Mobile (did not force close) and my laptop client show encrypted video, but my decryption still not yet success.

[tmoldovan8x8]

Hi @alexivaner, if I see correctly in the decryptor you rely on the fact that encrypted_frame size - 1 = frame size, like in this loop for (size_t i = unencrypted_bytes; i < frame.size(); i++) { frame[i] = encrypted_frame[i] ^ fake_key_; } I think it's safer to use encrypted_frame.size() - 1, it will save you from potential future problems. The code seems to be fine, maybe this causes one byte to be copied unproperly or something. Some more debugging should fix this. One think that helped me a lot was just sending the index instead of the actual byte value in the encryptor, for (size_t i = 0; i < unencrypted_bytes; i++) { encrypted_frame[i] = i; } This way I could see better in the decryptor if I get some consecutive numbers it's ok, if not, keep debugging. Anyhow, it's just a suggestion.

[alexivaner]

Hi @alexivaner, if I see correctly in the decryptor you rely on the fact that encrypted_frame size - 1 = frame size, like in this loop for (size_t i = unencrypted_bytes; i < frame.size(); i++) { frame[i] = encrypted_frame[i] ^ fake_key_; } I think it's safer to use encrypted_frame.size() - 1, it will save you from potential future problems. The code seems to be fine, maybe this causes one byte to be copied unproperly or something. Some more debugging should fix this. One think that helped me a lot was just sending the index instead of the actual byte value in the encryptor, for (size_t i = 0; i < unencrypted_bytes; i++) { encrypted_frame[i] = i; } This way I could see better in the decryptor if I get some consecutive numbers it's ok, if not, keep debugging. Anyhow, it's just a suggestion.

Hi after debugging, you're correct my decryptor logic is not actually wrong and it's better to use encrypted_frame.size() - 1. But looks like previously in PeerConnectionObserver.java I add my frame decryptor using RTPReceiver to add setFrameDecryptor in onAddTrack and in onAddStream. After debugging, I found that this decryptor was only decrypting my audio, not my video. The result is the video remain shown as a funny noise.

So here is my solution, I add both setFrameDecryptor and setFrameEncryptor in PeerConnection.java inside webrtc source code then I called peerConnection.setFrameEncryptor(new OurFrameEncryptor()); on PeerConnectionObserver.java (addStream function) and for peerConnection.setFrameDecryptor(new OurFrameDecryptor()) on PeerConnectionObserver.java (onIceCandidate function)

Now both video and audio could decrypt really well and I could show the original video and original audio after decryption. Thank you for all of your solutions and suggestions, means a lot.

[zxk69809]

hi @alexivaner, I have encountered the same problem as you. Please tell me about peerconnection How did Java modify it? Thank you very much;

[alexivaner]

hi @alexivaner, I have encountered the same problem as you. Please tell me about peerconnection How did Java modify it? Thank you very much;

First, I modify webrtc -> PeerConnection.java You can see here line 1304, I add setFrameEncryptor and setFrameDecryptor function in peerConnection.java then you can call RTP Sender and RTP Receiver from there.

Then you can go to react-native-webrtc and go to PeerConnectionObserver.java here and add peerConnection.setFrameEncryptor(yourEncryptor) in addStream line 80, you can also add your peerConnection.setFrameDecryptor there.