Filled side_ data not received correctly（javacv 1.5.9）

[chenhao523]

my app:project A send side_data To B I used mediamtx as the RTSP server
java code: A:

        private static String inputUrl = "rtsp://localhost:8554/stream";
        private static String outputUrl = "rtsp://localhost:8554/stream1";
        FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(inputUrl);
        grabber.setOption("rtsp_transport", "tcp");

        grabber.start();
        FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(outputUrl, grabber.getImageWidth(), grabber.getImageHeight());
        recorder.setVideoCodec(AV_CODEC_ID_H264);
        recorder.setOption("rtsp_transport", "tcp");
        recorder.setPixelFormat(AV_PIX_FMT_YUV420P);
        recorder.setFormat("rtsp");
        recorder.start(grabber.getFormatContext());
        AVPacket packet = null;
        while ((packet = grabber.grabPacket()) != null) {

            BytePointer retrievedData = av_packet_new_side_data(packet, AV_PKT_DATA_NEW_EXTRADATA, extradata.length);
            retrievedData.put(seiBytes,0,seiBytes.length);
            // Consistent with the 'extradata' filled in
            System.out.println(Arrays.toString(av_packet_get_side_data(packet, AV_PKT_DATA_NEW_EXTRADATA, new SizeTPointer()).getStringBytes()));
            recorder.recordPacket(packet);
            avcodec.av_packet_unref(packet);
        }

B:

        private static String inputUrl = "rtsp://localhost:8554/stream1";
        FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(inputUrl);
        grabber.setOption("rtsp_transport", "tcp");
        // retrievedData is null
        BytePointer retrievedData = av_packet_get_side_data(avPacket, AV_PKT_DATA_NEW_EXTRADATA, new SizeTPointer());
        

How should I modify the code so that the A can transmit the side correctly_ Data to B

[saudet]

Please try to use the getVideoSideData()/setVideoSideData() methods instead.

[chenhao523]

Do you mean to use setVideoSideData() under FFmpegFrameRecorder and setVideoSideData() under FFmpegFrameGrabber for transfer?

This is the Java code for service A:

            private static String outputUrl = "rtsp://localhost:8554/stream";
            private static String inputFolder = "xxxxxx";

            FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(outputUrl, imageWidth, imageHeight)
            recorder.setVideoCodec(AV_CODEC_ID_H264);
            recorder.setFrameRate(25); 
            recorder.setGopSize(1);
            recorder.setOption("rtsp_transport", "tcp");
            recorder.setPixelFormat(AV_PIX_FMT_YUV420P);
            recorder.setImageHeight(imageHeight);
            recorder.setImageWidth(imageWidth);
            recorder.setFormat("rtsp");
            recorder.start();
           //Convert all images in the folder into a Frame for sending
            for (int i = 0; i < imageFiles.length; i++) {
                        //
                        File imageFile = imageFiles[i];
                        // Record the image file name and add it to the recorder
                        String imageFileName = imageFile.getName();
                        ByteBuffer buffer = ByteBuffer.wrap(imageFileName.getBytes(StandardCharsets.UTF_8));
                        System.out.println("StandardCharsets.UTF_8.decode(byteBuffer):"+StandardCharsets.UTF_8.decode(buffer));
                        String key = "test" + i;
                        recorder.setVideoSideData(key,buffer);
                        Buffer test = recorder.getVideoSideData(key);
                        if (test instanceof ByteBuffer byteBuffer) {
                            System.out.print(key + " contents: ");
                            while (byteBuffer.hasRemaining()) {
                                byte b = byteBuffer.get();
                                System.out.print(b + " ");
                            }
                            System.out.println();  
                        } else {
                            System.out.println("No data found for key: " + key);
                        }
                        BufferedImage read = ImageIO.read(imageFile);
                        frame = converter.convert(read);
                        recorder.record(frame);
            }

But the test didn't print anything out

This is the Java code for service B:

private static String inputUrl = "rtsp://localhost:8554/stream";
FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(inputUrl);
grabber.setOption("rtsp_transport", "tcp");
grabber.start();
AVPacket packet = null;
        while ((packet = grabber.grabPacket()) != null) {
            Map<String, Buffer> videoSideData = grabber.getVideoSideData();
            System.out.println(videoSideData);
}

But videoSideData didn't print anything out How should I use getVideoSideData()/setVideoSideData() correctly?

[saudet]

Please try to call setVideoSideData() before start()

[chenhao523]

But I want each frame to transmit a different side_ data, how should I modify the code

[chenhao523]

            String seiData = "testSeiData";
            String testKey = "test";
            byte[] seiBytes = seiData.getBytes(StandardCharsets.UTF_8);

            ByteBuffer buffer = ByteBuffer.allocate(seiBytes.length);
            buffer.put(seiBytes);

            recorder.setVideoSideData(testKey,buffer);
            recorder.start();

            Buffer testValue = recorder.getVideoSideData(testKey);
            if (testValue instanceof ByteBuffer byteBuffer) {
                System.out.print(testValue + " contents: ");
                while (byteBuffer.hasRemaining()) {
                    byte b = byteBuffer.get();
                    System.out.print(b + " ");
                }
                System.out.println();
            } else {
                System.out.println("No data found for key: " + testValue);
            }

Recorder. getVideoSideData (testKey) still obtains empty

[saudet]

But I want each frame to transmit a different side_ data, how should I modify the code

I see, that's not implemented, yet, but contributions are welcome!

[chenhao523]

project A:

        AVPacket packet = null;
        while ((packet = grabber.grabPacket()) != null) {

            BytePointer retrievedData = av_packet_new_side_data(packet, AV_PKT_DATA_NEW_EXTRADATA, extradata.length);
            retrievedData.put(seiBytes,0,seiBytes.length);
            // Consistent with the 'extradata' filled in
            System.out.println(Arrays.toString(av_packet_get_side_data(packet, AV_PKT_DATA_NEW_EXTRADATA, new SizeTPointer()).getStringBytes()));
            recorder.recordPacket(packet);
            avcodec.av_packet_unref(packet);
        }

project B:

        AVPacket packet = null;
        while ((packet = grabber.grabPacket()) != null) {
          BytePointer retrievedData = av_packet_get_side_data(avPacket, AV_PKT_DATA_NEW_EXTRADATA, new SizeTPointer());
        }

Side_ data cannot be sent, so side_data Does data only exist in one's own program。

[saudet]

I don't think grabPacket()/recordPacket() works with RTSP. That's also something that needs to be done. Like I said, contributions are welcome!