Frame Synchronization on OAK-FFC-4P Board with Dual AR0234 Camera Modules

[tarekmuallim]

Hello, I'm currently utilizing an OAK-FFC-4P board equipped with two AR0234 camera modules. Specifically, one camera is interfaced with the CAM_A port, while the other is connected to CAM_D. Upon assessing the time interval between these two frames, I observed a delay of approximately 7.8 milliseconds. Is there a software-based approach to minimize this time gap? Additionally, could you direct me to any comprehensive documentation that outlines the process of synchronizing frames externally when employing an OAK-FFC-4P board with two AR0234 camera modules?"

[jakaskerl]

Hi @tarekmuallim You can use sequence based syncing to minimize the gap (source).

Alternatively/preferably you could hardware-synchronize the frames to achieve an even lower (sub-ms) accuracy. A guide is available here.

Thanks

[tarekmuallim]

Hello @jakaskerl

Thank you for your prompt response.

I've reviewed the documents you provided. From my understanding, there's already a synchronization clock generated on the OAK-FFC board. To establish synchronization, I believe I need to connect a wire from the FSYNC output on the board to the FSIN input on the camera module.

Specifically, I am utilizing the AR0234 module in conjunction with an OAK-FFC-4P board. However, I couldn't locate a designated FSIN input on the AR0234 module board. Is it possible that the XVS input serves this purpose? Unfortunately, I couldn't find any documentation that clarifies the inputs for the camera module.

Additionally, if you could kindly direct me to the specific location on the OAK-FFC-4P board where I should connect the sync input to the camera, it would be greatly appreciated.

Lastly, I have a question regarding supplying an external sync clock to the camera modules. Could you inform me of the required voltage for this purpose, and does it vary depending on the camera modules in use?

Once again, thank you for your assistance.

[Erol444]

Hi @tarekmuallim , So you would need to first set both cameras to input:

cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 4LANE
cam_D.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 4LANE

And then connect the FSYNC signal + GND via PSRBS connector (required 1.8V):

Pinout:

So from user perspective, left to right:

5V
STROBE
Module reset
Boot select
FSIN_2LANE
FSIN_4LANE
GND

[tarekmuallim]

Hi @Erol444

Thanks for your replay.

I'm currently facing some confusion as I can't seem to locate the PSRBS connector on my board.

Additionally, I'd appreciate it if you could confirm whether the XVS input on the AR0234 camera module functions as the FSIN input, as I couldn't find any input labeled as FSIN on the camera module board.

[Erol444]

Hi @tarekmuallim , Looks like you have an older version of the board. Yes, XVS == FSIN. See here on how to access FSIN on older version of OAK-FFC-4P: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/sync_frames/#oak-ffc-4p-fsin

[tarekmuallim]

Hi @Erol444

I appreciate your response. I will review the link you provided. Regarding my board, it is R4M1E4. Could you kindly direct me to the PDF files containing the assembly drawing and schematic specific to this board version? Thank you once again.

[tarekmuallim]

Hi @Erol444

I attempted to verify the wire indicated in the documentation that is supposed to carry the FSIN signal using an oscilloscope. I examined the area on the board labeled as "FSIN1." To my surprise, there was no discernible signal. I was unable to detect any clock or trigger signals. I also examined "FSIN2", "FSIN3", "FSIN4", and so forth with the same result.

Furthermore, I inspected the FSIN testpoint on the cable, and once again, there was no signal detected.

Is there something specific that I may have overlooked in this process?

[Erol444]

Hi @tarekmuallim , Schematics can be found here: https://github.com/luxonis/depthai-hardware/tree/master/DD2090FFC_DepthAI_4-camera_FFC Have you enabled the FSIN output in the API, and you are using OV9x82 as the FSIN output, as specified here?

[tarekmuallim]

I'm currently using the OV9782 camera for this test. After connecting the cameras to sockets B and C, I observed that the time difference between the frames is reduced to 0.01ms in this configuration. Subsequently, I enabled the FSIN output using the following code:

pipeline = dai.Pipeline()
colorLeft = pipeline.create(dai.node.ColorCamera)
colorRight = pipeline.create(dai.node.ColorCamera)
colorLeft.setBoardSocket(dai.CameraBoardSocket.CAM_B)
colorRight.setBoardSocket(dai.CameraBoardSocket.CAM_C)
colorLeft.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT)
colorRight.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) 
config = dai.Device.Config()
config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT, dai.BoardConfig.GPIO.Level.HIGH)

with dai.Device(config) as device:
    device.startPipeline(pipeline)

However, in this setup, I noticed that the time difference between the frames increased to 3ms, and I was unable to detect any FSIN signal at the board or at the testpoint on the cable.

When I attempted the same setup with the AR0234 camera and enabled the FSIN output, I encountered an error. [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'

[themarpe]

@tarekmuallim at the moment AR0234 likely only works in EXTERNAL_TRIGGER mode, compared to FSYNC OUTPUT/INPUT. Do you mind trying with that?

[tarekmuallim]

Experimenting with the AR0234 camera in EXTERNAL_TRIGGER mode, I set the external trigger for the left and right color cameras as follows:

colorLeft.initialControl.setExternalTrigger(1, 0)
colorRight.initialControl.setExternalTrigger(1, 0)

This resolved the error. I did not receive any frames because I didn't supply a trigger signal. However, my primary concern is the absence of the FSIN signal on the board. I'm puzzled as to why it's not being generated.

[themarpe]

@tarekmuallim FSIN is usually generated by one of the imagers (usually LEFT imager in L/R combination) and can also not be present when manually driving sensors via External Trigger mode

[tarekmuallim]

Hi @themarpe @Erol444

To clarify the situation, let me reiterate the issue. I thing we have encountered some confusion, so I'll provide a clear explanation.

AR0234 appears to operate exclusively in EXTERNAL_TRIGGER mode. Consequently, I've switched to using the OV9782 for my current testing purposes. The camera modules are connected to both the left and right sockets, and I have implemented the code specified below.

pipeline = dai.Pipeline()
colorLeft = pipeline.create(dai.node.ColorCamera)
colorRight = pipeline.create(dai.node.ColorCamera)
colorLeft.setBoardSocket(dai.CameraBoardSocket.CAM_B)
colorRight.setBoardSocket(dai.CameraBoardSocket.CAM_C)
colorLeft.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT)
colorRight.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) 
config = dai.Device.Config()
config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT, dai.BoardConfig.GPIO.Level.HIGH)

with dai.Device(config) as device:
    device.startPipeline(pipeline)

My objective was to detect a clock signal on the FSIN testpoint within the left camera cable or on the board itself. Regrettably, no signal was detected. I am attempting to follow the instructions outlined in the link provided below to address this issue. https://discuss.luxonis.com/d/934-ffc-4p-hardware-synchronization

[themarpe]

@tarekmuallim Correct, AR0234 on rvc2 works in ext trigger now, and not in FSync (CC: @alex-luxonis ) It also does not provide FSync output.

WRT OV97 in the above mentioned configuration is looks like it should work. @alex-luxonis any ideas on this?

[tarekmuallim]

Hi @alex-luxonis @themarpe I haven't received any response or updates. Can you please provide me with some information?

[alex-luxonis]

@tarekmuallim Sorry for the delay...

For OV9782/OV9282, by default when connected on ports B and C that share the same I2C bus, the cameras are started in broadcast mode (I2C write to both), that will ensure they will be kept synced without the need for a sync signal, as long as they're clocked from the same source, which is the case with Luxonis camera modules, where the clock is provided by the OAK-FFC-4P baseboard. It's important to be the same source, and not separate oscillators of the same frequency -- drift over time would occur here.

I'm currently using the OV9782 camera for this test. After connecting the cameras to sockets B and C, I observed that the time difference between the frames is reduced to 0.01ms in this configuration.

^ here it's likely here that the cameras were streaming started with the default broadcast mode. The 0.01ms difference is expected (actually due to the SoC FW taking the timestamp 2 times in sequence, while the real timestamp difference should be very close to 0).

Some third party camera modules (Arducam, Raspberry Pi) do have clock oscillators on board, and then hardware sync would be needed. Also needed when connected to different sockets on FFC-4P, where different I2C buses are used. Your code above looks good for enabling HW sync between the 2x OV9782. A note that the sync signal (on FSIN pin) as generated by OV9782/OV9282 is a very short active-high pulse, just slightly over 10us. May need to adjust the measuring equipment to notice it. If the signal still seems to be missing, possible to post a picture with the setup, including the camera modules?

For the AR0234 issue, I'll follow up in a new comment.

[l4es]

Dear @alex-luxonis,

I am interested in this issue too. Currently we have 3 set of AR0234 based modular camera using OAK-FFC-4P waiting for this evolution. I would like to know if there's any progress in the firmware side, please ?

Thanks and best regards, Khang

[alex-luxonis]

Yes, the firmware was just updated earlier today (develop branch, and we'll release soon 2.23.0.0), it improves the operation/stability in fsync input mode for AR0234, and other sensors as well.

Previously this error happened if the fsync signal was not being provided within some time (approx 1 second) after enabling the sensor, it should be no longer an issue now: [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0' (technical detail: MSS CPU was stuck handling continuously triggered MIPI DPHY error interrupts coming from the sensor, and couldn't communicate with the main CSS CPU, but didn't actually crash)

---

AR0234 sensors can be configured like: camNodeA.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) ... However, if only AR0234 sensors are present on the device, they don't have the hardware capability to generate a sync signal (while OV9782/OV9282 have this option). So either the signal has to be provided externally, or RVC2 GPIOs can be configured to output it. We don't have this option by default in FW yet, but can be done through DeviceConfig, or a Script node for more options. I'll follow up with an example.

[l4es]

Hi @alex-luxonis,

Could you share where would the aforementioned example be found, please?

[alex-luxonis]

Modified cam_test.py: https://github.com/luxonis/depthai-python/compare/main...alex-luxonis:depthai-python:fsync_gpio_ffc_4p (https://github.com/alex-luxonis/depthai-python/commit/edc5666997f49f3a5df2dc3a7af2d4b8e7da7589)

Mainly added a new Script node with the content:

import time
import GPIO

# Script static arguments
fps = %f

calib = Device.readCalibration2().getEepromData()
prodName  = calib.productName
boardName = calib.boardName
boardRev  = calib.boardRev

node.warn(f'Product name  : {prodName}') 
node.warn(f'Board name    : {boardName}') 
node.warn(f'Board revision: {boardRev}')

revision = -1
# Very basic parsing here, TODO improve
if len(boardRev) >= 2 and boardRev[0] == 'R':
    revision = int(boardRev[1])
node.warn(f'Parsed revision number: {revision}')

# Defaults for OAK-FFC-4P older revisions (<= R5)
GPIO_FSIN_2LANE = 41  # COM_AUX_IO2
GPIO_FSIN_4LANE = 40
GPIO_FSIN_MODE_SELECT = 6  # Drive 1 to tie together FSIN_2LANE and FSIN_4LANE

if revision >= 6:
    GPIO_FSIN_2LANE = 41  # still COM_AUX_IO2, no PWM capable
    GPIO_FSIN_4LANE = 42  # also not PWM capable
    GPIO_FSIN_MODE_SELECT = 38  # Drive 1 to tie together FSIN_2LANE and FSIN_4LANE
# Note: on R7 GPIO_FSIN_MODE_SELECT is pulled up, driving high isn't necessary (but fine to do)

# GPIO initialization
GPIO.setup(GPIO_FSIN_2LANE, GPIO.OUT)
GPIO.write(GPIO_FSIN_2LANE, 0)

GPIO.setup(GPIO_FSIN_4LANE, GPIO.IN)

GPIO.setup(GPIO_FSIN_MODE_SELECT, GPIO.OUT)
GPIO.write(GPIO_FSIN_MODE_SELECT, 1)

period = 1 / fps
active = 0.001

node.warn(f'FPS: {fps}  Period: {period}')

withInterrupts = False
if withInterrupts:
    node.critical(f'[TODO] FSYNC with timer interrupts (more precise) not implemented')
else:
    overhead = 0.003  # Empirical, TODO add thread priority option!
    while True:
        GPIO.write(GPIO_FSIN_2LANE, 1)
        time.sleep(active)
        GPIO.write(GPIO_FSIN_2LANE, 0)
        time.sleep(period - active - overhead)

And configured the enabled cameras in sync mode:

# Num frames to capture on trigger, with first to be discarded (due to degraded quality)
# Note: Only OV9282/9782 supports dropping frames. For AR0234 the arguments passed here have no effect
# Note2: in this mode it's best to set sensor FPS as max supported (e.g 60) to avoid missed triggers,
# as exposure starts immediately after trigger and is not overlapped with previous frame MIPI readout
#cam[c].initialControl.setExternalTrigger(1, 0)

# Note: setFrameSyncMode takes priority over setExternalTrigger (if both are set)
cam[c].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

Getting with 3 AR0234 attached on A,B,C ports and cable FSIN soldered to TRIGGER pin of Arducam module python3 utilities/cam_test.py -cams cama,c camb,c camc,c -cres 1200 -ds 2 -fps 30 -show

[mxid] [3.1] [4.141] [Script(8)] [warning] Product name  : OAK-FFC 4P
[mxid] [3.1] [4.141] [Script(8)] [warning] Board name    : DD2090
[mxid] [3.1] [4.141] [Script(8)] [warning] Board revision: R5M1E5
[mxid] [3.1] [4.141] [Script(8)] [warning] Parsed revision number: 5
[mxid] [3.1] [4.142] [Script(8)] [warning] FPS: 30.0  Period: 0.03333333333333333
...
FPS:  28.10| 28.30  28.30| 28.30  28.29| 28.30 
[cama ,  100, 1159028.071043] Exp: 30.000 ms, ISO:  198, Lens pos:  -1, Color temp: 5386 K
[camb ,  100, 1159028.071054] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
[camc ,   99, 1159028.071063] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
FPS:  28.43| 28.38  28.43| 28.38  28.43| 28.38 
[cama ,  101, 1159028.104990] Exp: 30.000 ms, ISO:  198, Lens pos:  -1, Color temp: 5380 K
[camb ,  101, 1159028.105001] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
[camc ,  100, 1159028.105012] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
FPS:  28.35| 28.38  28.40| 28.38  28.48| 28.38 
[cama ,  102, 1159028.139839] Exp: 30.000 ms, ISO:  198, Lens pos:  -1, Color temp: 5374 K
[camb ,  102, 1159028.139851] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
[camc ,  101, 1159028.139862] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
FPS:  28.54| 28.38  28.36| 28.38  28.37| 28.38