Camera source docs

[hamishwillee]

This adds docs for setting camera source. Follows on from https://github.com/mavlink/mavlink/pull/2079

@rmackay9 , @nexton-winjeel, @julianoes

We may have run into a snag unless we can assume that streams are independent of the source - can we make that assumption? I mean if I have a particular stream id, can I assume that the same URL and resolution will be published if the source is changed - albeit with data from a different sensor.

If the assumption is that they can be dependent then we're in trouble, because changing the source would make the current stream invalid. You'd lose video because the ground station will be expecting data at the URL with given resolution etc. It wouldn't know your intent to switch to some other stream.

If they can be dependent I think we'd have to use the stream ID option for setting the source.

This will need modifications based on responses.

[nexton-winjeel]

The payloads I've dealt with that have multiple sources and a single stream are mostly independent. More specifically, for the various fields in VIDEO_STREAM_INFORMATION:

• stream_id: No change
• count: No change
• type: No change
• flags: May change (if the source swaps to thermal)
• framerate: May change
• resolution_h: May change (stream resolution might be fixed to source resolution)
• resolution_v: May change
• bitrate: No change
• rotation: May change
• hfov: Likely to change
• name: No change
• uri: No change

[nexton-winjeel]

The other problem with https://github.com/mavlink/mavlink/pull/2079 is that there is no way for a GCS to determine the number of sources available. Either need a new field in CAMERA_INFORMATION or a new message similar to VIDEO_STREAM_INFORMATION.

[hamishwillee]

@nexton-winjeel So you're saying that generally you can change the source and the stream URL stays the same. In which case we have no problem with the stream still being where GCS expects it to be, but the resolution etc might not be set up correctly?

Perhaps we should require that any source-change that affects a stream should force the affected VIDEO_STREAM_INFORMATION to re-emit?

W.r.t advertising the source I was thinking you'd try to set one that didn't exist it would error. But yes, you could have this information, and potentially in both CAMERA_INFORMATION and VIDEO_STREAM_INFORMATION.

Thoughts?

[nexton-winjeel]

So you're saying that generally you can change the source and the stream URL stays the same. In which case we have no problem with the stream still being where GCS expects it to be, but the resolution etc might not be set up correctly?

Yeah, generally the URL and encoding stays the same, so GCS will still receive.

Perhaps we should require that any source-change that affects a stream should force the affected VIDEO_STREAM_INFORMATION to re-emit?

That would help.

W.r.t advertising the source I was thinking you'd try to set one that didn't exist it would error.

The problem is, with the current implementation, I can't dynamically build a UI that will only show the valid choices for the source.

But yes, you could have this information, and potentially in both CAMERA_INFORMATION and VIDEO_STREAM_INFORMATION.

Preferably not both -- single source of truth please.

[hamishwillee]

But yes, you could have this information, and potentially in both CAMERA_INFORMATION and VIDEO_STREAM_INFORMATION.

Preferably not both -- single source of truth please.

Kind of for different purposes. It should go in CAMERA_INFORMATION then. I was thinking that you might need it in the stream info to know what source is used, but you know it can only be one source per camera.

[hamishwillee]

FYI @auturgy was suggesting in dev call that he has seen cameras with multiple sensors that can stream from those simultaneously. So you could have video streams for IR and for Visible light at the same time from one camera, and presumably also picture in picture. Also that you could have the camera allowing photo capture from those sensors and not affecting the video.

I said "but didn't we say for that case you need to implement different cameras"? Probably not if you want to allow the PIP case.

I am somewhat sure that what we have done will not work well for all these cases, but I have no idea how to resolve it.

[nexton-winjeel]

Kind of for different purposes. It should go in CAMERA_INFORMATION then.

Actually, you're right:

• CAMERA_INFORMATION should report how many sources are available.
• VIDEO_STREAM_INFORMATION should report which source is being used in the stream.

[nexton-winjeel]

FYI @auturgy was suggesting in dev call that he has seen cameras with multiple sensors that can stream from those simultaneously.

In that case, you just report multiple streams, yeah? And per the above, add a field to VIDEO_STREAM_INFORMATION to report source being used?

However, this also implies that MAV_CMD_SET_CAMERA_SOURCE needs an additional stream field so that we can set the source for each stream.

[hamishwillee]

@nexton-winjeel

Yes to "VIDEO_STREAM_INFORMATION should report which source is being used in the stream."

Not so sure on "CAMERA_INFORMATION should report how many sources are available.". A number is useful if you support features in order. Would be better as a bitmask of supported sources?

However, this also implies that MAV_CMD_SET_CAMERA_SOURCE needs an additional stream field so that we can set the source for each stream.

Maybe, but I'd be tempted to add those to the video start/stop capture commands and also add source fields to VIDEO_STREAM_STATUS. If sources are dependent then setting a source on one stream might then force all the other streams to change, which the GCS would pick up through VIDEO_STREAM_STATUS. Either way, the stream and any recording continues based on its setting.

Then MAV_CMD_SET_CAMERA_SOURCE might reasonably be removed and we put the settings into the image capture and the video capture. For a source-independent camera that would start capture on the specified source. For a source dependent camera it would silently make any other captures ongoing switch to the new source.

(my concern about using MAV_CMD_SET_CAMERA_SOURCE is that it sets the source for video and still. But if you can have multiple streams being captured in both video and still then which one does it apply to? Yes you could add the stream ID to set that here, but you're still indicating only one still capture. Wheras if this is in the image/video capture on an independent source camera you can set up multiple captures on whatever you want.) I may be missing something

[julianoes]

The way I understand it is that a stream doesn't change when a source changes but what's transmitted on the stream might change, and with it maybe the resolution, etc., depending on the implementation. Cameras with multiple streams would probably not implement various sources because they don't have to as they can always stream both.