add joystick to metadata type

[baumanta]

I would like to reserve the id 6 in the component meta data type for mavlink joysticks. The idea would be that a joystick which communicates over mavlink can report its brand, numbers of axes and buttons, etc.

[baumanta]

@bkueng @JonasVautherin

[hamishwillee]

What is the problem that you're trying to solve?

• You can get the supported axis from MANUAL_CONTROL since the supported axis have INT16_MAX if not supported.
• It is true you can't get which buttons might be set, just whether they are or not. What would you do with this information if you had it. If this is generally useful should it be considered as a part of this, or a second MAVLink message?
• What would you do with the brand metadata if you had it?

Basically, why do we need the information, and if we do, why metadata and not a message? It is tempting to shove everything into metadata but you'd now be requiring that a mavlink joystick now support sending component metadata (rather than just messages). This might also be pluggable, which means you will need to support the not-yet-accepted dynamic metadata updates.

In addition, last time we discussed MANUAL_CONTROL we realised that joystick handling probably needs a pretty thorough overhaul. Specifically, if you read https://mavlink.io/en/services/manual_control.html#future-extensions you can see we're concerned that the whole approach for mapping buttons in the flight stack. There was some thinking that it might be better to get the GCS to map buttons internally to mavlink messages. This would allow a much more lightweight

Not opposed at all, but would love clarification on your thinking. Also on timeframes you would plan on testing this.

[baumanta]

@hamishwillee thanks for the great questions! So the problem I have is kind of the opposite from the MANUAL_CONTROL message. Where the MANUAL_CONTROL is for sending inputs to the drone, I want to solve the communication between the joystick and QGC. Imagine you have a "joystick" (I use joystick as a term describing an assembly of joysticks and buttons having a number of axes and button states). But this "joystick" is not mounted directly to the device QGroundcontrol is running on. E.g. it could be connected to a computer which is in turn connected to a tablet where QGC is running. So it will not appear as a hid-device for QGC to be able to communicate with it directly. In order to interface QGC with the joystick, e.g. for assigning functioonalities to axes and buttons, we would like the joystick to behave as a mavlink component. Similar to other mavlink components (e.g. camera), it would emit a heartbeat, it would be configured by QGC, and it reports its state.

[hamishwillee]

Thanks @baumanta. Ah, so this is a mavlink joystick for connecting a joystick to QGC (instead of say a USB joystick). QGC then gets messages from the joystick, and forwards them/modifies them to control the device exactly as now?

I think I need more diagrams :-). My thinking below.

So when you connect a USB joystick you get information about what stick axes are supported and what buttons. QGC setup screen then shows these. If I move the sticks I see them move on the screen. In the setup screen I would then see the axes and buttons presented and if I move the sticks or press a button I see them light up.

I can then assign a function to the button (say). On PX4 this is mapped to a particular button in the MANUAL_CONTROL array that PX4 understands means that function.

• The component metadata would be a reasonable interface for telling us what buttons and sticks are supported. So we can construct the UI.
• But if you just want to do that, you could have a message with fields indicating number of buttons present and axes - no need for for component metadata, or necessarily for a joystick to be a problem mavlink component (though that is very desirable). You also don't need to set parameters.
• How were you planning on simulating button presses so the user can tell what physical button maps to a button press? Sending MANUAL_CONTROL with the GCS as a target? QGC then forwards as needed?

I think you only need component metadata if you need to set parameters. You'd only need that if the joystick needs to hold/retain information - and I think you only need that if it is sending control messages direct to the vehicle.

[JonasVautherin]

I think you only need component metadata if you need to set parameters.

I believe this is the point here: the "mavlink joystick" has a "component information" definition that lists the joystick params. A param can be e.g. a stick, a switch, a button. And the joystick can run standalone (it does not need QGC to send anything to the drone).

So an example workflow could be this:

1. QGC downloads the component information from the joystick. From that it knows what kind of buttons the joystick has, and what kind of actions the joystick can execute. For instance, the joystick has 2 sticks, 4 buttons, and knows the actions "takeoff", "RTL", "hold".
2. QGC receives JOYSTICK_STATE messages from the joystick, that say which button is in which state (so that QGC can show it on the UI).
3. QGC can set the joystick params to decide which button does what action ("button1 -> takeoff").
4. The joystick works standalone, i.e. after QGC has set "button1 -> takeoff", QGC can be disconnected, and the joystick will still send "takeoff" to the drone when "button1" is pressed.

It is similar to the USB joystick, except that with the USB joystick, QGC has to send the mavlink messages to the drone, and in this case the joystick can run standalone (and QGC is only here to show UI feedback and set the joystick params).

Does that make sense? What do you think about this use-case? This is why we believe that it is a good fit for the "component information", but maybe we are overthinking it?

[hamishwillee]

@JonasVautherin I don't think you're overthinking it.

• I think you're making me guess to much to be able to assess it fairly :-). Hence "I think I need more diagrams".
• What does JOYSTICK_STATE message look like?
• The more you tell me the closer I think it is to a good use case.
• BUT my take on this is that if we can avoid component metadata then we do so. It isn't even out of WIP yet. So I hope to see thinking around "what would I do if I didn't have component metadata". Then we can compare options.

The only concern I have off the top of my head is how the mapping of a button press in the joystick happens. I think you are saying that the joystick knows how to send a takeoff command. So you'd assign button X to the takeoff command, and THAT is what would get send by the joystick to the flight stack - rather than button presses in the manual control message. Right? I'm supportive of that approach.

The joystick would also send MANUAL_CONTROL for the joystick but NOT send any particular buttons in the array as set. Right?

How would the joystick be mapped to a function? For example, say it is a gimbal controller - where/how does the system know that?

[JonasVautherin]

and THAT is what would get send by the joystick to the flight stack - rather than button presses in the manual control message. Right?

Correct!

The joystick would also send MANUAL_CONTROL for the joystick but NOT send any particular buttons in the array as set. Right?

Spot on :blush:

How would the joystick be mapped to a function? For example, say it is a gimbal controller - where/how does the system know that?

The component information exposes it. I am not very familiar with it so I am not sure yet how to write that with JSON (I don't see "options" in the example here), but my understanding is that it's supposed to be functionally equivalent to the camera definition, where I would think of this:

<parameter name="BTN_0" type="uint32" default="0">
    <description>Button 0 action</description>
    <options>
        <option name="Arm" value="0" />
        <option name="Takeoff" value="1" />
        <option name="ReturnToLaunch" value="3" />
        <option name="Land" value="4" />
        <option name="StartMission" value="5" />
        <option name="PauseMission" value="7" />
    </options>
</parameter>

So QGC can now decide to set BTN_0 to 1, and now the joystick knows that button 0 should make the drone takeoff. QGC can expose that to the user in the UI, just like for the camera.

[hamishwillee]

@JonasVautherin

1. Sounds like a really good use of the component metadata system to me.
2. Is there any point reserving the id until you have the rest of it planned out - i.e. why not do it all at the same time? IMO we should just hold off merging this until you're "ready".
3. When this "all" goes in we still need diagrams of message flow and description of the JOYSTICK_STATE message. Essentially enough for full implementations.
4. Think about what is/is not captured from https://docs.qgroundcontrol.com/master/en/SetupView/Joystick.html#advanced-options
5. There is no json format yet that is a precise map to camera definition. Beat implemented actuator control using parameters, but they aren't as directly exposed (in most cases) as proposed here. Most of the concepts from camera definition should be reusable though.

When I asked about "mapping function" I meant "how does the system know that this joystick is for controlling a gimbal rather than the vehicle movement"? I.e. IN this example you have commands mapped for a vehicle control joystick - ie pausing a mission. For a gimbal joystick you might have capture image. You can select what you want from the commands so that's not a problem. But what about what stick positions control?

Presumably the joystick gets the system ID of the system it is part of as a setting too.

Is there a concept of a setting mode, and activated mode? (Just trying to think through workflows.)

Earlier I asked about "could we do this using messages". The short answer is yes.

• You could have an API for querying joystick information - "emit your buttons" and it would emit all buttons, with an array indicating the ids of commands or modes it understands, and the parameter that sets them.
• You could then have a setter command that specifies the mapping between a button and its command.
• Ditto for things like setting the target etc.

I much prefer the flexibility of using component metadata for this. Though it does mean that a mavlink joystick will have to implement parameter, mavftp and component metadata protocols. Note, metadata not information! I guess it might be easy-ish with MAVSDK server plugins.

Lastly, depending on when you want to do this, we should discuss in the dev call to make sure everyone is on board. I suspect that they will be, but much better to get buy in to novel approaches in advance. (which you're doing thanks :-).

[JonasVautherin]

When I asked about "mapping function" I meant "how does the system know that this joystick is for controlling a gimbal rather than the vehicle movement"?

Good question :thinking:.

But what about what stick positions control?

I was hoping that this would be encoded in the component information JSON, somehow :grin:

Is there a concept of a setting mode, and activated mode? (Just trying to think through workflows.)

Is that needed? Some kind of "transaction" you mean? I was fine with the idea that whenever you set a parameter, then it is changed right away.

component metadata

Component metadata? What's that? :sweat_smile:

[hamishwillee]

rather than the vehicle movement"?

Good question 🤔.

The "ignoring the cost" approach would be to have a new component_target in MANUAL_CONTROL. This would be set to the device ID of a particular gimbal through the config API if you wanted to talk to a particular gimbal or whatever.

The GCS could do the heavy lifting of working out what things are available to control or you could discover this yourself. But I guess GCS is best way because it is updated more regularly.

But what about what stick positions control?

I was hoping that this would be encoded in the component information JSON, somehow 😁

For sure. I guess when you go through all the functions required of a joystick and GCS mapping we'll have a clearer picture.

Is there a concept of a setting mode, and activated mode? (Just trying to think through workflows.)

Is that needed? Some kind of "transaction" you mean? I was fine with the idea that whenever you set a parameter, then it is changed right away.

Yes. I was concerned that you might do a mapping and it apply immediately, causing the vehicle to crash say. But the joystick doesn't have to be enabled to configure it (I think). Either way, no different that the behaviour now.