qgroundcontrol
qgroundcontrol copied to clipboard
mavlink
Downloadable JSon file for firmware listings
Currently the urls to the various firmware versions are hardcoded into QGC. Also QGC needs to jump through hoops to determine things like the versions of those firmware files. It would be much better if these were dynamically determined at runtime instead of hardcoded in. I'm proposing using a standard json file format to represent the available firmware versions. These json files could then be downloaded from multiple places such that QGC understands the full set available. The combination of all files would represent the full list of firmware available tto QGC. This gets QGC out of the business of understanding build system locations and custom parsing logic for different firmware providers.
Below is the proposed format. Keys which are shows as "foo|bar" represent the options which can be put into the key. I'm hoping the keys are self explanatory.
Each individual firmware entry is represented like this:
{
"version": 1,
"stable": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore",
"stack": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover",
"version": "#.#.##",
"url": "http://foo.px4|hex"
},
...
],
"olderVersions": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore"
"stack": "PX4|ArduPilot",
"releaseType": "Stable|Beta|Developer",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover",
"version": "#.#.##",
"url": "http://foo.px4|hex"
},
...
]
}
At this point the important thing is the set of information required to identify the firmware a versions. There are many ways this could be represented in a json hierarchy. I'm open to just about anything that has the needed set of information in it.
Note: There is nothing here that is QGC specific. This should work for any ground station.
@LorenzMeier @lvale FYI: Proposal to standardize downloadable firmware information across PX4, ArduPilot, AeroCore. Can someone include in the AeroCore folks?
Following our previous conversation:
{
"version": 1.1,
"stable": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"stack": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
"olderVersions": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2"
"stack": "PX4|ArduPilot",
"releaseType": "Stable|Beta|Developer",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
]
}
Looks good to me - is the "|" symbol intended as part of the format or is it separating examples? Because for PX4 the vehicle type makes no sense.
Means "or" to seperate examples. The other important question is whethere the PX4 and ArduPilot build systems can generate something like this. Since your going to want to automate this.
Just separated older releases from the developing versions and uniformization amongst levels.
{ "version": 1.2,
"stable": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"releaseType": "Stable",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
"previous": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2"
"autopilot": "PX4|ArduPilot",
"releaseType": "Stable",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
"developer": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2"
"autopilot": "PX4|ArduPilot",
"releaseType": "Beta|Master",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
]
}
Seems ok to me. "Developer" should be "developer". Also I think format "version" should change to "formatVersion" to not confuse it with the firmware version key.
@DonLakeFlyer Any reason to have Navio explicit there? We have so many Linux boards supported and this will only increase over time. I think just leaving it as Linux would be enough. Or is this field informational only?
@lucasdemarchi I added Navio. Was the only one at firmware.diydrones, the others aren't.
example: http://firmware.diydrones.com/Copter/stable/
Instead of "stable", "previous" and "developer", how about using names consistent with FIRMWARE_VERSION_TYPE defined in MAVLink (http://mavlink.org/messages/common) and then we could get rid of the releaseType field.
You're talking names (dev, alpha, beta, rc, official) not change to numbers right? Make sense.
@DonLakeFlyer, yes, that's right. The version numbers could remain as they are... although could the version be any string? for example I just did a beta release called "Copter-3.3.4-rc2-pixracer". In either case, I don't really need that full string to be in the file. the version could be just "3.3.4" and that would be fine.
although could the version be any string
Don't see why not. These string will be shown to the user though.
@rmackay9 @DonLakeFlyer
Tried to accommodate and compile all the suggestions above on this:
If we close this, then @peterbarker will get to move on the Ardupilot side with still to be decided location of this file
{
"version": 1.3,
"official": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
"rc": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
"beta": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
"alpha": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
"dev": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
}
There is no way to tell which builds are the latest "official" as opposed to old "official"'s. There is a missing level of hierarchy. I can see what people want. Let me take a stab at re-working it.
The versions numbers are just strings (which I think is a good ting). I don't want to depend on those being real numbers to do comparison.
we could potentially make the vehicleType consistent with the MAVLink MAV_TYPE. So "Octo" would become "Octorotor". I don't really mind either way, just a thought.
Hmm. My thinking there was to have it match ArduPilot terminology would be better and more human understandable since that naming shows up in the UI as well. But I do like the idea of instead matching mavlink terminology better since it makes it more mavlink generic. Let's go with mavlink for everything.
So I think we have this:
{
"formatVersion": 1,
"official:rc:beta:alpha:dev": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
"previous": [
"official:rc:beta:alpha:dev": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf"
},
...
],
]
I changed to using ":" as an "or" operator for explaining the format. So at the top level you have the latest version of all official, rc, beta and alpha releases. Inside each of those can have multiple firmwares listed. At this level of the hierarchy boardType+autopilot+vehicleType form the unique key. There should only be a single firmware listed for each of those keys
Then under "previous" are all previous versions you want to expose, categorized first by official:rc:beta:alpha releases. Here the unique key expands to include +version as far as what would be considered a duplicate entry.
I didn't change to using MAV_TYPE because I couldn't figure out how to deal with X8 versus Octo.
Another thing to note "formatVersion" is an int. We just need to indicate a breaking change with a bump of formatVersion. No need for tracking point releases I don't think.
Sorry, I'm late to this party.
I'm not convinced by the structure yet. You've pulled out one axis to be "special" - the package release type. That could just as easily have been another entry in each description.
The structure is currently very flat. This means that a GCS attempting to find the correct package to download will need to filter the list that the JSON turns into - e.g. my @possibilities = grep { $_{'releasetype'} eq 'official' && $_{'vehicleType'} eq 'rover' } \@lotsofthings. (Note that adding "releaseType" to this is natural). Adding a "latest" (boolean) entry would allow you to do away with the "top-level" object and "previous" list of objects.
An alternate structure could allow the GCS to walk directly to a list of possibilities:
{
"version": 0.1,
"ArduPilot": {
"Official": {
"Copter": {
"X8": {
"PX4FMUv2": {
"latest": {
"version": "#.#.##",
"url": "http://xyzzy..."
},
"#.#.##": {
"url": "http://xyzzy..."
}
}}}}}}}}} (roughly).
I only mention that latter for completeness. Structured documents went out in favour of searching/filtering (thanks Google!).
So, I'd like to recommend:
{
"formatVersion": 1,
"binaries": [
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf",
"releaseType": "official:rc:beta:alpha:dev",
"latest": 0
},
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf",
"releaseType": "official:rc:beta:alpha:dev",
"latest": 1
},
...
]
}```
There are two aspects to the usage of this file:
- Easy to read by a ground station
- Easy to write by a build system
For #1: The special casing of latest releases allows a ground station to quickly find them without needing to read the entire file contents.
For #2 I think the special casing may make it easier to generate as well. The reason being you could keep a file which has older releases in it, then merge that together with the latest. Then you wouldn't have to keep tweaking the "latest" key everytime a new set of latest comes out.
I think both of these may be minor issues. I'm fine with just having a single list with no special callouts if other folks are.
@peterbarker Moving the example for the ArduPilot issue so the discussion doesn't fork. If you put the two examples side by side you can see the differences and what is missing. Anything extra is fine, but all the fields are needed.
{
"firmware": [
{
"mav-type": "GROUND_ROVER",
"format": "hex",
"url": "https://firmware.diydrones.com/Rover/latest/apm1/APMrover2.hex",
"mav-firmware-version-type": "DEV",
"mav-autopilot": "ARDUPILOTMEGA",
"platform": "apm1",
"git-sha": "705d3d567abfe8d86cbc5f3120bc6b09b879476f",
"latest": 1
}
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf",
"releaseType": "official:rc:beta:alpha:dev",
"latest": 0
},
They have almost the same information. It's jsut missing version amd then key names and wording differences. I don't care too much about the key names. Randy's suggestion was to go with values names that match mavlink naming which I think makes sense. So ARDUPILOTMEGA makes sense with respect to that.
Can we close out on this? Latest version is here:
{
"boardType": "PX4FMUV1|PX4FMUV2|PX4FMUV4|AeroCore|NAVIO|PXF|APM1-1280|APM1|APM2",
"autopilot": "PX4|ArduPilot",
"vehicleType": "Default|Quad|X8|Hexa|Octo|Y|Y6|Heli|Plane|Rover|AntennaTracker",
"version": "#.#.##",
"url": "http://foo.px4|hex|elf",
"releaseType": "official:rc:beta:alpha:dev",
"latest": 0|1
},
...
These are the required fields. Any extra fields are fine.
I'd like to get QGC cut over to this.
This is what I'm currently generating:
{
"mav-firmware-version": "3.4.0-FIRMWARE_VERSION_TYPE_DEV",
"mav-type": "HELICOPTER",
"format": "px4",
"url": "http://fred/Copter/2016-03/2016-03-21-17:03/PX4-heli/ArduCopter-v4.px4",
"mav-firmware-version-type": "DEV",
"mav-autopilot": "ARDUPILOTMEGA",
"platform": "PX4-v4",
"git-sha": "df7720cb0bc6a7096c69b340cb141d705c338242",
"latest": 0
},
Anything prepended with "mav" should be coming directly from the mavlink specifications. The "mav-autopilot" is a good example of that - previously "ArduPilot", not "ARDUPILOTMEGA" (as that's what the definition says).
I believe "platform" and "format" are vital - they disambiguate for some things in ArduPilot.
@DonLakeFlyer Does this look OK? If so I can put in a PR for the script to generate the above output.
platform: Looks the same as boardType in my proposal, correct? Of so, this is ok. format: QGC just goes off file extension. So it doesn't need it. mav-type: What are the defines for Y6/X8? Those don't seem to be in MAV_TYPE. mav-firmware-version: Don't want to have to parse our version number from the string. Could it just be version number only?
Other than the above, this is workable.