MAVSDK-Java
MAVSDK-Java copied to clipboard
mavlink
v2.1.3 Gremsy Vio camera payload fails to download camera definition due to "unsupported protocol" error
2024-11-26 12:38:58.851 20122-20563 Mavsdk com.dronesense.pilot.blue I Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-11-26 12:38:58.852 20122-20563 Mavsdk com.dronesense.pilot.blue I Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-11-26 12:38:58.853 20122-20563 Mavsdk com.dronesense.pilot.blue E Error while downloading text, curl error code: Unsupported protocol
2024-11-26 12:38:58.853 20122-20563 Mavsdk com.dronesense.pilot.blue E Failed to download camera definition.
2024-11-26 12:38:58.854 20122-20563 Mavsdk com.dronesense.pilot.blue I Downloaded file, result Error
2024-11-26 12:38:58.854 20122-20563 Mavsdk com.dronesense.pilot.blue D Failed to fetch camera definition!
We build curl without ssl, I think it won't support https: https://github.com/mavlink/MAVSDK/blob/main/third_party/curl/CMakeLists.txt#L12
Would you have a way to use http instead?
A weird thing in your case is that the camera definition is downloaded from the internet. Shouldn't it be served by the drone? Or is it somehow a simulator setup?
I tried to add openssl with https support in https://github.com/mavlink/MAVSDK/pull/2386 and I gave up trying to get it building for all platforms. If you want to waste a few days staring at CI playing with dependencies, feel free. I find it utterly frustrating.
@JonasVautherin
We build curl without ssl, I think it won't support https: https://github.com/mavlink/MAVSDK/blob/main/third_party/curl/CMakeLists.txt#L12
Would you have a way to use http instead?
A weird thing in your case is that the camera definition is downloaded from the internet. Shouldn't it be served by the drone? Or is it somehow a simulator setup?
It's a real drone and payload. Nope, they for some reason decided it was a good idea to have the file fetched from Github every time; here's their documentation: https://docs.gremsy.com/payloads/vio/camera-setting-menu/camera-definition-file-download - pretty sure I have no way of changing this behavior...
For generate the camera setting menu, the QGroundControl app need to download the camera definition file from Gremsy github server. The download path will be sent by the Vio automatically.
@julianoes
I tried to add openssl with https support in mavlink/MAVSDK#2386 and I gave up trying to get it building for all platforms. If you want to waste a few days staring at CI playing with dependencies, feel free. I find it utterly frustrating.
Oh boy...sorry to hear about that. By any chance, were we able to get it to work on Android at least?
Pretty sure Android was one of the broken ones: https://github.com/mavlink/MAVSDK/actions/runs/11588668231/job/32262783180#step:6:1416
Is the file available with http though? In that case, you can just intercept (but not in Java) the message containing the URL and change https to http.
Or what we could also try is to just parse https and use http to download. If it works good, otherwise no hard done, I'd think...
Whoops it got resolved automatically because I mentioned it in the PR. Let me reopen it.
Trying to backport it to v2.12 so that we can maybe release it before v3: https://github.com/mavlink/MAVSDK/pull/2454
@julianoes
Pretty sure Android was one of the broken ones: https://github.com/mavlink/MAVSDK/actions/runs/11588668231/job/32262783180#step:6:1416
That's unfortunate. Thanks for the info!
Is the file available with http though? In that case, you can just intercept (but not in Java) the message containing the URL and change https to http.
Or what we could also try is to just parse https and use http to download. If it works good, otherwise no hard done, I'd think...
Hmm... I don't think Github lets you download using HTTP anymore; here's what happens when I take that URL and do a wget with "HTTP" instead of "HTTPS":
$ wget http://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
--2024-12-03 09:50:55-- http://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
Resolving github.com (github.com)... 140.82.112.4
Connecting to github.com (github.com)|140.82.112.4|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml [following]
--2024-12-03 09:50:56-- https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/697574913/7809a88c-be05-4ff3-8f7e-5a72ae3e69ee?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241203T145056Z&X-Amz-Expires=300&X-Amz-Signature=953f4511aeadde975534734960e4d0aeedd6555d6f07228f74fd7e0b997a79d1&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dvio_camera_f1_def.xml&response-content-type=application%2Foctet-stream [following]
--2024-12-03 09:50:56-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/697574913/7809a88c-be05-4ff3-8f7e-5a72ae3e69ee?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241203T145056Z&X-Amz-Expires=300&X-Amz-Signature=953f4511aeadde975534734960e4d0aeedd6555d6f07228f74fd7e0b997a79d1&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dvio_camera_f1_def.xml&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16600 (16K) [application/octet-stream]
Saving to: ‘vio_camera_f1_def.xml’
@JonasVautherin Thanks!
Can you try with mavsdk_server:2.1.4? Just pushed it to MavenCentral. It should support HTTPS
@JonasVautherin - Thanks for the updated release! With 2.1.4 it does get a bit further, but runs into this issue,
2024-12-03 10:55:50.389 2182-4091 Mavsdk com.dronesense.pilot.blue I Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-12-03 10:55:50.392 2182-4091 Mavsdk com.dronesense.pilot.blue I Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-12-03 10:55:50.452 2182-4091 Mavsdk com.dronesense.pilot.blue E Error while downloading text, curl error code: Problem with the SSL CA cert (path? access rights?)
2024-12-03 10:55:50.452 2182-4091 Mavsdk com.dronesense.pilot.blue E Failed to download camera definition.
2024-12-03 10:55:50.456 2182-4091 Mavsdk com.dronesense.pilot.blue I Downloaded file, result Error
2024-12-03 10:55:50.456 2182-4091 Mavsdk com.dronesense.pilot.blue D Failed to fetch camera definition!
Oh yeah, that's a good question: in order to properly leverage HTTPS, curl (openssl?) needs to have access to certificates. Usually they are found on the system, but I honestly don't know how we should do that with our static library 🤔.
An easy way would be to disable the verification (see e.g. here). We could justify it by saying that downloading a wrong camera_definition.xml is not a security issue 😅. Ideally we would have a way to feed certificates to mavsdk_server though?
Would you mind looking into libcurl to see if there is a way to give it certificates? Apparently the executable has some options for that (see here), but in our case we use libcurl (not the executable). The next question would be whether we can read system certificates on Android...
I had not realized before, but it isn't as easy as just enabling https in curl 🙈.
EDIT: here I see stuff like this:
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1);
curl_easy_setopt(curl, CURLOPT_CAINFO, "./ca.crt");
So if we had read access to the system certificates on Android, we could maybe just use that? I checked on my device and I have certificates in /system/etc/security/cacerts, for instance.
Oh maybe we could just set CURLOPT_CAPATH to /system/etc/security/cacerts on Android (with an ifdef in mavsdk)?
And it seems like this answer does it at build time!
I got this to work on Android by recompiling libcurl and configuring the default search path for certificates. This can be done by passing the option:
--with-ca-path=/system/etc/security/cacerts to ./configure
or
-DCURL_CA_PATH=/system/etc/security/cacerts to cmake
@rayw-dronesense: would you be able to try this PR? https://github.com/mavlink/MAVSDK/pull/2459
@JonasVautherin - I guess it makes it further now... but,
2024-12-03 11:56:06.591 6622-7070 Mavsdk com.dronesense.pilot.blue I Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-12-03 11:56:06.591 6622-7070 Mavsdk com.dronesense.pilot.blue I Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-12-03 11:56:06.671 6622-7070 Mavsdk com.dronesense.pilot.blue E Error while downloading text, curl error code: SSL peer certificate or SSH remote key was not OK
2024-12-03 11:56:06.673 6622-7070 Mavsdk com.dronesense.pilot.blue E Failed to download camera definition.
2024-12-03 11:56:06.679 6622-7070 Mavsdk com.dronesense.pilot.blue I Downloaded file, result Error
Does your device have /system/etc/security/cacerts?
Yes it does,
~ % adb shell ls /system/etc/security/cacerts
00673b5b.0
02756ea4.0
02b73561.0
03f2b8cf.0
04f60c28.0
052e396b.0
08aef7bb.0
0d5a4e1c.0
0d69c7e1.0
10531352.0
111e6273.0
119afc2e.0
124bbd54.0
12d55845.0
1676090a.0
17b51fe6.0
1dac3003.0
1dcd6f4c.0
1df5a75f.0
1e1eab7c.0
1e8e7201.0
1eb37bdf.0
1f58a078.0
21855f49.0
219d9499.0
23f4c490.0
262ba90f.0
27af790d.0
2add47b6.0
2d9dafe4.0
2fa87019.0
33815e15.0
33815e15.1
343eb6cb.0
35105088.0
3929ec9f.0
399e7759.0
3a3b02ce.0
3ad48a91.0
3c58f906.0
3c6676aa.0
3c860d51.0
3c9a4d3b.0
3d441de8.0
3e7271e8.0
40dc992e.0
418595b9.0
450c6e38.0
455f1b52.0
48a195d8.0
4be590e0.0
4e18c148.0
5046c355.0
524d9b43.0
52b525c7.0
559f7c71.0
57692373.0
58a44af1.0
5a250ea7.0
5a3f0ff8.0
5cf9d536.0
5e4e69e7.0
5f47b495.0
60afe812.0
6187b673.0
63a2c897.0
6645de82.0
67495436.0
69105f4f.0
6e8bf996.0
6fcc125d.0
72f369af.0
75680d2e.0
76579174.0
7672ac4b.0
7999be0d.0
7a819ef2.0
7d453d8f.0
81b9768f.0
82223c44.0
8470719d.0
85cde254.0
86212b19.0
87753b0d.0
882de061.0
89c02a45.0
8d6437c3.0
91739615.0
9282e51c.0
9339512a.0
9479c8c3.0
9576d26b.0
95aff9e3.0
961f5451.0
9685a493.0
9772ca32.0
9ab62355.0
9c3323d4.0
9d6523ce.0
9dbefe7b.0
9f533518.0
a0bc6fbb.0
a2c66da8.0
a2df7ad7.0
a3896b44.0
a7605362.0
a7d2cf64.0
a81e292b.0
ab5346f4.0
aeb67534.0
b0ed035a.0
b0f3e76e.0
b3fb433b.0
b7db1890.0
b872f2b4.0
bc3f2570.0
bdacca6f.0
bf64f35b.0
c491639e.0
c51c224c.0
c7e2a638.0
c90bc37d.0
cb156124.0
cb1c3204.0
ccc52f49.0
cf701eeb.0
d06393bb.0
d16a5865.0
d18e9066.0
d4c339cb.0
d5727d6a.0
d59297b8.0
d66b55d9.0
d6e6eab9.0
d7746a63.0
d8317ada.0
dbc54cab.0
dc99f41e.0
dfc0fe80.0
e268a4c5.0
e442e424.0
e48193cf.0
e60bf0c0.0
e775ed2d.0
e8651083.0
ea169617.0
ed39abd0.0
ee7cd6fb.0
ee90b008.0
f61bff45.0
f80cc7f6.0
fac084d7.0
facacbc6.0
fb126c6d.0
fde84897.0
ff783690.0
Update: clean and rebuild didn't change anything. Still got Error while downloading text, curl error code: SSL peer certificate or SSH remote key was not OK
Just for fun I tried disabling SSL verification per the above post...and that got it past the download definition no problem. But getPossibleSettings... errors out,
2024-12-03 12:41:55.354 21290-21576 Mavsdk com.dronesense.pilot.blue I Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-12-03 12:41:55.357 21290-21576 Mavsdk com.dronesense.pilot.blue I Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-12-03 12:41:55.514 21290-21576 Mavsdk com.dronesense.pilot.blue I Downloaded file, result Success
2024-12-03 12:41:55.514 21290-21576 Mavsdk com.dronesense.pilot.blue D Successfully loaded camera definition
2024-12-03 12:41:55.514 21290-21576 Mavsdk com.dronesense.pilot.blue E tinyxml2::Parse failed: Error=XML_ERROR_EMPTY_DOCUMENT ErrorID=13 (0xd) Line number=0
2024-12-03 12:41:55.601 21290-21408 Mavsdk com.dronesense.pilot.blue E Unknown setting to get: CAM_MODE
2024-12-03 12:41:55.601 21290-21408 Mavsdk com.dronesense.pilot.blue E Unknown setting to set: CAM_MODE
Yeah okay, looks like the error is correct - it's getting an empty string for some reason.
std::thread([this, camera_information]() {
std::string content{};
const auto result = fetch_camera_definition(camera_information, content);
if (result == Camera::Result::Success) {
LogDebug() << "Successfully loaded camera definition";
LogDebug() << "RAWR: THE CONTENT IS " << content;
The output is empty,
2024-12-03 12:49:40.988 8277-8568 Mavsdk com.dronesense.pilot.blue D RAWR: THE CONTENT IS
2024-12-03 12:49:40.988 8277-8568 Mavsdk com.dronesense.pilot.blue E tinyxml2::Parse failed: Error=XML_ERROR_EMPTY_DOCUMENT ErrorID=13 (0xd) Line number=0
Got it. Also had to enable "Follow Location" to allow it to handle redirects,
curl_easy_setopt(curl.get(), CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(curl.get(), CURLOPT_FOLLOWLOCATION, 1L);
Then it started to work,
2024-12-03 13:29:47.293 2230-4159 Mavsdk com.dronesense.pilot.blue I Downloaded file, result Success
2024-12-03 13:29:47.293 2230-4159 Mavsdk com.dronesense.pilot.blue D Successfully loaded camera definition
2024-12-03 13:29:47.293 2230-4159 Mavsdk com.dronesense.pilot.blue D RAWR: THE CONTENT IS <?xml version="1.0" encoding="UTF-8" ?>
<mavlinkcamera>
<definition version="5">
<model>Vio Payload</model>
<vendor>Gremsy</vendor>
</definition>
<parameters>
<!-- control = 0 tells us this should not create an automatic UI control -->
<parameter name="CAM_MODE" type="uint32" default="1" control="0">
<description>Camera Mode</description>
So now it's just a matter of having the certs validate the proper way I guess
Did you try with CURLOPT_SSL_VERIFYPEER=1 and CURLOPT_FOLLOWLOCATION=1? Maybe the error "SSL peer certificate or SSH remote key was not OK" happens on the empty string because the redirection was not followed?
Did you try with CURLOPT_SSL_VERIFYPEER=1 and CURLOPT_FOLLOWLOCATION=1? Maybe the error "SSL peer certificate or SSH remote key was not OK" happens on the empty string because the redirection was not followed?
Just tried it. Same Error while downloading text, curl error code: SSL peer certificate or SSH remote key was not OK error
Just to be sure: are you sure that your Android device has up-to-date certificates? I.e. it's not an Android system from 6 years ago that never received updates? 😅
Just to be sure: are you sure that your Android device has up-to-date certificates? I.e. it's not an Android system from 6 years ago that never received updates? 😅
Actually, it is an Android system from 6 years ago - it's a Herelink controller running Android 7.1.2. Just to verify however, I did try loading up Chrome.apk on there and tried the Github page and it loaded fine there.
Actually, it is an Android system from 6 years ago - it's a Herelink controller running Android 7.1.2.
Haha, yeah I was a bit suspicious of that 🙈😅
Just to verify however, I did try loading up Chrome.apk on there and tried the Github page and it loaded fine there.
The problem here is that Chrome does not necessarily use the system certificates; I could imagine that they embed their own (which is good in your case because otherwise Chrome wouldn't work properly on the Herelink, I think).
Would you be able to run the test on a more up-to-date Android system? It would be interesting to see if we have a bug in our solution or if the problem is Herelink's outdated certificates 🤔.
Thanks a lot for all the testing, by the way!
Would you be able to run the test on a more up-to-date Android system? It would be interesting to see if we have a bug in our solution or if the problem is Herelink's outdated certificates 🤔.
Unfortunately, I don't have a way to connect the drone to anything else 😢
I had initially posted earlier that running the logic on S23 independent of drone worked. However, that was too early to call, because I had forgotten to remove the "disable SSL verification" flag before running the test. Will re-test it with flag removed. Apologies for the confusion.
Yeah, after removing the "disable verification" flag, running on S23 results in the same error as before,
2024-12-04 14:10:53.421 17866-17939 Mavsdk com.dronesense.pilot.blue I RAWR Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-12-04 14:10:53.485 17866-17956 Mavsdk com.dronesense.pilot.blue E Error while downloading text, curl error code: SSL peer certificate or SSH remote key was not OK
2024-12-04 14:10:53.485 17866-17956 Mavsdk com.dronesense.pilot.blue E RAWR Failed to download camera definition.
Some contents from my original post:
I was able to test this logic on an S23 phone independent of a drone.
HttpLoader http_loader; std::string uri = "https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml"; LogInfo() << "RAWR Downloading camera definition from: " << uri; std::string camera_definition_out = ""; if (!http_loader.download_text_sync(uri, camera_definition_out)) { LogErr() << "RAWR Failed to download camera definition."; } else { LogInfo() << "RAWR SUCCESS " << camera_definition_out; }I put this inside mavsdk_impl.cpp.
I guess that means, this use case could be instrumentally tested.... 😉
@rayw-dronesense sorry, I'm not quite following. Are you saying we have the same problem on the S23, so this is not a Herelink problem as such?