Download Mission triggers User callback queue overflown error

[ankitatcodingnebula]

Hello,

I am running into an issue where calling the downloadMission() api leads to User callback queue overflown error error on the mavsdk console causing the System's Core plugin connection state to disconnect

The issue occurs with a large mission uploaded to Gazebo headless docker instance. There are 2 different instances of error I see:

1. On an area of 160 acre with around 2000 waypoints - User callback queue overflown error
2. On an area of 10 acre with around 500 waypoints - User callback queue too slow

Below is the code I am using:

mav.getMission().downloadMission().subscribe({ MissionProto.MissionPlan plan -> println('Downloaded')}, {})

I was initially using the blockingGet() api and I thought the error could be because this blocking api. But the results are same with subscribe() too

mav.getMission().downloadMission().blockingGet()

Please advise.

Thank You

[JonasVautherin]

Did the error message show this link, and did you have a look? https://mavsdk.mavlink.io/main/en/cpp/troubleshooting.html#user_callbacks

[ankitatcodingnebula]

Did the error message show this link, and did you have a look? https://mavsdk.mavlink.io/main/en/cpp/troubleshooting.html#user_callbacks

Hi @JonasVautherin ,

Yes I did and that's why I changed the blockingGet() call to subscribe() but it has no effect. I am doing nothing inside the subscribe onNext call. Just doing a console print.

mav.getMission().downloadMission().subscribe({ MissionProto.MissionPlan plan -> println('Downloaded')}, {})

This basic code throws the error on mavsdk for large waypoints

[JonasVautherin]

Got it :ok_hand:. Sounds like a bug then. I'll try to have a look one of these days, but if you feel like debugging it, it's happening here.

Also maybe @julianoes has an intuition on that?

[julianoes]

@ankitatcodingnebula it would be helpful if you could provide a minimal example to reproduce this.

[julianoes]

Nevermind. We were able to test it with an example:

    System drone = new System();

    List<Mission.MissionItem> cycle = new ArrayList<>();
    cycle.add(generateMissionItem(47.398039859999997, 8.5455725400000002));
    cycle.add(generateMissionItem(47.398036222362471, 8.5450146439425509));
    cycle.add(generateMissionItem(47.397825620791885, 8.5450092830163271));
    cycle.add(generateMissionItem(47.397832880000003, 8.5455939999999995));

    List<Mission.MissionItem> missionItems = new ArrayList<>();
    for (int i = 0; i < 300; i++) {
        missionItems.addAll(cycle);
    }

    Mission.MissionPlan missionPlan = new Mission.MissionPlan(missionItems);
    logger.debug("About to upload " + missionItems.size() + " mission items");

    CountDownLatch latch = new CountDownLatch(1);
    drone.getMission()
        .setReturnToLaunchAfterMission(true)
        .andThen(drone.getMission().uploadMission(missionPlan)
            .doOnComplete(() -> logger.debug("Upload succeeded")))
        .andThen(drone.getMission().downloadMission()
            .doOnSubscribe(disposable -> logger.debug("Downloading mission"))
            .doAfterSuccess(disposable -> logger.debug("Mission downloaded")))
        .toCompletable()
        .andThen((CompletableSource) cs -> latch.countDown())
        .subscribe();

    try {
      latch.await();
    } catch (InterruptedException ignored) {
      // This is expected
    }

However, we can't see the problem happening in mavsdk_server. Which version of mavsdk_server are you using?

[JonasVautherin]

To add to @julianoes comment above, I tried to upload it to the Auterion cloud simulator (with a higher delay), and it also seems to work:

> Task :runMission
10:32:20.683 [main] DEBUG io.mavsdk.example.RunMission - Starting example: mission...
10:32:20.951 [main] DEBUG io.mavsdk.example.RunMission - About to upload 1200 mission items
10:40:43.367 [grpc-default-executor-2] DEBUG io.mavsdk.example.RunMission - Upload succeeded
10:40:43.370 [grpc-default-executor-2] DEBUG io.mavsdk.example.RunMission - Downloading mission
10:48:28.871 [grpc-default-executor-2] DEBUG io.mavsdk.example.RunMission - Mission downloaded

Note how it took 8 minutes to upload and download in this case (it's much faster when running on a faster link obviously).

[ankitatcodingnebula]

To add to @julianoes comment above, I tried to upload it to the Auterion cloud simulator (with a higher delay), and it also seems to work:

> Task :runMission
10:32:20.683 [main] DEBUG io.mavsdk.example.RunMission - Starting example: mission...
10:32:20.951 [main] DEBUG io.mavsdk.example.RunMission - About to upload 1200 mission items
10:40:43.367 [grpc-default-executor-2] DEBUG io.mavsdk.example.RunMission - Upload succeeded
10:40:43.370 [grpc-default-executor-2] DEBUG io.mavsdk.example.RunMission - Downloading mission
10:48:28.871 [grpc-default-executor-2] DEBUG io.mavsdk.example.RunMission - Mission downloaded

Note how it took 8 minutes to upload and download in this case (it's much faster when running on a faster link obviously).

Thanks @JonasVautherin @julianoes

I am using v0.39 64 bit version on Windows machine. Please note that I do not see this issue with macOS version.

Thanks for the snippet above. I will try the same code and respond in few hours. And also will provide my sample with issue.