probe
probe copied to clipboard
ooni
all: incorrect progress estimates in percents in log
Describe the bug
Progress estimates in percents are incorrect To Reproduce
Steps to reproduce the behavior.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
Running circumvention tests
iplookup: using avast
sessionresolver: https://dns.google/dns-query...
- Device: pc
- OS: windows 11
- OONI Probe version: 3.7 Additional context
Add any other context about the problem here.
Logs depicting behavior.
2022-05-30 19:42:31.517619+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "processing https://doh-fi.blahdns.com/dns-query";
percentage = "0.7580645161290323";
};
}
2022-05-30 19:42:33.057562+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "dnscheck: measure https://95.216.212.177/dns-query: ok";
percentage = "0.7";
};
}
........
2022-05-30 19:42:40.358453+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "processing dot://doh-fi.blahdns.com/dns-query";
percentage = "0.7629032258064516";
};
}
2022-05-30 19:42:41.968269+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "dnscheck: measure dot://95.216.212.177/dns-query: ok";
percentage = "0.7";
};
}
........
2022-05-30 19:50:19.040735+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "processing https://dnsnl-noads.alekberg.net/dns-query";
percentage = "0.9854838709677419";
};
}
2022-05-30 19:50:22.647250+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "dnscheck: measure https://51.15.124.208/dns-query: ok";
percentage = "0.7";
};
}
........
2022-05-30 19:50:29.237998+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "processing https://doh.tiarap.org/dns-query";
percentage = "0.9903225806451613";
};
}
2022-05-30 19:50:32.285116+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "dnscheck: measure https://104.21.30.162/dns-query: ok";
percentage = "0.55";
};
}
2022-05-30 19:50:32.632051+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "dnscheck: measure https://172.67.173.59/dns-query: ok";
percentage = "0.7";
};
}
2022-05-30 19:50:37.963476+0100 ooniprobe[7108:130841] Got event: {
key = "status.progress";
value = {
message = "dnscheck: measure https://[2606:4700:3030::ac43:ad3b]/dns-query: network_unreachable";
percentage = "0.85";
};
}
It seems it's better to systematically test all platforms and check their behavior. It seems not all platforms behave in the same way and this is because we have different code for handling this kind of events.
| product | group | monotonic progress |
|---|---|---|
| cli | websites | ✔️ |
| cli | im | ✔️ |
| cli | performance | ✔️ |
| cli | circumvention | ✔️ |
| cli | middlebox | ✔️ |
| cli | experimental | ✔️ |
| android | websites | ✔️ |
| android | im | ✔️ |
| android | circumvention | ✔️ |
| android | performance | ✔️ |
| android | experimental | ❌ stunreachability |
| ios | websites | ✔️ |
| ios | im | ✔️ |
| ios | circumvention | ❌ tor |
| ios | performance | ❌ seems completely broken |
| ios | experimental | ❌ |
| deskop | websites | ✔️ |
| desktop | im | ✔️ |
| desktop | circumvention | ✔️ |
| desktop | performance | ✔️ |
| desktop | middleboxes | ✔️ |
| desktop | experimental | ✔️ |
Table 1. Checking whether progress is monotonic by platform
Once I have finished testing all platforms and have completed the table, I can then understand the real issue and then plan on how to fix it. The https://github.com/ooni/probe-cli/pull/785 diff apparently fixed dnscheck for cli and android.
So, the bottom line here seems that we have issues with mobile only. That's certainly a bit surprising but it may be this is happening because we're using a different codepath for running mobile experiments that mimics the old MK API.
Having implemented fixes, it's now time to test again!
| product | group | monotonic progress |
|---|---|---|
| cli | websites | ✔️ |
| cli | im | ✔️ |
| cli | performance | ✔️ |
| cli | circumvention | ✔️ |
| cli | middlebox | ✔️ |
| cli | experimental | ✔️ |
| android | websites | ✔️ |
| android | im | ✔️ |
| android | circumvention | ✔️ |
| android | performance | ✔️ |
| android | experimental | ✔️ |
| ios | websites | N/A |
| ios | im | N/A |
| ios | circumvention | N/A |
| ios | performance | N/A |
| ios | experimental | N/A |
| desktop | websites | ✔️ |
| desktop | im | ✔️ |
| desktop | circumvention | ✔️ |
| desktop | performance | ✔️ |
| desktop | middleboxes | ✔️ |
| desktop | experimental | ✔️ |
I cannot test iOS without releasing, but at least it seems all other platforms have been fixed. I'll bless a new release and then test whether the issue still persist with iOS.
Release v3.15.1, now let's test iOS:
| product | group | monotonic progress |
|---|---|---|
| ios | websites | ✔️ |
| ios | im | ❌ jumps back and forth (why?) |
| ios | circumvention | ❌ jumps back and forth (why?) |
| ios | performance | ❌ jumps back and forth (why?) |
| ios | experimental | N/A |
I don't know what to do 😕. I have confirmed these tests are using ooni/probe-cli v3.15.1, which has fixed the issue completely for Android. Since the engine's codebase is the same, I think the problem is outside my domain. I will ask @aanorbel to run tests with Android and iOS version to confirm he also sees the progress bar jumping on iOS and everything seems WAI for Android.
I feel like a did what was possible for me to do to improve the situation for v3.15.x. We'll see what is the root cause on iOS and perhaps we need to apply further changes to ooni/probe-cli (but I am not sure). So, I am going to move this issue to v3.16.x for now.
We can get similar results on the iOS application as with the Android application without much change. I am looking into the iOS implementations right now.
So, to summarize, @aanorbel confirmed to me that we have fixed Android. We still have issues with iOS and it is yet unclear to us why we have such issues. On a positive note, iOS is the only product we still need to fix.
I am going to add @aanorbel as an assignee so that he can investigate the iOS related aspects of this issue.
@bassosimone , can we revisit this issue to determine the current state and do something to close the issue?
@aanorbel the progress bar is still misbehaving on iOS as I am about to document on https://github.com/ooni/probe/issues/2509. I'll send you a video in private that shows the progress bar misbehaving.