Use Spansh search results for API calls requiring bulk system data? (since EDSM API is quite rate-limited)

[splatpope]

What's Wrong (please be as specific as possible)

Plotting a ship route for station services in the Navigation Monitor tab can be exceedingly slow and inefficient, leading to some instances in which the search doesn't conclude in a timely manner (we're talking upwards of 10 minutes without a response, and we will see below that this delay is most likely to be one hour long), mostly when searching for tech brokers. For the other services, searches tend to resolve successfully, but sometimes after more than 20 seconds of unresponsiveness.

The fact that there are no timeouts on those searches, or that they simply cannot be cancelled can be very damageable to user experience regarding this specific feature. These are separate issues that are arguably even more critical.

This particular issue seems to stem from a cooldown issue with EDSM.

---

I will rather describe below a situation in which I'm in the Nu Pictoris system, looking for the nearest Guardian Tech Broker. According to EDSM, which to my understanding is the source of the data used by these searches, the proper route should be a direct jump to the Latuba system, then supercruise to the Soukup City station. This intended target is well within the advertised limits of the search feature.

Expected

A two waypoint route in the relevant list view, the first of which is the current system "Nu Pictoris", the second of which being the "Latuba" system, "Soukup City" station.

Observed

The search does not conclude and is stuck, as evidenced by the "Search" button displaying a looping animation, and the text-to-speech program warning me that a route search is in progress when trying to setup a new, easier search (i.e. nearest fuel star).

The latter symptom can not appear when using EDDI directly, as the search interface does not allow for plotting a new route while a query is in progress; thus, this appeared when using EDDI with VoiceAttack, but note that it can be safely concluded that VA isn't implicated in the issue.

Steps to reproduce

While in the Nu Pictoris system, ask the "Plot Ship Route" feature of the "Navigation Monitor" tab to find the nearest Guardian or Human Technology Broker. To truly reproduce my particular experience, search for all other services than tech brokers beforehand.

Configuration

• Version [Please give full version numbers here, not just "latest" or "current"]
  • EDDI Version: 4.0.2

My Investigation

Investigation Notes

One thing I did was to read the verbose logs. The one attached contains output for multiple successful searches originating from the Nu Pictoris system. The last one is the guardian tech broker search mentioned above.

One can easily see that the search gets a list of systems from EDSM, then for each and all of them, it gets the list of its celestial bodies from EDSM in separate requests then parses the contents. This seems consistent with the capabilities of EDSM's API.

Some time during this process, EDSM complains about too many requests and the search seems to be stuck for the duration of EDSM's cooldown, i.e. one hour (!!!).

It seems that all the relevant code for this feature is in NavigationService/QueryResolvers/ServiceQueryResolvers.cs

I do see that query results are supposed to be cached to a SQLite database, which seems to explain why the other searches seemed to become much faster after the first time doing them. (Although this first time is still exceedingly long). One can probably deduce that since tech broker searches have a much larger range, it is normal that more queries have to be made than for the other types of searches, these queries being logically not cached yet.

The aforementioned database cache is probably why the log doesn't show similar output for the previous successful searches, as I had performed them multiple times before during testing, in a previous instance of EDDI.

Considering all of the above, if the amount of EDSM requests generated by the searches is actually reasonible, then the problem might be with EDSM itself and its absurdly long cooldown. (to be honest, I'm 50/50 on this)

EDDI Logs

[Any log files providing additional details about what happened. These are located at %APPDATA%/EDDI and have the extension ".log"] eddi.log

Player journals

[These are not required if EDDI Logs are verbose but may be helpful otherwise. They are located at [your Saved Games folder]/Frontier Developments/Elite Dangerous/. Please only provide excerpts or attach files from the play session where the issue occurred.]

---

All of this notwithstanding, I do not understand why a full search has to be performed in a cube around the player to find specific services.

On EDSM's website, it is possible to search for specific services, simultaneously sorting the results by distance to commander (by supplying the current commander position). Sadly, it seems that their API does not support crafting such a query.

One could directly use EDSM's (or inara's for that matter) station service search directly from their site, using web crawling. Obviously, this solution is highly sensitive to their website's layout.