/universe/names is occasionally unexpectedly returning 404 for valid ids

[eve-n0rman]

Bug

When posting a particular set of ids to /universe/names for name resolution, ESI is frequently returning 404 for valid IDs

Request

curl -X POST "https://esi.evetech.net/latest/universe/names/?datasource=tranquility" -i -H "accept: application/json" -H "Content-Type: application/json" -d "[ 621, 30001198, 2116133890, 98169165]"

Response

When not working:

{"error":"Ensure all IDs are valid before resolving."}%

When working:

[{"category":"character","id":2116133890,"name":"Cody McClaren"},{"category":"corporation","id":98169165,"name":"Brave Newbies Inc."},{"category":"inventory_type","id":621,"name":"Caracal"},{"category":"solar_system","id":30001198,"name":"GE-8JV"}]%

Status Code

404

Headers

When not working

HTTP/1.1 404 Not Found
Date: Tue, 17 Mar 2020 21:41:22 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 54
Connection: keep-alive
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,Authorization,If-None-Match,X-User-Agent
Access-Control-Allow-Methods: POST,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Type,Warning,ETag,X-Pages,X-ESI-Error-Limit-Remain,X-ESI-Error-Limit-Reset
Access-Control-Max-Age: 600
Allow: OPTIONS,POST
Strict-Transport-Security: max-age=31536000
X-Esi-Error-Limit-Remain: 99
X-Esi-Error-Limit-Reset: 38
X-Esi-Request-Id: 6976d04b-c889-4e60-9df1-29743d5ffacd

When working

HTTP/1.1 200 OK
Date: Tue, 17 Mar 2020 21:41:27 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 248
Connection: keep-alive
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,Authorization,If-None-Match,X-User-Agent
Access-Control-Allow-Methods: POST,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Type,Warning,ETag,X-Pages,X-ESI-Error-Limit-Remain,X-ESI-Error-Limit-Reset
Access-Control-Max-Age: 600
Allow: OPTIONS,POST
Strict-Transport-Security: max-age=31536000
X-Esi-Error-Limit-Remain: 99
X-Esi-Error-Limit-Reset: 33
X-Esi-Request-Id: f82c4fc6-d057-489e-be9a-44335200c293

Body

When not working:

{"error":"Ensure all IDs are valid before resolving."}%

When working:

[{"category":"character","id":2116133890,"name":"Cody McClaren"},{"category":"corporation","id":98169165,"name":"Brave Newbies Inc."},{"category":"inventory_type","id":621,"name":"Caracal"},{"category":"solar_system","id":30001198,"name":"GE-8JV"}]%

Expected

200

Checklist

Check all boxes that apply to this issue:

• [x] Bug description is provided
• [x] Request path is provided
• [x] Response status code is provided
• [x] Response headers are provided
• [x] Response body is provided
• [x] Expected response is provided

[lukasni]

Expanding a bit on this, it seems to be particular sets of IDs that reliably trigger this behaviour. Some sets I'm testing with will never 404, while others reliably trigger errors every other request or so.

Having encountered the issue, I did the following investigation:

Initial bad ids. This list would quite reliably trigger 404 errors. [30003098,95883265,98311523,498125261,670,1228904765,98143374,99000285,11176,2613]

I split the list in two halves, eliminating the first half which now reliably returned, while the second half continued to exhibit the issue.

Remaining bad ids [1228904765, 98143374, 99000285, 11176, 2613]

Repeating that process left us with the following list, which once again would cause 404 errors: [99000285, 11176, 2613]

Further attempts with that list lead to the final combination of [99000285, 2613] which can still cause these issues. No other subsets of that last list would cause the error, nor does either value when used individually.

Here are two sequential requests with the response headers, one successful, the other 404. Note that the requests are identical and only one second apart.

iex(159)> ExEsi.Request.Hackney.request(:post, "https://esi.evetech.net/latest/universe/names/", "[99000285, 2613]")
{:ok,
 %{
   body: "[{\"category\":\"alliance\",\"id\":99000285,\"name\":\"Good Sax\"},{\"category\":\"inventory_type\",\"id\":2613,\"name\":\"Mjolnir Fury Light Missile\"}]",
   headers: [
     {"Date", "Tue, 12 May 2020 04:44:30 GMT"},
     {"Content-Type", "application/json; charset=UTF-8"},
     {"Content-Length", "133"},
     {"Connection", "keep-alive"},
     {"Access-Control-Allow-Credentials", "true"},
     {"Access-Control-Allow-Headers",
      "Content-Type,Authorization,If-None-Match,X-User-Agent"},
     {"Access-Control-Allow-Methods", "POST,OPTIONS"},
     {"Access-Control-Allow-Origin", "*"},
     {"Access-Control-Expose-Headers", 
      "Content-Type,Warning,ETag,X-Pages,X-ESI-Error-Limit-Remain,X-ESI-Error-Limit-Reset"},
     {"Access-Control-Max-Age", "600"},
     {"Allow", "OPTIONS,POST"},
     {"Strict-Transport-Security", "max-age=31536000"},
     {"X-Esi-Error-Limit-Remain", "100"}, 
     {"X-Esi-Error-Limit-Reset", "30"},
     {"X-Esi-Request-Id", "cf67cd8f-f1bd-41fc-a12a-400b1161ce2b"}
   ],
   status_code: 200
 }}
iex(160)> ExEsi.Request.Hackney.request(:post, "https://esi.evetech.net/latest/universe/names/", "[99000285, 2613]")
{:ok,
 %{
   body: "{\"error\":\"Ensure all IDs are valid before resolving.\"}",
   headers: [
     {"Date", "Tue, 12 May 2020 04:44:31 GMT"},
     {"Content-Type", "application/json; charset=UTF-8"},
     {"Content-Length", "54"},
     {"Connection", "keep-alive"},
     {"Access-Control-Allow-Credentials", "true"},
     {"Access-Control-Allow-Headers",
      "Content-Type,Authorization,If-None-Match,X-User-Agent"},
     {"Access-Control-Allow-Methods", "POST,OPTIONS"},
     {"Access-Control-Allow-Origin", "*"},
     {"Access-Control-Expose-Headers",
      "Content-Type,Warning,ETag,X-Pages,X-ESI-Error-Limit-Remain,X-ESI-Error-Limit-Reset"},
     {"Access-Control-Max-Age", "600"},
     {"Allow", "OPTIONS,POST"},
     {"Strict-Transport-Security", "max-age=31536000"},
     {"X-Esi-Error-Limit-Remain", "99"},
     {"X-Esi-Error-Limit-Reset", "29"},
     {"X-Esi-Request-Id", "8b049999-dd7c-46b2-b360-cb4163663ab3"}
   ],
   status_code: 404
 }}

[ErikKalkoken]

I recognized that behavior too. My current workaround is a divide and conquer approach:

When I get a 404 I divide the list of IDs into halfs and try again for both. Then recursively re-try again for each half that returns 404 until the list only contains only one item or I reach a maximum number of recursions.

[lukasni]

I considered that, but it's not really necessary, just retrying with the full set with a bit of backoff generally gets me a good result in 2-3 attempts.

[lukasni]

Sorry about that, I may have accidentally the wrong button ^^

[gitAskur]

universe/names is a chimera of an endpoint route that mixes so many different name contexts together that the backend is entirely inefficient and stuff like this happens because we need to protect the backend from frivolous requests, coupled with the complete lack of context of what type of name is being resolved just makes this route an error-prone mess.

As we complete the internal migration tasks, I doubt we will reinstate this riyte but I'm going to bring that up to the squad before closing this to make sure that´s not just some daydream I had.

As a workaround, please try not supplyig typeIDs into universe/names but rather use the universe/types/{type_id} route for types. Same goes for any itemID name resolution you know the type of - it is in all cases better to use the contextualized routes, i.e. pretty much any other universe endpoint route other than universe/names.

Hopefully if the endpoint only gets valid owner names supplied, it wil not have a problem resolving them.

[guiguilechat]

But then, assuming we can do the specific path calls for a type, what information does this endpoint add ? Would it be better to just remove it ?

[gitAskur]

Yes it would. This endpoint is a bastardman that should never have been as far as I care.

[guiguilechat]

So it does not add anything to the information provided ? I think it helps people get the batch names for several entities, but this can be also done with correct parallel requests on the corresponding universe/X paths. Also it seems to me inefficient to use a POST request to GET static information.

[CarbonAlabel]

Keep in mind the /universe/names/ endpoint is currently relied on by other endpoints for determining entity types, see https://github.com/esi/esi-issues/issues/839 and https://github.com/esi/esi-issues/issues/879.

[gitAskur]

This is why this endpoint should never have existed. It allows us to give you identities without context, forcing you to use this endpoint as a catch-all for getting the identities entity context. We should fix those ambiguous returns rather than maintain this bastardmanroute, and then consider all future releases that are somehow ambiguous as broken rather than supported by this.

[jamesrobb]

Still seems like an issue. Internal issue created: TTC-3825

[jamesrobb]

@eve-n0rman have you noticed any particular circumstances when this happens? Time of day, for example.

[lukasni]

@jamesrobb It doesn't appear to be linked to time of day but specific ID combinations, specifically mixing character/corps and similar dynamic data with static data like types and solar systems. See my reply above for a more detailed analysis.

I'll also add some logging for those to my application that is most likely to encounter the issue to give you some time of day data.

EDIT: The following gist from my current log, I wasn't logging the payload previously, adding that now. Will need a little time before I can provide any meaningful data. https://gist.github.com/lukasni/0ae38e0d122b7f7775e6bb4cb5e9b655

[mateuszkrasucki]

@lukasni @eve-n0rman @guiguilechat @CarbonAlabel this should be fixed now. Please confirm.

[antihax]

This appears to be occurring again since around noon today and resulted in at least four of my ESI nodes getting banned since I was using this to identify entities out of notifications.

Valid character https://esi.evetech.net/latest/characters/92792768/

 curl -X POST "https://esi.evetech.net/latest/universe/names/?datasource=tranquility" -H "accept: application/json" -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d "[92792768]"
{"error":"Ensure all IDs are valid before resolving."}