esi-issues icon indicating copy to clipboard operation
esi-issues copied to clipboard

Endpoint to determine if character IDs are valid/exist

Open PrometheusSatyen opened this issue 3 years ago • 7 comments

Feature Request

There is an issue where certain character IDs which despite being referenced in mails or other entities, no longer exist in the database (or perhaps exist in a bugged state preventing them from being fetched). I would like a way to determine valid/invalid character IDs without producing errors (and thus potentially falling afoul of rate limits).

Use case

Currently I employ a strategy whereby I take a list of character IDs requiring ID to name resolution, and use the /universe/names endpoint to attempt to resolve them. If I receive a 400 I split the list into 10 groups and then try to resolve them, I repeat this process until I have narrowed down and logged the problematic IDs, which I store in a table and never attempt to fetch again.

It would be very helpful if either:

  • The /universe/names endpoint simply failed softly on invalid IDs and didn't return them. I could then deduce the IDs are invalid by the fact they were not returned.
  • A new endpoint could be added which when given a list of IDs, returns a list of the IDs which are valid/invalid, without throwing any errors.

Authentication

A public route, no auth required.

Example return

For proposal 2 above:

{
    "invalid": [
        1234,
        12345
    ]
}

Checklist

Check all boxes that apply to this issue:

  • [X] Feature request description is provided
  • [X] Use case exists
  • [X] Feature requires a new route
  • [X] Feature adds data to existing route
  • [ ] Feature requires new auth scope
  • [ ] Feature can reuse existing scope
  • [X] Feature does not require auth
  • [ ] Meta feature, applies to all routes

PrometheusSatyen avatar Mar 11 '21 16:03 PrometheusSatyen

I'd say your current method works well enough. You could also try verifying your IDs against the known ranges. Only concern is older IDs before the IDs became sequential, and I can't think that verifying those would justify creating a whole new endpoint.

Also, if you are consistently getting invalid IDs, perhaps you should consider finding a way to correct the source of said IDs.

cvweiss avatar Mar 11 '21 20:03 cvweiss

Also, if you are consistently getting invalid IDs, perhaps you should consider finding a way to correct the source of said IDs.

These IDs are directly from ESI. I have a list of character IDs which did exist and were valid at some point in the past, but are now invalid. The most common place I find them is in mails and notifications since those can go back a long time.

It was indicated to me there are a number of reasons this can happen including certain GDPR related requests.

As such it is impossible to determine if these IDs are valid without trial and error against the API. In my opinion, it should not be necessary to cause errors in order to use the EVE API, since errors can cause rate limits.

PrometheusSatyen avatar Mar 11 '21 22:03 PrometheusSatyen

Since there are multiple ways to do that (including character's public info endpoint) I guess the gist of request here is to have batch verification functionality?

mateuszkrasucki avatar Mar 15 '21 12:03 mateuszkrasucki

Since there are multiple ways to do that (including character's public info endpoint) I guess the gist of request here is to have batch verification functionality?

The key functionality from my point of view is the ability to verify character IDs without producing (potentially a lot of) errors. Batch functionality that guarantees a 200 response would fulfil that.

PrometheusSatyen avatar Mar 15 '21 12:03 PrometheusSatyen

Understood, what you want is a way of verifying validity of character IDs without triggering errors that count against error limit.

mateuszkrasucki avatar Mar 15 '21 15:03 mateuszkrasucki

Understood, what you want is a way of verifying validity of character IDs without triggering errors that count against error limit.

Exactly yes :)

PrometheusSatyen avatar Mar 15 '21 15:03 PrometheusSatyen

I think this feature would be worse as it would allow and encourage scrapping. Instead, IMO all ids that have existed should be considered valid.

guiguilechat avatar Mar 16 '21 15:03 guiguilechat