Celeo
Expose result header information.
It seems currently the API doesn't expose the headers returned from esi.
For simple requests this doesn't matter, but when querying paged results with many pages (like orders in jita) it's necessary to inspect header data to get consistent data.
The reason for this is that when requesting pages 1-100+ with one request per page, different requests can return data from different snapshots when done near expiry time.
To my knowledge this is best avoided by one of these:
- Check that responses for all pages have the same last changed time.
- Fire of one request and use the header info to time the query such that it's "guaranteed" to not overlap the data expiration time.
- All of the above.
Either way one needs to inspect the header data to avoid the problem of different pages representing different snapshots.
The easiest way would probably be to have *_response methods like get_op_response that returns the response object instead of just the esi data. But I understand if that's beyond the complexity you want to take this package to.
For reference I currently use esipy for this, but it using pyswagger, and pyswagger being broken with python 3.10+ I'm looking for alternatives.
+1. Also looking to move away from esipy and Preston seems to be a good alternative. Need to dig in a bit more, but it looks like we don't get paginated results outside of the default first page:
In [128]: orders = auth.get_op('get_markets_region_id_orders',order_type="sell",region_id=10000067,page=1)
In [129]: print(orders[999])
{'duration': 365, 'is_buy_order': False, 'issued': '2023-10-03T11:07:23Z', 'location_id': 60008398, 'min_volume': 1, 'order_id': 1605899268, 'price': 25000000.0, 'range': 'region', 'system_id': 30005229, 'type_id': 2825, 'volume_remain': 595, 'volume_total': 595}
In [130]: orders = auth.get_op('get_markets_region_id_orders',order_type="sell",region_id=10000067,page=12)
In [131]: print(orders[999])
{'duration': 365, 'is_buy_order': False, 'issued': '2023-10-03T11:07:23Z', 'location_id': 60008398, 'min_volume': 1, 'order_id': 1605899268, 'price': 25000000.0, 'range': 'region', 'system_id': 30005229, 'type_id': 2825, 'volume_remain': 595, 'volume_total': 595}
In [132]:
Thanks for the interest, and thanks for bringing this up!
Yup, finding some way to bridge that update time to ensure data consistency definitely sounds like something I'll want to fix.
Same with pagination in general - I'll take a look.
The library isn't adding the query parameters, which results in always getting the first page.
I published an update that adds URL query params from the kwargs not used in URL variable replacement.
Thus,
orders = preston.get_op(
'get_markets_region_id_orders',
order_type = "sell",
region_id = 10000067,
page = 1
)
will yield a URL of https://esi.evetech.net/v1/markets/10000067/orders/?order_type=sell&page=1.
Hey, hate to wake a sleeping bear, but i stumbled over this issue too. I either need the maxPage header data to be exposed to loop with the existing function, or the existing function to support pagination.
Trying to get market data from a private structure via an authenticated call, gives back 41 pages i need to pass threw :(
Would https://github.com/Celeo/Preston/pull/32 work?
data = p.get_op('get_markets_region_id_orders', region_id=10000012, order_type='all', page=1)
print(f'Got {len(data)} entries')
print(p.stored_headers[0])
# {'Date': 'Sun, 13 Oct 2024 17:42:24 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Headers': 'Content-Type,Authorization,If-None-Match,X-User-Agent', 'Access-Control-Allow-Methods': 'GET,HEAD,OPTIONS', 'Access-Control-Allow-Origin': '*', 'Access-Control-Expose-Headers': 'Content-Type,Warning,ETag,X-Pages,X-ESI-Error-Limit-Remain,X-ESI-Error-Limit-Reset', 'Access-Control-Max-Age': '600', 'Allow': 'GET,HEAD,OPTIONS', 'Cache-Control': 'public', 'Content-Encoding': 'gzip', 'Etag': '"3374c3659f3ab9b24a7707080ccbc541711c8b101c7a40077b88c0c8"', 'Expires': 'Sun, 13 Oct 2024 17:43:16 GMT', 'Last-Modified': 'Sun, 13 Oct 2024 17:38:16 GMT', 'Strict-Transport-Security': 'max-age=31536000', 'Vary': 'Accept-Encoding', 'X-Esi-Error-Limit-Remain': '100', 'X-Esi-Error-Limit-Reset': '36', 'X-Esi-Request-Id': '4bb8ea86-559b-488c-9446-fe1eac52bc6b', 'X-Pages': '12'}
I'd be pushing the latest headers to the start of that list, so [0] would be the most recent. I'm not sure if people would need more than just the most recent request headers. If not, I'll drop the list idea.