Feature Request

The path is https://esi.evetech.net/ui/#/Market/get_markets_region_id_history.

The days without a sale are not returned in this path. example, getting the jormungandr sales in jita I get no entry for 2019-04-08

Use case

When I try to get the volumic sales of an item, I use the median value to avoid peaks . So I fetch this path and take the median of the volumes. However this means that even if there was only one day of sale, I will have a median of this value ; while actually the median should be 0.

ATM How I fixed it is, by getting the min and max dates of the orders, and simulating those days with a zero. Instead of 365 (one day) values this avoids both adding days when the item was just released, and being subject to a modification of the market period modification by CCP. However I can't be sure whether this item was released or not before the first sale. In case of the jormungandr, the first sale after the 2018-04-06 (which should not be present) is the 2018-04-23. does it mean the item was released on this date ? I can't tell by looking at the result only.

Checklist

Check all boxes that apply to this issue:

• [x] Feature request description is provided
• [x] Use case exists
• [ ] Feature requires a new route
• [x] Feature adds data to existing route
• [ ] Feature requires new auth scope
• [x] Feature can reuse existing scope
• [ ] Feature does not require auth
• [ ] Meta feature, applies to all routes

There's no way to make a promise in the spec that days without a sale will be artificially injected so that things can be correctly averaged like you want to, so I'm not a fan of that as a solution. It falls into the same camp as things like providing implicit sorting, it's a psuedo-undocumented feature that could cause problems if it becomes a defacto standard since there's no way to have it be tested for correct behavior against the spec.

It sounds like you're dealing with this correctly. The market order history entries have a date, and you're using the dates provided to enrich and pad the data in a way which lets you run the type of analysis you wish to run. That's a robust way of handling the data, situation normal?

Yeah, average of no value is difficult to compute and I also guessed that if the data is not stored in monolith that would not be possible.

The second issue I have with my code is that since the resource does not provide today's date(it's in the headers, not in the resource) I may lack the period range (eg if last sale was 6 month ago and first one was 6month and 1 day ago I will consider the range is 2days - while it should be 6months)

Yeah, average of no value is difficult to compute and I also guessed that if the data is not stored in monolith that would not be possible.

It is possible, it's just not efficient. Padding the data with zeros can be done as easily in your code as on the eve server, but the eve server has limited CPU resources to spend on the various tasks it has. It makes sense to only spend that resource on jobs which cannot be done downstream. It also makes the data transmitted much larger, which has many downsides particularly for mobile applications.

When it comes to today's date, unless you're processing data that was retrieved in the past, you don't even need to check the headers, you can just use your languages date function to get the current date.

Maybe add an optional &padding=true parameter to the path to add 0-volume, 0-everything entries to the list ? As I wrote before if the monolith does not know when an item is added to the game it may become difficult though.

Is this more about determining when an item was authored?

That's a more complex story. The server doesn't know when things were authored, it just knows the things it's been told exist. Also authoring dates can be a little deceptive. Some things were authored /years/ before they were used. They can idle in a changelist without being checked in, or be checked in and deployed but unpublished. There's also the case of checked in and published but no one cares about the item until one day it's changed and suddenly it matters.

It's a very fuzzy thing to try to quantify when an item was added to the game.

that's the java code I use once I get sent the new history cache (to create the new list of volumes, sorted by value desc)

		List<Long> newList = cache.stream().map(data -> data.volume).sorted((l1, l2) -> Long.compare(l2, l1))
				.collect(Collectors.toList());
		// pad with 0 for the days that do not appear.
		if (cache.size() > 1) {
			SimpleDateFormat myFormat = new SimpleDateFormat("yyyy-MM-dd");
			try {
				Date date1 = myFormat.parse(cache.get(0).date);
				Date date2 = myFormat.parse(cache.get(cache.size() - 1).date);
				long periodDays = TimeUnit.DAYS.convert(date2.getTime() - date1.getTime(), TimeUnit.MILLISECONDS);
				for (int i = newList.size(); i < periodDays + 1; i++) {
					newList.add(0l);
				}
			} catch (ParseException e) {
				e.printStackTrace();
			}
		}

The cache is ensured to be sorted by the dates (it's an intermediate structure from what is received by the cache manager that handles the fetch on the esi)

... It also makes the data transmitted much larger, which has many downsides particularly for mobile applications.

Not only for mobile applications. If you're scraping history data a large portion of region/type pairs returns basically empty markets. Sure, compression alleviates the issue a little and responses are being cached. While ESI servers can cache the responses daily fairly easily once generated, high-volume consumers (e.g. EVE-Tools/static-data) of the data would need to parse a pretty big payload (including complex string-based datetime data types) for basically empty markets.

Maybe add an optional &padding=true parameter to the path to add 0-volume, 0-everything entries to the list ?

The authoring problem is not really solved. However, you solved the problem yourself already but I see that the documentation (especially on the swagger page) is lacking in that regard. So maybe updating the documentation about certain guarantees (not) given by the API could be a reasonable compromise.

close with a won't-fix ?