CS-SI
Providers chaining for download
Hello,
I think that enabling a sort of provider chaining in case of the preffered one(s) fail(es) could be a real advantage. This redundancy would allow EODAG to be used in some usecase where products are downloaded automatically and where those drawbacks (not responding servers) have to be tackled.
Let me explain my usecase by an example:
I set PEPS as my preffered provider, and I want to download S2-L1C products.
For any reason, the PEPS server doesn't respond.
In this case, it would be very nice to try to download these products on an other provider.
The chaining could be done by selecting providers with decreasing priority.
Thanks a lot and keep up the good work :)
Hello Rémi, thank you for this feature request. This behaviour seems natural, and we will plan to implement it
As this is related to #164, we'd suggest that here the user can do dag.download_all(products, fallback_providers=["theia"]). The method would first start to download the products with their associated provider (the one that returned them from a search), and if that fails, would fallback to trying to download them with the providers listed in fallback_providers.
We'd use fallback_providers instead of providers (like in search) since dag.download_all(products, providers=["theia", "peps"]) could be confusing if the products were obtained with a different provider (e.g. sobloo) than the first listed one (should it download with sobloo or instead with theia).
We would also not rely on the set_preferred_provider/priority mechanisms here. We can imagine a situation whereby a user would start to download products from a provider and actually download them from AWS (and pay) if they have it configured (YAML, env vars), without being aware of such mechanisms. Passing a fallback_providers parameter seems more explicit/safe.
I gave some thought about this issue and believe that @remi-braun you could help me clarify what you would be interested in.
The fallback mechanism could be implemented in two different ways:
- Try to download products from a provider and if that fails at some point, try to download the remaining products (potentially all of them) from another set of providers (sequentially if they also fail). The idea here would be to try to download the exact same products as the ones that were asked to be downloaded (those passed to
dag.download_all(products)). - Try to download products from a provider and if that fails at some point, try to download the products obtained from the same search made to another provider.
For example:
- A search on peps ended up with 10 products. The downloads fail at some point. The fallback mechanism tries to download the remaining products from sobloo, if that fails again or at some point, from mundi.
- A search on peps ended up with 10 products. The downloads fail at some point. The fallback mechanism makes the same search on sobloo and gets 12 products, it tries to download them. If that fails it makes the same search on mundi and tries to download them.
To implement 1., we could use a feature we don't expose so much in the API documentation but that is exposed in the CLI: we can look for a product given its ID and provider. As a test I ran the following snippet to check what kind of ID we get on a search of S2_MS1_L1C products with different providers:
import datetime
import dateutil
import time
from eodag import EODataAccessGateway
from eodag.utils.logging import setup_logging
setup_logging(verbose=1)
DAG = EODataAccessGateway('/home/maxime/TRAVAIL/06_EODAG/01_eodag/eodag/tests/resources/user_conf.yml')
SEARCH_CRITERIA = dict(
productType='S2_MSI_L1C',
start='2020-08-08',
end='2020-08-16',
geom=[151, 73, 152, 74],
items_per_page=30,
)
def sort_product_by_start_date(product):
"""Get product start date"""
start_date = product.properties.get("startTimeFromAscendingNode")
if not start_date:
# Retrieve year, month, day, hour, minute, second of EPOCH start
epoch = time.gmtime(0)[:-3]
start_date = datetime.datetime(*epoch).isoformat()
return dateutil.parser.parse(start_date)
def search_and_get_id(provider):
DAG.set_preferred_provider(provider)
products, estimate = DAG.search(**SEARCH_CRITERIA)
print(f'Got {len(products)} products.')
filter_prods = sorted(products, key=sort_product_by_start_date, reverse=True)
try:
product = filter_prods[0]
except IndexError:
print("No prods, skip")
return
p_id = product.properties["id"]
return p_id
providers = DAG.available_providers(product_type=SEARCH_CRITERIA["productType"])
provider_id = {}
for provider in providers:
p_id = search_and_get_id(provider)
if p_id:
provider_id[provider] = p_id
print(provider_id)
It returned:
{
"astraea_eod":"S2A_OPER_MSI_L1C_TL_EPAE_20200815T043159_A026884_T56XMG_N02.09",
"aws_eos":"S2B_MSIL1C_20200816T023549_N0209_R089_T56XMG_20200816T041557",
"creodias":"S2A_MSIL1C_20200815T021611_N0209_R003_T56XMH_20200815T043159",
"earth_search":"S2A_56XMG_20200815_0_L1C",
"mundi":"S2A_MSIL1C_20200815T021611_N0209_R003_T56XMH_20200815T043159",
"peps":"S2A_MSIL1C_20200815T021611_N0209_R003_T56XMG_20200815T043159",
"sobloo":"S2A_MSIL1C_20200815T021611_N0209_R003_T56XMH_20200815T043159"
}
The IDs of these products don't seem to always follow the same format. This could be due to us not mapping properly the ID to retrieve from the response we get (which seems to be the case for earth_search), or some differences between the providers. Anyway, in most cases they seem to follow the same naming conventing so that could work (hopefully) most of the time.
As for 2., we would need to handle what happens if some products were first downloaded with the preferred provider before it failed. I guess that in that case the smart thing to do would be to (1) make a search with the fallback provider, (2) try to see if among the products found some of them have already been downloaded or not, if yes, remove them from the list of products to download (3) try to download the products from the fallback provider.
Whch one of 1. and 2. makes the more sense to you?
I would go for the first solution 😄
Another usecase can also be when some queried products are offline for the lead provider, they can be queried on another available providers.
(I implemented on my side, as what I need are the products, from whatever provider. This can be very useful when combining free and AWS-based providers)
The main problem a got so far is if sth is wrong with the lead provider that makes it return an empty query (ie. ONDA for now) , it won't download anything. But this case may be handled separatly (or directly by the user).
Do you now why the same query can results in different results for different providers ?
Thanks for your answer!
I think I also prefer the first solution :+1:
The other use case you mention would indeed be interesting to support. It seems that what we're talking about here could be called fallback_providers_on_error, while there could be another option fallback_providers_on_offline. We could open another issue for that.
Am I right if I say that your main issue is with whenever a provider doesn't return products in a search rather than whenever it fails at downloading them after the search is made (except the issue of offline products we just mentioned)? I would expect (perhaps naively) that if a search return products with a given provider, downloading them afterwards would be possible in most cases. Have you already observed cases when it wasn't true?
If we're talking more about the search side, it seems to me that it's not to complicated to handle that:
try:
products = dag.search_all(...) # or products, estimate = dag.search(raise_errors=True, ...)
except Exception:
dag.set_preferred_provider("some_other_provider")
products = dag.search_all(...)
Yes you are right about the search side. The trick is two handle this automatically (searching only with registered providers, that can query this particular product type), but it is fine on my side.
And I agree with you, an online product queried can almost always be downloaded. But in some rare cases I think it can fail (just for example, what happen when querying with an AWS-based provider with an AWS account that cannot spend money ?). But it is relatively rare and I don't even bothered to manage this case (I do not use AWS based-providers for now)
Eventually, the main issues I have faced (but that are taken care of on my side) and could be managed by EODAG are:
- offline queried product
- provider that is down and raises an Exception on querying
The case provider down + empty query cannot be solved I think...
Yes you are right about the search side. The trick is two handle this automatically (searching only with registered providers, that can query this particular product type), but it is fine on my side.
Yes I think we should document a little more how to list which products are available for a given provider, or which providers provide a given product.
And I agree with you, an online product queried can almost always be downloaded. But in some rare cases I think it can fail (just for example, what happen when querying with an AWS-based provider with an AWS account that cannot spend money ?). But it is relatively rare and I don't even bothered to manage this case (I do not use AWS based-providers for now)
I also don't know what happens in that AWS case! But we seem to agree that search (OK) and download (FAIL) is a rare case if eodag has been configured properly (credentials).
The case provider down + empty query cannot be solved I think...
Do you mean that you search for products and that the result you get is empty (SearchResult([])) despite the provider being down? dag.search has a raise_errors argument that is False by default. If an error occurs while trying to search for products the error is just logged and dag.search returns an empty result.
offline queried product
I'll open a new issue for this case. I might ping you there to ask you a few questions since you implemented already implemented it. And PRs are welcomed too!
provider that is down and raises an Exception on querying
So after all the interesting use case here would be to have a fallback mechanism for searching instead of downloading. Let's see if we need to implement it or just document it:
try:
products = dag.search_all(...)
except Exception:
dag.set_preferred_provider("fallback_provider")
try:
products = dag.search
except Exception:
print("Today is a bad day apparently!")
results = []
Do you mean that you search for products and that the result you get is empty (SearchResult([])) despite the provider being down? dag.search has a raise_errors argument that is False by default. If an error occurs while trying to search for products the error is just logged and dag.search returns an empty result.
For example, currently ONDA is in this case: for some normal queries that give sth for another provider, ONDA return an empty list, even with raise_errors=True
And PRs are welcomed too!
Sadly my implementation is pretty specific and are most of the times wrappers rather than code modification. But ofc if I have sth pretty to add, I won't hesitate 😉
So after all the interesting use case here would be to have a fallback mechanism for searching instead of downloading. Let's see if we need to implement it or just document it:
It could be really nice to wrap these code lines and throw a nice wrapping EODAG exception!
The idea of passing a list fallback searching providers seems really promising imo
The symetry with an hypothetical implementation of fallback_if_offline could be great too.
I think the main issue of letting the user do this provider cascade maybe the chaining of authenticated and usable provider, which is not trivial.
For example in my usecase, I have a recursive function that does more or less:
def search(
start: Union[datetime, str] = None,
end: Union[datetime, str] = None,
aoi: Polygon = None,
**keywords) -> (SearchResult, int):
"""
Overload of the Eodag search function:
- 500 products maximum
- only one page (so 500 products queried max) -> if a user want to see the second page, he can add the keyword
- product type specified by the config file
- raises errors
Args:
start (Union[datetime, str]): Starting datetime (if str, in isoformat; with no microseconds !)
end (Union[datetime, str]): Ending datetime (if str, in isoformat; with no microseconds !)
aoi (Union[Polygon, None]): Polygon if AOI, None if tile
**keywords: All other accepted keywords
Returns:
(SearchResult, int): Query and number of products
"""
# -- Not really done here, but I display these lines for the example --
curr_prov, _ = dag.get_preferred_provider()
ok_providers = dag.available_providers(product_type=product_type)
# This is really not trivial, or maybe i am missing sth, see hereunder
registered_providers = registered_providers(dag)
# --
# Query all wanted products
try:
dag.search(items_per_page=500,
productType=product_type,
start=start, end=end,
geom=aoi,
raise_errors=True,
**keywords)
# If we got a request error, that means the current provider is broken
# Retry the query with another one
except RequestError as ex:
# Set the current provider as broken
down_providers.append(curr_prov)
# Get all registered providers that are not broken
other_providers = [prov for prov in registered_providers.keys()
if (prov not in down_providers and prov in ok_providers)]
# Set the first provider with highest priority that is not broken as preferred
if len(other_providers) > 0:
new_prov = other_providers[0]
dag.set_preferred_provider(new_prov)
return self.search(start, end, aoi, **keywords)
# Else
raise QueryError(ex) from ex
# AttributeErrors often means a bad configuration
except AttributeError as ex:
raise MisconfiguredError(f"{curr_prov} has missing credentials in {os.environ['EODAG_CFG_FILE']}") from ex
# Gather here all other exceptions
except Exception as ex:
raise QueryError(f"Something went wrong during the query:\n{ex}") from ex
def registered_providers(eo_gateway: EODataAccessGateway) -> OrderedDict:
"""
Get all registered providers and their priority and sort them by decreasing priority
Args:
eo_gateway (EODataAccessGateway): EODataAccessGateway class
Returns:
OrderedDict: provider names and their priorities, sorted by decreasing priority
"""
registered_prov = {}
# Loop over provider configurations
for prov_name, prov_cfg in eo_gateway.providers_config.items():
# Provider needs to be accepted by eodownload
if prov_name in Provider.to_value_list():
# Get where the credentials are stored
if hasattr(prov_cfg, "api"):
auth = prov_cfg.api
elif hasattr(prov_cfg, "auth"):
auth = prov_cfg.auth
else:
raise TypeError(f"Unknown provider config: {prov_cfg}")
# A provider is registered if credentials are set and not null
if hasattr(auth, "credentials") and all([val != "" for val in auth.credentials.values()]):
registered_prov[prov_name] = prov_cfg.priority
return OrderedDict(sorted(registered_prov.items(), key=lambda t: t[1], reverse=True))
For example, currently ONDA is in this case: for some normal queries that give sth for another provider, ONDA return an empty list, even with raise_errors=True
When it comes to trying to download products, we send a request if the response contains a given code error, that was configured as a valid Authentication error code (e.g. 403 for theia), we're able to raise a specific AuthenticationError. Maybe such thing could be done for when a search is made. Do you have an example of a request failing with onda right now?
It could be really nice to wrap these code lines and throw a nice wrapping EODAG exception!
Indeed catching this bare Exception isn't the best thing.
The idea of passing a list fallback searching providers seems really promising imo The symetry with an hypothetical implementation of fallback_if_offline could be great too.
Ok so I'll also open an issue for the idea of a fallback mechanism on the "search" side.
Thanks a lot for your sharing your code! It's super interesting to see how you use eodag. I'll have a look at it in more details next week.
Do you now why the same query can results in different results for different providers ?
I can't tell for sure now but I think I have observed that a few times already. I'd have to double check though. As to why if that's the case, no clue!
It has happen some times for me, some providers seem consistent with each other (peps ~= scihub ~= creodias) unlike others seem to act differently (like sobloo).
From what I remember is that once I got 10 products for scihub, creodias and peps and 12 on sobloo. Next time I get one of these I will report it to you