USEPA
TADA_GetATTAINS returning 'There are no ATTAINS catchments associated with these WQP observations.' on some large spatial region datapulls
Describe the bug
I expect there to be catchments and AU associated with a WQP TADA data retrieval for WQP observations found in the entire state of Oregon, but none are being returned when you run TADA_GetATTAINS.
We see TADA_Overviewmap() below.
To Reproduce
TADA_Example <- TADA_DataRetrieval( startDate = "2022-01-01", endDate = "2024-12-31", organization = "OREGONDEQ", characteristicName = c("Enterococcus", "Escherichia", "Escherichia coli", "Fecal Coliform", "Total Coliform"), ask = FALSE)
TADA_ATTAINS <- TADA_GetATTAINS(TADA_Example, return_nearest = TRUE)
Expected behavior
I would expect there to be at least some WQP Monitoring Sites that would fall within a catchment and AU if the results are pretty representative of all sites in Oregon.
A smaller data query in Oregon returned a TADA_GetATTAINS output - an AU was matched to those sites successfully with a smaller spatial extent.
@wokenny13 - you could look at the ATTAINS geospatial layer directly to get an idea of what should be appearing in that area in OR. If I'm remembering correctly, OR's GIS submission was very large so I am wondering if there may be some issue when there are many different features to be returned by TADA_GetATTAINS.
Hm - I think that ATTAINS is currently having a hard time handling large API queries. Decreasing L262's id_chunks <- split(assessment_unit_ids, ceiling(seq_along(assessment_unit_ids) / 1000)) to something lower than 1000 might help?
I just tried 10 and it seemed to solve the issue, albeit it is much slower...
Perhaps a value in between, like 200 could work? A large enough value that can make it still run at a reasonable speed would be good to use.
This was also occurring for a few other states (can't recall which). But if 200 is still too large, modifying this number could be a good spot to look at in the future.
Would using a tryCatch help to still return ATTAINS catchment that are associated with WQP observations for those that have been found? And a message that indicates which monitoring locations the error has occurred due to handling of large API queries to help identify this problem when it pops up?
Hm - I think that ATTAINS is currently having a hard time handling large API queries. Decreasing L262's
id_chunks <- split(assessment_unit_ids, ceiling(seq_along(assessment_unit_ids) / 1000))to something lower than 1000 might help?I just tried 10 and it seemed to solve the issue, albeit it is much slower...
I tried to lower this to 10 for the Oregon example, but still run into the same issue. Lowering this number did help with other examples that had this problem, but even when lowering it to 10, I still seem to run into the same error message:
Depending on your data's observation count and its spatial range, the ATTAINS pull may take a while. although coordinates are longitude/latitude, st_intersects assumes that they are planar There are no ATTAINS catchments associated with these WQP observations.
If I reduce the aoi in Oregon though, I am able to successfully get WQP sites matched to ATTAINS AUs.
Hm - I think that ATTAINS is currently having a hard time handling large API queries. Decreasing L262's
id_chunks <- split(assessment_unit_ids, ceiling(seq_along(assessment_unit_ids) / 1000))to something lower than 1000 might help? I just tried 10 and it seemed to solve the issue, albeit it is much slower...I tried to lower this to 10 for the Oregon example, but still run into the same issue. Lowering this number did help with other examples that had this problem, but even when lowering it to 10, I still seem to run into the same error message:
Depending on your data's observation count and its spatial range, the ATTAINS pull may take a while. although coordinates are longitude/latitude, st_intersects assumes that they are planar There are no ATTAINS catchments associated with these WQP observations.If I reduce the aoi in Oregon though, I am able to successfully get WQP sites matched to ATTAINS AUs.
I wonder if we could also automatically split up the aoi for an entire state into smaller chunks before querying ATTAINS, and then join the layers back together? @hillarymarler Do you know if ATTAINS has any summary service that could be used to understand how many features would be returned from a state or county (AOI) before actually pulling the geospatial features?
@cristinamullin - I am not sure if ATTAINS has any summary service for the geospatial features, but I will take a look at it and see what I can find out.
@cristinamullin and @wokenny13 - it looks there are some options for understanding how many features would be returned. Do you know where you'd be wanting to implement this? As part of TADA_GetATTAINS? Of in fetchATTAINS?
Have we tried reducing the AOI size at which the sites need to be split into clusters? I'm giving this a try now to see if it helps w/ the OR example. Update: It did not work.
I'm also still exploring some options for getting number of features.
I added a draft version of the fetchATTAINScount function to get counts of the features within a bbox from each ATTAINS layer. It has not yet been incorporated into fetchATTAINS.
@hillarymarler @wokenny13 could this be related to the issue Ben raised here: https://github.com/USEPA/TADAShinyJoinToAU/issues/1