software-review rsi: Efficiently Retrieve and Process Satellite Imagery

Submitting Author Name: Mike Mahoney Submitting Author Github Handle: @mikemahoney218 Repository: https://github.com/Permian-Global-Research/rsi/ Version submitted: Submission type: Standard Editor: @jhollist Reviewers: @mdsumner, @OldLipe

Due date for @mdsumner: 2024-06-19

Due date for @OldLipe: 2024-07-09 Archive: TBD Version accepted: TBD Language: en

Paste the full DESCRIPTION file inside a code block below:

Package: rsi
Title: Efficiently Retrieve and Process Satellite Imagery
Version: 0.2.0.9000
Authors@R: c(
    person("Michael", "Mahoney", , "[email protected]", role = c("aut", "cre"),
           comment = c(ORCID = "0000-0003-2402-304X")),
    person("Permian Global", role = c("cph", "fnd"))
  )
Description: Downloads spatial data from spatiotemporal asset catalogs 
    ('STAC'), computes standard spectral indices from the Awesome Spectral 
    Indices project (Montero et al. (2023) <doi:10.1038/s41597-023-02096-0>) 
    against raster data, and glues the outputs together into predictor bricks. 
    Methods focus on interoperability with the broader spatial ecosystem; 
    function arguments and outputs use classes from 'sf' and 'terra', and data 
    downloading functions support complex 'CQL2' queries using 'rstac'.
License: Apache License (>= 2)
Depends: 
    R (>= 4.0)
Imports: 
    future.apply,
    glue,
    httr,
    jsonlite,
    lifecycle,
    proceduralnames,
    rlang,
    rstac,
    sf,
    terra,
    tibble
Suggests: 
    curl,
    knitr,
    progressr,
    rmarkdown,
    testthat (>= 3.0.0),
    withr
Config/testthat/edition: 3
Config/testthat/parallel: true
Encoding: UTF-8
LazyData: true
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.0
URL: https://github.com/Permian-Global-Research/rsi, https://permian-global-research.github.io/rsi/
BugReports: https://github.com/Permian-Global-Research/rsi/issues
VignetteBuilder: knitr

Scope

Please indicate which category or categories from our package fit policies this package falls under: (Please check an appropriate box below. If you are unsure, we suggest you make a pre-submission inquiry.):
- [x] data retrieval
- [ ] data extraction
- [x] data munging
- [ ] data deposition
- [ ] data validation and testing
- [ ] workflow automation
- [ ] version control
- [ ] citation management and bibliometrics
- [ ] scientific software wrappers
- [ ] field and lab reproducibility tools
- [ ] database software bindings
- [x] geospatial data
- [ ] text analysis
Explain how and why the package falls under these categories (briefly, 1-2 sentences):

This package supports (spatial) data retrieval from APIs implementing the OGC STAC API standard, processing the downloaded data (including automated masking, compositing and rescaling) computing spectral indices from those data, and wrangling the outputs into formats useful for modeling and visualization.

Who is the target audience and what are scientific applications of this package?

Anyone with a need to download and process spatial data, particularly remote sensing data, particularly satellite-based earth observation rasters. We've used rsi to automate the entire data preparation process of forest carbon and structure models (not yet published), but the package is broadly useful to anyone working in Earth surface modeling.

Are there other R packages that accomplish the same thing? If so, how does yours differ or meet our criteria for best-in-category?

Yes:

rstac is a fantastic package for querying and downloading data from STAC APIs. rstac does not implement the other elements of rsi (compositing, masking, rescaling, computing spectral indices, wrangling the outputs), and rsi provides a "higher level" method for downloading data from STAC APIs (powered by lower-level rstac functions); rsi also uses a faster download method.
gdalcubes and sits both provide higher-level approaches for accessing data from STAC APIs, organized around data cube models. There is more substantial overlap between these packages and rsi. The biggest difference is that rsi, very intentionally, does not provide a new data model. To quote the vignette:

A core difference between rsi and these packages is that rsi does not have a data model: rsi is focused entirely on finding the bits of data you want from remote endpoints, and getting those bits on your local machine for you to process with your normal spatial data tooling. There are no new classes in rsi (other than the band mapping objects), and the outputs of functions are local rasters. This is an approach that fits better in my head than the more abstract delayed computations in some other packages; at the same time, it’s possible that this approach can be less efficient, downloading more data at finer resolutions than is actually needed for a given task.

There are a few minor things that I think rsi does better than other approaches (we sign items right before each one is downloaded, for instance, whereas some other packages sign items before starting to download the entire set, meaning the signature can expire causing large downloads to fail), but this difference I think is mostly a matter of taste. I'm very familiar with local GDAL, terra, and sf, and so rsi tries to get users back to working with local GDAL, terra, and sf as fast as possible.

(If applicable) Does your package comply with our guidance around Ethics, Data Privacy and Human Subjects Research?

NA

If you made a pre-submission inquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.

NA

Explain reasons for any pkgcheck items which your package is unable to pass.

~I have never successfully gotten the CI item to be a check, and I have no idea why. I'm using the standard usethis functions to structure my CI, for what it's worth!~ Apparently this is a local-only issue.

I addressed failing covr in https://github.com/ropensci/software-review/issues/636#issuecomment-2028086514

Technical checks

Confirm each of the following by checking the box.

[x] I have read the rOpenSci packaging guide.
[x] I have read the author guide and I expect to maintain this package for at least 2 years or to find a replacement.

This package:

[x] does not violate the Terms of Service of any service it interacts with.
[x] has a CRAN and OSI accepted license.
[x] contains a README with instructions for installing the development version.
[x] includes documentation with examples for all functions, created with roxygen2.
[x] contains a vignette with examples of its essential functions and uses.
[x] has a test suite.
[x] has continuous integration, including reporting of test coverage.

Publication options

[x] Do you intend for this package to go on CRAN?
[ ] Do you intend for this package to go on Bioconductor?
[ ] Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:

MEE Options

[ ] The package is novel and will be of interest to the broad readership of the journal.
[ ] The manuscript describing the package is no longer than 3000 words.
[ ] You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)
(Scope: Do consider MEE's Aims and Scope for your manuscript. We make no guarantee that your manuscript will be within MEE scope.)
(Although not required, we strongly recommend having a full manuscript prepared when you submit here.)
(Please do not submit your package separately to Methods in Ecology and Evolution)

Code of conduct

[x] I agree to abide by rOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

Mar 30 '24 13:03 mikemahoney218

Thanks for submitting to rOpenSci, our editors and @ropensci-review-bot will reply soon. Type @ropensci-review-bot help for help.

Mar 30 '24 13:03 ropensci-review-bot

:rocket:

Editor check started

:wave:

Mar 30 '24 13:03 ropensci-review-bot

Another quick note -- I will not be able to transfer this repository to rOpenSci if accepted. I had asked on Slack and was told by Yani that this was acceptable, though it's only mentioned in the book here. I want to flag this at the start, in case it turns out to be an issue!

Mar 30 '24 13:03 mikemahoney218

Checks for rsi (v0.2.0.9000)

git hash: 694d2a5f

:heavy_check_mark: Package is already on CRAN.
:heavy_check_mark: has a 'codemeta.json' file.
:heavy_check_mark: has a 'contributing' file.
:heavy_check_mark: uses 'roxygen2'.
:heavy_check_mark: 'DESCRIPTION' has a URL field.
:heavy_check_mark: 'DESCRIPTION' has a BugReports field.
:heavy_check_mark: Package has at least one HTML vignette
:heavy_check_mark: All functions have examples.
:heavy_check_mark: Package has continuous integration checks.
:heavy_multiplication_x: Package coverage failed
:heavy_check_mark: R CMD check found no errors.
:heavy_check_mark: R CMD check found no warnings.
:eyes: Function names are duplicated in other packages

Important: All failing checks above must be addressed prior to proceeding

(Checks marked with :eyes: may be optionally addressed.)

Package License: Apache License (>= 2)

1. Package Dependencies

Details of Package Dependency Usage (click to open)

The table below tallies all function calls to all packages ('ncalls'), both internal (r-base + recommended, along with the package itself), and external (imported and suggested packages). 'NA' values indicate packages to which no identified calls to R functions could be found. Note that these results are generated by an automated code-tagging system which may not be entirely accurate.

type	package	ncalls
internal	base	175
internal	rsi	34
internal	methods	3
internal	stats	2
internal	tools	2
imports	rlang	10
imports	terra	10
imports	rstac	5
imports	glue	4
imports	future.apply	2
imports	httr	1
imports	sf	1
imports	jsonlite	NA
imports	lifecycle	NA
imports	proceduralnames	NA
imports	tibble	NA
suggests	progressr	1
suggests	curl	NA
suggests	knitr	NA
suggests	rmarkdown	NA
suggests	testthat	NA
suggests	withr	NA
linking_to	NA	NA

Click below for tallies of functions used in each package. Locations of each call within this package may be generated locally by running 's <- pkgstats::pkgstats(<path/to/repo>)', and examining the 'external_calls' table.

base

names (20), c (19), class (10), lapply (9), length (8), vapply (8), args (6), formals (6), list (6), mget (6), tryCatch (5), file.path (4), for (4), max (4), min (4), options (4), tempfile (4), ifelse (3), url (3), all (2), call (2), character (2), drop (2), eval (2), is.null (2), nrow (2), paste (2), paste0 (2), unlist (2), with (2), col (1), data.frame (1), dirname (1), get (1), grep (1), mapply (1), merge (1), ncol (1), numeric (1), readLines (1), replicate (1), seq_len (1), source (1), str2lang (1), suppressWarnings (1), t (1), tempdir (1), tolower (1), toupper (1), vector (1)

rsi

build_progressr (5), spectral_indices (3), extract_urls (2), remap_band_names (2), alos_palsar_mask_function (1), calc_scale_strings (1), calculate_indices (1), check_indices (1), check_type_and_length (1), default_query_function (1), download_web_indices (1), filter_bands (1), filter_platforms (1), get_alos_palsar_imagery (1), get_dem (1), get_landsat_imagery (1), get_naip_imagery (1), get_rescaling_formula (1), get_sentinel1_imagery (1), get_sentinel2_imagery (1), get_stac_data (1), is_pc (1), landsat_mask_function (1), maybe_sign_items (1), set_gdalwarp_extent (1), spectral_indices_url (1)

rlang

arg_match (4), caller_env (2), warn (2), exec (1), new_environment (1)

terra

rast (5), sprc (2), crs (1), nlyr (1), predict (1)

rstac

assets_url (2), items_datetime (2), stac_search (1)

glue

glue (4)

methods

is (3)

future.apply

future_lapply (2)

stats

predict (1), setNames (1)

tools

file_ext (1), R_user_dir (1)

httr

user_agent (1)

progressr

progressor (1)

sf

st_bbox (1)

NOTE: Some imported packages appear to have no associated function calls; please ensure with author that these 'Imports' are listed appropriately.

2. Statistical Properties

This package features some noteworthy statistical properties which may need to be clarified by a handling editor prior to progressing.

Details of statistical properties (click to open)

The package has:

code in R (100% in 15 files) and
1 authors
3 vignettes
5 internal data files
11 imported packages
21 exported functions (median 19 lines of code)
62 non-exported functions in R (median 15 lines of code)

Statistical properties of package structure as distributional percentiles in relation to all current CRAN packages The following terminology is used:

loc = "Lines of Code"
fn = "function"
exp/not_exp = exported / not exported

All parameters are explained as tooltips in the locally-rendered HTML version of this report generated by the checks_to_markdown() function

The final measure (fn_call_network_size) is the total number of calls between functions (in R), or more abstract relationships between code objects in other languages. Values are flagged as "noteworthy" when they lie in the upper or lower 5th percentile.

measure	value	percentile	noteworthy
files_R	15	73.0
files_vignettes	3	92.4
files_tests	9	89.6
loc_R	1437	77.1
loc_vignettes	453	75.7
loc_tests	853	84.6
num_vignettes	3	94.2
data_size_total	26424	76.4
data_size_median	4831	74.3
n_fns_r	83	71.4
n_fns_r_exported	21	68.8
n_fns_r_not_exported	62	73.1
n_fns_per_file_r	3	55.1
num_params_per_fn	7	85.3
loc_per_fn_r	15	46.1
loc_per_fn_r_exp	19	44.7
loc_per_fn_r_not_exp	15	49.5
rel_whitespace_R	10	63.3
rel_whitespace_vignettes	32	76.2
rel_whitespace_tests	13	73.5
doclines_per_fn_exp	51	64.2
doclines_per_fn_not_exp	0	0.0	TRUE
fn_call_network_size	36	59.4

2a. Network visualisation

Click to see the interactive network visualisation of calls between objects in package

3. `goodpractice` and other checks

Details of goodpractice checks (click to open)

3a. Continuous Integration Badges

GitHub Workflow Results

id	name	conclusion	sha	run_number	date
8486854009	Lock Threads	success	694d2a	156	2024-03-30
8482543797	pages build and deployment	success	1de384	65	2024-03-29
8482287996	pkgdown	success	694d2a	135	2024-03-29
8482287992	R-CMD-check	success	694d2a	131	2024-03-29
8482287990	R-CMD-check-hard	success	694d2a	131	2024-03-29
8482287998	test-coverage	success	694d2a	131	2024-03-29

3b. `goodpractice` results

`R CMD check` with rcmdcheck

rcmdcheck found no errors, warnings, or notes

Test coverage with covr

ERROR: Test Coverage Failed

Cyclocomplexity with cyclocomp

The following functions have cyclocomplexity >= 15:

function	cyclocomplexity
get_stac_data	44
stack_rasters	28
check_type_and_length	25

Static code analyses with lintr

lintr found the following 127 potential issues:

message	number of times
Avoid library() and require() calls in packages	4
Lines should not be more than 80 characters.	123

4. Other Checks

Details of other checks (click to open)

:heavy_multiplication_x: The following 2 function names are duplicated in other packages:

- calculate_indices from ClusterStability
- sign_planetary_computer from rstac

Package Versions

package	version
pkgstats	0.1.3.11
pkgcheck	0.1.2.21

Editor-in-Chief Instructions:

Processing may not proceed until the items marked with :heavy_multiplication_x: have been resolved.

Mar 30 '24 14:03 ropensci-review-bot

Guessing covr fails due to not setting my custom is_covr environment variable: https://github.com/Permian-Global-Research/rsi/commit/c62e6e9aa08c06a54094edc6c3df994260d8aa17

This environment variable is used to skip a test on my covr CI. The tl;dr is that rsi executes some code in a minimal environment to protect against malicious code downloaded from the internet, which prevents covr from injecting its tracking inside of that minimal environment. I still want the file to be tested (and the other pieces of the file to be counted in coverage), though, so I wrapped the local environment section in nocov and added this environment variable.

You can see my code coverage report at https://app.codecov.io/gh/Permian-Global-Research/rsi and my CI workflow for this at https://github.com/Permian-Global-Research/rsi/blob/main/.github/workflows/test-coverage.yaml

Mar 30 '24 14:03 mikemahoney218

@ropensci-review-bot assign @jhollist as editor

Apr 01 '24 20:04 ldecicco-USGS

Assigned! @jhollist is now the editor

Apr 01 '24 20:04 ropensci-review-bot

@mikemahoney218 Been swamped these last few days. I will work on digging through this today and tomorrow and get back to you soon and should hopefully be ready to start finding reviewers.

I am looking forward to this review. Does look like an interesting package!

Apr 04 '24 14:04 jhollist

No worries, and thanks for the update!

Apr 04 '24 14:04 mikemahoney218

@ropensci-review-bot check rsi

Apr 11 '24 14:04 jhollist

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@ropensci-review-bot help

Apr 11 '24 14:04 ropensci-review-bot

@ropensci-review-bot check package

Apr 11 '24 14:04 jhollist

Thanks, about to send the query.

Apr 11 '24 14:04 ropensci-review-bot

:rocket:

Editor check started

:wave:

Apr 11 '24 14:04 ropensci-review-bot

Just want to flag that I'm still expecting (your version of) covr to fail, due to https://github.com/ropensci/software-review/issues/636#issuecomment-2028086514

The core issue is that calculate_indices() is basically intended to run code from a random site on the internet -- a trusted site, but still a security risk. As such, that downloaded code is run inside a minimal environment that prevents injecting any unexpected code or functions. Unfortunately, covr works by injecting unexpected functions into your source code, and then counting how many times its functions get run -- then calculate_indices() doesn't allow those to execute, causing the function to fail.

As a result, I toggle the tests that hit this code path using a custom is_covr variable, which isn't set by your covr check (because I just made it up, I don't know of a supported way to do this) and so your covr check fails.

I don't want to disable the whole .R file from covr, because covr can instrument the rest of the file, and I don't want to drop these tests (or make them off by default) because I'd like R CMD check to check this function. I've got a live coverage report running via GHA and hopefully viewable at: https://app.codecov.io/gh/Permian-Global-Research/rsi?branch=main

Apr 11 '24 15:04 mikemahoney218