software-review Download and Parse Public Data Released by B3 Exchange

Submitting Author Name: Marcelo S. Perlin Submitting Author Github Handle: @msperlin Other Package Authors Github handles: (comma separated, delete if none) @wilsonfreitas Repository: https://github.com/wilsonfreitas/rb3 Version submitted: 0.0.3 Submission type: Standard Editor: @emilyriederer Reviewers: TBD

Archive: TBD Version accepted: TBD Language: en

Paste the full DESCRIPTION file inside a code block below:

Package: rb3
Title: Download and Parse Public Data Released by B3 Exchange
Description: Download and parse public files released by B3 and convert them
    into useful formats and data structures common to data analysis
    practitioners.
Version: 0.0.3
Authors@R: c(person("Wilson", "Freitas",
                    email = "[email protected]",
                    role = c("aut", "cre")),
             person("Marcelo", "Perlin",
                    email = "[email protected]",
                    role = "aut"))
License: MIT + file LICENSE
LazyData: true
Depends:
    R (>= 4.1.0),
Imports:
    bizdays,
    stringr,
    proto,
    cli,
    readr,
    dplyr,
    rvest,
    httr,
    jsonlite,
    purrr,
    ascii,
    rlang,
    methods,
    yaml,
    digest,
    base64enc
Suggests:
    testthat,
    knitr,
    DT,
    miniUI,
    shiny,
    xtable,
    rmarkdown,
    ggplot2,
    covr,
    scales,
    magrittr,
    tibble,
    tidyr,
    fixedincome
Collate:
    'rb3-package.R'
    'util.R'
    'transmute.R'
    'fields.R'
    'handlers.R'
    'marketdata.R'
    'download-data.R'
    'file.R'
    'convert_to.R'
    'scraper-cdi.R'
    'scraper-futures.R'
    'scraper-yc.R'
    'scraper-cotahist.R'
    'addin-show-templates.R'
    'addin-display-template.R'
    'readers.R'
    'downloaders.R'
    'zzz.R'
BugReports: https://github.com/wilsonfreitas/rb3/issues
URL: https://github.com/wilsonfreitas/rb3,
    http://wilsonfreitas.github.io/rb3/
VignetteBuilder: knitr
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.1.2
Config/testthat/edition: 3
Encoding: UTF-8

Scope

Please indicate which category or categories from our package fit policies this package falls under: (Please check an appropriate box below. If you are unsure, we suggest you make a pre-submission inquiry.):
- [X] data retrieval
- [X] data extraction
- [ ] data munging
- [ ] data deposition
- [ ] data validation and testing
- [ ] workflow automation
- [ ] version control
- [ ] citation management and bibliometrics
- [ ] scientific software wrappers
- [ ] field and lab reproducibility tools
- [ ] database software bindings
- [ ] geospatial data
- [ ] text analysis
Explain how and why the package falls under these categories (briefly, 1-2 sentences):

The package downloads and organizes raw financial data directly from B3, the main financial exchange in Brazil. These datasets are not available in any other way.

Who is the target audience and what are scientific applications of this package?

Academic researchers and practioners of financial markets.

Are there other R packages that accomplish the same thing? If so, how does yours differ or meet our criteria for best-in-category?

No.

(If applicable) Does your package comply with our guidance around Ethics, Data Privacy and Human Subjects Research?

Yes.

If you made a pre-submission inquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.
Explain reasons for any pkgcheck items which your package is unable to pass.

Technical checks

Confirm each of the following by checking the box.

[X] I have read the rOpenSci packaging guide.
[X] I have read the author guide and I expect to maintain this package for at least 2 years or to find a replacement.

This package:

[X] does not violate the Terms of Service of any service it interacts with.
[X] has a CRAN and OSI accepted license.
[X] contains a README with instructions for installing the development version.
[X] includes documentation with examples for all functions, created with roxygen2.
[X] contains a vignette with examples of its essential functions and uses.
[X] has a test suite.
[X] has continuous integration, including reporting of test coverage using services such as Travis CI, Coveralls and/or CodeCov.

Publication options

[X] Do you intend for this package to go on CRAN?
[ ] Do you intend for this package to go on Bioconductor?
[ ] Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:

MEE Options

[ ] The package is novel and will be of interest to the broad readership of the journal.
[ ] The manuscript describing the package is no longer than 3000 words.
[ ] You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)
(Scope: Do consider MEE's Aims and Scope for your manuscript. We make no guarantee that your manuscript will be within MEE scope.)
(Although not required, we strongly recommend having a full manuscript prepared when you submit here.)
(Please do not submit your package separately to Methods in Ecology and Evolution)

Code of conduct

[X] I agree to abide by rOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

May 11 '22 11:05 msperlin

Thanks for submitting to rOpenSci, our editors and @ropensci-review-bot will reply soon. Type @ropensci-review-bot help for help.

May 11 '22 11:05 ropensci-review-bot

Error (500). The editorcheck service is currently unavailable

May 11 '22 11:05 ropensci-review-bot

@wilsonfreitas

May 11 '22 11:05 msperlin

Checks for rb3 (v0.0.3)

git hash: cbb35b8a

:heavy_check_mark: Package is already on CRAN.
:heavy_multiplication_x: does not have a 'codemeta.json' file.
:heavy_multiplication_x: does not have a 'contributing' file.
:heavy_check_mark: uses 'roxygen2'.
:heavy_check_mark: 'DESCRIPTION' has a URL field.
:heavy_check_mark: 'DESCRIPTION' has a BugReports field.
:heavy_check_mark: Package has at least one HTML vignette
:heavy_multiplication_x: These functions do not have examples: [display_template, rb3, show_templates].
:heavy_multiplication_x: Package uses global assignment operator ('<<-').
:heavy_check_mark: Package has continuous integration checks.
:heavy_multiplication_x: Package coverage is 74.6% (should be at least 75%).
:heavy_check_mark: R CMD check found no errors.
:heavy_check_mark: R CMD check found no warnings.

Important: All failing checks above must be addressed prior to proceeding

Package License: MIT + file LICENSE

1. Package Dependencies

Details of Package Dependency Usage (click to open)

The table below tallies all function calls to all packages ('ncalls'), both internal (r-base + recommended, along with the package itself), and external (imported and suggested packages). 'NA' values indicate packages to which no identified calls to R functions could be found. Note that these results are generated by an automated code-tagging system which may not be entirely accurate.

type	package	ncalls
internal	base	231
internal	rb3	174
internal	stats	26
internal	utils	19
internal	graphics	15
imports	methods	14
imports	httr	9
imports	purrr	9
imports	stringr	7
imports	jsonlite	5
imports	bizdays	4
imports	proto	4
imports	rvest	4
imports	dplyr	3
imports	readr	2
imports	ascii	1
imports	digest	1
imports	base64enc	1
imports	cli	NA
imports	rlang	NA
imports	yaml	NA
suggests	shiny	9
suggests	miniUI	2
suggests	testthat	NA
suggests	knitr	NA
suggests	DT	NA
suggests	xtable	NA
suggests	rmarkdown	NA
suggests	ggplot2	NA
suggests	covr	NA
suggests	scales	NA
suggests	magrittr	NA
suggests	tibble	NA
suggests	tidyr	NA
suggests	fixedincome	NA
linking_to	NA	NA

Click below for tallies of functions used in each package. Locations of each call within this package may be generated locally by running 's <- pkgstats::pkgstats(<path/to/repo>)', and examining the 'external_calls' table.

base

list (40), class (16), for (10), if (10), c (9), lapply (9), format (8), is.null (7), names (7), url (7), attr (6), F (6), length (6), as.Date (5), colnames (5), do.call (5), paste0 (4), structure (4), UseMethod (4), by (3), file.path (3), readLines (3), seq (3), as.data.frame (2), as.integer (2), body (2), match (2), options (2), parent.frame (2), Reduce (2), seq_along (2), sign (2), strptime (2), sub (2), sum (2), Sys.Date (2), table (2), try (2), as.character (1), as.numeric (1), as.POSIXct (1), basename (1), charToRaw (1), data.frame (1), dim (1), environment (1), evalq (1), exists (1), get (1), gsub (1), missing (1), order (1), print (1), rbind (1), rep (1), strftime (1), switch (1), t (1), tempdir (1)

rb3

handler (30), fields (11), server (6), download_data (5), width (5), composite (4), match_regex (4), retrieve_template (4), transmuter (4), get (3), query_cdi (3), read_fwf (3), read_marketdata (3), take (3), cachedir (2), code2month (2), fields_description (2), fields_handlers (2), fields_names (2), fields_widths (2), flatten_names (2), iter_rules (2), pass_thru_handler (2), rule_result (2), show_templates (2), unformat (2), apply_rule (1), apply_rule.class_rule (1), apply_rule.predicate_rule (1), apply_rule.regex_rule (1), as_dbl (1), as.data.frame.fields (1), cdi_get (1), clearcache (1), convert_to (1), cotahist_bdrs_get (1), cotahist_equity_get (1), cotahist_equity_options_get (1), cotahist_funds_get (1), cotahist_funds_options_get (1), cotahist_get (1), cotahist_index_options_get (1), cotahist_indexes_get (1), cotahist_units_get (1), cotahist_units_options_get (1), csv_read_file (1), curve_download (1), curve_read (1), datetime_download (1), display_template (1), field (1), fields_description.fields (1), fields_handlers.fields (1), fields_names.fields (1), fields_widths.fields (1), filter_equity_data (1), format_equity (1), format_options (1), futures_get (1), futures_mget (1), fwf_read_file (1), get_single_yc (1), idi_get (1), json_read_file (1), just_download_data (1), keys (1), match_class (1), match_predicate (1), maturity2date (1), mcsv_read_file (1), mfwf_read_file (1), new_field (1), new_part (1), options_open_interest_read (1), parse_columns (1), print.fields (1), print.parts (1), put (1), register (1), save_resource (1), settlement_prices_download (1), settlement_prices_read (1), simple_download (1), single_futures_get (1), stock_indexes_composition_download (1), stock_indexes_composition_reader (1), to_dbl (1), unzip_recursive (1)

stats

df (25), pt (1)

utils

data (15), read.table (2), methods (1), unzip (1)

graphics

text (10), lines (5)

methods

setGeneric (4), as (3), is (3), representation (2), initialize (1), setRefClass (1)

httr

content (3), GET (3), parse_url (2), POST (1)

purrr

map_chr (5), map_lgl (3), map_int (1)

shiny

dialogViewer (4), runGadget (2), shinyApp (2), h3 (1)

stringr

str_replace (2), str_sub (2), str_detect (1), str_match (1), str_pad (1)

jsonlite

fromJSON (3), toJSON (2)

bizdays

bizseq (2), add.bizdays (1), bizdayse (1)

proto

proto (4)

rvest

read_html (2), html_element (1), html_table (1)

dplyr

bind_rows (2), tibble (1)

miniUI

miniPage (2)

readr

read_file (1), read_rds (1)

ascii

ascii (1)

base64enc

base64encode (1)

digest

digest (1)

2. Statistical Properties

This package features some noteworthy statistical properties which may need to be clarified by a handling editor prior to progressing.

Details of statistical properties (click to open)

The package has:

code in R (100% in 18 files) and
2 authors
2 vignettes
no internal data file
16 imported packages
25 exported functions (median 4 lines of code)
200 non-exported functions in R (median 9 lines of code)

Statistical properties of package structure as distributional percentiles in relation to all current CRAN packages The following terminology is used:

loc = "Lines of Code"
fn = "function"
exp/not_exp = exported / not exported

All parameters are explained as tooltips in the locally-rendered HTML version of this report generated by the checks_to_markdown() function

The final measure (fn_call_network_size) is the total number of calls between functions (in R), or more abstract relationships between code objects in other languages. Values are flagged as "noteworthy" when they lie in the upper or lower 5th percentile.

measure	value	percentile	noteworthy
files_R	18	78.3
files_vignettes	2	85.7
files_tests	18	95.7
loc_R	1345	75.6
loc_vignettes	203	50.1
loc_tests	276	62.4
num_vignettes	2	89.2
n_fns_r	225	91.3
n_fns_r_exported	25	73.6
n_fns_r_not_exported	200	93.2
n_fns_per_file_r	7	78.2
num_params_per_fn	1	1.6	TRUE
loc_per_fn_r	8	20.0
loc_per_fn_r_exp	4	4.3	TRUE
loc_per_fn_r_not_exp	9	27.1
rel_whitespace_R	12	66.1
rel_whitespace_vignettes	17	26.9
rel_whitespace_tests	30	68.1
doclines_per_fn_exp	42	52.8
doclines_per_fn_not_exp	0	0.0	TRUE
fn_call_network_size	134	83.7

2a. Network visualisation

Click to see the interactive network visualisation of calls between objects in package

3. `goodpractice` and other checks

Details of goodpractice and other checks (click to open)

3a. Continuous Integration Badges

GitHub Workflow Results

id	name	conclusion	sha	run_number	date
2306339093	pages build and deployment	success	38384b	33	2022-05-11
2306316808	pkgdown	success	cbb35b	36	2022-05-11
2306316813	R-CMD-check	success	cbb35b	54	2022-05-11
2306316802	test-coverage	success	cbb35b	54	2022-05-11

3b. `goodpractice` results

`R CMD check` with rcmdcheck

R CMD check generated the following check_fail:

no_import_package_as_a_whole

Test coverage with covr

Package coverage: 74.58

Cyclocomplexity with cyclocomp

No functions have cyclocomplexity >= 15

Static code analyses with lintr

lintr found the following 3 potential issues:

message	number of times
Lines should not be more than 80 characters.	3

Package Versions

package	version
pkgstats	0.0.4.30
pkgcheck	0.0.3.19

Editor-in-Chief Instructions:

Processing may not proceed until the items marked with :heavy_multiplication_x: have been resolved.

May 11 '22 12:05 ropensci-review-bot

@msperlin The build failure in the checks above is because

Error: 'load_builtin_calendars' is not an exported object from 'namespace:bizdays'

That generally means you're not locally using latest versions of all dependency packages. Please let us know when you've fixed that, and the other issues identified above. Thanks.

Edit: I see that it was actually our system which needed updating to the new bizdays released yesterday on CRAN. Checks have been updated accordingly.

May 11 '22 12:05 mpadge

Hello @msperlin and many thanks for your submission.

We are discussing whether the package is in scope and need a bit more information.

I see you are the author of other packages such as GetDFDData2 already on CRAN. Could you please help to clarify the differences between these two packages? Is the former focused only on financial statements whereas this package provides more granular time series? While I'm sure your two packages are not "competing", I want to fully understand what makes this package novel and how it is best in class.

Similarly, I see a B3 data listed on general services like Yahoo Finance which I know can be accessed through a number of packages including yfR, quantmod, and tidyquant. Could you elaborate on why none of these packages are in a "comparison set"?

Thank you!

May 12 '22 00:05 emilyriederer

Hi @emilyriederer

Please find my replies below:

I see you are the author of other packages such as GetDFDData2 already on CRAN. Could you please help to clarify the differences between these two packages? Is the former focused only on financial statements whereas this package provides more granular time series? While I'm sure your two packages are not "competing", I want to fully understand what makes this package novel and how it is best in class.

The different between GetDFPDAta2 and rb3 is in their scope and audience. Package GetDFPData2 is related to financial documents released by companies to the exchange and its audience is mostly business analysts. This includes sales, profit, and many other financial information on a annual basis. Meanwhile, you cant think of rb3 as an interface to all trade data available at the b3 website, including prices of many different type of contracts such as options, equities (companies) and futures. ITs audience is more related to traders and market participants.

While there is some relationship between datasets from rb3 (stock prices) and GetDFPData2 (financial statements) for equities, there is none for other types of markets.

Similarly, I see a B3 data listed on general services like Yahoo Finance which I know can be accessed through a number of packages including yfR, quantmod, and tidyquant. Could you elaborate on why none of these packages are in a "comparison set"?

The data is indeed simillar, but only for equities (companies). Yahoo finance, for example, does not provide historical prices for futures or option contracts. Package rb3 also provides access to historical yield curves, which is a very unique (and rare) set of data in finance.

May 12 '22 19:05 msperlin

@wilsonfreitas I just fixed the code for codemeta and contributing file. The other problems are deeper in the code. Can you please have a look?

May 12 '22 20:05 msperlin

Hi @emilyriederer and @msperlin

I added the examples to the functions show_templates and display_templates. rb3 is not a function, it is just the tag @name being used in the package documentantion, I changed that.

And with respect to <<- operator, it is used with a ReferenceClass, and accoding to documentation this is the suggest way to modify objects fields. So, I believe that this is not an issue.

May 13 '22 01:05 wilsonfreitas

Furtherly, the test coverage is now 84%.

May 13 '22 11:05 wilsonfreitas

Thanks @wilsonfreitas, our check system has just been updated to do what above link says: Allow global assignment operator if in RefClass statement. Your package no longer fails that check :+1:

And FYI, another new check has just been added which in your case reports this:

4. Other Checks

:heavy_multiplication_x: The following function name is duplicated in other packages:

- download_data from admixr, FedData, fgeo.x, fHMM, hkdatasets, pRecipe, Rnumerai

That's currently a "trial" function so you don't necessarily need to respond, but feel free to change the name if you want.

May 13 '22 12:05 mpadge

Hi @mpadge

Thanks @wilsonfreitas, our check system has just been updated to do what above link says: Allow global assignment operator if in RefClass statement. Your package no longer fails that check 👍

This is great!

download_data has been renamed to download_marketdata.

May 13 '22 13:05 wilsonfreitas

@ropensci-review-bot check package

May 13 '22 22:05 emilyriederer

Thanks, about to send the query.

May 13 '22 22:05 ropensci-review-bot

:rocket:

Editor check started

:wave:

May 13 '22 22:05 ropensci-review-bot