[REVIEW]: corrselect: Exhaustive variable subset selection based on correlation and association matrices

[editorialbot]

Submitting author: @gcol33 (Gilles Colling) Repository: https://github.com/gcol33/corrselect Branch with paper.md (empty if default branch): main Version: v3.0.2 Editor: @Nikoleta-v3 Reviewers: @dansmith01, @danStich Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c379b3d830d4c1430e4bd7a152ca60a7"><img src="https://joss.theoj.org/papers/c379b3d830d4c1430e4bd7a152ca60a7/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c379b3d830d4c1430e4bd7a152ca60a7/status.svg)](https://joss.theoj.org/papers/c379b3d830d4c1430e4bd7a152ca60a7)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@dansmith01 & @danStich, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @Nikoleta-v3 know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @dansmith01

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.32614/CRAN.package.corrselect is OK
- 10.1145/362342.362367 is OK
- 10.1111/j.1600-0587.2012.07348.x is OK
- 10.1214/009053607000000505 is OK
- 10.1214/09-AOAS312 is OK
- 10.1126/science.1205438 is OK
- 10.1186/1471-2105-9-559 is OK
- 10.1111/j.1467-9868.2005.00503.x is OK
- 10.1093/biostatistics/kxp008 is OK
- 10.1007/978-3-642-17517-6_36 is OK

🟡 SKIP DOIs

- None

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[editorialbot]

Repository Analysis Report

Code breakdown

github.com/AlDanial/cloc v 1.98  T=0.25 s (750.4 files/s, 314132.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                            38           1254             10          22312
Markdown                        40           2528              0           7925
CSS                              8           3213             68           7458
JavaScript                      13           2117           1975           7110
R                               33           1258           2403           5192
SVG                             17              2              1           4059
Rmd                             10           2323           3704           2443
C++                             11            164            127            988
Text                             2             79              0            244
YAML                             3             17              3            126
C/C++ Header                    11             37             11            112
TeX                              1              8              0             99
Python                           1             18             15             96
XML                              1              1              0             32
JSON                             1              0              0              1
-------------------------------------------------------------------------------
SUM:                           190          13019           8317          58197
-------------------------------------------------------------------------------

Commit Count by Author

91	gilles colling
    11	Gilles Colling

Repository History

Repository age: First commit on August 05, 2025

Code distribution (top 3 48-hour windows):

Period	Insertions	% of Total	Signal	View Changes
Nov 24, 2025 - Nov 26, 2025	108524	68.8%	🟠	View diff
Aug 04, 2025 - Aug 06, 2025	25923	16.4%	🟢	View diff
Sep 08, 2025 - Sep 10, 2025	10252	6.5%	🟢	View diff

GitHub Activity Metrics

Metric	Count
GitHub stars	0
Forks	0
GitHub contributors	2
Releases	1
Total issues	0
Total pull requests	0
Unique commenters/reviewers (excluding authors)	0

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 844

✅ The paper includes a Statement of need section

[editorialbot]

License info:

🟡 License found: Other (Check here for OSI approval)

[Nikoleta-v3]

Hey @dansmith01, @danStich (@gcol33) 👋🏻 this is the review thread for the paper. All of our communications will take place here from now on.

As a reviewer, your first step is to create a checklist for your review by entering:

@editorialbot generate my checklist

at the top of a new comment in this thread.

These checklists contain the JOSS requirements ✅ As you go through the submission, please check off any items that have been satisfied. The first comment in this thread also includes links to the JOSS reviewer guidelines.

You’re both familiar with the JOSS process, but just as a quick reminder: reviewers are encouraged to submit issues and pull requests directly on the software repository. When doing so, please mention https://github.com/openjournals/joss-reviews/issues/9539 so a link to this thread is created (and I can keep track of everything that’s happening). Alternatively, feel free to leave your reviews as comments on this issue!

We aim to complete reviews within about 2–4 weeks. Please let me know if either of you needs more time. We can also use EditorialBot to set automatic reminders if you’ll be away for a known period.

Feel free to ping me (@Nikoleta-v3) anytime if you have questions or concerns 😄🙋🏻

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[dansmith01]

Review checklist for @dansmith01

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/gcol33/corrselect?
• [x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@gcol33) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[dansmith01]

@gcol33, can you confirm that you'd like us to review version 2.0.1 of corrselect? I see the development version is 3.0.2.

[gcol33]

Thanks for the question. The version submitted for review is 2.0.1. Since submitting, I have continued development and added new functions and a user interface, which are included in the development version (3.0.2). If appropriate, I can update the submission so the review covers the latest stable version. Please let me know which you prefer.

[dansmith01]

@gcol33 I'd suggest having us review the most recent version of corrselect that satisfies the requirements given in the review checklist, but ultimately it is your decision.

[gcol33]

Thanks, that works for me. What is the correct procedure on the JOSS side for updating the submission? Should I tag a new release and update the paper in the same repository, or is there a specific step needed for the review to switch to the newer version?

[dansmith01]

@gcol33 , @Nikoleta-v3 would need to comment here the command @editorialbot set v1.2.3 as version, substituting 1.2.3 of course. What version of corrselect would you like reviewed?

[gcol33]

Version 3.0.2, which is the latest cran version. I updated the paper.md file accordingly

[Nikoleta-v3]

Hello both, yes reviewing the most recent version sounds best 👍🏻

[Nikoleta-v3]

@editorialbot set v3.0.2 as version

[editorialbot]

Done! version is now v3.0.2

[Nikoleta-v3]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[dansmith01]

Review of corrselect

I have reviewed the corrselect package and the accompanying manuscript. The package addresses a clear need in the field—selecting variables to reduce multicollinearity while preserving the "admissible set" of all possible valid subsets, rather than just a single greedy solution.

I have already opened six issues on the repository regarding specific bugs and improvements. Below are my broader comments on the submission, structured by JOSS requirements and best practice suggestions.

1. JOSS Requirements

• Comparison with State-of-the-Art: The "Statement of Need" correctly identifies caret::findCorrelation() as a common greedy approach. To strengthen this comparison, it would be beneficial to briefly clarify the practical difference for the user: caret returns a single subset based on heuristic removal (typically removing the variable with the highest mean correlation), whereas corrselect (in exact mode) returns all maximal subsets. This distinction is the core value proposition of your package and could be emphasized more strongly.

• Graph Theory & Dependencies:

  • Theoretical Context: The problem the package solves is mathematically equivalent to finding all Maximal Cliques in the complement graph (or Independent Sets in the correlation graph). Explicitly stating this helps users with a computer science background immediately understand the underlying logic.
  • Implementation & igraph: I commend the author for providing custom C++ implementations of the Bron-Kerbosch and ELS algorithms, which keeps the package lightweight. However, because igraph is used in the vignettes, users might assume the package is merely a wrapper around igraph::maximal.cliques(). It would be beneficial to explicitly state in the paper that corrselect implements these algorithms natively. Citing igraph (Csardi & Nepusz, 2006) as the standard general-purpose baseline allows you to effectively contrast your specialized implementation against it.

• Performance Claims: The documentation mentions the "Exact mode" is recommended for $p \le 100$. It is important to briefly mention the computational complexity (NP-hard in the general case) so users understand why the exact mode might hang on larger datasets without a high threshold.

2. Best Practices & Opinions (Constructive Feedback)

• Additional Literature:

  • FCBF (Fast Correlation Based Filter): You might consider referencing Yu, L., & Liu, H. (2003). Feature selection for high-dimensional data: A fast correlation-based filter solution. This is a seminal work on correlation-based filter methods that uses a greedy approach. Citing it allows you to contrast your exhaustive search against the standard greedy baseline established in the literature.

• Functionality/Testing: The current test suite covers many standard scenarios, but I recommend adding two specific edge-case tests to ensure robustness:

  1. The "Identity" Case: Test a pure identity matrix (all off-diagonals = 0). Assert that the output is exactly one subset containing all variables.
  2. The "Perfect Duplicate" Case: Test a matrix with two perfectly correlated variables ($r=1.0$) without force_in. Assert that the algorithm correctly separates them into different valid subsets (i.e., verifying that the floating-point comparison strictly excludes $r > threshold$).

• Documentation: In paper.md, it would be helpful to include a very short code snippet demonstrating the "All Subsets" output from corrSelect(), as this visualizes the key differentiator from caret and other greedy methods.

Summary of Suggested Citations for paper.md

• Yu, L., & Liu, H. (2003). Feature selection for high-dimensional data: A fast correlation-based filter solution. Proceedings of the 20th International Conference on Machine Learning (ICML-03).
• Csardi, G., & Nepusz, T. (2006). The igraph software package for complex network research. InterJournal.

Conclusion

In conclusion, corrselect represents a valuable contribution to the R ecosystem, filling a specific niche for researchers who need rigorous, exhaustive variable selection rather than heuristic approximations. The package is well-implemented, with a core in C++ for performance, and the documentation is generally clear. With the minor additions suggested above regarding context and testing, I believe this package will be a robust tool for fields dealing with high-dimensional, collinear data.

[gcol33]

Thank you for the thorough and constructive review. I have addressed all points raised.

1. JOSS Requirements

Comparison with State-of-the-Art
The comparison with caret has been strengthened in the Statement of Need. The text now explicitly states:

“Where caret returns one subset, corrselect in exact mode might reveal a dozen equally valid alternatives. Having the full set of options helps when domain knowledge should guide final variable selection, or when researchers need to assess the sensitivity of their conclusions to predictor choice.”

Graph Theory and Dependencies
Additional graph-theoretic context was added to the Statement of Need:

“This is equivalent to finding all maximal cliques in the compatibility graph, a well-studied problem in computer science.”

The paper states that algorithms are implemented natively in C++ (not as wrappers around external libraries such as igraph), with the appropriate igraph citation included.

Performance Claims
Benchmark data were added with explicit reference to the NP-hard nature of the problem:

“On a modern desktop (Intel i9-14900K) with correlations clustered near the threshold, p = 100 completed in under 1 second, p = 150 in approximately 19 seconds, p = 175 in about 3 minutes, and p = 200 in roughly 17 minutes.”

2. Best Practices

FCBF Citation
The Yu and Liu (2003) FCBF citation was added. I clarified that FCBF addresses a related but distinct problem, namely selecting features correlated with a target variable while removing redundancy. This distinction is important because FCBF is a supervised filter method, whereas corrselect’s model-agnostic functions are unsupervised.

Edge-case Tests
Both requested edge-case tests were added: the identity matrix case and the perfect-correlation case.

Code Snippet
Added a code example demonstrating the output of corrSelect().

Additional Changes

I reorganized the Functionality section to better distinguish between model-agnostic methods (correlation-based, unsupervised) and model-based methods (VIF-based, supervised). This clarifies the package scope and helps users identify the appropriate function for their use case.

[danStich]

Hi @gcol33, I just wanted to let you know that I will start reviewing formally today and should have all feedback to you by Friday.

[gcol33]

@danStich Thanks for letting me know :)

[dansmith01]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[dansmith01]

@gcol33, @Nikoleta-v3 - looks great!

[danStich]

Review checklist for @danStich

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/gcol33/corrselect?
• [x] License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@gcol33) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[danStich]

@gcol33 Thanks for the opportunity to review this. I took some time to work through the package this week. I also used it to vet several existing data sets from my own teaching and research. I think this is a valuable contribution and could definitely see it being useful for graduate students.

I posted a few issues in the GitHub repository (pasted here for convenience).

https://github.com/gcol33/corrselect/issues/10 https://github.com/gcol33/corrselect/issues/9 https://github.com/gcol33/corrselect/issues/8 https://github.com/gcol33/corrselect/issues/7

Please take these only as suggestions - none of them are acceptance-blockers. Mostly just a few things that came up from an 'end-user' perspective, some of which was during rolling internet outages :)

[gcol33]

@danStich Thanks for the thorough review and for testing the package with your own datasets. It's great to hear it's been useful, and I appreciate the perspective :)

I've replied to all four issues; the changes will be in v3.0.6.