Support for Exporting Template and Namespaced Functions in Rcpp

[ManosPapadakis95]

trafficstars

Enhancement: Support for Exporting Template and Namespaced Functions in Rcpp

Rcpp has been an incredible tool for bridging R and C++, and its flexibility has inspired me to explore new ways to extend its functionality. Building on the existing framework, I have added support for exporting template functions and functions inside namespaces, expanding the possibilities for advanced C++ integration.

Key Features

1. Support for Exporting Template Functions

• The Rcpp::export attribute can now be applied to explicit template instantiations.
• The function must have an explicit instantiation in the .cpp file for it to be correctly recognized.
• Works seamlessly with Rcpp::interfaces(cpp).

2. Support for Exporting Functions Inside Namespaces

• Functions within namespaces can now be exported, as long as a forward declaration is provided with cpp_name in Rcpp::export.
• This ensures proper symbol resolution while allowing structured code organization.

3. Handling of Declaration Specifiers

• The system now accounts for one and only one declaration specifier:
  • template → Used for explicit template instantiations.
  • inline → Supports functions defined inside headers.
• Combinations like "static inline" are not supported (e.g., static is explicitly disallowed).
• An inline function can still be forward-declared without the inline keyword, and it will work seamlessly.

4. New includes Attribute for Header File Support

• Since Rcpp does not scan for header files, a new Rcpp::includes attribute has been introduced.
• This allows specifying one or more header file paths, ensuring necessary definitions are available in the generated .cpp file.
• This is especially important for template functions, as the compiler needs access to their full definitions in order to instantiate them correctly.

Example Usage

// [[Rcpp::includes("helpers.h")]]

// [[Rcpp::export(name = "A1", cpp_name = "A1")]]
template NumericVector A<NumericVector>(size_t n);

// [[Rcpp::export(name = "A2", cpp_name = "A2")]]
template NumericVector A<NumericVector>(NumericVector x);

// [[Rcpp::export(name = "A3", cpp_name = "A3")]]
NumericVector Test::G();

Generated Rcpp Export File

// A<NumericVector>
template NumericVector A<NumericVector>(size_t n);
RcppExport SEXP _testRcpp_A1(SEXP nSEXP) {
BEGIN_RCPP
    Rcpp::RObject rcpp_result_gen;
    Rcpp::RNGScope rcpp_rngScope_gen;
    Rcpp::traits::input_parameter< size_t >::type n(nSEXP);
    rcpp_result_gen = Rcpp::wrap(A<NumericVector>(n));
    return rcpp_result_gen;
END_RCPP
}

// A<NumericVector>
template NumericVector A<NumericVector>(NumericVector x);
RcppExport SEXP _testRcpp_A2(SEXP xSEXP) {
BEGIN_RCPP
    Rcpp::RObject rcpp_result_gen;
    Rcpp::RNGScope rcpp_rngScope_gen;
    Rcpp::traits::input_parameter< NumericVector >::type x(xSEXP);
    rcpp_result_gen = Rcpp::wrap(A<NumericVector>(x));
    return rcpp_result_gen;
END_RCPP
}

// Test::G
NumericVector Test::G();
RcppExport SEXP _testRcpp_A3() {
BEGIN_RCPP
    Rcpp::RObject rcpp_result_gen;
    Rcpp::RNGScope rcpp_rngScope_gen;
    rcpp_result_gen = Rcpp::wrap(Test::G());
    return rcpp_result_gen;
END_RCPP
}

static const R_CallMethodDef CallEntries[] = {
    {"_testRcpp_A1", (DL_FUNC) &_testRcpp_A1, 1},
    {"_testRcpp_A2", (DL_FUNC) &_testRcpp_A2, 1},
    {"_testRcpp_A3", (DL_FUNC) &_testRcpp_A3, 0},
    {NULL, NULL, 0}
};

Testing & Future Considerations

I have run extensive tests to ensure this new feature works correctly. The ability to export template and namespaced functions integrates well with the existing Rcpp framework. However, given the complexity of C++ templates and namespaces, there might still be edge cases that I haven’t encountered.

Would this be a feature that the maintainers would be interested in? If so, I would be happy to prepare a pull request.

[eddelbuettel]

Intriguing, and potentially very interesting. Paging @jjallaire who is of course the main author of the 'Attributes' code.

[Enchufa2]

Thanks, @ManosPapadakis95, this is a very interesting proposal. It sounds like you have some working prototype, but I cannot find it in your list of repositories. Would you mind pointing us to it? It would be great to have a look to be able to evaluate and discuss the implementation complexity before going for the PR.

[ManosPapadakis95]

Yes, of course. Sorry, but I forgot to push the commits. They are now available. All the commits I have can be found here.

[eddelbuettel]

Sorry for keeping you waiting on this! As you may have seen we were quite busy establishing a new baseline of C++11 removing some older (now unused) code paths from that compilation standard was not (widely or at all) availble.

Good news is your commit rebase cleanly. That may be something you want to do. (Also it may be easier for you to catch up / play along if you changes are on a branch and you treat your master as a pure copy of our master. I sometimes forget that to when working on PRs for other repos).

With the rebased package, compileAttributes() still works on your included test package which also installs. So the question is how do we best decompose this to review / discuss? Are there in fact multiple proposals here? As in your points 1 to 4 above?

[ManosPapadakis95]

Yes, of course, Dr. Eddelbuettel. I was in a rush to make the commits and forgot that I pushed to the master branch. I have already updated the link in my previous comment to include the new branch as you suggested.

I believe, but I am not entirely sure, that the best way to structure this is:

• cpp_name can work with simple forward-declared functions (including namespaced functions) and is standalone within Rcpp’s current framework. Here are three examples:

// [[Rcpp::export(name = "A3", cpp_name = "A3")]]
NumericVector Test::G(); //The cpp_name field is necessary

//or

// [[Rcpp::export(name = "A3", cpp_name = "A3")]]
NumericVector G(); //No need for the cpp_name but it is accepted

//or

// [[Rcpp::export(name = "A3", cpp_name = "A3")]]
NumericVector G(){} //No need for the cpp_name but it is accepted

• Explicit template instantiations require declaration specifiers, cpp_name, and Rcpp::includes to function correctly. Here is an example:

// [[Rcpp::includes("helpers.h")]]

// [[Rcpp::export(name = "A1", cpp_name = "A1")]]
template NumericVector A<NumericVector>(size_t n); //The cpp_name field and the Rcpp::includes are necessary

• Declaration specifiers only serve as a mechanism to identify and store keywords (e.g., template, inline) within the Function class.

I would say that these are not multiple separate proposals, but rather interdependent components of a single enhancement.

This is just my opinion, though. I'm happy to adjust things based on your suggestions.

[eddelbuettel]

That's helpful. But what might be even better is more examples. Even for the simple declaration. I see some of these used in your testpkg (in the initial version, may have changed) but having more examples may help. We may need them as unit tests anyway.

In short, I see nothing fundamental that stands in the way of a PR like this but it would be good to have more motivation.

[ManosPapadakis95]

That makes a lot of sense. I’ll prepare more examples and update the test package accordingly.

I completely understand if the motivation may not seem pressing from your perspective. In my case, I was exploring the idea of creating a new package that acts as a C++ header-only library. I wanted an easier way to expose template functions to R and use the Rcpp mechanism without writing the intermediate wrappers by hand. Using these enhancements, I was able to reduce the boilerplate in my future package by over 60%, which was a big win in terms of maintainability and clarity.

I’ll commit the additional examples and test updates and follow up here once that’s done.

[eddelbuettel]

Oh I completely see where it could be useful -- fully agreed. We have some use cases where Rcpp Modules, and some where it could not get used, and something like this could definitely help. As would an example packages of course.

[ManosPapadakis95]

Apologies for the silence. It took me some time to create a package with examples (including a mix of all the attribute fields), and it was helpful because I discovered two bugs related to Rcpp::includes and complex template function names. You can find the commits here.

I have also created two packages:

1. testRcpp: This package exports various functions using intermediate wrapped functions or directly with Rcpp::export.
2. testRcppNew: This package exports various functions using the new enhancements (for template or namespaced functions) or directly with Rcpp::export.

I will continue testing the new features, but for now, I wanted to share my progress with these small packages.

[eddelbuettel]

Thanks. Might it make sense to put the example package(s) into a separate repo for easier access and study?

[ManosPapadakis95]

As requested, the packages are placed in separate repositories..

1. testRcpp
2. testRcppNew

[eddelbuettel]

I sent you a quick PR for testRcpp. Feel free to ignore or to adapt partially, but I have a pretty strong dislike of using namespace ... in package code. Explicit use is preferred.

[ManosPapadakis95]

Please proceed with any modifications you deem necessary; I will implement the same changes in testRcppNew.

[Enchufa2]

(Sorry for disappearing, but I'll probably have some time next week to take a look)

[eddelbuettel]

I just noticed that when I run compileAttributes() in your testRcppNew I get a change to RcppExports.cpp that seems odd.

diff --git i/src/RcppExports.cpp w/src/RcppExports.cpp
index bdc1385..13359a3 100644
--- i/src/RcppExports.cpp
+++ w/src/RcppExports.cpp
@@ -3,7 +3,7 @@
 
 #include <Rcpp.h>
 #include "helpers.h"
-#include "Str.h"
+#include "helpers.h"#include "Str.h"
 
 using namespace Rcpp;
 

I am now under R 4.5.0 but I doubt that has an effect.

[ManosPapadakis95]

That's one of the two bugs I mentioned earlier. I've already updated and rebased my branch.

[eddelbuettel]

My bad. I have a checkout here but I was stuck in your older master and not the new branch. All good once I get to that branch.

[eddelbuettel]

There is still a lot in testRcppNew and I am wondering how we can best break it down to bitesized changes.

And I may be too dense here. I see how helpers.h has template arguments, but by using only Rcpp::NumericVector to instantiate we do not really show why templates are useful.

[eddelbuettel]

@ManosPapadakis95 Are you aware that RcppExports.cpp will have include a file PACKAGENAME_types.h, if present?

Using that approach, I can build your testRcppNew package with the Rcpp::includes() directive to include helpers.h. I may make a quick PR just to show you.

[ManosPapadakis95]

Regarding your comment about PACKAGENAME_types.h, I must admit I was not aware of it, and when I initially did my research, I failed to find a solution. That’s why I implemented the Rcpp::includes. But now I can see that as long as the types file is named PACKAGENAME_types.h, no matter where it is placed under the include or src directory, it will be added to the RcppExports.cpp. Again, truly sorry for taking up your time on this.

So I removed the Rcpp::includes from the testRcppNew and performed these steps to adapt to the types mechanism:

1. Moved Str.h under the include directory.
2. Created a new file named testRcppNew_types.h under the include directory.

Now, compilation succeeds without Rcpp::includes under branch.

Personally, I would love not to expose Str.h under the include directory and keep it under the src directory. Maybe letting users define additional types files would be good enough? For example:

testRcppNew/
├── include/
│   ├── headers.h
│   └── testRcppNew_types.h
├── src/
│   ├── Str.cpp
│   ├── Str.h
│   └── Str_types.h
├── R/
│   └── ...

With this, I could keep the existing Str.h and Str.cpp intact, while having a dedicated Str_types.h under src/ for type definitions. But this is a matter for another potential enhancement, which I would gladly help with if you ever consider allowing it.

[ManosPapadakis95]

So, I believe we should first review the cpp_name, which simply changes the name of the exported wrapper function in RcppExports.cpp. The template instantiation and the namespaced functions require the cpp_name.

Therefore, it might be useful to break the package into separate branches for each case:

1. Usage of cpp_name.
2. Usage of template instantiations with cpp_name.
3. Usage of namespaced functions with cpp_name.

Regarding your comment about instantiating only Rcpp::NumericVector — it is indeed challenging to design truly generic algorithms, which is why I initially added the sample function instantiated for both NumericVector and IntegerVector. I wanted to show you my current progress and give you an idea of what I intended to do with the templates. Of course, I will add more examples as you suggested.

[eddelbuettel]

No worries, there are some nuggest hidden away in the Rcpp Attributes vignettes.

One thing I would change now too is headers.h. It's not a great name. If it defines a wrapper for the different C++ library RNGs (which is a great example idea!) maybe call the file random_wrapper.h or something that evokes from the name what is in it? (Also the random example may not need all those generators defined. It may make it harder to see the bigger picture.)

Splitting use of cpp_name into three distinct examples seems like a good idea.

[ManosPapadakis95]

You're completely right about the headers.h naming — I'll rename it to something more descriptive like random_wrapper.h. I'll also split the usage of cpp_name into three distinct, focused examples as suggested. I'll make the same changes to both packages so they can be compared more easily.

[ManosPapadakis95]

I have implemented all the suggested changes.

[eddelbuettel]

When I

• remove all references to kIncludesAttribute from src/attributes.cpp in your Rcpp branch 'feature-template-exports'
• move Str.h to inst/include in testRcppNew
• source Str.h in inst/include/testRcppNew_types.h via an #include

then all is well and we are a little simpler. Good enough?

[ManosPapadakis95]

Of course. I have already made these changes for testRcppNew and they are available in the branch cpp_name-namespaced-templated-functions.
I will also remove the references to Rcpp::includes. Apologies for that — I was quite busy and it slipped my mind.

[eddelbuettel]

Sounds good. Maybe also merge from that branch into the main branch of testRcppNew ?

[ManosPapadakis95]

I already done it.

[ManosPapadakis95]

Also, removed references of the new attribute.

[eddelbuettel]

The comments are getting shorter and shorter and I am not sure I follow at all times. Maybe you can add reference to your repository and or commits?

Also, for what it is worth, while looking into #1377, I found that your current branch would not run compileAttributes() against a very vanilla new test package. I think I was current to your branch. I reverted to 1.0.14.12 ie what we have in master here.