Kenneth Benoit

OK let's drill down and try to interpret the specific clues from the UBSAN code. First, the error seems to be occurring twice, during the vignette execution and during the...

Just got back the comments from CRAN, after resubmitting yesterday evening. Same problem. quanteda.Rcheck/build_vignettes.log:/data/gannet/ripley/R/test-clang/Rcpp/include/Rcpp/proxy/AttributeProxy.h:74:32: runtime error: downcast of address 0x7ffe88447cf0 with insufficient space for an object of type 'Rcpp::Vector' quanteda.Rcheck/build_vignettes.log:/data/gannet/ripley/R/test-clang/Rcpp/include/Rcpp/proxy/AttributeProxy.h:74:32:...

I've had nothing but frustration on my mac trying to set up this container. Not RcppArmadillo but another dependency.

OK, so it was something in RcppArmadillo that was causing the issue? And you think the UBSAN problems will be gone now?

Am I right in understanding that you are not able to replicate the UBSAN problems shown by CRAN when running your testing setup on master? If it's catching the same...

If I resubmit and we have the same refusal based on the same warnings, I feel we could get into trouble with CRAN. Does anyone out there have experience with...

The `tokens_annotate()` seems to be not radical, but supplementary. By not changing `tokens_lookup()`, it's actually more conservative. There are several dependent packages that are built on `tokens_lookup()` so if possible,...

So this needs to be changed into a new `tokens_annotate()` function, leaving `tokens_lookup()` unchanged? @koheiw do you want to modify this PR to add that function?

That would be the perfect solution!

Sounds good too. I was thinking mainly that we would want a way to append the annotation to the token itself, so adding `append_key = TRUE` to `tokens_annotate()`.