dataMaid icon indicating copy to clipboard operation
dataMaid copied to clipboard

Minor suggestion: add option to sanitize atypical unicode

Open owasow opened this issue 4 years ago • 1 comments

I was working with some survey data in which a number of open-ended text responses included atypical unicode characters that broke latex compilation (though could work with xelatex). These text strings tended to be nonsense input from users so it wasn't essential to include them in the codebook but I found it hard to find the right code to strip them out before running dataReporter. Ultimately, it turned out to be easy with the following code gsub('[^\x20-\x7E]', '', text) but it took me a while to locate this particular solution on StackOverflow (see link below). https://stackoverflow.com/questions/38828620/how-to-remove-strange-characters-using-gsub-in-r

I wonder if an option/argument to automatically sanitize character strings would make sense given that this will likely be an issue for a wide range of data sets.

owasow avatar Feb 22 '21 05:02 owasow

Thanks for reaching out!

We are a bit hesitant about implementing features that alters the data even though it's just for the sake of formatting.

Would you mind providing a bit more info about the encoding problem your were having? What did the text strings look like?

It sounds like you have found a good workaround. But as an alternative, you could also run makeDataReport with standAlone = FALSE - this gives you a .rmd document without the YAML header - and then make the YAML header yourself. This would allow you to choose which latex engine to use, include a .tex header and more.

annennenne avatar Feb 25 '21 10:02 annennenne