utils

utils copied to clipboard

User data can include invalid characters, e.g. "\xF6" which is an ö in ISO-8859-1, but invalid for UTF-8.

Invalid characters can be replaced with a "?", by using mb_convert_encoding($string, 'UTF-8', 'UTF-8'); but this needs to be done on every individual string (tricky with large arrays), and mbstring is a non-default extension (so not always available).

Would it be possible to use the JSON_INVALID_UTF8_SUBSTITUTE flag to json_encode(), so invalid characters can be replaced with \0xfffd (i.e. �), and a usable value is returned (rather than getting an exception).

I recently ran into this issue where JSON_INVALID_UTF8_SUBSTITUTE is silently ignored. That's 30 minutes of my life im never getting back! So, i just updated all the calls to: json_encode($data, JSON_INVALID_UTF8_SUBSTITUTE | JSON_THROW_ON_ERROR) which is functionally equivalent since PHP 7.3