JSON UTF-8 encoding error

[tatref]

Hi,

From time to time, I hit the following error:

Error: (0) json_encode(): Invalid UTF-8 sequence in argument
URL: /nagvis/server/core/ajax_handler.php?mod=Map&act=getMapObjects&show=automap&sources=automap&header_menu=1&hover_menu=1&context_menu=1&zoom=100&header_template=default&root=%3C%3C%3Cmonitoring%3E%3E%3E&render_mode=undirected&backend_id=ndomy_1&width=1024&height=800&iconset=std_medium&filter_by_state=0&child_layers=-1&margin=50&rankdir=LR&overlap=scale&label_show=1&url_target=_self&_ajaxid=1591281732
File: /usr/local/dtb/addons/nagvis/share/server/core/classes/NagVisMap.php
Line: 117
#0 [internal function]: nagvisExceptionErrorHandler(2, 'json_encode(): ...', '/usr/local/dtb/...', 117, Array)
#1 /usr/local/dtb/addons/nagvis/share/server/core/classes/NagVisMap.php(117): json_encode(Array)
#2 /usr/local/dtb/addons/nagvis/share/server/core/classes/CoreModMap.php(325): NagVisMap->parseObjectsJson()
#3 /usr/local/dtb/addons/nagvis/share/server/core/classes/CoreModMap.php(105): CoreModMap->getMapObjects()
#4 /usr/local/dtb/addons/nagvis/share/server/core/functions/index.php(120): CoreModMap->handleAction()
#5 /usr/local/dtb/addons/nagvis/share/server/core/ajax_handler.php(59): require('/usr/local/dtb/...')
#6 {main}

• Centos 6
• PHP 5.3.3
• Nagvis 1.9.20
• happens in automap with backend ndomy
• character set in the database is latin1, this is the default with NDO (see https://github.com/NagiosEnterprises/ndoutils/blob/master/db/mysql.sql)

I think that some check is returning garbage data from time to time, but I don't known how to identify which one. I already spent some time making requests in the database without any success.

Any idea is very welcome!

[tatref]

The function producing the error is:

    return json_encode($arrRet);

Possible fixes:

1. PHP 7.2 introduces the following flags for json_encode: JSON_INVALID_UTF8_IGNORE, JSON_INVALID_UTF8_SUBSTITUTE

2. Use iconv prior to json_encode (I know nothing about PHP...):

    array_walk_recursive($arrRet, function(&$value) {
        $new_value = iconv('UTF-8', 'UTF-8//IGNORE', $value);
        if ("$new_value" !== "$value") {
            error_log("Corrected UTF-8 string: $value => $new_value");
            $value = $new_value;
        }
   
    });
    return json_encode($arrRet);
   

For the moment, I'm testing the iconv code provided

3. make sure the data is properly encoding when making the requests from the DB

4. if no proper fix is possible, at least show the item producing the error