std::map and std::unordered_map serialization broken for keys of type std::u16string

[mehecip]

Description

I am using a std::map<std::u16string, std::string> container. I would expect that the serialized form for each of the items in the container is an json object {"key" : "value"}, instead it is a vector of vectors [["key", "value"]].

Manually converting the std::u16string to std::string in a to_json function does not help. Not converting it, makes it worse (the key would be an array of bytes - I suppose this is the root problem - 16bit chars are converted to 2 x 8 bit chars?).

Reproduction steps

#include <iostream>
#include <nlohmann/json.hpp>
#include <ostream>
#include <map>
#include <sstream>


#include <codecvt>
#include <locale>


std::string u16string_to_string(const std::u16string& input) {
    // Create a converter for UTF-16 to UTF-8
    std::wstring_convert<std::codecvt_utf8_utf16<char16_t>, char16_t> converter;
    return converter.to_bytes(input);
}
namespace std {
void to_json(nlohmann::json& j, const std::u16string& wide_str) {
    std::cout << "to_json  std::u16string called \n" ;
    j = u16string_to_string(wide_str);
}
}
int main() {
    std::map<std::u16string, std::string> ustrmap = {{std::u16string(u"key"), "value"}};
    nlohmann::json j = ustrmap;
    std::cout << "u16string serialization " << j << "\n";

    std::map<std::string, std::string> strmap = {{"key", "value"}};
    j = strmap;
    std::cout << "string serialization " << j;
    return 1;
}

Expected vs. actual results

actual result: [["key","value"]]
expected result: {"key":"value"}

Minimal code example

Error messages

Compiler and operating system

gcc12.2, gcc trunk

Library version

3.9.1, 3.11.1

Validation

• [ ] The bug also occurs if the latest version from the develop branch is used.
• [ ] I can successfully compile and run the unit tests.

[nlohmann]

Using the to_json does not work, because the keys only need to be converted to object keys, but not basic_json. We use this code to determine if a type can be used as object key:

// type trait to check if KeyType can be used as an object key (without a BasicJsonType)
// see is_usable_as_basic_json_key_type below
template<typename Comparator, typename ObjectKeyType, typename KeyTypeCVRef, bool RequireTransparentComparator = true,
         bool ExcludeObjectKeyType = RequireTransparentComparator, typename KeyType = uncvref_t<KeyTypeCVRef>>
using is_usable_as_key_type = typename std::conditional <
                              is_comparable<Comparator, ObjectKeyType, KeyTypeCVRef>::value
                              && !(ExcludeObjectKeyType && std::is_same<KeyType,
                                   ObjectKeyType>::value)
                              && (!RequireTransparentComparator
                                  || is_detected <detect_is_transparent, Comparator>::value)
                              && !is_json_pointer<KeyType>::value,
                              std::true_type,
                              std::false_type >::type;

// type trait to check if KeyType can be used as an object key
// true if:
//   - KeyType is comparable with BasicJsonType::object_t::key_type
//   - if ExcludeObjectKeyType is true, KeyType is not BasicJsonType::object_t::key_type
//   - the comparator is transparent or RequireTransparentComparator is false
//   - KeyType is not a JSON iterator or json_pointer
template<typename BasicJsonType, typename KeyTypeCVRef, bool RequireTransparentComparator = true,
         bool ExcludeObjectKeyType = RequireTransparentComparator, typename KeyType = uncvref_t<KeyTypeCVRef>>
using is_usable_as_basic_json_key_type = typename std::conditional <
    is_usable_as_key_type<typename BasicJsonType::object_comparator_t,
    typename BasicJsonType::object_t::key_type, KeyTypeCVRef,
    RequireTransparentComparator, ExcludeObjectKeyType>::value
    && !is_json_iterator_of<BasicJsonType, KeyType>::value,
    std::true_type,
    std::false_type >::type;

As there is no built-in way to convert a std::u16string to a std::string, we fall back to converting the map to a list of pairs.

I am not sure how to improve this situation.

[mehecip]

Hi,

thanks for the answer. So my only, naive, option is to implement this:

namespace std {

void to_json(nlohmann::json& j, const std::map<std::u16string, std::string>& m) {   
    for (const auto& [key, value] : m) {
        j[u16string_to_string(key)] = value; 
    }   
}
}

which results in

u16string serialization {"key":"value"}
string serialization {"key":"value"}

[nlohmann]

Yes, this looks reasonable.

[gregmarr]

Since you aren't allowed to add things to the std namespace, you need to use the adl_serializer method instead: https://json.nlohmann.me/features/arbitrary_types/#how-do-i-convert-third-party-types

[mehecip]

Since you aren't allowed to add things to the std namespace, you need to use the adl_serializer method instead: https://json.nlohmann.me/features/arbitrary_types/#how-do-i-convert-third-party-types

Makes sense, thanks!