depthai-core icon indicating copy to clipboard operation
depthai-core copied to clipboard

env vars with non-ascii paths fail on Windows

Open diablodale opened this issue 2 years ago • 0 comments

depthai paths are also stored in environment variables. Therefore apis that read env vars need to use Windows unicode apis wchar to get the full UTF-16 which can then be converted into dai::path or UTF-8.

Found while fixing #352 PR #384 enables Unicode paths in API signatures, if/ofstream, and spdlog. A second PR should enable Unicode from environment variables...which naturally needs the first PR's fixes.

Setup

  • all on Windows

Discussion

The os manages the storage of the different char sets (ascii, utf-8, utf-16, etc.). When an app needs the env var value, it calls an os API. On Windows...it needs to call the UTF-16 wide character APIs for environment variables to get Unicode text.

A Windows customer in a country/language who's glyphs do not fit in ASCII, e.g. Japan, China, UAE, Germany, etc. stores a filename or path, using their native language, into env vars that depthai reads. depthai calls the API to get the env var.

// runtime fail: underlying utility calls ascii OS api, Windows provides best-fit value, the path is corrupt/unusable
auto fwBinaryPath = utility::getEnv("DEPTHAI_DEVICE_BINARY");

// compile fail: underlying utility calls the ascii OS api which returns a std::string, it should return a Unicode std::wstring
std::wstring fwBinaryPath = utility::getEnv("DEPTHAI_DEVICE_BINARY");

I believe the utility::getEnv() call chain should always return wstring on Windows. Making the caller responsible for managing any unicode/ascii/number convert. After the getEnv() is fixed, then this is possible...

// works: the getEnv() code has been updated to return wstring
auto fwBinaryPath = utility::getEnv("DEPTHAI_DEVICE_BINARY");

// works: the getEnv() code has been updated to return wstring and dai::Path can be directly constructed with it
dai::Path fwBinaryPath = utility::getEnv("DEPTHAI_DEVICE_BINARY");

There ?may? be some cascading work to ensure things that parse the env var wstring are not assuming char. Templates and/or the typical MSVC overloads that can accept wchar in functions can likely address these.

Quick look at spdlog's code, they are partially aware of this issue. I see macro defines like SPDLOG_WCHAR_TO_UTF8_SUPPORT, SPDLOG_WCHAR_FILENAMES but I don't see they dealt with Windows for the env vars.

Originally posted by @diablodale in https://github.com/luxonis/depthai-core/issues/352#issuecomment-1049062049

diablodale avatar Apr 11 '22 19:04 diablodale