AcademySoftwareFoundation
[BUG] fstream_mingw.h crashed at runtime
Describe the bug
I get runtime crash at fstream_mingw.h with mingw gcc.
However, it only happened when dynamically linking OIIO to OSL and static linking libstdc++ and libgcc for both, with mingw.
OpenImageIO version and dependencies
OpenImageIO 2.5.9.0 OpenShadingLanguage master
To Reproduce
Build OIIO dll with mingw with "-static-libstdc++ -static-libgcc" Build OSL dll with mingw with "-static-libstdc++ -static-libgcc" Link to OSL dll and call compile().
I had this issue with different mingw compilers including the one shipped with Debian 12. However, I had no problems using fstream_mingw.h in minimal tests.
Evidence
The problem stems from:
std::ios_base::_M_init()
when it calls:
std::locale::operator=(std::locale const&) ()
0 0x00007ffc8004f123 in ntdll!RtlIsZeroMemory () from C:\Windows\SYSTEM32\ntdll.dll #1 0x00007ffc80057ee2 in ntdll!RtlpNtSetValueKey () from C:\Windows\SYSTEM32\ntdll.dll #2 0x00007ffc800581ca in ntdll!RtlpNtSetValueKey () from C:\Windows\SYSTEM32\ntdll.dll #3 0x00007ffc8005de51 in ntdll!RtlpNtSetValueKey () from C:\Windows\SYSTEM32\ntdll.dll #4 0x00007ffc7ff75bf0 in ntdll!RtlGetCurrentServiceSessionId () from C:\Windows\SYSTEM32\ntdll.dll #5 0x00007ffc7ff747b1 in ntdll!RtlFreeHeap () from C:\Windows\SYSTEM32\ntdll.dll #6 0x00007ffc7fc49c9c in msvcrt!free () from C:\Windows\System32\msvcrt.dll #7 0x00007ffc4308bbe0 in std::locale::_Impl::~_Impl() () from libOpenImageIO_Util_d.dll #8 0x00007ffc4308a7b3 in std::locale::_Impl::_M_remove_reference() () from libOpenImageIO_Util_d.dll #9 0x00007ffc4308bfbe in std::locale::operator=(std::locale const&) () from libOpenImageIO_Util_d.dll #10 0x00007ffc430b60ad in std::ios_base::_M_init() () from libOpenImageIO_Util_d.dll #11 0x00007ffc430b7cb1 in std::basic_ios<char, std::char_traits<char> >::init(std::basic_streambuf<char, std::char_traits<char> >*) () from libOpenImageIO_Util_d.dll #12 0x00007ffc42fc2335 in OpenImageIO_v2_5::basic_ofstream<char, std::char_traits<char> >::open_internal ( this=0xc3e8a0, path=L"node_output_surface.oso", mode=std::_S_out) at OpenImageIO/src/include/OpenImageIO/fstream_mingw.h:267 #13 0x00007ffc42fc2408 in OpenImageIO_v2_5::basic_ofstream<char, std::char_traits<char> >::open (this=0xc3e8a0, path=L"node_output_surface.oso", __mode=std::_S_out) at OpenImageIO/src/include/OpenImageIO/fstream_mingw.h:294 #14 0x00007ffc42f7b66f in OpenImageIO_v2_5::Filesystem::open (stream=..., path=..., mode=std::_S_out) at OpenImageIO\src\libutil\filesystem.cpp:608 #15 0x00007ffc2e137ae7 in OSL_v1_14_1::pvt::OSLCompilerImpl::compile(OpenImageIO_v2_5::basic_string_view<char, std::char_traits<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, OpenImageIO_v2_5::basic_string_view<char, std::char_traits<char> >) () from liboslexec.dll
IF YOU ALREADY HAVE A CODE FIX: There is no need to file a separate issue
I did make a patch that optionally uses boost_nowide for mingw which fixed the issue. However, it seems boost is no longer a dependency for OIIO so this workaround doesn't apply to master without optionally re-introducing boost dependency for OIIO.
Not sure why boost_nowide solved the issue, it probably uses the same GCC specific headers to support UTF-16 filenames in Windows.
diff --git a/src/cmake/compiler.cmake b/src/cmake/compiler.cmake
index ce79003b2..5ce71d14c 100644
--- a/src/cmake/compiler.cmake
+++ b/src/cmake/compiler.cmake
@@ -269,6 +269,18 @@ if (USE_LIBCPLUSPLUS AND CMAKE_COMPILER_IS_CLANG)
endif ()
+###########################################################################
+# Option to use boost_nowide for mingw builds instead of OIIO's own
+# fstream_mingw.h. Using it solved a runtime crash when dynamic linking
+# OIIO and static linking libgcc and libstdc++. Both OIIO and any client
+# code using either OIIO::ofstream or OIIO::ifstream need to link to
+# boost_nowide with this option set.
+option (USE_BOOST_NOWIDE "Use boost_nowide instead of fstream_mingw.h to support UTF-8 on Windows for MinGW builds only" OFF)
+if (USE_BOOST_NOWIDE)
+ message (STATUS "Using boost_nowide for mingw builds")
+ add_definitions (-DOIIO_BOOST_NOWIDE)
+endif ()
+
+
###########################################################################
# For gcc >= 5, allow an option to force which version of the C++ ABI to
# use (mostly this affects the implementation of std::string).
diff --git a/src/cmake/externalpackages.cmake b/src/cmake/externalpackages.cmake
index 3f73cd266..8615c7382 100644
--- a/src/cmake/externalpackages.cmake
+++ b/src/cmake/externalpackages.cmake
@@ -53,6 +53,9 @@ if (MSVC)
endif ()
set (Boost_COMPONENTS thread)
+if (OIIO_BOOST_NOWIDE)
+ list (APPEND Boost_COMPONENTS nowide)
+endif ()
if (NOT USE_STD_FILESYSTEM)
list (APPEND Boost_COMPONENTS filesystem)
endif ()
diff --git a/src/include/OpenImageIO/filesystem.h b/src/include/OpenImageIO/filesystem.h
index dbaec7c6a..a1b8be0c8 100644
--- a/src/include/OpenImageIO/filesystem.h
+++ b/src/include/OpenImageIO/filesystem.h
@@ -34,7 +34,11 @@
#if defined(_WIN32) && defined(__GLIBCXX__)
# define OIIO_FILESYSTEM_USE_STDIO_FILEBUF 1
-# include <OpenImageIO/fstream_mingw.h>
+# ifndef OIIO_BOOST_NOWIDE
+# include <OpenImageIO/fstream_mingw.h>
+# else
+# include <boost/nowide/fstream.hpp>
+# endif
#endif
@@ -51,8 +55,14 @@ OIIO_NAMESPACE_BEGIN
// of ifstream::open or ofstream::open. To properly support UTF-8 encoding on MingW we must
// use the __gnu_cxx::stdio_filebuf GNU extension that can be used with _wfsopen and returned
// into a istream which share the same API as ifsteam. The same reasoning holds for ofstream.
-typedef basic_ifstream<char> ifstream;
-typedef basic_ofstream<char> ofstream;
+# ifndef OIIO_BOOST_NOWIDE
+ typedef basic_ifstream<char> ifstream;
+ typedef basic_ofstream<char> ofstream;
+# else
+// Alternatively, use Boost nowide instead of directly using __gnu_cxx::stdio_filebuf
+ typedef boost::nowide::ifstream ifstream;
+ typedef boost::nowide::ofstream ofstream;
+# endif
#else
typedef std::ifstream ifstream;
typedef std::ofstream ofstream;
diff --git a/src/include/OpenImageIO/oiioversion.h.in b/src/include/OpenImageIO/oiioversion.h.in
index 9e2f84d69..58f4f3513 100644
--- a/src/include/OpenImageIO/oiioversion.h.in
+++ b/src/include/OpenImageIO/oiioversion.h.in
@@ -198,5 +198,7 @@ namespace @PROJ_NAME@ = @PROJ_NAMESPACE_V@;
// Was the project built with TBB support?
#cmakedefine01 OIIO_TBB
+// Was the project built with Boost_nowide support?
+#cmakedefine01 OIIO_BOOST_NOWIDE
#endif /* defined(OPENIMAGEIO_VERSION_H) */
diff --git a/src/libutil/CMakeLists.txt b/src/libutil/CMakeLists.txt
index f873b3eed..288fef84f 100644
--- a/src/libutil/CMakeLists.txt
+++ b/src/libutil/CMakeLists.txt
@@ -20,6 +20,7 @@ target_link_libraries (OpenImageIO_Util
${GCC_ATOMIC_LIBRARIES}
${OPENIMAGEIO_IMATH_DEPENDENCY_VISIBILITY}
${OPENIMAGEIO_IMATH_TARGETS}
+ $<TARGET_NAME_IF_EXISTS:Boost::nowide>
PRIVATE
$<TARGET_NAME_IF_EXISTS:Boost::filesystem>
$<TARGET_NAME_IF_EXISTS:Boost::thread>
diff --git a/src/libutil/filesystem.cpp b/src/libutil/filesystem.cpp
index 0485f6aa9..faba8dd11 100644
--- a/src/libutil/filesystem.cpp
+++ b/src/libutil/filesystem.cpp
@@ -45,7 +45,11 @@ namespace filesystem = boost::filesystem;
using boost::system::error_code;
#endif
-
+#ifdef OIIO_FILESYSTEM_USE_STDIO_FILEBUF
+#ifdef OIIO_BOOST_NOWIDE
+#include <boost/nowide/fstream.hpp>
+#endif
+#endif
OIIO_NAMESPACE_BEGIN
@@ -544,10 +548,15 @@ FILE*
Filesystem::fopen(string_view path, string_view mode)
{
#ifdef _WIN32
+# if !defined(OIIO_FILESYSTEM_USE_STDIO_FILEBUF) || !defined(OIIO_BOOST_NOWIDE)
// on Windows fopen does not accept UTF-8 paths, so we convert to wide char
std::wstring wpath = Strutil::utf8_to_utf16wstring(path);
std::wstring wmode = Strutil::utf8_to_utf16wstring(mode);
return ::_wfopen(wpath.c_str(), wmode.c_str());
+# else
+ // Use boost_nowide with mingw to pass in UTF-8
+ return boost::nowide::fopen(std::string(path).c_str(), std::string(mode).c_str());
+# endif
#else
// on Unix platforms passing in UTF-8 works
return ::fopen(std::string(path).c_str(), std::string(mode).c_str());
@@ -584,7 +593,7 @@ void
Filesystem::open(OIIO::ifstream& stream, string_view path,
std::ios_base::openmode mode)
{
-#ifdef _WIN32
+#if defined(_WIN32) && (!defined(OIIO_FILESYSTEM_USE_STDIO_FILEBUF) || !defined(OIIO_BOOST_NOWIDE))
// Windows std::ifstream accepts non-standard wchar_t*
// On MingW, we use our own OIIO::ifstream
std::wstring wpath = Strutil::utf8_to_utf16wstring(path);
@@ -601,7 +610,7 @@ void
Filesystem::open(OIIO::ofstream& stream, string_view path,
std::ios_base::openmode mode)
{
-#ifdef _WIN32
+#if defined(_WIN32) && (!defined(OIIO_FILESYSTEM_USE_STDIO_FILEBUF) || !defined(OIIO_BOOST_NOWIDE))
// Windows std::ofstream accepts non-standard wchar_t*
// On MingW, we use our own OIIO::ofstream
std::wstring wpath = Strutil::utf8_to_utf16wstring(path);
@@ -616,7 +625,7 @@ Filesystem::open(OIIO::ofstream& stream, string_view path,
int
Filesystem::open(string_view path, int flags)
{
-#ifdef _WIN32
+#if defined(_WIN32) && (!defined(OIIO_FILESYSTEM_USE_STDIO_FILEBUF) || !defined(OIIO_BOOST_NOWIDE))
// on Windows _open does not accept UTF-8 paths, so we convert to wide
// char and use _wopen.
std::wstring wpath = Strutil::utf8_to_utf16wstring(path);
