Appleseed + Gaffer + Windows crashes

[ericmehl]

Hi @est77, I've been working on a fork of Gaffer for Windows and I'm having trouble getting Appleseed working with it. I'm on the 2.0.5-beta branch that @johnhaddon recently updated the master project to.

I just finished a big reorganization of my branches to hopefully make it easier for others to get the right code to do their own builds with. Would you be able to lend a hand to help get Appleseed and Gaffer Windows working?

The branches for my Windows builds are:

• GafferDependencies: https://github.com/hypothetical-inc/gafferDependencies/tree/msvc2017
• Cortex (if you need that separately): https://github.com/hypothetical-inc/cortex/tree/msvc2017
• Gaffer: https://github.com/hypothetical-inc/gaffer/tree/msvc2017

The GafferDependencies and Gaffer forks both have releases also to help jumpstart things (I'm including Cortex in the GafferDependencies release). I'm using MSVC 2017.

In my debugging I've traced it back to the aligned_free function being called. The calling functions of aligned_free have varied, including texture allocation / deallocation, renderer destructors, AOV allocators and others which is where I start to lose the thread of the problem.

Any help you could offer is much appreciated!

[dictoon]

Hi @ericmehl, I'd be happy to help but my free time is very limited and if possible I would rather not embark on building Gaffer myself, even though it looks like you made things a lot easier.

On Windows, appleseed overrides global new and delete to ensure that allocated memory blocks are always 16-byte aligned. Could this somehow interfer with Gaffer?

Could you also share a few more details:

• Are you building appleseed yourself? If so, what is your CMake command line?
• Does your standalone build of appleseed work correctly? Do all unit tests pass?
• Could you provide a couple callstacks of the crashes?

Finally, in appleseed/main/allocator.cpp, there are a couple #define that allow to track memory allocations and deallocations, maybe that would allow finding places where memory is somehow allocated with the default, non-overridden new operator but deallocated with appleseed's delete operator, or vice versa.

[ericmehl]

Thanks for the information about the new and delete and the debugging tips. It looks like the delete is where it's crashing.

I tried using the appleseed.cli application and I'm getting crashes there too, so I don't think it's related to Gaffer. I haven't been able to get the tests running, perhaps it's crashing for those too. I did get the benchmarks running - does that suggest anything interesting to you?

Maybe it's related to Boost? It seems like it often crashes when freeing a string created as part of boost::filesystem functions. We're using 1.61 for Gaffer so I'm building against that.

I am building Appleseed myself, here's the cmake command I'm using to build Appleseed:

cmake ^
-G "Visual Studio 15 2017 Win64" ^
-D CMAKE_BUILD_TYPE=RELEASE ^
-D CMAKE_VERBOSE_MAKEFILE:BOOL=ON ^
-D PYTHON_INCLUDE_DIR=M:\gaffer_build\include ^
-D WITH_OSL=ON ^
-D WITH_CLI=ON ^
-D WITH_STUDIO=OFF ^
-D WITH_TOOLS=OFF ^
-D WITH_PYTHON=ON ^
-D WITH_TESTS=ON ^
-D USE_STATIC_BOOST=OFF ^
-D USE_STATIC_OIIO=OFF ^
-D USE_STATIC_EXR=OFF ^
-D USE_STATIC_OSL=OFF ^
-D USE_EXTERNAL_ZLIB=ON ^
-D USE_EXTERNAL_EXR=ON ^
-D USE_EXTERNAL_PNG=ON ^
-D USE_EXTERNAL_XERCES=ON ^
-D USE_EXTERNAL_OSL=ON ^
-D USE_EXTERNAL_OIIO=ON ^
-D USE_EXTERNAL_ALEMBIC=ON ^
-D WARNINGS_AS_ERRORS=OFF ^
-D USE_SSE=ON ^
-D CMAKE_PREFIX_PATH=M:\gaffer_build ^
-D CMAKE_INSTALL_PREFIX=M:\gaffer_build\appleseed ^
..
cmake --build . --config "Visual Studio 15 2017 Win64" --target install

As part of the build I am patching cmake/config/win-vs.txt to account for some LLVM files our LLVM build is not creating:

diff --git a/./cmake/config/win-vs.txt b/./cmake/config/win-vs.txt
--- ./cmake/config/win-vs.txt
+++ ./cmake/config/win-vs.txt
@@ -297,10 +297,10 @@
 
 macro (link_against_osl target)
     set (LLVM_LIBS
-        LLVMLTO LLVMLinker LLVMipo LLVMVectorize LLVMBitWriter LLVMTableGen LLVMDebugInfo LLVMOption LLVMX86Disassembler LLVMX86AsmParser
+        LLVMLTO LLVMLinker LLVMipo LLVMVectorize LLVMBitWriter LLVMTableGen LLVMOption LLVMX86Disassembler LLVMX86AsmParser
         LLVMX86CodeGen LLVMSelectionDAG LLVMAsmPrinter LLVMX86Desc LLVMX86Info LLVMX86AsmPrinter LLVMX86Utils LLVMIRReader LLVMBitReader
-        LLVMAsmParser LLVMMCDisassembler LLVMMCParser LLVMInstrumentation LLVMInterpreter LLVMMCJIT LLVMJIT LLVMCodeGen LLVMObjCARCOpts
-        LLVMScalarOpts LLVMInstCombine LLVMTransformUtils LLVMipa LLVMAnalysis LLVMRuntimeDyld LLVMExecutionEngine LLVMTarget LLVMMC
+        LLVMAsmParser LLVMMCDisassembler LLVMMCParser LLVMInstrumentation LLVMInterpreter LLVMMCJIT LLVMCodeGen LLVMObjCARCOpts
+        LLVMScalarOpts LLVMInstCombine LLVMTransformUtils LLVMAnalysis LLVMRuntimeDyld LLVMExecutionEngine LLVMTarget LLVMMC
         LLVMObject LLVMCore LLVMSupport
     )

Here's an example stack trace when running appleseed.cli with one of the test scenes:

ntdll.dll!00007ffd56ca2503()
ucrtbase.dll!00007ffd5355df7b()
[Inline Frame] appleseed.dll!?A0x9f442a7e::delete_impl() Line 505
appleseed.dll!operator delete[](ptr) Line 539
[Inline Frame] appleseed.dll!std::_Deallocate(_Ptr, _Bytes) Line 207
[Inline Frame] appleseed.dll!std::allocator<char>::deallocate(_Count, ) Line 992
[Inline Frame] appleseed.dll!std::basic_string<char,std::char_traits<char>,std::allocator<char>>::_Tidy_deallocate() Line 3992
[Inline Frame] appleseed.dll!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::{dtor}() Line 2460	
appleseed.dll!renderer::ProjectFileReader::load_project_file(project_filepath=0x0000013f1e333460, schema_filepath=0x0000013f1acbdbd0, options=0, event_counters={...}, search_paths=0x0000000000000000) Line 3401
appleseed.dll!renderer::ProjectFileReader::read(project_filepath=0x0000013f1e333460, schema_filepath=0x0000013f1acbdbd0, options=0) Line 3253
appleseed.cli.exe!`anonymous namespace'::load_project(project_filepath) Line 417
appleseed.cli.exe!`anonymous namespace'::render(project_filename) Line 467
appleseed.cli.exe!main(argc=2, argv) Line 718
[Inline Frame] appleseed.cli.exe!invoke_main() Line 78
appleseed.cli.exe!__scrt_common_main_seh() Line 288
kernel32.dll!00007ffd561781f4()
ntdll.dll!00007ffd56cfa251()

I also enabled the memory tracking you described and scanned it for memory addresses being deallocated that didn't have a corresponding allocation. It didn't turn up anything, does that sound like the kind of analysis you had in mind?

[dictoon]

Hi Eric,

I'm on winter vacation and will have lots of free time over the next couple weeks. If you're still working on this (or planning to) I would love to help.

Regarding some of your points:

I tried using the appleseed.cli application and I'm getting crashes there too

Well that's interesting. We should definitely start investigating what happens with appleseed.cli then (rather than focus on Gaffer).

I haven't been able to get the tests running, perhaps it's crashing for those too

So I assume you tried appleseed.cli.exe --run-unit-tests (you can add --verbose-unit-tests to get more feedback)? Does it crash right away? Or on a specific test?

[dictoon]

I did get the benchmarks running

Do you mean that appleseed.cli.exe --run-unit-benchmarks works fine?

I am building Appleseed myself, here's the cmake command I'm using to build Appleseed

One thing of note is that you're using shared versions of Boost, OIIO, OpenEXR and OSL, which isn't the default setup.

Are you compiling the rest of the dependencies yourself or are you relying on our prebuilt binaries for those (which are found here)?

[ericmehl]

hi @dictoon, that would be great if you could spare some time to look into it! I haven't had a chance to pursue it since I last posted on the thread.

I'm compiling the other dependencies myself. The Gaffer dependencies are already pretty extensive and include all the libraries Appleseed uses (albeit some are different versions) so ideally I'd like to reuse those.

I tried building with the recommended static instructions from the Appleseed site but I was having trouble getting it to work. I think it was because some things like boost were getting mixed up with other versions on my machine that I'm using for Gaffer development. I didn't go very far down that path since I'd prefer to use the shared dependencies.

But maybe the static route is a compromise we need to strike. If you have time, can you try a normal Appleseed build installed into the latest Gaffer Windows release? That is here: https://github.com/hypothetical-inc/gaffer/tree/0.55.1.0 You will need to enable Appleseed by commenting out the configuration lines in the bin/gaffer.bat file.

Thanks for any help you can offer! I would love to get Appleseed going on Windows Gaffer and I think a number of other people do as well.