Executing shader using fmod() sometimes crashes on windows

[MasterZap]

Problem

In OSL on 3ds Max (Windows), compiling attached shader and running it crashes. The crash is in LLVM.

Expected behavior: It should compile and work fine. If you change all isntances of fmod() to mod(), it works.

Actual behavior: There is a crash in LLVM when executing the shader

Steps to Reproduce

1. In 3ds Max OSL shader, unzip and load attached shader herringboneTilling.zip
2. Watch OSL crash

Versions

• OSL branch/version: OSL-1.11.16.0
• OS: Windows 10
• C++ compiler: MSVC 2019
• LLVM version: llvm-13.0.0
• OIIO version: OpenImageIO-2.3.13.0

[MasterZap]

The crash in OSL is here, which is strange. The cause is that module()->getFunction(name) seems to return nullptr in this case, when name is “osl_fmod_dvdvf”?

[MasterZap]

Here is a video showing me debugging the problem https://www.dropbox.com/s/xqnxaeqdkdckslo/osl-crash-with-fmod.mp4?dl=0

[fpsunflower]

Would you be able to try against the latest version of the library? It looks like the way we codegen fmod has changed a bit.

[MasterZap]

@lgritz So based on last nights TSC discussion, it seems to be that we need an implementation for osl_fmod_dvdvf ?

I think the reason I see that and you don't, is that all the OSL in max goes through a wrapper shader and that wrapper shader calls calculatenormal() in some cases, which, I guess, would trigger a derivative evaluation.... (wild guess?)

[fpsunflower]

In theory there are only two versions of fmod: float fmod(float, float) and vector fmod(vector,vector)

So it should be impossible to get vector fmod(vector,float). Something odd must be happening at the oslc level for that to happen (or perhaps later on, but I seem to remember that oslc is the one doing overload resolution. Can you find where in the shader the fmod() call is? That might help narrow it down. For example I wonder if something like fmod(vector, 3) would properly cast the int all the way up to a vector or something like that.

[lgritz]

That's not quite right, Chris.

I think I've found the missing piece. Stay tuned.

[fpsunflower]

Ah I didn't realize that fmod was different from the others. Good catch!