Critical: std::shared_ptr wrong type dispatch on MSVC Release

[deadlocklogic]

1. Repro

#define SOL_ALL_SAFETIES_ON 1
#include <sol/sol.hpp>

struct ClassA
{
    virtual ~ClassA() = default;

    static std::shared_ptr<ClassA> makeShared()
    {
        return std::make_shared<ClassA>();
    }
};

class ClassB
{
};

int main()
{
    sol::state lua;

    lua.open_libraries(sol::lib::base);

    lua["call"] = [](sol::object ptr)
    {
        if ((ptr.is<std::shared_ptr<ClassB>>()))
        {
            return "ClassB"; // Actual behavior: should not get in here in our test!!!
        }
        if ((ptr.is<std::shared_ptr<ClassA>>()))
        {
            return "ClassA"; // Expected behavior
        }
        return "nullptr";
    };
    lua.new_usertype<ClassA>("ClassA", "makeShared", &ClassA::makeShared);
    lua.new_usertype<ClassB>("ClassB");

    lua.script(R"(
do
  local instance = ClassA.makeShared()
  assert(call(instance) == "ClassA")
end
)",
               [](lua_State*, sol::protected_function_result pfr)
               {
                   sol::error err = pfr;
                   std::cout << err.what() << std::endl;
                   return pfr;
               });

    return 0;
}

2. Tests fails unexpectedly (Expected behavior: ClassA, Actual behavior: ClassB). Notice 1: this behavior is on MSVC Release (Debug/RelWithDebInfo work fine) Notice 2: std::unique_ptr behaves correctly.

3. Compiler/IDE: Visual Studio Language: C++ sol2 version: latest @ThePhD 👀.

Thanks.

[totalgee]

I can repro that. In Release (where your assertion fails), note that it passes if you change the order of your checks (so it also passes the case for shared_ptr<ClassA>. Hmm, the reason is that in fact, ptr.is returns true for any shared_ptr type, such as shared_ptr<int>!

    // This seems to return true in Release, for sol::object of any shared_ptr<> type
    bool result = ptr.is<std::shared_ptr<int>>();

[deadlocklogic]

@totalgee Exactly, weirdly enough. It is sad that these bugs exist in a nice library.