libobs: Fix memory leak when load_module_exports failed

[norihiro]

Description

Release mod.module when an error is detected after the module was successfully opened.

Also add a warning message that the module is failed to load exports.

Motivation and Context

When reading plugin files and load_module_exports failed, the object should be released before returning an error code but was not released. This is a memory leak.

How Has This Been Tested?

Added a random shared object into the plugin directory and checked the module is released.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• [x] My code has been run through clang-format.
• [x] I have read the contributing document.
• [x] My code is not on the master branch.
• [x] The code has been tested.
• [x] All commit messages are properly formatted and commits squashed where appropriate.
• [x] I have included updates to all appropriate documentation.

[notr1ch]

I believe this is intentional behavior - see the comment in free_module: https://github.com/obsproject/obs-studio/blob/de01cfc063d78723bccde96b4c0bc78ada702965/libobs/obs-module.c#L573-L575

What these "issues" are needs investigating as ideally we should be closing modules after they are no longer needed. This was mentioned also in https://github.com/obsproject/obs-studio/pull/5883

[norihiro]

According to above comment, I close this issue. The amount of the allocated memory would be not so large and not increasing over the time. And, ideally the failed plugin should be removed by users instead of cleanup by the code.