OBS does not de-initialize encoders on RTMP stream start fail

[tt2468]

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

30.1.2

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/6iizipF9Li5buy3e

OBS Studio Crash Log URL

No response

Expected Behavior

If the stream output fails to start, allocated encoders should be de-allocated if they are not being used.

Current Behavior

The encoders sit initialized but not started, as obs_output_begin_data_capture() was never called. In the case of NVENC, this results in an NVENC session lingering around taking up a slot until OBS is shut down or a stream is successfully started.

nvidia-smi:

    Encoder Stats
        Active Sessions                   : 1
        Average FPS                       : 0
        Average Latency                   : 0

Steps to Reproduce

1. Load OBS
2. Configure an RTMP or WHIP stream with an invalid stream key/bearer token
3. Attempt to start stream, it should fail with the "unable to access channel" error message
4. Observe that the encoder was initialized but never freed.

Anything else we should know?

No response

[tytan652]

OBS does not de-initialize encoders on RTMP stream start fail

2. Configure an RTMP or WHIP stream with an invalid stream key/bearer token

Is it RTMP-only or it concerns RTMP and WHIP or any protocol (or even any encoded output) ?

[tt2468]

I only validated this for RTMP and WHIP, though others are likely affected also. It seems as though the issue mainly stems from:

• output start callback called
  • it initializes encoders
  • it creates a new thread for the connect process
    • connect thread fails, does obs_output_signal_stop() with a fail code, and never truly starts
      • libobs fails to clean up properly