Fix issue where deadlock can occur over logging._lock

[DylanRussell]

Description

We (google) saw a deadlock occur when logging.config.dictConfig is called after the OTEL LoggingHandler is attached to the root logger.

This happened b/c dictConfig acquires logging._lock and then clearsHandlers which then calls shutdown on the OTEL LoggingHandler, which calls flush.

flush triggered an export call to our exporter. Deep inside our exporter we spin up a new thread to handle auth, and that thread also tried to acquire logging._lock resulting in a deadlock..

To fix this I updated LoggingHandler.flush to execute force_flush in a separate thread, and not to block/wait before returning.. This should be reasonable because we don't return the result of the force flush anyway, so why block there.

This seems to reliably fix the deadlock, but I think it's technically possible for this new thread to spin up and reach the lock before logging.config.dictConfig releases it's lock..

Another simple fix is to set flushOnClose to true, so that the OTEL LogHandler.flush is not called during shutdown. This seems fine to me as well because we call shutdown on exit anyway. Either of these solutions seem fine.

Also considered making our exporter async, but we don't have support for async exporter's in this repo yet.

Type of change

• [x ] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

I've added a unit test to show how the deadlock happens.. I don't think that test should actually be submitted because of the chance a deadlock can occur and lock up the test suite..

Does This PR Require a Contrib Repo Change?

• [ ] Yes. - Link to PR:
• [x ] No.

Checklist:

• [ x] Followed the style guidelines of this project
• [ ] Changelogs have been updated
• [ x] Unit tests have been added
• [x ] Documentation has been updated