Fix race condition in ConcurrentMultiSpanProcessor.shutdown()

[jomcgi]

Description

Fixed error during shutdown: 'cannot schedule new futures after shutdow'. Modified shutdown sequence to handle executor already being in shutdown state. Added direct processor shutdown when executor is unavailable. Added regression test for executor shutdown race condition.

Fixes #4461

Type of change

• [x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• [x] tox -e test-opentelemetry-sdk

Does This PR Require a Contrib Repo Change?

• [x] No.

Checklist:

• [x] Followed the style guidelines of this project
• [x] Changelogs have been updated
• [x] Unit tests have been added
• [x] Documentation has been updated

[aabmass]

Check my comment on #4461 I'm not sure this fixes the problem

[jomcgi]

I've rebased the PR onto main to resolve conflicts.

Check my comment on #4461 I'm not sure this fixes the problem

@aabmass the reproducer from #4461 generates an error in main for Python >= 3.9. I can't reproduce on this branch, it works as expected.

Do you think we should be looking at an alternative solution?