add core state lock deadlock detection config option

[troyready]

Previously, a build flag could be specified to enable the go-deadlock library, but this change makes it a regular service config option.

It's possible that this is still too a bit too conservative of an approach: because go-deadlock has a disable option that is about as efficient as possible we could just use that instead of this conditional assignment.

[troyready]

Testing here might also be too clever by half: I didn't want to punt on adding a test but also wanted to avoid mocking away too much from go-deadlock's logging/exit. There's perhaps a better approach in using go-deadlock's config options to not exit instead.

[troyready]

Ah, sorry for the obtuseness @ncabatoff -- I had drawn some incorrect conclusions due to errant test results. I've simplified the implementation as discussed.

[troyready]

I'm pretty sure the failing race condition test is expected due to the deadlock induced in the new test. I'm torn about best way to fix it:

• Could disable the test execution during CircleCI's race-condition checking (e.g. a new environment variable set during those tests that causes the specific unit test to short-circuit)
• Alternatively, could scale the new test back and not actually induce the deadlock. Maybe checking the config option is good enough?

[jasonodonnell]

Closing in favor of #18604 .