Add integration and end to end testing for hot reload of config

[aaronburtle]

We have unit testing that covers the hot reload scenario in testing, but not integration testing or end to end testing. This issue seeks to address that missing test coverage.

One thing to consider is that with this kind of testing there is a race condition created between the modification of the file, the file watcher detecting that modification, the updating of the config, and the follow up request that utilizes the hot reloaded config. Effective testing needs to solve this race condition.

[seantleonard]

Moving this + hot reload related tasks to post GA -> tracking via 1.2rc milestone.