Perform indexed searches even as infrastructure changes

[nateiam]

As a Shepherd user I want to maintain my ability to perform indexed searches even if the underlying network infrastructure changes.

Tech Notes

• Currently Herd will lose it's connectivity to ElasticSearch if the ES infrastructure (incl ELB) is restarted and if AWS swaps IPs on the ELB. We believe this is due to the fact that the ES Transport Client caches IP addresses.
• The fix is currently understood to involve refreshing he ES Transport client programatically at pre-determined intervals and/or upon any failure
  • It was attempted to modify the DNS caching configuration but that did not fix the issue

Acceptance Criteria

• Indexed search succeeds after ES infrastructure rebuild
• Indexed search succeeds after AWS swaps ELB IP addresses