Use persistent / persistentRecursive Watches for Helix

[narendly]

Is your feature request related to a problem? Please describe. A potential scalability issue with installing a lot of Watches every time they get triggered.

Describe the solution you'd like Use a new ZK feature: https://issues.apache.org/jira/browse/ZOOKEEPER-1416

Additional context

ZK Watches, previously, were a hassle to work with because we use them to "subscribe" to changes, but the subscription would expire (once fired, the watches would be gone and will have to be reset).

Helix implements this subscription feature in the Controller's callback (See subscribeChildChanges() and subscribeDataChanges()). This is an expensive operation because it happens every time an event gets queued.

A lot of this could be done using persistent and persistentRecursive watches and thereby reducing the compute load/heap pressure, which has been a scalability issue for Helix Controller.

This will shift the watch-reinstall to the ZK server and cut down on much of the redundant server-client interaction that we have currently.

See: https://issues.apache.org/jira/browse/ZOOKEEPER-1416