istio
Consider separating Telemetry from Checks
Telemetry and Checks have vastly different runtime requirements.
Check: High Availability, Low Latency, Logical Unary Call Telemetry: Moderate availability, Very high data volume, Logical data stream
Check and Telemetry have orthogonal audience.
- Create 2 different mixer binaries.
MixerTelemeterandMixerAdmissionControl?
We can look at this as a deployment concern and create a different deployment each for Telemetry and AdmissionControl. However it leaves out some potentially fundamental changes like thinking of telemetry as a Queue consumer. kafka anyone ;-)
There are some things we should consider:
- ability to share caches for preprocess adapters across check and report
- increased realized cost of deployment and management
- forcing adapters to only be check or report
- potential increase in admin/operator cognitive load
This seems like something we go a long time without necessarily having to face. There are approaches to managing Check/Quota and Report within the same binary that may scale quite well and avoid the cost of further fragmentation of Istio services. Certainly, we have some experience with a combined binary running at quite large scale. In the meantime, I don't think there is anything that prevents building Mixer with different adapters (and potentially using routing rules or similar to achieve similar effect).
You mean instead of making one RPC to check, and deriving telemetry from same data - have 2 RPCs ? I don't think it'll help performance.
Allowing sidecar to be configured to optionally make batched reports to a separate service ( but can be same binary ) may be useful in some large deployments.
On Wed, Jun 28, 2017 at 6:18 PM, Douglas Reid [email protected] wrote:
There are some things we should consider:
- ability to share caches for preprocess adapters across check and report
- increased realized cost of deployment and management
- forcing adapters to only be check or report
- potential increase in admin/operator cognitive load
This seems like something we go a long time without necessarily having to face. There are approaches to managing Check/Quota and Report within the same binary that may scale quite well and avoid the cost of further fragmentation of Istio services. Certainly, we have some experience with a combined binary running at quite large scale. In the meantime, I don't think there is anything that prevents building Mixer with different adapters (and potentially using routing rules or similar to achieve similar effect).
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/istio/mixer/issues/845#issuecomment-311835437, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFI6iXDVpud0K4OtNANttxk3CFXQRU2ks5sIvt2gaJpZM4OF-zE .
@costinm Right now, Mixer exposes both Check and Report out of the same binaries. Mandar is proposing that since what Check and Report actually do is fairly different, maybe we should have different binaries or at least different instances, tailored for each.
In this mode, the proxy would issue the same number of RPCs, it's just that Check and Report would go to different services.
Big +1 on allowing Check and Report to use different services, i.e. sidecar to allow pointing to different clusters. Also +1 on having separate service protos.
But I don't see the need to have 2 different binaries. In small installs having a single binary providing both services ( and maybe also including the DS ?), in large installs we can split the RPCs to different services. A lot of initialization / config is shared.
On Thu, Jun 29, 2017 at 7:33 AM, Martin Taillefer [email protected] wrote:
@costinm https://github.com/costinm Right now, Mixer exposes both Check and Report out of the same binaries. Mandar is proposing that since what Check and Report actually do is fairly different, maybe we should have different binaries or at least different instances, tailored for each.
In this mode, the proxy would issue the same number of RPCs, it's just that Check and Report would go to different services.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/istio/mixer/issues/845#issuecomment-311985567, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFI6vH072JjxTVR9L_vfGd2FFzvR1diks5sI7XAgaJpZM4OF-zE .
There's no reason to have separate service protos, Mixer exposes a single logical service. Mandar is proposing that that service be implemented by two separate binaries.
IMO we should hold off on splitting until we better understand Mixer performance in general, and specifically how Report affects Check. Then we can evaluate splitting the binary with some data in hand. Even then, given that we can trivially route per method, it's not clear that the Check and Report binaries need to actually be different: the overhead of unused adapters in Mixer is minimal, one could easily use a single Mixer image deployed into two pools. That also avoids nastiness around splitting up config, dealing with adapters that exist on both the Check and Report paths, the extra operational overhead of building and deploying two things instead of one, etc.
