CSMO-ICP
CSMO-ICP copied to clipboard
ibm-cloud-architecture
Added available entropy timeseries panel
*Lack of entropy can be a potential cause of performance/quality problems for a platform, especially on Linux VM systems. /dev/random is a blocking process and is a main source of entropy. low Entropy is a problem (e.g consistent access to a random number generator) for encryption/private keys/tls/ etc… especially on cloud vms where there are very few sources of random behavior (e.g. no keyboard, mouse movement, etc …). Would be nice to have this graphic in the dashboard across nodes to monitor if entropy runs low. Usually anything under 1000 is considered bad and under 200 is horrible. Symptoms when that low include sloooow ssh entry to the machine, blocking processes that pile up causing memory pressure and ultimately OOM, etc … This panel can help validate mitigation strategies such as implementation of the haveged entroy aggregator daemon, Hardware based random number generators (RNG), etc ...
@joshisa - just a clarification question - do you find that entropy is more significant for VMs running containers/kubernetes/ICP or is it a generic "good thing to measure"?
@RobertJBarron Definitely an issue for VMs in general to measure. Platforms such as k8s, etc ... that perform alot of encryption, TLS, etc ... activities tend to deplete more rapidly -- but other workloads (e.g. heavily loaded SSL websites, etc ...) can experience the same risk of rapid depletion. Because of the "non-random" normalized nature of most input streams flowing into a VM env, randomness is a rare and valuable commodity -- So I feel its a "good thing to measure" in general for any VM based deployments.
Here's a few references that helped me grok the importance of tuning this for my deploys:
- https://github.com/coreos/coreos-kubernetes/issues/701
- http://giovannitorres.me/increasing-entropy-on-virtual-machines.html
- https://www.digitalocean.com/community/tutorials/how-to-setup-additional-entropy-for-cloud-servers-using-haveged
- https://wiki.openstack.org/wiki/VirtEntropyProvision
Sorry for the delay - as soon as I get the cycles I will pull this in and take a look at it. This is great stuff and the collaboration @RobertJBarron and I are hoping for!