Logging solution

[daurnimator]

trafficstars

I have a bit of a pet hate for kibana and to a less degree, elastic-search. Lets set up loki (to be used from grafana).

TODO list:

• [x] collect logs (fluent-bit vs promtail?)
• [ ] rotate logs (? is this automatic?)
• [x] make logs searchable
• [x] collect node logs (e.g. dmesg)
• [ ] look into @fatalbanana 's ideas below (boltdb?)
• [ ] set up ingress for local logcli usage

[fatalbanana]

Loki is cool- however the microservices-style deployment can get into a broken state if a k8s worker vanishes for example (ie. bad things will happen if ingesters do not remove themselves from the hash ring (in Consul or etcd)).

Using the monolith-style deploy avoids this problem; however it can't be scaled out & you may more likely lose logs in failure scenarios.

Support for writing boltdb indexes to the object store was recently added: it's probably worth trying that if you'd prefer not using locally attached storage (probably a good idea?)- since there is nothing like BigTable nearby.

If you (eventually?) need to go for the distributed setup I'd suggest trying memberlist for hash ring- though I'm not convinced it will solve something besides removing a dependency.

Not sure why you'd be tempted to reach for fluent-bit but in general I'd recommend using promtail.

[daurnimator]

basic (monolithic) loki setup done via #32

[daurnimator]

collect node logs (e.g. dmesg)

done via 5d8b683825a9fce478e68299be4473149f5e376b