[Roadmap] Logging and Debugging

[simon-mo]

We plan to start working on aggregating logs from Clipper containers using fluentd. This can be divided in the few PRs. @rkooo567 will leads the effort.

PR 1: Pipe logs to fluentd, use the stdout export

• [x] clipper_admin should start the fluentd container and configure it correctly
• [x] When clipper_admin start any containers, it should correctly configure the logging driver
• [x] There should be a flag in start_clipper: use_fluetnd_log=False

PR 2: Pipe log from fluentd to sqlite

• [ ] Configure a custom fluentd container that use sqlite3 output plugin to send the logs to a sqlite database
• [ ] The database should be a standalone container.
• [ ] Add a simple API to clipper connection to query the log: clipper_conn.query_log(sql_query)

PR 3: Add debug effort

• [ ] Add extensive logging for the model container's rpc.py, etc.
• [ ] When a container is not responding or timeout, query the sqlite, respond to user.

PR 4: Kubernetes Port

[rkooo567]

The first PR's config build is not complete yet. https://github.com/ucbrise/clipper/pull/652/files/2ea53ce78f3f7725f8619b3e5aac4fd0b635a5eb#diff-4e9780876d0c353566121aba8ecdd09bR137

# clipper_admin/clipper_admin/docker/logging/fluentd.py
# Logging-TODO: Currently, it copies the default conf from clipper_fluentd.conf.
#               We need a way to customize it.

This should be handled in PR2 as we need customization for integrating with sqlite

[rkooo567]

@simon-mo Will begin to work on part 2 from May 18th.

[rkooo567]

#735 handles the second part of the PR. @simon-mo Please review it whenever you are free!