Make API server stateless: move chart data to MongoDB

[prydonius]

Currently chart data is stored in memory and fetched via a goroutine at some interval. This is problematic because:

• data is not shared between replicas of the Monocular API server, this means there is the potential for two instances to be out-of-sync
• some data is stored on the filesystem (icons, READMEs) and this can be lost when the pods are restarted
• it's not possible to manually refresh repos if the API server is scaled out (https://github.com/kubernetes-helm/monocular/issues/344#issuecomment-325323237)

The Monocular project is now using a MongoDB data to store repository and user data. This should be extended to charts so we can make the API server truly stateless and therefore more scalable. I propose:

• extracting the repo refresh job from the API server to a standalone tool that writes to a MongoDB database
• run this as a scheduled job (e.g. Kubernetes CronJob)
• a mechanism for kicking off a manual refresh (e.g. service that creates a Kubernetes Job)

[prydonius]

One idea for this is to have a controller that tracks a CRD (AppRepositories) that will:

• create a Kubernetes CronJob to perform the sync when an AppRepository is created
• delete a Kubernetes CronJob when an AppRepository is deleted

It's not clear to me how to trigger a manual refresh using the CRD/Controller method. One idea could be to update an arbitrary field in the CRD (e.g. lastRequestedManualSyncAt: <timestamp>) which would trigger the controller to run a one-time Job to sync.

A better option might be to create an AppRepositorySyncRequest CRD as well, which we may need to garbage collect after some time.

[migmartri]

create a Kubernetes CronJob to perform the sync when an AppRepository is created delete a Kubernetes CronJob when an AppRepository is deleted

WRT using cronjobs, we need to double check its availability in their API servers, for example in <1.8 it is an alpha feature that many clusters might not have enabled.

So as an alternative we could make the controller be the one that triggers jobs every x minutes based on a configurable value in the repo CRD.

[prydonius]

@migmartri yeah, thanks for pointing it out. I talked about this with @arapulido and our plan is to support N and N-1. Kubernetes 1.9 will be out soon and there's still a few weeks until this change will be put into place, so I think we are okay to use CronJobs.