[Epic] Kelp-as-a-Service

[nikhilsaraf]

This is the master-list of issues required to make Kelp-as-a-Service a reality, written in order of precedence.

For specific tasks, we should create new issues and link there here next to the specified item.

Essential Tasks

• [x] add support for multitenancy on the backend (issue)
• [x] [1] Add support to run backend and UI server without launching frontend browser (issue)
• [x] [3] quit function only available when in kaas mode (issue)
• [x] [13] OAuth or other login mechanism to receive and persist namespace token on client (issue)
• [ ] [8] delegated signing to third-party server (issue)
• [ ] [3] support delegated signing to third-party server in Kelp GUI (issue)
• [ ] [8] Setup monitoring and alerting for KaaS services (automatically provided by k8s container resource monitoring?)

Not essential for v1.0

• [ ] [5] improve log management (rotation?) for bot services (k8s cluster-level logging will help?)
• [ ] [2] manual building of Docker images for Kelp binaries, upload once to DockerHub
• [ ] [5] k8s deployment in various environments (local/staging/prod)
• [ ] [8] encrypted secret keys (HSM?, encryption?)

May be required later (but noting here anyway)

• [ ] [5] database schema for users table to isolate requests, and bots, based on the context of a namespace (orgID or userID, or both?)
• [ ] [3] automated building of Docker images for Kelp binaries, uploaded to DockerHub on every push to master
• [ ] [13] idempotent trader bot strategies: make sure all trading strategies operate in an idempotent manner
  • do we need any of this right now because buysell is already idempotent?
  • maybe write to database to maintain statelessness of bots?
  • may need that data layer for state management in the common bot infra after all?
  • persist in-memory state after each run as a temporary solution to idempotent bots?

[brujoand]

Sorry for adding this here, but I'm testing a few approaches of getting this to work with the current code base, and my problems seem to touch on a bit off everything in this list so I thought this would be the best place for it.

I created a simple docker container based on alpine that only installs bash and fetches the latest kelp binary and uploads the image to docker hub. I had to create a dummy xdg-open script to trick kelp into not launching the browser, but even then the UI webpage doesn't work properly because it fails to fetch certain resources. The last output from kelp is '2021/03/07 08:12:12 Serving frontend and API server on HTTP port: 8000', the webpage appears to be working until I try adding a new bot where I just end up at /new and an empty page. During all the interaction the browser console shows 'Uncaught (in promise) TypeError: NetworkError when attempting to fetch resource.' every second. The url it's trying to hit is 'http://localhost:8000/api/v1/fetchKelpErrors' but it seems the entire API endpoint is unresponsive.

Regarding #347 and naming of the deamon mode, I'd suggest having two options instead of one, namely 'server' and 'gui'. Where the current 'server' flag would be renamed to 'gui' and actually launches a gui either through electron or by opening a browser window. Then the 'server' option would simply run kelp as a deamon process. The reason why I think these are completely unrelated run modes is that the deamon mode would probably only be used on remote installs, whereas 'gui' would be for local use. And eventually that might be the default way for users to use the app locally.

In any case, my plan is to deploy this on k8s using helm so whenever that becomes relevant I could help on that. But right now I'm just trying to see if there is a way to make this work and also to ping on this process as it seems to have been quiet for a while.

[nikhilsaraf]

Hi @brujoand thanks for commenting on this issue, it's the right place.

1. You can now run kelp without bringing up the browser or electron by adding the following cli flag --enable-kaas For example: ./bin/kelp server --enable-kaas

2. Can you provide more information about the network error you are getting in relation to the /fetchKelpErrors endpoint. Log files and/or screenshots will help. the more info you can provide the better.

3. splitting up the commands into server and gui makes sense. I can include it in this list of items above, although not a priority right now. see --enable-kaascli option for now.

4. The backend is now multi-tenant so supports multiple users from a data structures and API perspective, however a fair amount of work needs to be done to fully introduce the concept of users into the system. That is being worked on but unfortunately not on a full-time basis. Will have more updates soon. I will try to keep this list of items in my original description up to date.