[Roadmap] Clipper 0.4.0 Roadmap

[simon-mo]

This issue thread is used to gather community feedback on next Clipper version.

• What major feature do you want to see in Clipper?
• What model deployer do you want to add?
• Which part of Clipper needs to be more battle-tested?
• etc

Please feel free to respond below!

[HarryAtMove]

What major feature do you want to see in Clipper?

Support for AWS ECS / Fargate

Dedicated endpoint and data type for image data as input; we would like to be able to pass bytes to the predict function instead of having to base64 encode. This would be possible if we could configure particular applications to specify the input is image data.

Which part of Clipper needs to be more battle-tested?

Clipper query front end needs to scale horizontally to accommodate throughput of tens to hundreds of requests per second.

[hyttysmyrkky]

What major feature do you want to see in Clipper?

• Support for ensemble methods (https://github.com/ucbrise/clipper/issues/450), at least as a plugin (+documentation how to implement that).

• Complete handling of deployments, so that no interaction is required between the user and Docker registry (or between the user and Kubernetes). I'm not sure if this https://github.com/ucbrise/clipper/pull/478 is the last missing step though. Then there is https://github.com/ucbrise/clipper/issues/342?

Which part of Clipper needs to be more battle-tested?

• Clipper Query frontend. There seems to be bugs (such as https://github.com/ucbrise/clipper/issues/518). Also the horizontal scaling, as @HarryAtMove said.

[pedrochavesmp]

Option to set the full endpoint name when registering an app (i.e., other than /predict).

[withsmilo]

I'm using Clipper as a prediction-serving module to deploy our ML models smoothly and support thousands of requests per second in the production environment.

What major feature do you want to see in Clipper?

• Add some following functions to manage model, app and link.
  • unlink_model_from_app(app_name, model_name)
    • We need to unlink model from app.
    • We applied parts of #254 PR to implement this function.
  • delete_versioned_model(model_name, model_version)
    • If we deployed many versions of the same model, Clipper has stored all the information of old versions. We need to a function to clean up some old versions if they are useless anymore.
    • We applied parts of #478 PR to implement this function.
• Support the password-required Redis

What model deployer do you want to add?

• Keras

Which part of Clipper needs to be more battle-tested?

• QueryFrontend
  • Expose http configurations to clipper_admin
    • Must be needed to tune http configurations in production environment.
    • I created it as a #524 PR already.
  • Adaptive Batching v2's bug (#518 issue)
    • I applied an workaround patch.
  • memory usage increase continuously. (#468 issue)
    • I will create a PR newly to solve this issue.
  • Memory leak occurred in QueryFrontend when all the user requests met SLO (#549 issue)
    • I applied an workaround patch.
  • SIGSEGV occurred in the SimpleWebServer very rarely (#551 issue)
    • After finished to merge #524 PR with upstream, I will create a new PR for this issue.

etc

• Need to refactor containers/python/rpc.py
• Upgrade folly to latest version

[withsmilo]

What major feature do you want to see in Clipper?

• Support that each model replica has multi RPCServices because IP address resource might be more valuable than memory and CPU in many cases.