[Serve] Support serve component callback

[sihanwang41]

Why are these changes needed?

• User can set environment variable to setup controller and http proxy callback function path.

test_callback.py

from starlette.middleware import Middleware
import logging

logger = logging.getLogger("ray.serve")

class ASGIMiddleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        logger.info("my_logger!")
        await self.app(scope, receive, send)


def proxy_callback():
    return [Middleware(ASGIMiddleware)]

set env variable to start ray serve.

RAY_SERVE_HTTP_PROXY_CALLBACK_IMPORT_PATH="test_callback.proxy_callback"

@shayanhoshyari manual tested for their use case.

Related issue number

Closes: #35046

Checks

• [ ] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• [ ] I've run scripts/format.sh to lint the changes in this PR.
• [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
  • [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in doc/source/tune/api/ under the corresponding .rst file.
• [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • [ ] Unit tests
  • [ ] Release tests
  • [ ] This PR is not tested :(

[shayanhoshyari]

I did not have much luck with this. I ran into some problems.

Here I fully put a completely standalone example https://github.com/shayanhoshyari/ray-playground/tree/main/http_proxy_callback with what I am trying to achieve and what is failing. Here I have the vscode workflow, but the logger and middlewares are also similar.

I tried the callback for init logging and debugger workflow. See the comments in main.py and readme.md. The summary of the issue is that I don't have a way to pass the env var that contains the path to the callback, and the information that the callback needs to the HTTPProxy.

I did not try the production workflow. My main requirement there is to be able to put the callback in runtime_env; e.g.

import_path: fruit:deployment_graph
runtime_env: {
    # Can HTTP Proxy pick this up?
    "py_modules" : "...", # code that contains fruit.
    "SERVE_HTTP_PROXY_CALLBACK_IMPORT_PATH" : "fruit:http_callback"
}

[sihanwang41]

I did not have much luck with this. I ran into some problems.

Here I fully put a completely standalone example https://github.com/shayanhoshyari/ray-playground/tree/main/http_proxy_callback with what I am trying to achieve and what is failing. Here I have the vscode workflow, but the logger and middlewares are also similar.

I tried the callback for init logging and debugger workflow. See the comments in main.py and readme.md. The summary of the issue is that I don't have a way to pass the env var that contains the path to the callback, and the information that the callback needs to the HTTPProxy.

I did not try the production workflow. My main requirement there is to be able to put the callback in runtime_env; e.g.

import_path: fruit:deployment_graph
runtime_env: {
    # Can HTTP Proxy pick this up?
    "py_modules" : "...", # code that contains fruit.
    "SERVE_HTTP_PROXY_CALLBACK_IMPORT_PATH" : "fruit:http_callback"
}

Hi @shayanhoshyari , this can be unblocked

1. Set the environment in your image directly.
2. Set env_var into your ray process start command. (https://docs.ray.io/en/latest/serve/dev-workflow.html?highlight=env_vars#testing-on-a-remote-cluster)

[sihanwang41]

@edoakes ping for merge, doc test failures are unrelated.