feat: test bento bundle

[asafalinadsg]

What does this PR address?

This PR adds the bentoml test command which enables testing a Bento for making sure it is correctly packaged. In order to use bentoml test the Bento builder must supply input/output example(s), this will be done in the bentofile.yaml in a dedicated section called endpoints_tests. Once specified, the Bento builder can call bentoml test <bento-tag> to test the specified bento, or alternatively call bentoml build --test which will build and test in one call (still not implemented).

The testing itself will invoke the bentoml containerize command to create an image, will launch a docker container from that image and then will interact with the docker through HTTP request to test the endpoints. This will cover all the critical functionality of Bentos; creating an environment from the Bento specification and the service's endpoints.

Example

Imagine I have the below, dummy, scikit-learn scaler model serving:

# init
model_name = 'test-scaler'

# load model's metadata
scaler_meta = bentoml.sklearn.get(f'{model_name}:latest')

# create service
scaler_runner = scaler_meta.to_runner()
scaler_service = bentoml.Service(model_name, runners=[scaler_runner])

# create endpoint for the service
@scaler_service.api(input=PandasDataFrame(), output=NumpyNdarray())
def my_predict_endpoint(input_df: pd.DataFrame) -> pd.DataFrame:
    res = scaler_runner.run(input_df)
    return res

The altered Bentofile.yaml will look as follows (note the new endpoints_tests section):

service: "serve_model:scaler_service"  # Same as the argument passed to `bentoml serve`
labels:
  owner: bentoml-team
  stage: dev
include:
  - "*.py"  # A pattern for matching which files to include in the bento
python:
  packages: # Additional pip packages required by the service
    - scikit-learn
    - pandas

## TESTING
endpoints_tests:
  my_predict_endpoint:
    - input: [[1, 2, 3, 4, 5]]
      output: [[-1.4638501094227998,-1.4638501094227998,-0.8783100656536799,-0.29277002188455997,-0.8783100656536799]]
    - input: [[1, 2, 3, 4, 5]]

As you can see, I made 2 tests for my_predict_endpoint endpoint:

1. one that takes both input and output.
2. one that takes only an input.

Now, when building a Bento, using bentoml build, we will get a Bento tag, somewhat similar to test-scaler:e45pn7bm7cr4rdsd.

Now we can test our newly created Bento, using the command bentoml test test-scaler:e45pn7bm7cr4rdsd, which will do the following:

1. Create a docker image from the given Bento (using the containerize command under the hood).
2. Run a docker container from the newly created docker image.
3. Run the 2 tests, specified in the Bentofile.yaml, against the my_predict_endpoint endpoint: i. The 1st test, which takes the input, send it to the endpoint, and check if the received result equals to the expected output specified in the test ([[-1.4638501094227998, ...]]). ii. The 2nd test, which takes the input, send it to the endpoint, and just check that the service didn't throw and error (unlike the 1st test, it doesnt do any checking on the responded output).

Issue

https://github.com/bentoml/BentoML/issues/2967

Before submitting:

• [x] Does the Pull Request follow Conventional Commits specification naming? Here are GitHub's guide on how to create a pull request.
• [x] Does the code follow BentoML's code style, both make format and make lint script have passed (instructions)?
• [x] Did you read through contribution guidelines and follow development guidelines?
• [ ] Did your changes require updates to the documentation? Have you updated those accordingly? Here are documentation guidelines and tips on writting docs.
• [ ] Did you write tests to cover your changes?

Who can help review?

@aarnphm @parano

Feel free to tag members/contributors who can help review your PR.

[pep8speaks]

Hello @asafalinadsg, Thanks for updating this PR.

• In the file bentoml/client/__init__.py:

Line 1:1: F401 'bentoml.client.test_client.Endpoint' imported but unused Line 2:1: F401 'bentoml.client.test_client.TestClient' imported but unused

• In the file bentoml/client/test_client/__init__.py:

Line 1:1: F401 'bentoml.client.test_client.utils.is_equal' imported but unused Line 2:1: F401 'bentoml.client.test_client.utils.get_test_data' imported but unused Line 3:1: F401 'bentoml.client.test_client.client.Endpoint' imported but unused Line 4:1: F401 'bentoml.client.test_client.client.TestClient' imported but unused

• In the file bentoml/client/test_client/client.py:

Line 83:9: E722 do not use bare 'except'

Comment last updated at 2022-09-13 12:56:08 UTC

[yubozhao]

@asafalinadsg this amazing

@ssheng Can we check this one out?

[codecov[bot]]

Codecov Report

Merging #2968 (e291039) into main (e291039) will not change coverage. The diff coverage is n/a.

:exclamation: Current head e291039 differs from pull request most recent head fb4ccf3. Consider uploading reports for the commit fb4ccf3 to get more accurate results

@@           Coverage Diff           @@
##             main    #2968   +/-   ##
=======================================
  Coverage   69.00%   69.00%           
=======================================
  Files         122      122           
  Lines       10162    10162           
=======================================
  Hits         7012     7012           
  Misses       3150     3150           

[aarnphm]

Hi there, one small request. Is it possible that you can move the client folder to bentoml instead of bentoml._internal?

I think it is nice to also expose the python api

from bentoml.client import testclient
...

[asafalinadsg]

I've added few more things. The bentofile.yaml file changed a bit: Notice the input / output option of local / remote (s3 etc...) files.

service: "serve_model:scaler_service"  # Same as the argument passed to `bentoml serve`
labels:
  owner: bentoml-team
  stage: dev
include:
  - "*.py"  # A pattern for matching which files to include in the bento
python:
  packages: # Additional pip packages required by the service
    - scikit-learn
    - pandas

# TESTING
tests:
  config:
    timeout: 10
  endpoints:
    my_predict_endpoint:
      - input: [[1, 2, 3, 4, 5]]
        output: [[1 ,2, 3, 4, 5]]
      - input: [[1, 2, 3, 4, 5]]
      - input: /my/local/file/path/input.json
        output: /my/remote/file/path/output.json

Also, I've added the option to bentoml build --test. This will create a Bento, as usual, plus run the tests right after the build. In other words:

bentoml build
bentoml test <my-bento>

is just like:

bentoml build --test

[asafalinadsg]

Notice the input / output option of local / remote (s3 etc...) files.

Regarding the above: The input / output option of local / remote files is using the fsspec package under the hood. This is a new dependency for bentoml, which is yet to be added to the PR. If possible, would be nice if someone can assist me with adding it.

[aarnphm]

Notice the input / output option of local / remote (s3 etc...) files.

We are currently using PyFilesystem2, would be great if we can use that. to avoid introducing more dependencies.

[asafalinadsg]

@aarnphm

I think the ability to include testing data under the bentofile.yaml is nice and convenient, but I'm not convinced yet if this is the way going forward.

Do you still consider adding testing as part of the bentofile.yaml? Are there any other alternative ways you consider for testing a bento?

[ssheng]

@asafalinadsg Sorry about the delay response. The team discussed internally about the proposal and we liked the idea of make bentos testable. We also recognize the challenges of building bentos without being able to test it in the environment (e.g. docker) that it is going to be run in. We propose a 2-step change to make the bentoml test feature more useful.

1. Making bentos testable through bentoml test. As you implemented in this PR, we can add a tests section in the bentofile.yaml for the bento creator to define the tests to be run against the deployed bento. However, instead of hard coding inputs and outputs in the bentofile.yaml, we think specifying a set of modules that can be pytest'ed against for better flexibility. So running bentoml test will first serve the bento, then run the list of modules with pytest. In the test modules, users should be able to call the deployed bento using a client, currently being developed (https://github.com/bentoml/Kitchen/issues/42).
2. Allowing the specification of environment during bentoml test through the --env argument. Valid environments include local which is the same as the behavior today, conda, or docker. If virtualenv is specified, the bentoml test command will containerize the bento and run the tests in a Docker environment. The --env argument can also be extended beyond bentoml test to bentoml serve, and behaves the same as above.

Would you like to help with the first step of the bentoml test command with this PR? We are happy to share more detailed thinking over a quick chat.

[asafalinadsg]

Hi @ssheng , thanks for your replay. Yes, I would like to help with the first step that you proposed. I want to make sure I understand your proposal, I'll connect with you through slack.

[aarnphm]

Hi there, any updates so far @asafalinadsg?

[asafalinadsg]

Hi there, any updates so far @asafalinadsg?

Hi @aarnphm Sorry, don't have any updates so far, didn't have the time to work on it. I'll try to let you know in advance whenever I start work on it.

[aarnphm]

@asafalinadsg if you don't have the bandwidth, I'm more than happy to take it over.

[asafalinadsg]

@aarnphm Sure thing, if thats ok with you then yea I don't mind you taking over. Thanks for the help!

[parano]

Hi @asafalinadsg we've recently introduced the BentoML serve python API, which should make writing functional tests with Bentos a lot easier: https://github.com/bentoml/BentoML/pull/3696

Closing this PR for now as we have other plans around developer workflow that overlap with this effort.