h2ogpt icon indicating copy to clipboard operation
h2ogpt copied to clipboard

Add test to the Client to validate parameters order with h2ogpt

Open this opened this issue 1 year ago • 0 comments

This PR,

  • Adds test_parameters_order test which asserts the API parameter order used in the Client matches with the order defined in h2oGPT.
    • eval_func_param_names, gen_hyper, no_default_param_names, and eval_func_param_names_defaults were refactored & moved from generate.py to parameters.py so that eval_func_param_names can be imported into the Client without causing import errors.
  • Refactors how we copy source files (e.g. enums.py) from h2oGPT to the Client.

Tested

Client tests

make H2OGPT_SERVER="http://0.0.0.0:7860" test

PASSED tests/test_client.py::test_text_completion
PASSED tests/test_client.py::test_text_completion_sync
PASSED tests/test_client.py::test_chat_completion
PASSED tests/test_client.py::test_chat_completion_sync
PASSED tests/test_client.py::test_parameters_order

With h2oGPT env

pytest client/tests

platform linux -- Python 3.10.12, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/sajith/src/h2ogpt/client, configfile: pyproject.toml
plugins: anyio-3.7.0, xdist-3.2.1, asyncio-0.21.0
asyncio: mode=strict
collected 5 items                                                                                                                                    

client/tests/test_client.py::test_text_completion 
------------------------------------------------------------------- live log setup -------------------------------------------------------------------
INFO     tests.conftest:conftest.py:17 Couldn't find a running h2oGPT server. Hence starting a one.
INFO     tests.conftest:conftest.py:45 Loading module 'generate' from '/home/sajith/src/h2ogpt/generate.py'.
INFO     numexpr.utils:utils.py:148 Note: NumExpr detected 20 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO     numexpr.utils:utils.py:160 NumExpr defaulting to 8 threads.
WARNING  accelerate.utils.modeling:modeling.py:706 The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.
WARNING  accelerate.utils.modeling:modeling.py:706 The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.
INFO     apscheduler.scheduler:base.py:444 Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO     apscheduler.scheduler:base.py:444 Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO     apscheduler.scheduler:base.py:885 Added job "clear_torch_cache" to job store "default"
INFO     apscheduler.scheduler:base.py:885 Added job "ping_gpu" to job store "default"
INFO     apscheduler.scheduler:base.py:171 Scheduler started
INFO     tests.conftest:conftest.py:30 h2oGPT server started at 'http://0.0.0.0:7860'.
PASSED                                                                                                                                         [ 20%]
client/tests/test_client.py::test_text_completion_sync PASSED                                                                                  [ 40%]
client/tests/test_client.py::test_chat_completion PASSED                                                                                       [ 60%]
client/tests/test_client.py::test_chat_completion_sync PASSED                                                                                  [ 80%]
client/tests/test_client.py::test_parameters_order 
------------------------------------------------------------------- live log setup -------------------------------------------------------------------
INFO     tests.conftest:conftest.py:45 Loading module 'parameters' from '/home/sajith/src/h2ogpt/parameters.py'.
PASSED

In a notebook with the Client wheel

image

this avatar Jul 05 '23 20:07 this