stbt run: Allow passing arguments to test case functions

[wmanley]

This is a counterpart to passing arguments to the old-style test case files as sys.argv, but for new-style test case functions.

It's also a lot nicer because now you don't need to implement argument parsing in the test-case itself, you just get the arguments passed through.

stbt run can now take pass arguments to test case functions with the --kwargs option.

Example: with this test case in tests/channel_test.py:

    def change_channel(channel_number):
        print "Changing channel to %i" % channel_number

you can pass the channel number 3 with:

    stbt run tests/channel_test.py::change_channel \
        --kwargs='{"channel_number": 3}'

Note: the --kwargs option takes a JSON formatted string and is passed as an argument after the test case name. This means that you can also pass the --kwargs option on the stbt batch command line for each test case:

    stbt batch run \
        tests/channel_test.py::change_channel --kwargs='{"channel_number": 2}' -- \
        tests/channel_test.py::change_channel --kwargs='{"channel_number": 3}' --

An argument could be made that we shouldn't override sys.argv in this case, or possibly produce an error if you attempt to pass additional arguments to test case functions. This would be a breaking change, but you could argue that it would be good to do it as it would allow an opportunity in the future to even more automatic argument parsing. Probably a bad idea for now though.

[wmanley]

Note: I chose a single JSON argument to allow passing types through to the functions rather than passing everything through as a string.

[drothlis]

My initial reaction to this was that (a) it's a homegrown/non-standard way of doing remote procedure calls from the command line, and (b) yet another divergence from existing test runners like nose or pytest.

Re. (a) I suppose the problem is that there simply isn't a good way of passing typed parameters from the command line or exec(3). In looking for existing solutions to this problem I came across systemd/dbus busctl tool, which looks like:

busctl call <service> <object> <interface> <method> <signature> <arguments>

e.g.:

busctl call ... ss "first string argument" "second string argument"
busctl call ... as 3 hello world foobar  # an array of 3 strings
busctl call ... a{sv} 3 One s Eins Two u 2 Yes b true  # a dictionary of string-to-variant mappings

I'm not suggesting we use that. If anything it's a counter-example to show how much better it is to use json.

I suppose the alternative would be to allow stbt to be used as a library, and write a server that listens for RPC requests and runs the requested function. That sounds like a lot more work.

Re. (b) after further thought I don't care -- neither nose nor pytest support passing arguments to test functions at all, so it's not like we're being incompatible with them. Nose only supports passing configuration via a config file which you then read explicitly in your test function.

I don't really have a point here, just rambling thoughts.

[wmanley]

neither nose nor pytest support passing arguments to test functions at all, so it's not like we're being incompatible with them.

I don't think this is true. We could use py.test's fixtures to pass the arguments in.

[drothlis]

neither nose nor pytest support passing arguments to test functions at all, so it's not like we're being incompatible with them.

I don't think this is true. We could use py.test's fixtures to pass the arguments in.

I meant that you can't pass arguments at run-time from the command line. Though I suppose you could write a pytest plugin that used the fixtures mechanism to provide arguments from the command line?

[wmanley]

An alternative would be to encode the expected types in the test scripts themselves and do the conversion based on that. For example you could parse:

def function1(self, arg1, arg2, arg3):
    """returns (arg1 / arg2) + arg3

    :param arg1: the first value
    :param arg2: the second value
    :param arg3: the third value
    :type arg1: int
    :type arg2: str
    :type arg3: dict
    """
    return arg1/arg2 + arg3

or

def function1(self, arg1, arg2, arg3):
"""
Args:
    arg1 (int): The first parameter.
    arg2 (str): The second parameter
    args (dict): The third parameter
"""

or even use PEP-484 stub files and know what types we are expecting and do conversion as appropriate.

[drothlis]

Yeah but then how do you specify the dict's contents on the command line? The --kwargs '{"json": ["arguments"]} idea seems to be the best idea so far.

[wmanley]

how do you specify the dict's contents on the command line?

I guess you would then require JSON. e.g from bash:

stbt run tests/epg.py::my_test string 79 '{"a more": "complex data structure"}'

or

stbt run tests/epg.py::my_test --param1=string --param2=79 --param3='{"a more": "complex data structure"}'

The --kwargs '{"json": ["arguments"]} idea seems to be the best idea so far.

I agree, but the question is: is it good enough to go in?

[drothlis]

is it good enough to go in?

Yeah go ahead. I certainly haven't come up with anything better in the last 6 weeks.

[drothlis]

It would be nice to also pass in positional arguments specified on the command line, even if we don't do any type conversions and pass them as strings:

stbt batch run tests/epg.py::my_test arg1 arg2 arg3

At least this would be consistent with the existing command-line interface. To keep backwards compatibility with functions that don't take arguments but that do look in sys.argv, stbt-run could use inspect.getargspec to determine whether the function takes any arguments, and if not, only pass the arguments via sys.argv.

[drothlis]

If I recall correctly, one of the motivations for allowing arguments as JSON is to make it easy to integrate stbt batch run with a REST API for running tests, where you're already specifying various parameters (which test to run, which remote control to use, etc) in JSON.