make test parameters visible to setup method

[granders]

A funky pattern has started to develop in kafka system tests, which means that setup isn't quite doing its job:

• parametrized tests allow for exploding a single test method into multiple tests
• these parameters are often used specify how services should be configured before they are started
• these parameters are not visible to the setup method
• therefore test authors have started writing helper methods such as start_kafka etc which are effectively shared setup logic that make use of these parameters, but which have to be called directly in every test method

The behavior we might want:

def TestClass(Test):
  def setUp(x, y):
     start_some_service(x, y)

  @parametrize(x=1, y=2)
  @parametrize(x=1, y=2)
  def cool_test(x, y):
    # specific values of x and y are supplied to the setup method!

[ewencp]

Not sure I understand why you should have this. This is pretty normal in unit test frameworks as well. setUp() is supposed to be test-agnostic, so it shouldn't depend on those parameters. It is a bit more verbose to have to call a (non-setUp()) method in each test, but it's ok for test code to be a bit more verbose and redundant for the sake of keeping the test cases clear, readable, and simple.

[granders]

@ewencp Are those frameworks allowing parametrization of their tests?

The point I'm making is that users now have started effectively awkwardly splitting setup logic into two methods, and this is unnecessary.

Here is what I see happening now:

def setup():
  zookeeper.start()

def start_kafka(security):
  kafka.security = security
  kafka.start()

@parametrize(security=SSL)
test1(security):
  start_kafka(security)
  ... test logic ...

@parametrize(security=PLAINTEXT)
test2(security):
  start_kafka(security)
  ... test logic

vs

def setup(security):
  kafka.security = security
  zookeeper.start()
  kafka.start()

@parametrize(security=SSL)
test1(security):
  ... test logic ...

@parametrize(security=PLAINTEXT)
test2(security):
  ... test logic

I'd argue that this is quite a bit more clear, readable, and simple. Although there is some implicit magic in how the parameters get injected into setup, to me this seems pretty intuitive.

[ewencp]

@granders They don't, but it doesn't really change anything. I just meant that if you were doing anything that requires a standard pattern in your test, but customization per test, then there will be some duplication, e.g. to mirror your example:

@Before
public void setUp() {
    this.zk = new EmbeddedZookeeper()
}

@Test
public void testPlaintext() {
    this.kafka = new KafkaServer(SecurityProtocol.PLAINTEXT);
    this.kafka.run();
   ....

@Test
public void testSsl() {
    this.kafka = new KafkaServer(SecurityProtocol.SSL);
    this.kafka.run();
   ....

If it's worth it, you would just refactor those lines into a method you could call. Part of the value of setUp is that it should also help you tell the difference between the thing you're testing failing and something unrelated breaking that is causing your test to fail. But other than that, it's really just a formalized way to express common code and avoid 1 method call.

In any case, think about what this change would mean when you don't have the exact same set of parameters. For example, take a look at benchmark_test in Kafka.