Improve testing of the REST interface

[suntzu86]

Currently we have two kinds of tests for the REST interface. For simple functions, we:

• call REST and call low level endpoints, verify they're the same for more complex stuff (e.g., select next point), we
• call REST and just verify that nothing explodes.

That second test is inadequate. Yes we trust that all the underlying stuff works, so there's no reason to fully re-test next points optimization, for example.

But to me, that means interface testing should look like: call from REST call C++ (or python libary) directly verify that results are identical

You could reverse points_to_sample and points_being_sampled somewhere in the REST interface and we would never know from tests. I think that's bad.

I think we need more kinds of tests:

1. verify data is flowing through the interface correctly (pretty dumb test but we have made these errors before) <--- the only UNIT test in this list

2. end-to-end verify that very 'coarse' cases work. This can be as dumb as a) run the case manually. record the result. compare against it in the future but preferably b) find some more automatic way (e.g., calling low level endpoints directly) that will evolve as the implementation evolves

3. end-to-end verify that 'real' cases work, more regression-style tests (perhaps run less frequently. these can be expensive)

LASTLY, I know there are links out there with names like "Why Integration Testing is Evil" and such--there may be even better techniques for this kind of stuff and we should look into it.

[sc932]

There is a lot of literature on testing (unit only, mock everything vs integration everything). Most of the sane voices seem to be in favor of a mixed approach. Basically moderation is everything, different tests have different bugs they catch. I like the action items in this ticket but vote for keeping (fast) integration tests.