Don't roll back failed test state

[JLDLaughlin]

Hi all--first, thank you so much for this test suite! It made developing an adapter (and feeling confident about it) so much easier.

Something I found a bit challenging, though, was the fact that test failures roll back any state created during the run. In some situations, I was unable to debug without poking around the objects and data created by the test. To do so, I turned on more aggressive logging in my underlying database, recreated the test state at the time of the failure, and debugged from there.

Would you all be interested in changing this behavior? Or in providing an optional flag to preserve local state on failure?

[jtcohen6]

@JLDLaughlin Thanks so much for mettle-testing the suite, and for opening the issue! You're right—it's frustrating to have lots of past test schemas cluttering up the db, and it's also frustrating to have those test schemas aggressively cleared when they'd be the best first stop for debugging.

I believe there already exists an optional --no-drop-schema flag that, when passed, tells the test suite not to drop the schema: https://github.com/fishtown-analytics/dbt-adapter-tests/blob/a411336e9493a38c228a876e19af24d70ec53d86/pytest_dbt_adapter/init.py#L13-L15

https://github.com/fishtown-analytics/dbt-adapter-tests/blob/a411336e9493a38c228a876e19af24d70ec53d86/pytest_dbt_adapter/spec_file.py#L659-L662

Would you be able to give that a go and see if it works as intended? If so, we should definitely document it in the README.

On top of that, I'm thinking about a -x/--fail-fast flag that would stop the testing suite at first sign of failure, and leave the database objects in place. Is that something you would have found helpful during adapter development?

[JLDLaughlin]

Awesome! The --no-drop-schema is exactly what I wanted, we'll try it out and report back here.

Also, -x/--fail-fast would be great addition! I also did this manually by commenting out tests in our .spec file.