kyua
kyua copied to clipboard
freebsd
Add Google Test support
This change adds fine grained execution of Google Test test programs.
First, the Google Test test is executed with --gtest_list_tests. Next,
based on the output from --gtest_list_tests, the testcases are run
individually. The output from each unique testcase is based on the
standard output it provides, per the test output protocol defined in
the GoogleTest docs on github [1], [2], and instrumented via various
demo programs I created, which can be found on GitHub
here.
This support is a very rough cut to provide an initial working integration effort. There're additional improvements that can be made by leveraging either the JSON or XML structured output format, instead of scraping standard output using beginning and ending sentinels to search for regular expressions. In order to do that though without reinventing the wheel, Kyua would need to rely on an external JSON or XML library.
This test interface supports fine grained execution of test programs
like the ATF test interface, but matches behavior of plain/TAP interfaces by
instead supporting metadata passing via $TEST_ENV_ prefixed environment
variables.
This support adds additional tests for verifying pass, fail, skip (will
be available in version 1.9.0 and is available in FreeBSD base's version
of Google Test), and disabled result determination, using mock output and a mock
test program (engine/googletest_helpers), for parity with other test formats
(ATF, plain, TAP). The helper test program purposely avoids relying on the
GoogleTest test infrastructure, in order to limit Kyua's dependencies.
- https://github.com/google/googletest/blob/master/googletest/docs/primer.md
- https://github.com/google/googletest/blob/master/googletest/docs/advanced.md
Signed-off-by: Enji Cooper [email protected]
:tada: One problem, though. When I use the new googletest test program, "kyua report" doesn't work.
> /home/somers/src/github/jmmv/kyua/kyua report
kyua: E: Column 'result_reason' is not a string (sqlite db: /home/somers/.kyua/store/results.usr_tests_sys_fs_fusefs.20190403-025838-736040.db).
🎉 One problem, though. When I use the new googletest test program, "kyua report" doesn't work.
> /home/somers/src/github/jmmv/kyua/kyua report kyua: E: Column 'result_reason' is not a string (sqlite db: /home/somers/.kyua/store/results.usr_tests_sys_fs_fusefs.20190403-025838-736040.db).
Hmmm... that's strange. I'll see if I can reproduce that.
"kyua report" still give me the same error even with your newest patch.
I suspect it's one of the cases involving the skipped or disabled status. Need to do some more digging here with your branch..
As I suspected, the issue is that GTEST_SKIP is being called without a reason:
$ sudo kyua test -k /usr/tests/sys/fs/fusefs/Kyuafile write:Write.append
write:Write.append -> skipped [0.017s]
Results file id is usr_tests_sys_fs_fusefs.20190405-171435-875837
Results saved to /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-171435-875837.db
1/1 passed (0 failed)
$ sudo kyua report --results-file=/root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-171435-875837.db
kyua: E: Column 'result_reason' is not a string (sqlite db: /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-171435-875837.db).
$ sudo kyua debug -k /usr/tests/sys/fs/fusefs/Kyuafile write:Write.append
Note: Google Test filter = Write.append
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Write
[ RUN ] Write.append
[ SKIPPED ] Write.append (13 ms)
[----------] 1 test from Write (13 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (13 ms total)
[ PASSED ] 0 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] Write.append
write:Write.append -> skipped
Wait... the fact that GTEST_SKIP is not properly emitting any messages is a bug in our (FreeBSD base) version of googletest too. Argh...
Hmmm... that didn't work :/...
$ kyua report --results-file=/root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-182049-970358.db
kyua: E: Column 'result_reason' is not a string (sqlite db: /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-182049-970358.db).
$ sqlite3 /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-182049-970358.db
SQLite version 3.27.1 2019-02-08 13:17:39
Enter ".help" for usage hints.
sqlite> SELECT * FROM test_results;
1|skipped|YOU HAVE 1 DISABLED TEST|1554488449986045|1554488449990167
2|skipped|YOU HAVE 1 DISABLED TEST|1554488449992589|1554488449997075
3|skipped|YOU HAVE 1 DISABLED TEST|1554488450000970|1554488450005545
4|skipped||1554488450007688|1554488450022533
5|passed||1554488450029400|1554488450039586
6|passed||1554488450042407|1554488450061054
7|passed||1554488450064575|1554488450077458
8|passed||1554488450080841|1554488450091814
9|passed||1554488450095204|1554488450107807
10|passed||1554488450110648|1554488450123577
11|passed||1554488450126588|1554488450139898
12|skipped||1554488450143172|1554488450153392
13|skipped||1554488450156506|1554488450165748
14|skipped||1554488450169125|1554488450178281
15|skipped||1554488450182709|1554488450193590
16|skipped|YOU HAVE 1 DISABLED TEST|1554488450196618|1554488450201177
17|skipped|YOU HAVE 1 DISABLED TEST|1554488450203280|1554488450207678
18|passed||1554488450209844|1554488450220627
As I suspected, the issue is that GTEST_SKIP is being called without a reason:
Shouldn't that be allowed? If you don't supply a reason to GTEST_SKIP, shouldn't it use a sane default, like ""?
Shouldn't that be allowed? If you don't supply a reason to
GTEST_SKIP, shouldn't it use a sane default, like""?
That’s what I’m trying to implement as a workaround, but it’s a waste of space on disk.
Ok. I'm starting to understand where the issue's coming from:
192 static model::test_result
193 parse_result(sqlite::statement& stmt, const char* type_column,
194 const char* reason_column)
195 {
196 try {
197 const model::test_result_type type =
198 store::column_test_result_type(stmt, type_column);
199 if (type == model::test_result_passed) {
200 if (stmt.column_type(stmt.column_id(reason_column)) !=
201 sqlite::type_null)
202 throw store::integrity_error("Result of type 'passed' has a "
203 "non-NULL reason");
204 return model::test_result(type);
205 } else {
206 return model::test_result(type,
207 stmt.safe_column_text(reason_column));
208 }
Well! It looks like store/write_transaction.cpp:425-426 does the field NULLing for me :D!
409 int64_t
410 store::write_transaction::put_result(const model::test_result& result,
411 const int64_t test_case_id,
412 const datetime::timestamp& start_time,
413 const datetime::timestamp& end_time)
414 {
415 try {
416 sqlite::statement stmt = _pimpl->_db.create_statement(
417 "INSERT INTO test_results (test_case_id, result_type, "
418 " result_reason, start_time, "
419 " end_time) "
420 "VALUES (:test_case_id, :result_type, :result_reason, "
421 " :start_time, :end_time)");
422 stmt.bind(":test_case_id", test_case_id);
423
424 store::bind_test_result_type(stmt, ":result_type", result.type());
425 if (result.reason().empty())
426 stmt.bind(":result_reason", sqlite::null());
427 else
428 stmt.bind(":result_reason", result.reason());
And... magic!
$ sudo kyua test -k /usr/tests/sys/fs/fusefs/Kyuafile write
write:AioWrite.DISABLED_aio_write -> skipped: YOU HAVE 1 DISABLED TEST [0.004s]
write:Write.DISABLED_direct_io_evicts_cache -> skipped: YOU HAVE 1 DISABLED TEST [0.004s]
write:Write.DISABLED_mmap -> skipped: YOU HAVE 1 DISABLED TEST [0.004s]
write:Write.append -> skipped [0.012s]
write:Write.append_direct_io -> passed [0.010s]
write:Write.direct_io_short_write -> passed [0.010s]
write:Write.direct_io_short_write_iov -> passed [0.012s]
write:Write.indirect_io_short_write -> passed [0.016s]
write:Write.write -> passed [0.011s]
write:Write.write_large -> passed [0.012s]
write:Write.write_nothing -> passed [0.010s]
write:WriteBack.close -> skipped [0.011s]
write:WriteBack.o_direct -> skipped [0.009s]
write:WriteBack.rmw -> skipped [0.010s]
write:WriteBack.writeback -> skipped [0.013s]
write:WriteThrough.DISABLED_update_file_size -> skipped: YOU HAVE 1 DISABLED TEST [0.007s]
write:WriteThrough.DISABLED_writethrough -> skipped: YOU HAVE 1 DISABLED TEST [0.004s]
write:WriteThrough.pwrite -> passed [0.010s]
Results file id is usr_tests_sys_fs_fusefs.20190405-223507-578754
Results saved to /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-223507-578754.db
18/18 passed (0 failed)
$ kyua report --results-file=/root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-223507-578754.db
===> Skipped tests
write:AioWrite.DISABLED_aio_write -> skipped: YOU HAVE 1 DISABLED TEST [0.004s]
write:Write.DISABLED_direct_io_evicts_cache -> skipped: YOU HAVE 1 DISABLED TEST [0.004s]
write:Write.DISABLED_mmap -> skipped: YOU HAVE 1 DISABLED TEST [0.004s]
write:Write.append -> skipped [0.012s]
write:WriteBack.close -> skipped [0.011s]
write:WriteBack.o_direct -> skipped [0.009s]
write:WriteBack.rmw -> skipped [0.010s]
write:WriteBack.writeback -> skipped [0.013s]
write:WriteThrough.DISABLED_update_file_size -> skipped: YOU HAVE 1 DISABLED TEST [0.007s]
write:WriteThrough.DISABLED_writethrough -> skipped: YOU HAVE 1 DISABLED TEST [0.004s]
===> Summary
Results read from /root/.kyua/store/results.usr_tests_sys_fs_fusefs.20190405-223507-578754.db
Test cases: 18 total, 10 skipped, 0 expected failures, 0 broken, 0 failed
Total time: 0.170s
@jmmv: I realize this is a large change, but I was wondering if there's a way that I could work with you to get this change reviewed and into the kyua project? I'm asking, because there's a chance that I might get a job in the next few months which will invalidate my individual CLA.
Plus, this change makes it markably easier to run Google Test programs in FreeBSD with Kyua.
@jmmv could you please review this change? If this gets merged and released before I finish my fusefs work, it will change how I organize the tests.
@emaste: given the lack of feedback in this PR, I think it's time to consider officially forking atf/kyua/lutok into a FreeBSD-org managed repo and install the ports from that repo. I'll reach out to you soon about this.