swig
swig copied to clipboard
jaeandersson
Fix unit tests (was: Broken unit tests on OS X)
@ibell @KrisThielemans @tesch1 I finally got the unit test framework working but I get loads and loads of errors. The list above isn't even complete. How do you usually run the unit tests, via "make check-matlab-test-suite" from the command line? Do all the unit tests not marked as broken in Examples/test-suite/matlab/Makefile.in pass for you? It might be a OS X thing (I'm on OS X 10.10 with Clang++ and MATLAB 2014B).
(with the changes https://github.com/tesch1/swig/commit/86695f1a41eeb5169684ed373e7e5b49e56cc855) MacOS 10.9.5
$ make check-matlab-test-suite RUNPIPE=\>\>/tmp/xxx
$ grep skipping /tmp/xxx | wc -l
306
$ grep failed /tmp/xxx | wc -l
53
$ grep passed /tmp/xxx | wc -l
63
$ matlab | grep R201
R2011b (7.13.0.564) 64-bit (maci64)
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.0 (clang-600.0.54) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin13.4.0
Thread model: posix
306 that aren't even tried!!!
Here's the difference against your list... I can send you the log with the detailed errors if you want to look at it, there's no nice way to attach it here, and it seems kind of spammy to paste the whole 80k file into a comment.
$ diff -y -W 70 mtfailed jafailed
array_member | abstract_typedef2
> abstract_virtual
> allprotected_not
> arrays_global
> char_strings
> class_ignore
> constructor_copy
contract contract
cpp_enum cpp_enum
cpp_namespace cpp_namespace
default_constructor default_constructor
> dynamic_cast
exception_order exception_order
friends friends
> fvirtual
grouping grouping
import_nomodule import_nomodule
imports | inherit_missing
li_attribute li_attribute
li_carrays li_carrays
li_carrays | member_funcptr_galore
li_cmalloc | member_pointer
li_cpointer <
li_cpointer <
li_std_vector <
minherit minherit
mod | namespace_spaces
multi_import <
namespace_typemap namespace_typemap
nested_structs | namespace_virtual_method
null_pointer null_pointer
overload_extend overload_extend
overload_extend <
overload_rename overload_rename
overload_simple overload_simple
overload_template overload_template
preproc <
preproc_constants preproc_constants
> refcount
reference_global_vars reference_global_vars
return_const_value <
smart_pointer_extend <
smart_pointer_member <
smart_pointer_multi <
smart_pointer_multi_typedef <
smart_pointer_not <
smart_pointer_overload <
smart_pointer_rename <
smart_pointer_simple <
smart_pointer_templatevariable <
smart_pointer_typedef <
struct_value <
template_default_arg template_default_arg
template_static | template_inherit
template_type_namespace | template_specialization_defarg
template_typedef <
template_typedef_cplx <
template_typedef_cplx2 template_typedef_cplx2
template_typedef_import | typedef_inherit
unions | using_private
virtual_poly virtual_poly
voidtest voidtest
indeed. a lot of the test break at compilation stage, also on Linux. I believe I indicated accordingly on swig-devel.Quite a lot of them break because of directors, which are currently not supported. William has indicated however that he wants the code to compile cleanly anyway (I guess returning errors to matlab then). There's also a few others which get repeatedly flagged-up. So, I'm somewhat optimistic that with a small amount of work, the fraction of tests that will work at compilation stage will increase dramatically.
The "skipping" means that the test is only compiled, but there is no corresponding runme to do additional tests in matlab itself.
A large fraction of the runme.m come straight from conversion from the Octave ones with an automatic process. As the commit-log https://github.com/jaeandersson/swig/commit/b74e35eb9b9c28fb811ae865d3147c1bed2b26bc says some of these still need manual editing, e.g. because they use Octave's in-file function support.
There's also a large fraction of the tests that don't have a runme, but I guess not more than octave.
Finally, I had problems with R2014a on Linux actually running the tests themselves. When called from make, matlab crashed (but when executing the test manually, it all worked fine). That seemed to be a matlab issue, so I hadn't reported this one on the list yet and I haven't had time to investigate this further.
so... happy bug fixing!
@tesch1 @KrisThielemans Thanks a lot, that makes it clearer. It appears as if many (not all) of the tests fail more violently for me, with segfaults in MATLAB from which the test suite cannot recover. I will try to get MATLAB up and running in my Ubuntu Virtualbox where I have better debuggers. So yeah, bugfixing...
Strangely, I had the same problem on Linux -- tests that crash and hang the test-suite because MATLAB throws up a prompt at the crash. My linux matlab version is newer though, so it might have more to do with particular matlab versions' handling errors.
here's some - https://github.com/tesch1/swig/commit/b219d4ee46f35c170496fa34118bae94585e87cc
I had to disable matlab testing for my library because on windows on errors matlab would go interactive, and if you are always throwing errors, you end up with one copy of the interpreter per test, which very quickly eats up whatever RAM you can throw at it.
On Sat, Nov 29, 2014 at 5:23 AM, michael tesch [email protected] wrote:
here's some - tesch1@b219d4e https://github.com/tesch1/swig/commit/b219d4ee46f35c170496fa34118bae94585e87cc
— Reply to this email directly or view it on GitHub https://github.com/jaeandersson/swig/issues/6#issuecomment-64947459.
yes, I don't know how to force matlab to exit on certain errors.
How do you run these on Windows by the way?
Nope, not yet, what's the protocol for running tests again? I can set up my windows box to run the tests. Do you use buildbot?
On Sat, Nov 29, 2014 at 2:49 PM, Kris Thielemans [email protected] wrote:
yes, I don't know how to force matlab to exit on certain errors.
How do you run these on Windows by the way?
— Reply to this email directly or view it on GitHub https://github.com/jaeandersson/swig/issues/6#issuecomment-64963035.
Note that eventually, we need to get the unit tests running on travis-ci, since that's what SWIG uses. In our project, we're also migrating from buildbot to travis-ci, though for details you have to ask @jgillis.
Will it be enough to have the tests just build? I dont see any way to run MATLAB under travis-ci...(?)
Doesn’t travis just use the existing make+configure stuff?
This is a conversation for swig-devel really.
From: michael tesch [mailto:[email protected]] Sent: 07 December 2014 15:46 To: jaeandersson/swig Cc: Thielemans, Kris Subject: Re: [swig] Broken unit tests on OS X (#6)
Will it be enough to have the tests just build? I dont see any way to run MATLAB under travis-ci...(?)
— Reply to this email directly or view it on GitHubhttps://github.com/jaeandersson/swig/issues/6#issuecomment-65941398.
Current situation for me (Virtualbox with Ubuntu 14.04 64 bit guest, gcc-4.7, MATLAB 2014b):
- passed: 25 tests
- failed: 14 tests
- skipped: 123 tests
Only tests until overload_simple considered, at that point MATLAB segfaults.
Test framework running much more smoothly on my Mac and Clang, but the tests are only compiled, not run. Probably an issue with my setup - I have a new OS installation. Anyway, I don't get segfaults all the time anymore and the compilation errors I get look mostly harmless, as in missing features (in particular directors) or MATLAB symbols commencing with underscore. Will try again in Ubuntu and GCC.
not sure where those underscores can come from. I don't see that at all on Ubuntu.
If the tests aren't running matlab, I guess your matlab wasn't found properly at configure time?
Not sure, could be the MATLAB version perhaps (I use 2015a). Anyway, I'll try to get at least the generation step working in the next couple of weeks - should be pretty straightforward.
I get warnings like this:
Warning: invalid MATLAB symbol '__str__' (function)
Symbols may not start with '_'. Maybe try something like this: %rename(u__str__) __str__;
Warning: invalid MATLAB symbol '__str__' (function)
Symbols may not start with '_'. Maybe try something like this: %rename(u__str__) __str__;
But it looks like a legitimate warning.
I get the following error as well:
partialchecking matlab testcase template_default2
*** Internal error. Inconsistent clientdata for type 'oss::Module< oss::traits,oss::UnaryPolarization,oss::Interface_tpl< oss::UnaryPolarization > > *'
*** '"template_default2.Module_UP2"' != '"template_default2.Module_UP1"'
swig: Swig/typesys.c:1629: SwigType_remember_clientdata: Assertion `0' failed.
make[4]: *** [matlab_cpp] Aborted (core dumped)
make[3]: *** [template_default2.cpptest] Error 2
partialchecking matlab testcase template_default_arg
@wsfulton - any idea what this could be about?
And this:
partialchecking matlab testcase template_specialization_defarg
*** Internal error. Inconsistent clientdata for type 'C< double,double > *'
*** '"template_specialization_defarg.C_d"' != '"template_specialization_defarg.C_dd"'
swig: Swig/typesys.c:1629: SwigType_remember_clientdata: Assertion `0' failed.
make[4]: *** [matlab_cpp] Aborted (core dumped)
make[3]: *** [template_specialization_defarg.cpptest] Error 2
the str error is because some of the tests define a str function (it is used for instance in python where this function is used to provide a pretty-print mechanism). MATLAB doesn't allow such names and so the run-time test fails. This might be a topic for the swig-devel mailing list on how to handle this best.
the clientdata errors are tougher. They have to do with how we store the type info in the clientdata. I think this has to do with the search for base classes, but I'm not sure (and don't have the time to help right now)
I haven't reviewed any changes in this branch, but sounds like the core runtime might have changed or the symbol tables are modified in some way (neither of which is advisable unless you know what you are doing or run the changes over all the other languages first).
I am not aware of any changes to the core runtime. I guess that should be detected if I do a diff with master? I could try to pull the latest changes to master and see if the problem remains. Maybe we simply pulled a bad master revision.
I don't think the SwigType_remember_clientdata problems have anything to do with master, but are a matlab problem how we handle the clientdata.
After fixing the usage of error() (see https://github.com/jaeandersson/swig/issues/30), my current status on Linux, MATLAB 2012a, gcc 4.6.3:
$ make check-matlab-test-suite >& check.log $ grep skipped check.log | wc -l 0 $ grep passed check.log | wc -l 59 $ grep failed check.log | wc -l 121
still some director errors of course, the usage of _ in the test-case mentioned above, non-existent SWIG_Matlab_ConvertPacked and SWIG_Matlab_NewPackedObj. After that, it starts to become harder...
With Pull Request #41, the commit on my fork mentioned in Issue #30 and the fix in Issue #39, the status becomes a bit better:
$ grep passed check.log | wc -l 80 $ grep failed check.log | wc -l 58
Large remaining ones are SWIG_Matlab_ConvertPacked, usage of swig_this and some generated code that uses deref etc
Tests still do not run until completion for me (last one tested is multi_import. But I also get 80 passed tests, along with 47 failed ones.
I thought multi_import was the last one. It's the last of MULTI_CPP_TEST_CASES, which is the last of NOT_BROKEN_TEST_CASES as we don't have EXTRA_TEST_CASES. no?
a closer look by running e.g. partialcheck shows that when a runme.m doesn't exist, then the Makefile doesn't write "failed" or "passed" (see run_testcase in test-suite/matlab/Makefile). So we're having quite a lot more compile-only tests that work. This seems somewhat confusing though. Not sure how other languages handle this. Python seems to report if there's a run_test or not.
also a bit strange is that we have 160 runme.m files but less tests that report something. probably there's some files that I copied from Octave that aren't in common.mk.
If I run partialcheck I only get one test failing in an ugly way:
MacBook-Pro:matlab jaeandersson$ make -s template_specialization_defarg.cpptest
checking matlab testcase template_specialization_defarg
*** Internal error. Inconsistent clientdata for type 'C< double,double > *'
*** '"template_specialization_defarg.C_d"' != '"template_specialization_defarg.C_dd"'
Assertion failed: (0), function SwigType_remember_clientdata, file ../../Source/Swig/typesys.c, line 1629.
make[1]: *** [matlab_cpp] Abort trap: 6
make: *** [template_specialization_defarg.cpptest] Error 2
@wsfulton - any idea what could be the reason for this?