roboptim-core Add Windows (MSVC) support

Add Windows (MSVC) support

Open bchretien opened this issue 9 years ago • 30 comments

Some people are interested in using RobOptim on Windows, so this is worth investigating.

@kayusawa could you make a summary of your tests regarding the compilation of RobOptim on Windows? Thanks! :-)

Oct 07 '14 11:10 bchretien

:+1: Some details on the installation of the dependencies would also be appreciated.

Oct 08 '14 07:10 francois-keith

Indeed, we could then make a detailed installation process on the website.

Oct 08 '14 09:10 bchretien

Dear Dr.Crétien

Thank you for the e-mail. I could compile RobOpitim on Windows by using VisualStudio 2012 and Intel Compiler few month ago. For the compilation, I prepared several windows tools and libraries as follows:

CMake (2.8.12)
pkg_config
doxgen
eigen (3.2.2)
boost (1.55)
log4cxx (0.10.0)
libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself. I'm sorry that I don't really remember the details, but a few minor modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has to be changed from "utf8-without BOM" to "utf8-with BOM". (It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake will work properly without modification. However, if you plan to use VisualStudio, there may be a few conflicts about MSVC definitions. For example, "boost::mpl::size_t" and "unsigned int size_t" will be ambiguous when calling Eigen sparse matrix. Actually there are few errors in my case, so I modified Eigen codes directly.

If you have any question, please contact me.

Best regards, Ko Ayusawa

Ko AYUSAWA, researcher CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT, Intelligent Systems Research Institute, National Institute of Advanced Industrial Science and Technology(AIST) Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien [email protected]:

Some people are interested in using RobOptim on Windows, so this is worth investigating.

@kayusawa https://github.com/kayusawa could you make a summary of your tests regarding the compilation of RobOptim on Windows? Thanks! :-)

— Reply to this email directly or view it on GitHub https://github.com/roboptim/roboptim-core/issues/76.

Oct 08 '14 10:10 kayusawa

Dear Dr. Chrétien

I'm so sorry that I misspelled your name.

If you want to need the pre-compiled binaries or modified codes of the third-party libraries, I can prepare and send you.

Best regards,

e-mail: [email protected]

2014-10-08 19:35 GMT+09:00 Ko AYUSAWA [email protected]:

Dear Dr.Crétien

Thank you for the e-mail. I could compile RobOpitim on Windows by using VisualStudio 2012 and Intel Compiler few month ago. For the compilation, I prepared several windows tools and libraries as follows:

CMake (2.8.12)

pkg_config

doxgen

eigen (3.2.2)

boost (1.55)

log4cxx (0.10.0)

libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself. I'm sorry that I don't really remember the details, but a few minor modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has to be changed from "utf8-without BOM" to "utf8-with BOM". (It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake will work properly without modification. However, if you plan to use VisualStudio, there may be a few conflicts about MSVC definitions. For example, "boost::mpl::size_t" and "unsigned int size_t" will be ambiguous when calling Eigen sparse matrix. Actually there are few errors in my case, so I modified Eigen codes directly.

If you have any question, please contact me.

Best regards, Ko Ayusawa

Ko AYUSAWA, researcher CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT, Intelligent Systems Research Institute, National Institute of Advanced Industrial Science and Technology(AIST) Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien [email protected]:

Some people are interested in using RobOptim on Windows, so this is worth investigating.

@kayusawa https://github.com/kayusawa could you make a summary of your tests regarding the compilation of RobOptim on Windows? Thanks! :-)

— Reply to this email directly or view it on GitHub https://github.com/roboptim/roboptim-core/issues/76.

Oct 08 '14 10:10 kayusawa

@kayusawa Hi, I would be interested in testing roboptim on windows. Could you please send me (or put on a server) those pre-compiled libraries?

Thanks. Le 8 oct. 2014 12:43, "kayusawa" [email protected] a écrit :

Dear Dr. Chrétien

I'm so sorry that I misspelled your name.

If you want to need the pre-compiled binaries or modified codes of the third-party libraries, I can prepare and send you.

Best regards,

Ko AYUSAWA, researcher CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT, Intelligent Systems Research Institute, National Institute of Advanced Industrial Science and Technology(AIST) Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-08 19:35 GMT+09:00 Ko AYUSAWA [email protected]:

Dear Dr.Crétien

Thank you for the e-mail. I could compile RobOpitim on Windows by using VisualStudio 2012 and Intel Compiler few month ago. For the compilation, I prepared several windows tools and libraries as follows:

CMake (2.8.12)

pkg_config

doxgen

eigen (3.2.2)

boost (1.55)

log4cxx (0.10.0)

libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself. I'm sorry that I don't really remember the details, but a few minor modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has to be changed from "utf8-without BOM" to "utf8-with BOM". (It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake will work properly without modification. However, if you plan to use VisualStudio, there may be a few conflicts about MSVC definitions. For example, "boost::mpl::size_t" and "unsigned int size_t" will be ambiguous when calling Eigen sparse matrix. Actually there are few errors in my case, so I modified Eigen codes directly.

If you have any question, please contact me.

Best regards, Ko Ayusawa

Ko AYUSAWA, researcher CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT, Intelligent Systems Research Institute, National Institute of Advanced Industrial Science and Technology(AIST) Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien [email protected]:

Some people are interested in using RobOptim on Windows, so this is worth investigating.

@kayusawa https://github.com/kayusawa could you make a summary of your tests regarding the compilation of RobOptim on Windows? Thanks! :-)

— Reply to this email directly or view it on GitHub https://github.com/roboptim/roboptim-core/issues/76.

— Reply to this email directly or view it on GitHub https://github.com/roboptim/roboptim-core/issues/76#issuecomment-58339865 .

Oct 08 '14 11:10 francois-keith

Dear Ayusawa-san,

Thanks for the quick answer. Don't worry about my name, I know it's a difficult one :-)

For UTF-8 with BOM, this may be something that can be solved by CMake when generating the MSVC project files. If the compiler is the issue, a fix similar to this one may be what we need, especially if that means we do not need to add a BOM to all the files that contain UTF-8 characters.

For pre-compiled libraries, these could be released for our next major release. For the dependencies, I guess we could provide them (log4cxx/libtool), especially if some patching is involved.

For the Eigen error, is it something that happened in RobOptim or Eigen itself?

Oct 08 '14 12:10 bchretien

Dear Chrétien-san, Keith-san

For UTF-8 with BOM, this may be something that can be solved by CMake when

generating the MSVC project files. If the compiler is the issue, a fix similar to this one https://github.com/chewing/libchewing/blob/master/CMakeLists.txt#L42 may be what we need, especially if that means we do not need to add a BOM to all the files that contain UTF-8 characters.

Thank you for the advice. I agree that we should solve it by the configuration of CMake.

For pre-compiled libraries, these could be released for our next major

release. For the dependencies, I guess we could provide them (log4cxx/ libtool), especially if some patching is involved.

I attached the pre-compiled log4cxx and libtool (both x86 and x64). Please rename ".lib.dat" to ".lib" inside "install.zip" (This is because the security problem of AIST mailing service.) I'm sorry that the path setting of the package config file of log4cxx has to be modified manually.

I also attached the modified code of them. Almost all the modification of log4cxx is just the definition of namespace. The original windows libtool does not call loadlibrary properly on VC2012, so I tried to modify and create the windows solution file. I also attach the simple memo of those installation.

For the Eigen error, is it something that happened in RobOptim or Eigen

itself?

I'm sorry that I almost forgot about where the problem occured. But I guess "boost/mpl/size_t" and "unsigned int size_t" were ambiguous when creating Eigen matrix array in Function of RobOptim . When I used Intel Compiler, there are no such kind of problems. I guess there is a way to solve it by using MSVC macro to control them. But at that time, I had to compile RobOptim immediately, so I modified "size_t" to "::size_t" in several places in Eigen codes and planed to revert later.

If there are problems, please don't hesitate to contact me.

Best regards, Ko ayusawa

e-mail: [email protected]

2014-10-08 21:40 GMT+09:00 Benjamin Chrétien [email protected]:

Dear Ayusawa-san,

Thanks for the quick answer. Don't worry about my name, I know it's a difficult one :-)

For UTF-8 with BOM, this may be something that can be solved by CMake when generating the MSVC project files. If the compiler is the issue, a fix similar to this one https://github.com/chewing/libchewing/blob/master/CMakeLists.txt#L42 may be what we need, especially if that means we do not need to add a BOM to all the files that contain UTF-8 characters.

For pre-compiled libraries, these could be released for our next major release. For the dependencies, I guess we could provide them (log4cxx/ libtool), especially if some patching is involved.

For the Eigen error, is it something that happened in RobOptim or Eigen itself?

— Reply to this email directly or view it on GitHub https://github.com/roboptim/roboptim-core/issues/76#issuecomment-58350948 .

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

open VisualStudio comand prompt (x86)
change directory to boost top directory
type as follows: $ bootstrap $ b2 --build-dir=_build\x86 --stagedir=stage\x86 address-model=32 link=static,shared runtime-link=shared -j 4
if you have intel compiler... $ "c:\Program Files (x86)\Intel\Composer XE\bin\compilervars.bat" ia32 vs2012 $ b2 --toolset="intel" --build-dir=_build\x86 --stagedir=stage\x86 address-model=32 link=static,shared runtime-link=shared -j 4

1. open VisualStudio comand prompt (x64) 2. change directory to boost top directory 3. type as follows: $ bootstrap $ b2 --build-dir=_build\x64 --stagedir=stage\x64 address-model=64 link=static,shared runtime-link=shared -j 4 4. if you have intel compiler... $ "c:\Program Files (x86)\Intel\Composer XE\bin\compilervars.bat" intel64 vs2012 $ b2 --toolset="intel" --build-dir=_build\x64 --stagedir=stage\x64 address-model=64 link=static,shared runtime-link=shared -j 4

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// <log4cxx(MSVC2012)>

download as follows:
- apache-log4cxx-0.10.0.zip
- apr-1.5.1-win32-src.zip
- apr-iconv-1.2.1-win32-src-r2.zip
- apr-util-1.5.3-win32-src.zip
from http://logging.apache.org/log4cxx/ http://ftp.yz.yamagata-u.ac.jp/pub/network/apache/apr/
extract them as follows: log4cxx\appatch-log4cxx-0.10.0 log4cxx\apr log4cxx\apr-util log4cxx\apr-iconv
call as follows:
- configure.bat
- configure-aprutil.bat
open "log4cxx\apache-log4cxx-0.10.0\projects\log4cxx.dsw"
add "log4cxx\apr-iconv\apriconv.dsp" in the solution file.
replace files as follows:
- loggingevent.h
- loggingevent.cpp
- propertiespatternconverter.cpp
- xmllayout.cpp
build log4cxx
liblog4cxx.pc has to be modified manually.

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

1. open libltdl.sln 2. build libltdl

Oct 08 '14 14:10 kayusawa

Note regarding CI support for Windows: I recently noticed that some open-source projects now use AppVeyor, which could very roughly be seen as Travis for Windows.

May 21 '15 14:05 bchretien

In the latest stable there seems alot of issues with a windows build, that aren't dependencies.

Mar 01 '16 13:03 phuicy

Windows support is not available yet, although we are slowly getting there (cc @aescande).

Mar 01 '16 13:03 bchretien

We have worked out quite few issues, but are still working on a lot of templating issues, e.g.:

40>C:\git\roboptim-core\include\roboptim/core/problem.hxx(118): error C2491: 'roboptim::Problem<T>::function' : definition of dllimport function not allowed

Will keep you informed. Is there an old binary of an older commit?

Mar 01 '16 13:03 phuicy

Thanks for looking into this. Alas I don't think anyone shared any binary yet.

Mar 01 '16 13:03 bchretien

Just putting my 2 cents here.

I have tried to compile roboptim-core on windows very recently (I must be 1-2 commits late wrt master). Here are the errors I found (as I remember them, so this is not as precise as I'd like it to be).

The error you're describing definition of dllimport function not allowed is due to the fact that some methods / classes are now templated (but weren't in the last version tested). The error can be suppressed by removing this macro for those classes.
Some errors will also pop up because of the macro ROBOPTIM_CORE_**_DEPRECATED, that must be placed *before the function declaration in windows.
Some of the unit tests will not compile in VS2008 because of template issues (I cannot tell you which ones right here).
The format of the output is not the same as in unix (e.g., infinity is not written inf, but #INF, and the E notation takes an extra 0: 1e03 versus 1e003), so some unit tests may fail.

Those are some leads. I will push that on my fork (it is still a work in progress) tonight if you want to have a look.

Mar 01 '16 13:03 francois-keith

Awesome, this is exactly what we are seeing. I think to solve the 1st issue the ROBOPTIM_DLLAPI has to moved to a pre-declaration of the class., as:

// function definition
void __declspec(dllimport) funcB() {}   // C2491

// function declaration
void __declspec(dllimport) funcB();   // OK

number 2: We set

#define ROBOPTIM_CORE_**_DEPRECATED

Mar 01 '16 13:03 phuicy

I don't think it is required to keep the ROBOPTIM_DLLAPI macro for the templated class (e.g. Problem) since the definition of the method is in the .hxx and those files are installed as well. So there is no point AFAIK in exporting the symbols.

I compiled the unit tests as well as roboptim/roboptim-core-plugin-eigen without having any symbol issue.

Mar 01 '16 14:03 francois-keith

Indeed, I'll clean this up a bit. I recently added the possibility to precompile most of the templated classes, since they are now only templated on the matrix traits (dense/sparse). However this is disabled by default since explicit template instantiation is a C++11 feature. I guess the error will appear in some other classes.

Mar 01 '16 14:03 bchretien

ROBOPTIM_CORE_DEPRECATED (which is equivalent to __declspec (deprecated) on Windows with MSVC) has to be at the beginning of function declarations/typedefs?

Mar 01 '16 15:03 bchretien

Yes, otherwise some compilation issues may raise.

Mar 01 '16 15:03 francois-keith

Ok, we can easily fix that then. I was also considering making a PR to jrl-cmakemodules to have the deprecated macros support version numbers, e.g.:

#define ROBOPTIM_CORE_DEPRECATED(X) __attribute__ ((deprecated))
...
typedef Foo Bar ROBOPTIM_CORE_DEPRECATED(3.2);

They use this for Gazebo, that way you can see exactly when a given function was deprecated when you get the warning while compiling.

Mar 01 '16 15:03 bchretien

And talking about Gazebo, they had the exact same problem:

#if defined(__GNUC__)
#define GAZEBO_DEPRECATED(version) __attribute__((deprecated))
#define GAZEBO_FORCEINLINE __attribute__((always_inline))
#elif defined(_WIN32)
// GAZEBO_DEPRECATED should be defined as something like
// __declspec(deprecated), but it needs to go *before* the function name,
// and we're putting GAZEBO_DEPRECATED *after* the function.
#define GAZEBO_DEPRECATED(version)
#define GAZEBO_FORCEINLINE __forceinline
#else
#define GAZEBO_DEPRECATED(version) ()
#define GAZEBO_FORCEINLINE
#endif

Mar 01 '16 15:03 bchretien

Bear in mind, getting Gazebo on windows is still not an easy experience. But this looks good. That commit (992e734) looks very similar to what we got.

Mar 01 '16 16:03 phuicy

@phuicy I pushed on https://github.com/francois-keith/roboptim-core the current state of my windows version. I tested it on MSVC 2012, with the following deps:

boost 1.55
libtools 1.5.26
eigen 3.2.0

@bchretien No Pull request for now, mainly because of the test issue and the fact that the result file is different between the Win32 and UNIX systems.

Mar 02 '16 00:03 francois-keith

@francois-keith good catch on the bug recently added to boost.cmake (see jrl-umi3218/jrl-cmakemodules#76). As for the differences when printing, there are some ideas there. The _set_output_format trick could be added to our test fixtures, thus it would be automatically applied to all the RobOptim tests (core, plugins...) by simply changing a single file.

Mar 02 '16 01:03 bchretien

@francois-keith well apparently _set_output_format is obsolete, still any alternative workaround should probably be applied in our test fixtures.

Mar 02 '16 01:03 bchretien

Complementary information here:

The removal of _set_output_format is by design. The standards-conforming two-digit exponent mode is now the default and the non-conforming, previously-default, three-digit exponent format is no longer supported. This is noted in the draft Visual C++ 2015 breaking changes documentation: https://msdn.microsoft.com/en-us/library/bb531344(v=vs.140).aspx. We have not yet completed the full updates to the documentation for Visual Studio 2015.

So we can just check the Visual C++ version, and use _set_output_format if an old version is detected.

Mar 02 '16 04:03 bchretien

Awesome. Thank you

Mar 02 '16 09:03 phuicy

Does anyone know how MSVC behaves with explicit template instantiations and symbol exports? The weird symbol exports I had all around the code solved this problem, i.e. gcc and clang behave differently (either one returns a type attributes ignored after type is already defined error, or the other one does not export the symbols).

Mar 07 '16 12:03 bchretien

AFAIK, you can export the specialization of templated methods or methods of templated class. I've put an example of the syntax in this repo : francois-keith/test_template_export@34e2e12

To be honest, it was much more tricky in my memory (cf stack-of-tasks/dynamic-graph@5721ccb98), so maybe I'm missing the point / the difficulty here.

Mar 10 '16 01:03 francois-keith

The problem is that gcc and clang behave quite differently, so I was wondering if msvc exhibited the same problem.

The following version works (at least the export) in clang but not gcc:

// In the header
template <typename T>
class Problem { ... };
...
extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixDense>;
extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixSparse>;

// In the implementation cpp
template class Problem<EigenMatrixDense>;
template class Problem<EigenMatrixSparse>;

With clang and the explicit template instantiation:

$ nm -CD src/libroboptim-core.so | grep "W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()"
00000000000f5170 W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()

Without it, the symbol is not exported as expected.

gcc error:

roboptim-core/include/roboptim/core/problem.hxx:656:46: error: type attributes ignored after type is already defined [-Werror=attributes]
   extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixSparse>;
                                              ^

The following version works with both gcc and clang (at least the export), but will fail with msvc:

// In the header
template <typename T>
class ROBOPTIM_CORE_DLLAPI Problem { ... };
...
extern template class Problem<EigenMatrixDense>;
extern template class Problem<EigenMatrixSparse>;

// In the implementation cpp
template class Problem<EigenMatrixDense>;
template class Problem<EigenMatrixSparse>;

With gcc we finally get:

$ nm -CD src/libroboptim-core.so | grep "W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()"
00000000000f9e10 W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()

EDIT: removed ROBOPTIM_CORE_DLLEXPORT from implementation.

Mar 10 '16 01:03 bchretien

Experimental Windows support is currently available on the dev branch. The next step would be to validate the changes with a solver plugin.

May 30 '16 22:05 bchretien

roboptim-core roboptim-core copied to clipboard

Add Windows (MSVC) support

e-mail: [email protected]

Best regards,

e-mail: [email protected]

e-mail: [email protected]

Best regards,

e-mail: [email protected]

e-mail: [email protected]

e-mail: [email protected]

roboptim-core
roboptim-core copied to clipboard