TileDB-MariaDB
TileDB-MariaDB copied to clipboard
TileDB-Inc
Add support for windows build
Sync'ing up some discussion about this w/ @bhadhy:
So, I reran the CI for this PR: https://github.com/TileDB-Inc/TileDB-MariaDB/pull/122 And I get a similar error about curl, except it fails earlier in the cmake detection step. One possibility is that the base directory path is too long. Another is that using TileDB itself as an externalproject does not quite work to build the sub-dependencies such as curl — I’ve never tried a build like that.
Could you try commenting/removing this line: https://github.com/TileDB-Inc/TileDB-MariaDB/blob/master/CMakeLists.txt#L67 And then do a clean cmake setup and build.
^ that will disable S3, which is the project that (immediately) needs curl. There might be another dependency in Azure, which you could also remove — let’s try to get a successful build going with minimal dependencies, and then we can figure out the problem with the deps.
Using the 0.10.0 release tag of TileDB-MariaDB, and turing off S3 and Azure build completes successfully. The steps are:
mkdir build_deps cd build_deps powershell -Command "Invoke-WebRequest https://netactuate.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-dep.zip -OutFile bison-2.4.1-dep.zip" powershell -Command "Invoke-WebRequest https://cfhcable.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-bin.zip -OutFile bison-2.4.1-bin.zip"
unzip bison-2.4.1-dep.zip unzip bison-2.4.1-bin.zip
git clone https://github.com/MariaDB/server.git -b 10.5 10.5 cd 10.5
git submodule add https://github.com/TileDB-Inc/TileDB-MariaDB.git storage/mytile
cd storage\mytile
git checkout tags/0.10.0 -b 0.10.0-branch
cd ....
git submodule update --init --recursive
Modify storage/mytile/CMakeLists.txt, -DTILEDB_S3=OFF, -DTILEDB_AZURE=OFF
mkdir builddir cd builddir
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Debug -SWITH_DEBUG=1 ..
cmake --build . 2>&1 | tee cmake.build.log
If I turn on S3, the build complains about long path names and fails. If I turn on Azure, there are additional issues:
- C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk\src\base64.cpp(212,45): error C3861: 'runtime_error': identifier not found It can be solved by adding #include stdexcept to base64.cpp
- However link fails looking for unresolved external symbol __imp___CrtDbgReportw
However link fails looking for unresolved external symbol __imp___CrtDbgReportw
This means there is a mismatch between the build types of some projects (eg some projects in debug mode, others release). I believe what's happening is the mytile/mariadb project is configured with -DCMAKE_BUILD_TYPE=Debug, but that setting is not passed in to the libtiledb externalproject. So libtiledb is building in Release. You could do either of these options:
- use
-DCMAKE_BUILD_TYPE=Releasefor the mytile project - add
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}on one of these lines of the mytile externalproject config for libtiledb. (for example how this should look, see this line in the main tiledb repo)
(the 2nd option is better for long-term maintainability, so would suggest making a PR with that change at some point - if this analysis is correct)
Regarding inclusion of stdexcept to base64.cpp. The https://github.com/Azure/azure-storage-cpplite/blob/master/src/base64.cpp already has the fix. However we are pulling 0.3.0 tag. And there is no newer tag. What do we do?
Regarding release vs debug build: I made the change you suggested to CMakeLists.txt by adding -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
The release build from top level built successfully. The steps were: cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --config Release 2>&1 | tee cmake.build.log
However doing the similar thing for debug ends up in cl /c /IC:\work\mariadb\10.5\builddir\include /IC:\work\mariadb\10.5\include /IC:\work\mariadb\10.5\sql /I"C:\work\mariadb\10.5\builddir\extra\pcre2\src\pcre2-build" /IC:\work\mariadb\10.5\builddir\extra\pcre2\src\pcre2\src /IC:\work\mariadb\10.5\builddir\extra\wolfssl /IC:\work\mariadb\10.5\extra\wolfssl\wolfssl /IC:\work\mariadb\10.5\extra\wolfssl\wolfssl\wolfssl /IC:\work\mariadb\10.5\zlib /IC:\work\mariadb\10.5\builddir\zlib /IC:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include /IC:\work\mariadb\10.5\storage\mytile\mytile /Zi /W3 /WX- /diagnostics:caret /MP /Od /Ob0 /D WIN32 /D _WINDOWS /D ENABLED_DEBUG_SYNC /D MYSQL_DYNAMIC_PLUGIN /D WIN /D _CRT_SECURE_NO_DEPRECATE /D _WIN32_WINNT=0x0A00 /D NOMINMAX /D WIN32_LEAN_AND_MEAN /D _CRT_NONSTDC_NO_WARNINGS /D DBUG_TRACE /D HAVE_CONFIG_H /D "CMAKE_INTDIR="Debug"" /D mytile_EXPORTS /D _WINDLL /D _MBCS /Gm- /EHsc /RTC1 /MT /GS /fp:precise /permissive- /Zc:wchar_t /Zc:forScope /Zc:inline /GR /std:c++14 /Fo"mytile.dir\Debug\" /Fd"mytile.dir\Debug\vc142.pdb" /Gd /TP /errorReport:queue /we4099 /we4700 /we4311 /we4477 /we4302 /we4090 /d2OptimizeHugeFunctions -Wno-error=non-virtual-dtor C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc "C:\work\mariadb\10.5\storage\mytile\mytile\mytile-discovery.cc" "C:\work\mariadb\10.5\storage\mytile\mytile\mytile-metadata.cc" "C:\work\mariadb\10.5\storage\mytile\mytile\mytile-range.cc" "C:\work\mariadb\10.5\storage\mytile\mytile\mytile-sysvars.cc" C:\work\mariadb\10.5\storage\mytile\mytile\mytile.cc C:\work\mariadb\10.5\storage\mytile\mytile\utils.cc cl : command line error D8021: invalid numeric argument '/Wno-error=non-virtual-dtor' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
CMakeLists.txt has following line. SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")
If I comment that out, I get following error C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: throw std::runtime_error("Attempting to set type with incorrect size."); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: ^ (compiling source file C:\work\mariadb\10.5\storage\mytile\mytile\utils.cc) [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
So I added #include stdexcept to type.h
Next I get following error C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2121: '#': invalid character: possibly the result of a macro expansion [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2121: HA_DO_INDEX_COND_PUSHDOWN); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2121: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: syntax error: 'if' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: HA_DO_INDEX_COND_PUSHDOWN); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2468,1): fatal error C1019: unexpected #else [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2468,1): fatal error C1019: #else [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj] C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2468,1): fatal error C1019: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
So I changed it to: ulong tile::mytile::index_flags(uint idx, uint part, bool all_parts) const { DBUG_ENTER("tile::mytile::index_flags"); #if MYSQL_VERSION_ID < 100500 DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE | HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN | HA_DO_INDEX_COND_PUSHDOWN); #else DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE | HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN | HA_DO_INDEX_COND_PUSHDOWN | HA_CLUSTERED_INDEX); #endif }
Now I am getting linker error LINK : fatal error LNK1104: cannot open file 'externals\install\lib\tiledb.dll.2.4' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
CMakeLists.txt has following line.
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")
This option is gcc-only, so it needs to be guarded like this:
if(MSVC)
set(CMAKE_CXX_FLAGS ...)
endif()
Also, just to double-check: when you run the cmake build with CMAKE_BUILD_TYPE=Debug, is it in a clean directory? If not, please try to start in a clean build directory (MSVC tries to separate debug vs release but it often doesn't really work with inter-dependent projects)
I changed storage/mytile/CMakeLists.txt to have MSVC guard as you suggested:
if(NOT MSVC)
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")
endif()
Then I also changed the TILEDB_LIBRARIES for MSVC so that it points to .lib instead of .dll.2.4
if(MSVC)
SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_STATIC_LIBRARY_SUFFIX}")
else()
SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_SHARED_LIBRARY_SUFFIX}.${TILEDB_VERSION_MAJOR}.${TILEDB_VERSION_MINOR}")
endif()
message(STATUS "TILEDB_LIBRARIES: ${TILEDB_LIBRARIES}")
Now the error I am seeing is following:
mytile-metadata.obj : error LNK2001: unresolved external symbol "class Time_zone * my_tz_OFFSET0" (?my_tz_OFFSET0@@3PEAVTime_zone@@EA) [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
mytile.obj : error LNK2001: unresolved external symbol "class Time_zone * my_tz_OFFSET0" (?my_tz_OFFSET0@@3PEAVTime_zone@@EA) [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
mytile.obj : error LNK2001: unresolved external symbol my_charset_ucs2_bin [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
mytile.obj : error LNK2001: unresolved external symbol my_charset_utf16_bin [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
mytile.obj : error LNK2001: unresolved external symbol my_charset_utf32_bin [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\builddir\storage\mytile\Debug\ha_mytile.dll : fatal error LNK1120: 4 unresolved externals [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
Google search makes me think that these symbols are defined in MYSQL. How do we add mysql library to the list of libraries to be linked?
Do you know what is "-SWITH_DEBUG=1"? It is specified in Line 66 https://github.com/TileDB-Inc/TileDB-MariaDB/blob/erica/ch922/windows-build/scripts/azure-windows.yml
Once again in storage/mytile/CMakeLists.txt, I dropped the MODULE_ONLY
if(WITH_EMBEDDED_SERVER)
MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
# Else build as a dynamic library
else()
# MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT MODULE_ONLY RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
endif()
Now the build goes past the link. It ends in following error while creating initial database:
Hardlink created for mysqld.exe <<===>> mariadbd.exe
Generating initdb.dep
Executing C:/work/mariadb/10.5/builddir/sql/Debug/mariadbd.exe --no-defaults --console --bootstrap --datadir=.
input file bootstrap.sql, current directory C:/work/mariadb/10.5/builddir/sql/data
CMake Error at C:/work/mariadb/10.5/cmake/create_initial_db.cmake:68 (MESSAGE):
Could not create initial database
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(238,5): error MSB8066: Custom build for 'C:\work\mariadb\10.5\builddir\CMakeFiles\bcccb9ef86ddfbdefa040957b508ecb6\initdb.dep.rule;C:\work\mariadb\10.5\builddir\CMakeFiles\af6190842cf923122927623d5a3c966f\initial_database.rule' exited with code 1. [C:\work\mariadb\10.5\builddir\sql\initial_database.vcxproj]
Do you know what is "-SWITH_DEBUG=1"? It is specified in Line 66 https://github.com/TileDB-Inc/TileDB-MariaDB/blob/erica/ch922/windows-build/scripts/azure-windows.yml
That sets whether to build in debug mode I guess, but it looks like it only applies to gcc: https://github.com/MariaDB/server/blob/64f7dffcc7e0e69c31d9a36c2090a26300e57c4c/storage/mroonga/CMakeLists.txt#L322-L330
Now the build goes past the link.
Good news, hopefully!
It ends in following error while creating initial database:
Is this part of the test suite?
Do you know what is "-SWITH_DEBUG=1"? It is specified in Line 66 https://github.com/TileDB-Inc/TileDB-MariaDB/blob/erica/ch922/windows-build/scripts/azure-windows.yml
That sets whether to build in debug mode I guess, but it looks like it only applies to gcc: https://github.com/MariaDB/server/blob/64f7dffcc7e0e69c31d9a36c2090a26300e57c4c/storage/mroonga/CMakeLists.txt#L322-L330
So it is really -SWITH_DEBUG and not -DWITH_DEBUG ?
So it is really -SWITH_DEBUG and not -DWITH_DEBUG ?
Ah - it should be -DWITH_DEBUG if you want to set it. I don't think -SWITH_DEBUG would have any effect (it might even break something b/c -S has a different meaning for cmake)
So it is really -SWITH_DEBUG and not -DWITH_DEBUG ?
Ah - it should be
-DWITH_DEBUGif you want to set it. I don't think-SWITH_DEBUGwould have any effect (it might even break something b/c-Shas a different meaning for cmake)
-SWITH_DEBUG is listed in the docs: https://mariadb.com/docs/reference/mdb/system-variables/debug/
-SWITH_DEBUG is listed in the docs: https://mariadb.com/docs/reference/mdb/system-variables/debug/
Far be it for me to argue with their docs, but I don't see how that would work. AFACT it's a cmake option here, and cmake -S sets the source path:
Usage
cmake [options] <path-to-source>
cmake [options] <path-to-existing-build>
cmake [options] -S <path-to-source> -B <path-to-build>
Specify a source directory to (re-)generate a build system for it in the
current working directory. Specify an existing build directory to
re-generate its build system.
Options
-S <path-to-source> = Explicitly specify a source directory.
-B <path-to-build> = Explicitly specify a build directory.
...
elsewhere it's documented as -DWITH_DEBUG: https://mariadb.com/kb/en/server-system-variables/#debugdebug_dbug
(that said, doesn't seem like the main problem here)
I am sorry. Looks like the same problems exists in both Release or Debug builds. I started from a completely clean folder to do a release build and encountered the same set of issues as I did for the debug build. Here are the steps I followed.
- Set environment
@echo off
set PATH=C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;%PATH%
set PATH=C:\work\utils\strawberry-perl-5.32.1.1-64bit\perl\bin;%PATH%
set PATH=C:\work\utils\diffutils-2.8.7-1-bin;%PATH%
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
set PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;%PATH%
- Add build dependencies
mkdir build_deps
cd build_deps
powershell -Command "Invoke-WebRequest https://netactuate.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-dep.zip -OutFile bison-2.4.1-dep.zip"
powershell -Command "Invoke-WebRequest https://cfhcable.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-bin.zip -OutFile bison-2.4.1-bin.zip"
unzip bison-2.4.1-dep.zip
unzip bison-2.4.1-bin.zip
set PATH=%PATH%;C:\work\mariadb\build_deps\bin
- checkout mariadb and tiledb-mariadb
git clone https://github.com/MariaDB/server.git -b 10.5 10.5
cd 10.5
git submodule add https://github.com/TileDB-Inc/TileDB-MariaDB.git storage/mytile
cd storage\mytile
git checkout tags/0.10.0 -b 0.10.0-branch
cd ..\..
git submodule update --init --recursive
3.5 Make following changes to storage\mytile\CMakeLists.txt
(i) Do not use with MSVC
if(NOT MSVC)
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor")
endif()
(ii) Propogate build type (do not mix debug and release)
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
(iii) Build without AWS
-DTILEDB_S3=OFF
(iv) With MSVC specify static libraries
if(MSVC)
SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_STATIC_LIBRARY_SUFFIX}")
else()
SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_SHARED_LIBRARY_SUFFIX}.${TILEDB_VERSION_MAJOR}.${TILEDB_VERSION_MINOR}")
endif()
(v) Remove MODULE_ONLY from MYSQL_ADD_PLUGIN
MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES})
Or we could probably say -DWITH_EMBEDDED_SERVER
- Debug build
mkdir builddir
cd builddir
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Debug -DWITH_DEBUG=1 ..
cmake --build . --config Debug 2>&1 | tee cmake.build.log
- Release build
mkdir builddir
cd builddir
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --config Release 2>&1 | tee cmake.build.log
Bugs: (i) base64.cpp
C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk\src\base64.cpp(108,28): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk-build\azure-storage-lite.vcxproj] [C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\ep_azuresdk.vcxproj]
Why it is not being patched?
Need to add
#include <stdexcept>
(ii) type.h
C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
Need to add
#include <stdexcept>
(iii) ha_mytile.cc
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: syntax error: 'if' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: HA_DO_INDEX_COND_PUSHDOWN); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
Fix it
#if MYSQL_VERSION_ID < 100500
DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE |
HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN |
HA_DO_INDEX_COND_PUSHDOWN);
#else
DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE |
HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN |
HA_DO_INDEX_COND_PUSHDOWN | HA_CLUSTERED_INDEX);
#endif
(iv) Could not create initial database Executing C:/work/mariadb/10.5/builddir/sql/Release/mariadbd.exe --no-defaults --console --bootstrap --datadir=. input file bootstrap.sql, current directory C:/work/mariadb/10.5/builddir/sql/data CMake Error at C:/work/mariadb/10.5/cmake/create_initial_db.cmake:68 (MESSAGE): Could not create initial database
Here are the steps I followed.
- Set environment
@echo off
set PATH=C:\Program Files\Git\bin;C:\Program Files\Git\usr\bin;%PATH%
set PATH=C:\work\utils\strawberry-perl-5.32.1.1-64bit\perl\bin;%PATH%
set PATH=C:\work\utils\diffutils-2.8.7-1-bin;%PATH%
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" x64
set PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;%PATH%
2. Add build dependencies
mkdir build_deps cd build_deps powershell -Command "Invoke-WebRequest https://netactuate.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-dep.zip -OutFile bison-2.4.1-dep.zip" powershell -Command "Invoke-WebRequest https://cfhcable.dl.sourceforge.net/project/gnuwin32/bison/2.4.1/bison-2.4.1-bin.zip -OutFile bison-2.4.1-bin.zip" unzip bison-2.4.1-dep.zip unzip bison-2.4.1-bin.zip
set PATH=%PATH%;C:\work\mariadb\build_deps\bin
3. checkout mariadb and tiledb-mariadb
git clone https://github.com/MariaDB/server.git -b 10.5 10.5
cd 10.5
git submodule add https://github.com/TileDB-Inc/TileDB-MariaDB.git storage/mytile
cd storage\mytile
git checkout tags/0.10.0 -b 0.10.0-branch
cd ..\..
git submodule update --init --recursive
3.5 Make following changes to storage\mytile\CMakeLists.txt
(i) Do not use with MSVC if(NOT MSVC) SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=non-virtual-dtor") endif()
(ii) Propogate build type (do not mix debug and release) -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} (iii) Build without AWS -DTILEDB_S3=OFF (iv) With MSVC specify static libraries if(MSVC) SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_STATIC_LIBRARY_SUFFIX}") else() SET(TILEDB_LIBRARIES "${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_SHARED_LIBRARY_SUFFIX}.${TILEDB_VERSION_MAJOR}.${TILEDB_VERSION_MINOR}") endif()
(v) Remove MODULE_ONLY from MYSQL_ADD_PLUGIN for both "WITH_EMBEDDED_SERVER" and "not WITH_EMBEDDED_SERVER" MYSQL_ADD_PLUGIN(mytile ${SOURCE_FILES} STORAGE_ENGINE DEFAULT RECOMPILE_FOR_EMBEDDED LINK_LIBRARIES ${TILEDB_LIBRARIES}) Building by specifying -DWITH_EMBEDDED_SERVER at top level had a lot of issues.
(vi) Copy tiledb.dll to buildir/sql/Release. It is needed by builddir/sql/Release/mariadbd.exe to create initial database. So that we do not have to have the dll in path. add_custom_command(TARGET tiledb POST_BUILD COMMAND ${CMAKE_COMMAND} -E copy ${EP_INSTALL_PREFIX}/${CMAKE_INSTALL_BINDIR}/${CMAKE_SHARED_LIBRARY_PREFIX}tiledb${CMAKE_SHARED_LIBRARY_SUFFIX} ${CMAKE_BINARY_DIR}/sql/Release )
- Debug build
mkdir builddir
cd builddir
cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Debug -DWITH_DEBUG=1 ..
cmake --build . --config Debug 2>&1 | tee cmake.build.log
5. Release build
mkdir builddir cd builddir cmake -DPLUGIN_TOKUDB=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_SPIDER=NO -DPLUGIN_SPHINX=NO -DPLUGIN_FEDERATED=NO -DPLUGIN_FEDERATEDX=NO -DPLUGIN_CONNECT=NO -DCMAKE_BUILD_TYPE=Release .. cmake --build . --config Release 2>&1 | tee cmake.build.log
Bugs:
(i) base64.cpp
C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk\src\base64.cpp(108,28): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\externals\src\ep_azuresdk-build\azure-storage-lite.vcxproj] [C:\work\mariadb\10.5\builddir\storage\mytile\externals\src\tiledb-build\ep_azuresdk.vcxproj]
Why it is not being patched?
Need to add
#include
(ii) type.h
C:\work\mariadb\10.5\builddir\storage\mytile\externals\install\include\tiledb\type.h(344,18): error C2039: 'runtime_error': is not a member of 'std' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
Need to add
#include
(iii) ha_mytile.cc
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: syntax error: 'if' [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: HA_DO_INDEX_COND_PUSHDOWN); [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
C:\work\mariadb\10.5\storage\mytile\mytile\ha_mytile.cc(2467,1): error C2059: ^ [C:\work\mariadb\10.5\builddir\storage\mytile\mytile.vcxproj]
Fix it
#if MYSQL_VERSION_ID < 100500 DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE | HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN | HA_DO_INDEX_COND_PUSHDOWN); #else DBUG_RETURN(HA_READ_NEXT | HA_READ_PREV | HA_READ_ORDER | HA_READ_RANGE | HA_KEYREAD_ONLY | HA_DO_RANGE_FILTER_PUSHDOWN | HA_DO_INDEX_COND_PUSHDOWN | HA_CLUSTERED_INDEX); #endif
- Create a package
cmake --build . --config Release --target package 2>&1 | tee cmake.package.log
Copy package mariadb-10.5.13-winx64.zip from C:\work\mariadb\10.5\builddir to C:\work\mariadb
Extract the contents to C:\work\mariadb\mariadb-10.5.13-winx64
Manually copy C:\work\mariadb\10.5\builddir\sql\Release\tiledb.dll to C:\work\mariadb\mariadb-10.5.13-winx64\bin
7. Open a new command prompt. Put the installation in path.
set PATH=C:\work\mariadb\mariadb-10.5.13-winx64\bin;%PATH%
- Create a default database
$ mysql_install_db.exe --datadir=c:\work\mariadb\db --default-user --password=secret
Running bootstrap
2021-09-23 9:35:59 0 [Note] C:\work\mariadb\mariadb-10.5.13-winx64\bin\mysqld.exe (mysqld 10.5.13-MariaDB) starting as process 31640 ...
Setting root password
Creating my.ini file
Creation of the database was successful
9. Modify the c:\work\mariadb\db\my.ini to following
[mysqld] port=3306 plugin-maturity=experimental #plugin-load-add=ha_mytile #Enable embedded mytile plugin-mytile=ON #default-storage-engine=MyTile mytile_delete_arrays=ON #default-time-zone=UTC
sql-mode=NO_ENGINE_SUBSTITUTION explicit-defaults-for-timestamp=1
#The following is to get rid of the harmless #"Deadlock found when trying to get lock" errors, see MDEV-12285. log-warnings=1
datadir=c:/work/mariadb/db [client] plugin-dir=C:/work/mariadb/mariadb-10.5.13-winx64/lib/plugin
- Start the mariadbd server
$ mariadbd.exe --defaults-file=c:\work\mariadb\db\my.ini --console
2021-09-23 9:47:15 0 [Note] mariadbd.exe (mysqld 10.5.13-MariaDB) starting as process 34736 ...
2021-09-23 9:47:15 0 [Note] InnoDB: Uses event mutexes
2021-09-23 9:47:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-09-23 9:47:15 0 [Note] InnoDB: Number of pools: 1
2021-09-23 9:47:15 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-09-23 9:47:15 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2021-09-23 9:47:15 0 [Note] InnoDB: Completed initialization of buffer pool
2021-09-23 9:47:15 0 [Note] InnoDB: 128 rollback segments are active.
2021-09-23 9:47:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-09-23 9:47:15 0 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-09-23 9:47:15 0 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB.
2021-09-23 9:47:15 0 [Note] InnoDB: 10.5.13 started; log sequence number 45106; transaction id 20
2021-09-23 9:47:15 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-09-23 9:47:15 0 [Note] InnoDB: Loading buffer pool(s) from c:\work\mariadb\db\ib_buffer_pool
2021-09-23 9:47:15 0 [Note] Server socket created on IP: '::'.
2021-09-23 9:47:15 0 [Note] InnoDB: Buffer pool(s) load completed at 210923 9:47:15
2021-09-23 9:47:15 0 [Note] mariadbd.exe: ready for connections.
Version: '10.5.13-MariaDB' socket: '' port: 3306 Source distribution
2021-09-23 9:55:38 4 [Note] TILEDB_POSITIVE_DELTA_MAX_WINDOW=128
11. Open another command prompt. Put the installation in path.
set PATH=C:\work\mariadb\mariadb-10.5.13-winx64\bin;%PATH%
- Use mariadb to connect to server and test
$ mariadb --user=root --password=secret
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 4
Server version: 10.5.13-MariaDB Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> CREATE DATABASE bookstore;
Query OK, 1 row affected (0.003 sec)
MariaDB [(none)]> USE bookstore;
Database changed
MariaDB [bookstore]> create table test_array ( dim0 integer DIMENSION=1 lower_bound="0" upper_bound="100" tile_extent="10", dim1 integer DIMENSION=1 lower_bound="0" upper_bound="100" tile_extent="10", attr1 varchar(255) filters="GZIP=-1" ) engine=MyTile array_type='SPARSE' coordinate_filters="NONE" offset_filters="POSITIVE_DELTA=128";
Query OK, 0 rows affected (0.021 sec)
MariaDB [bookstore]> insert into test_array (dim0, dim1, attr1) values (1, 1, "cell 1"), (1, 2, "cell 2"), (2, 1, "cell 3");
Query OK, 3 rows affected (0.036 sec)
Records: 3 Duplicates: 0 Warnings: 0
MariaDB [bookstore]> select * from test_array;
+------+------+--------+
| dim0 | dim1 | attr1 |
+------+------+--------+
| 1 | 1 | cell 1 |
| 1 | 2 | cell 2 |
| 2 | 1 | cell 3 |
+------+------+--------+
3 rows in set (0.012 sec)
MariaDB [bookstore]>
