server icon indicating copy to clipboard operation
server copied to clipboard

MDEV-30432: Refactor connect to use libcurl instead of cpprestsdk/curl

Open an3l opened this issue 1 year ago • 2 comments

  • Windows BB of this MDEV depends on MDEV-33420 so apply changes of PR#3068 before
  • Remove GetRest occurance (cpprestsdk references)
  • Drop execv(curl) references
  • Link to libcurl (Unix/Windows)
  • Use IMPORTED target, so that INCLUDE_DIRECTORIES be called implicity
  • Add libcurl feature to ConnectSE to issue HTPP requests

Check linkage

  • The libcurl is linked in dynamic library
$ ldd storage/connect/ha_connect.so |grep libcurl
	libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007fd815d28000)

Check linkage on Windows

  • MDBF-644
>> cmake ../server -A x64 -DPLUGIN_ROCKSDB=NO -DMYSQL_MAINTAINER_MODE=ERR -Wno-dev -DCURL_LIBRARY=C:/ProgramData/chocolatey/lib/curl/tools/curl-7.85.0-win64-mingw/lib/libcurl.dll.a -DCURL_INCLUDE_DIR=C:/ProgramData/chocolatey/lib/curl/tools/curl-7.85.0-win64-mingw/include
-- Found CURL: C:/ProgramData/chocolatey/lib/curl/tools/curl-7.85.0-win64-mingw/lib/libcurl.dll.a (found version "7.85.0")
The following features have been enabled:
 * CONNECT_REST, Support for REST API in the CONNECT storage engine
 
>> # Only building connect module 
>> cmake --build . --verbose --config Debug --target connect -- -m
>> dumpbin /dependents storage\connect\Debug\ha_connect.dll
Image has the following dependencies:
    ODBC32.dll
    IPHLPAPI.DLL
    libcurl-x64.dll  # <<< here we are << 
    server.dll
    KERNEL32.dll
    USER32.dll
    ole32.dll
    OLEAUT32.dll
    ADVAPI32.dll
    WS2_32.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-filesystem-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-environment-l1-1-0.dll
    api-ms-win-crt-locale-l1-1-0.dll
    api-ms-win-crt-utility-l1-1-0.dll

Check without libcurl

Remove the library and check the patch
Install DB and run server locally :heavy_check_mark:
$ sudo apt remove libcurl4-openssl-dev
$ cmake --build . --target connect
[  0%] Built target uca-dump
[  8%] Built target mysqlservices
[  8%] Built target GenUnicodeDataSource
[ 58%] Built target mysys
[ 75%] Built target strings
[ 75%] Built target dbug
[ 75%] Built target comp_err
[ 75%] Built target GenError
make[3]: *** No rule to make target '/usr/lib/x86_64-linux-gnu/libcurl.so', needed by 'storage/connect/ha_connect.so'.  Stop.
make[3]: *** Waiting for unfinished jobs....
[ 75%] Building CXX object storage/connect/CMakeFiles/connect.dir/tabrest.cpp.o
/home/anel/GitHub/mariadb/server/src/connect-curl-11.4/storage/connect/tabrest.cpp:39:10: fatal error: curl/curl.h: No such file or directory
   39 | #include 
      |          ^~~~~~~~~~~~~
compilation terminated.

Check in container

  • This patch closes MDEV-26727 (tested with Docker):
buildbot@0013f8b295b9:/$ sudo mariadb --user root
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 11.4.0-MariaDB-1:11.4.0+maria~deb11 mariadb.org binary distribution
MariaDB [test]> create table cusers3 engine=connect http='http://jsonplaceholder.typicode.com/users' table_type=json;
Query OK, 0 rows affected, 1 warning (0.179 sec)

MariaDB [test]> select * from cusers3;
+----+--------------------------+------------------+---------------------------+-------------------+---------------+----------------+-----------------+-----------------+-----------------+-----------------------+---------------+--------------------+------------------------------------------+--------------------------------------+
| id | name                     | username         | email                     | address_street    | address_suite | address_city   | address_zipcode | address_geo_lat | address_geo_lng | phone                 | website       | company_name       | company_catchPhrase                      | company_bs                           |
+----+--------------------------+------------------+---------------------------+-------------------+---------------+----------------+-----------------+-----------------+-----------------+-----------------------+---------------+--------------------+------------------------------------------+--------------------------------------+
|  1 | Leanne Graham            | Bret             | [email protected]         | Kulas Light       | Apt. 556      | Gwenborough    | 92998-3874      | -37.3159        | 81.1496         | 1-770-736-8031 x56442 | hildegard.org | Romaguera-Crona    | Multi-layered client-server neural-net   | harness real-time e-markets          |
|  2 | Ervin Howell             | Antonette        | [email protected]         | Victor Plains     | Suite 879     | Wisokyburgh    | 90566-7771      | -43.9509        | -34.4618        | 010-692-6593 x09125   | anastasia.net | Deckow-Crist       | Proactive didactic contingency           | synergize scalable supply-chains     |
|  3 | Clementine Bauch         | Samantha         | [email protected]        | Douglas Extension | Suite 847     | McKenziehaven  | 59590-4157      | -68.6102        | -47.0653        | 1-463-123-4447        | ramiro.info   | Romaguera-Jacobson | Face to face bifurcated interface        | e-enable strategic applications      |
|  4 | Patricia Lebsack         | Karianne         | [email protected] | Hoeger Mall       | Apt. 692      | South Elvis    | 53919-4257      | 29.4572         | -164.2990       | 493-170-9623 x156     | kale.biz      | Robel-Corkery      | Multi-tiered zero tolerance productivity | transition cutting-edge web services |
|  5 | Chelsey Dietrich         | Kamren           | [email protected]  | Skiles Walks      | Suite 351     | Roscoeview     | 33263           | -31.8129        | 62.5342         | (254)954-1289         | demarco.info  | Keebler LLC        | User-centric fault-tolerant solution     | revolutionize end-to-end systems     |
|  6 | Mrs. Dennis Schulist     | Leopoldo_Corkery | [email protected]   | Norberto Crossing | Apt. 950      | South Christy  | 23505-1337      | -71.4197        | 71.7478         | 1-477-935-8478 x6430  | ola.org       | Considine-Lockman  | Synchronised bottom-line interface       | e-enable innovative applications     |
|  7 | Kurtis Weissnat          | Elwyn.Skiles     | [email protected]    | Rex Trail         | Suite 280     | Howemouth      | 58804-1099      | 24.8918         | 21.8984         | 210.067.6132          | elvis.io      | Johns Group        | Configurable multimedia task-force       | generate enterprise e-tailers        |
|  8 | Nicholas Runolfsdottir V | Maxime_Nienow    | [email protected]      | Ellsworth Summit  | Suite 729     | Aliyaview      | 45169           | -14.3990        | -120.7677       | 586.493.6943 x140     | jacynthe.com  | Abernathy Group    | Implemented secondary concept            | e-enable extensible e-tailers        |
|  9 | Glenna Reichert          | Delphine         | [email protected]   | Dayna Park        | Suite 449     | Bartholomebury | 76495-3109      | 24.6463         | -168.8889       | (775)976-6794 x41206  | conrad.com    | Yost and Sons      | Switchable contextually-based project    | aggregate real-time technologies     |
| 10 | Clementina DuBuque       | Moriah.Stanton   | [email protected]    | Kattie Turnpike   | Suite 198     | Lebsackbury    | 31428-2261      | -38.2386        | 57.2232         | 024-648-3804          | ambrose.net   | Hoeger LLC         | Centralized empowering task-force        | target end-to-end models             |
+----+--------------------------+------------------+---------------------------+-------------------+---------------+----------------+-----------------+-----------------+-----------------+-----------------------+---------------+--------------------+------------------------------------------+--------------------------------------+
10 rows in set (0.126 sec)

Check container from CLI

$ docker exec -it connect-patch sudo mariadb -uroot -e "use test; create table cusers4 engine=connect http='http://jsonplaceholder.typicode.com/users' table_type=json; select * from cusers4;"
+----+--------------------------+------------------+---------------------------+-------------------+---------------+----------------+-----------------+-----------------+-----------------+-----------------------+---------------+--------------------+------------------------------------------+--------------------------------------+
| id | name                     | username         | email                     | address_street    | address_suite | address_city   | address_zipcode | address_geo_lat | address_geo_lng | phone                 | website       | company_name       | company_catchPhrase                      | company_bs                           |
+----+--------------------------+------------------+---------------------------+-------------------+---------------+----------------+-----------------+-----------------+-----------------+-----------------------+---------------+--------------------+------------------------------------------+--------------------------------------+
|  1 | Leanne Graham            | Bret             | [email protected]         | Kulas Light       | Apt. 556      | Gwenborough    | 92998-3874      | -37.3159        | 81.1496         | 1-770-736-8031 x56442 | hildegard.org | Romaguera-Crona    | Multi-layered client-server neural-net   | harness real-time e-markets          |
|  2 | Ervin Howell             | Antonette        | [email protected]         | Victor Plains     | Suite 879     | Wisokyburgh    | 90566-7771      | -43.9509        | -34.4618        | 010-692-6593 x09125   | anastasia.net | Deckow-Crist       | Proactive didactic contingency           | synergize scalable supply-chains     |
|  3 | Clementine Bauch         | Samantha         | [email protected]        | Douglas Extension | Suite 847     | McKenziehaven  | 59590-4157      | -68.6102        | -47.0653        | 1-463-123-4447        | ramiro.info   | Romaguera-Jacobson | Face to face bifurcated interface        | e-enable strategic applications      |
|  4 | Patricia Lebsack         | Karianne         | [email protected] | Hoeger Mall       | Apt. 692      | South Elvis    | 53919-4257      | 29.4572         | -164.2990       | 493-170-9623 x156     | kale.biz      | Robel-Corkery      | Multi-tiered zero tolerance productivity | transition cutting-edge web services |
|  5 | Chelsey Dietrich         | Kamren           | [email protected]  | Skiles Walks      | Suite 351     | Roscoeview     | 33263           | -31.8129        | 62.5342         | (254)954-1289         | demarco.info  | Keebler LLC        | User-centric fault-tolerant solution     | revolutionize end-to-end systems     |
|  6 | Mrs. Dennis Schulist     | Leopoldo_Corkery | [email protected]   | Norberto Crossing | Apt. 950      | South Christy  | 23505-1337      | -71.4197        | 71.7478         | 1-477-935-8478 x6430  | ola.org       | Considine-Lockman  | Synchronised bottom-line interface       | e-enable innovative applications     |
|  7 | Kurtis Weissnat          | Elwyn.Skiles     | [email protected]    | Rex Trail         | Suite 280     | Howemouth      | 58804-1099      | 24.8918         | 21.8984         | 210.067.6132          | elvis.io      | Johns Group        | Configurable multimedia task-force       | generate enterprise e-tailers        |
|  8 | Nicholas Runolfsdottir V | Maxime_Nienow    | [email protected]      | Ellsworth Summit  | Suite 729     | Aliyaview      | 45169           | -14.3990        | -120.7677       | 586.493.6943 x140     | jacynthe.com  | Abernathy Group    | Implemented secondary concept            | e-enable extensible e-tailers        |
|  9 | Glenna Reichert          | Delphine         | [email protected]   | Dayna Park        | Suite 449     | Bartholomebury | 76495-3109      | 24.6463         | -168.8889       | (775)976-6794 x41206  | conrad.com    | Yost and Sons      | Switchable contextually-based project    | aggregate real-time technologies     |
| 10 | Clementina DuBuque       | Moriah.Stanton   | [email protected]    | Kattie Turnpike   | Suite 198     | Lebsackbury    | 31428-2261      | -38.2386        | 57.2232         | 024-648-3804          | ambrose.net   | Hoeger LLC         | Centralized empowering task-force        | target end-to-end models             |
+----+--------------------------+------------------+---------------------------+-------------------+---------------+----------------+-----------------+-----------------+-----------------+-----------------------+---------------+--------------------+------------------------------------------+--------------------------------------+

Basing the PR against the correct MariaDB version

  • [x] This is a new feature and the PR is based against the latest MariaDB development branch.
  • [ ] This is a bug fix and the PR is based against the earliest maintained branch in which the bug can be reproduced.

PR quality check

  • [x] I checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
  • [x] For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

an3l avatar Jan 11 '24 14:01 an3l

before I forget - need to revert #2466 as part of this.

grooverdan avatar Jan 30 '24 13:01 grooverdan

Windows BB depends on https://github.com/MariaDB/server/pull/3068, so I included it into PR only to pass BB. Please first merge https://github.com/MariaDB/server/pull/3068 (without closing PR) and I will rebase current PR.

an3l avatar Mar 07 '24 09:03 an3l