maplibre-native
maplibre-native copied to clipboard
maplibre
Run render tests in CI
This pull request adds a linux ci workflow with the aim of running the render tests in ci, see #350.
I added one test to the ignores list because it failed locally on my linux laptop.
Good point, thanks @ntadej. I was browsing around and saw the node workflow too. What is the difference between platform/node and platform/linux?
Good point, thanks @ntadej. I was browsing around and saw the node workflow too. What is the difference between platform/node and platform/linux?
As far as I understand linux wraps node and adds linux-specific items on top of it. Maybe node can actually be disabled, but e.g. darwin behaves the same way.
The render test artifact html file can be downloaded here: https://github.com/maplibre/maplibre-gl-native/suites/7393552170/artifacts/301090085
The following render test currently fail on the ubuntu runner:
2022-07-17T18:34:10.1241586Z * failed render-tests/fill-visibility/visible
2022-07-17T18:34:36.1808443Z * failed render-tests/runtime-styling/source-add-vector-inline
2022-07-17T18:34:40.0896321Z * failed render-tests/runtime-styling/set-style-source-add-vector-inline
2022-07-17T18:34:44.5957103Z * failed render-tests/runtime-styling/source-add-vector-url
2022-07-17T18:34:48.4943337Z * failed render-tests/runtime-styling/set-style-source-add-vector-url
These work fine on my ubuntu laptop. Maybe the problem is related to xvfb? Maybe I can debug this with GitHub Codespaces...
Hm, codespaces does not compile. If I run:
git submodule update --init --recursive --depth 1
sudo apt update
sudo apt install ccache cmake ninja-build pkg-config xvfb libcurl4-openssl-dev libglfw3-dev libuv1-dev g++-10 libc++-9-dev libc++abi-9-dev libpng-dev libgl1-mesa-dev libgl1-mesa-dri libjpeg-turbo8 libicu66
cmake . -B build -G Ninja -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER=gcc-10 -DCMAKE_CXX_COMPILER=g++-10
I get the following error:
-- The CXX compiler identification is GNU 10.3.0
-- The C compiler identification is GNU 10.3.0
-- Check for working CXX compiler: /usr/bin/g++-10
-- Check for working CXX compiler: /usr/bin/g++-10 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/gcc-10
-- Check for working C compiler: /usr/bin/gcc-10 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Configuring GL-Native with OpenGL renderer backend
-- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "7.68.0")
-- Found the following ICU libraries:
-- i18n (optional)
-- Found ICU: /usr/include (found version "66.1")
-- Found the following ICU libraries:
-- uc (optional)
-- Found ICU: /usr/include (found version "66.1")
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR)
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.16/Modules/FindJPEG.cmake:100 (find_package_handle_standard_args)
platform/linux/linux.cmake:4 (find_package)
CMakeLists.txt:1095 (include)
-- Configuring incomplete, errors occurred!
See also "/workspaces/maplibre-gl-native/build/CMakeFiles/CMakeOutput.log".
The commands were taken from the linux-ci.yml workflow where stuff compiles.
Maybe github actions runner and codespaces are not identical environments...
The Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR) on codespaces can be fixed by installing libjpeg-dev. So the full steps on codespaces are:
git submodule update --init --recursive --depth 1
sudo apt update
sudo apt install ccache cmake ninja-build pkg-config xvfb libcurl4-openssl-dev libglfw3-dev libuv1-dev g++-10 libc++-9-dev libc++abi-9-dev libpng-dev libgl1-mesa-dev libgl1-mesa-dri libjpeg-turbo8 libicu66 libjpeg-dev
cmake . -B build -G Ninja -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER=gcc-10 -DCMAKE_CXX_COMPILER=g++-10
cmake --build build -j $(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null)
xvfb-run -s "-ac -screen 0 1280x1024x24" ./build/mbgl-render-test-runner --manifestPath metrics/linux-clang8-release-style.json &> render-test.log
sed -i 's/\x1b\[[0-9;]*m//g' render-test.log
cat render-test.log|grep "* failed"
Results in:
* failed render-tests/fill-visibility/visible
* failed render-tests/runtime-styling/source-add-vector-url
* failed render-tests/runtime-styling/source-add-vector-inline
* failed render-tests/runtime-styling/set-style-source-add-vector-inline
* failed render-tests/runtime-styling/set-style-source-add-vector-url
And here is the summary
1216 passed (92.7%)
10 passed but were ignored (0.8%)
80 ignored (6.1%)
5 failed (0.4%)
1 errored (0.1%)
We could ignore the tests which fail in CI. There are only 5 failing tests and one which errors, so ignoring them is not a big deal. I could open an issue to fix them sometime in the future. Like that we would at least have some render tests in CI to start with...
Strange that the new linux render test workflow did not run. Let me close and re-open to trigger ci...
3 tests errored just now. I saw in codespaces that tests randomly error. Rerunning them worked fine afterwards. Don't know how to handle errored tests...
A typical error seems to be related to file size: error: Invalid size for actual image
This should be ready for review now. My conclusion is that xvfb on ubuntu is not stable enough to run the 1k+ render and query tests for MapLibre GL Native. Therefore, I now use macOS which has a graphics card I think to run the render tests.
In the future, we might want to remove the node ci render tests on ubuntu and only keep the macOS render tests in node for the same reason, i.e. because xvfb on GitHub ubuntu runners is not reliable enough.
The docs are bad I know but they are still better than no docs. Ideally the test instructions would be in a test folder...