build system improvements

[benoit-pierre]

Rework the build system to make it faster and better.

TODO:

• [x] PR fbink build system patches
• [x] resolve libk2pfopt situation
• ~[ ] update docker images (chrpath+meson)~

Highlights:

• move most of it to CMake: allow for parallelize each external project build steps as much as possible (e.g. all download / patch steps can be executed in parallel).
• add convenience targets for building koreader libraries and each external project build steps (e.g.: make koreader-cre to build only libkoreader-cre.so and all its dependencies, make leptonica-re to rebuild leptonica, etc…). Top level targets can be serialized and are forwarded to CMake (so its possible to do make clean zlib for example).
• change the build tree layout: keep the downloads in thirdparty/*/build/…, move the external projects build directories under the machine specific build directory: reduce the depth of the build tree, and simplify clean rules.
• “install” all external projects to the same staging directory under the build directory: simplify dependency lookups in external projects (including CMake).
• always use a CMake toolchain file: including when not cross-compiling, as way to pass parameters to CMake and simplify building CMake based external projects.
• use CMake to build external projects when possible (faster).
• use Ninja as default generator (faster).
• add missing luajson dependency, don't use luarocks for building/installing LUA external projects: faster builds, no need for network, all dependencies explicitly accounted for, simpler install tree (no need for the rocks directory anymore).
• use -l as as well as -j when running make / ninja to manage system load.
• rework libk2pdfopt / leptonica / tesseract situation: leptonica & tesseract are build by their respective external projects.
• changing an external project source file will automatically trigger its build step during the next build.

Build times (on my machine: -j8 / -j8 -l8, from scratch: after make distclean):

	master	PR
emulator-debug (from scratch)	11:05	4:30 (÷ 2.5)
emulator-debug (from cache)	4:04	0:38 (÷ 6.4)
android-arm (from scratch)	9:58	4:27 (÷ 2.2)
android-arm (from cache)	3:44	0:34 (÷ 6.6)
kindlepw2 (from scratch)	15:50	6:51 (÷ 2.3)
kindlepw2 (from cache)	5:21	0:50 (÷ 6.4)

Disk usage (downloads + build tree(s))

	master	PR
emulator-debug	291M + 2.3G = 2.6G	155M + 1.6G = 1.7G
android-arm	286M + 1.4G = 1.6G	150M + 1.0G = 1.2G
kindlepw2	326M + 1.5G = 1.9G	176M + 1.2G = 1.3G

Notes:

• minimum supported CMake version 3.15 (already mandated by zsync2's cpr submodule).
• using make as a CMake generator is still supported (with some limitations: no proper build by-products support), but since meson does not support it, I think we might as well drop support for it.
• need this patch to the top koreader project:

 Makefile | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git i/Makefile w/Makefile
index e3092b174..bdd6bfcaf 100644
--- i/Makefile
+++ w/Makefile
@@ -77,7 +77,7 @@ DOCKER:=$(shell grep -q docker /proc/1/cgroup 2>/dev/null && echo 1)
 INSTALL_FILES=reader.lua setupkoenv.lua frontend resources defaults.lua datastorage.lua \
 		l10n tools README.md COPYING
 
-all: $(if $(ANDROID),,$(KOR_BASE)/$(OUTPUT_DIR)/luajit)
+all:
 	$(MAKE) -C $(KOR_BASE)
 	install -d $(INSTALL_DIR)/koreader
 	rm -f $(INSTALL_DIR)/koreader/git-rev; echo "$(VERSION)" > $(INSTALL_DIR)/koreader/git-rev
@@ -85,12 +85,12 @@ ifdef ANDROID
 	rm -f android-fdroid-version; echo -e "$(ANDROID_NAME)\n$(ANDROID_VERSION)" > koreader-android-fdroid-latest
 endif
 ifeq ($(IS_RELEASE),1)
-	$(RCP) -fL $(KOR_BASE)/$(OUTPUT_DIR)/. $(INSTALL_DIR)/koreader/.
+	bash -O extglob -c '$(RCP) -fL $(KOR_BASE)/$(OUTPUT_DIR)/!(cache|cmake|history|staging|thirdparty) $(INSTALL_DIR)/koreader/'
 else
 	cp -f $(KOR_BASE)/ev_replay.py $(INSTALL_DIR)/koreader/
 	@echo "[*] create symlink instead of copying files in development mode"
 	cd $(INSTALL_DIR)/koreader && \
-		bash -O extglob -c "ln -sf ../../$(KOR_BASE)/$(OUTPUT_DIR)/!(cache|history) ."
+		bash -O extglob -c 'ln -sf ../../$(KOR_BASE)/$(OUTPUT_DIR)/!(cache|cmake|history|staging|thirdparty) .'
 	@echo "[*] install front spec only for the emulator"
 	cd $(INSTALL_DIR)/koreader/spec && test -e front || \
 		ln -sf ../../../../spec ./front
@@ -130,9 +130,6 @@ ifeq ($(IS_RELEASE),1)
 	rm -rf $(INSTALL_DIR)/koreader/data/{cr3.ini,cr3skin-format.txt,desktop,devices,manual}
 endif
 
-$(KOR_BASE)/$(OUTPUT_DIR)/luajit:
-	$(MAKE) -C $(KOR_BASE)
-
 $(INSTALL_DIR)/koreader/.busted: .busted
 	ln -sf ../../.busted $(INSTALL_DIR)/koreader
 
@@ -143,7 +140,9 @@ $(INSTALL_DIR)/koreader/.luacov:
 testfront: $(INSTALL_DIR)/koreader/.busted
 	# sdr files may have unexpected impact on unit testing
 	-rm -rf spec/unit/data/*.sdr
-	cd $(INSTALL_DIR)/koreader && ./luajit $(shell which busted) \
+	eval "$$($(LUAROCKS_BINARY) path)" && cd $(INSTALL_DIR)/koreader && \
+		env TESSDATA_DIR=$(INSTALL_DIR)/koreader/data \
+		./luajit "$$(which busted)" \
 		--sort-files \
 		--output=gtest \
 		--exclude-tags=notest $(BUSTED_OVERRIDES) $(BUSTED_SPEC_FILE)
@@ -629,4 +628,5 @@ static-check:
 doc:
 	make -C doc
 
+.NOTPARALLEL:
 .PHONY: all clean doc test update

---

This change is