td
td copied to clipboard
tdlib
Support skipping non-essential parts of the build
This significantly speeds up build times for projects which just want to use tdlib as-is. On a single core this should cut about 5 minutes; on 4 cores it cuts a bit more than a minute. All this for no measurable cost.
Without skipping:
$ cmake -DCMAKE_BUILD_TYPE=Release -DTD_SKIP_BENCHMARK=OFF -DTD_SKIP_TEST=OFF -DTD_SKIP_TG_CLI=OFF ..
( ... regular output ... )
$ time /usr/bin/time make install -j4 DESTDIR=../../td_destdir
( ... regular output ... )
1414.69user 55.66system 7:01.60elapsed 348%CPU (0avgtext+0avgdata 2752144maxresident)k
0inputs+1210120outputs (0major+21767636minor)pagefaults 0swaps
real 7m1,604s
user 23m34,699s
sys 0m55,665s
With skipping:
$ cmake -DCMAKE_BUILD_TYPE=Release -DTD_SKIP_BENCHMARK=ON -DTD_SKIP_TEST=ON -DTD_SKIP_TG_CLI=ON ..
( ... regular output ... )
$ time /usr/bin/time make install -j4 DESTDIR=../../td_destdir
( ... regular output ... )
1118.16user 42.62system 5:56.22elapsed 325%CPU (0avgtext+0avgdata 2751196maxresident)k
0inputs+891912outputs (0major+17310062minor)pagefaults 0swaps
real 5m56,230s
user 18m38,163s
sys 0m42,629s
This reduces build time only by 15% in your example. I'm not sure that such small speed up worth build setup complication, especially given build time can be improved much more without any change. You need to build TDLib only once, so build time should be irrelevant for ordinary usages, and for development purposes it is important to check that everything is built correctly. I also wouldn't call tests a non-essential part.
To build a specific target it is better to explicitly specify it, for example, cmake --build . --target tdjson -- -j4 or make -j4 tdjson. Install target wouldn't work this way, but it is easy to manually copy libtdjson.so or any other needed file.
build setup complication
I don't really see how this is a complication.
should be irrelevant for ordinary usages
It saves 5 minutes on Travis every time we need to rebuild. (Also because Travis only goes to -j 2.)
For the cost of 5 additional lines, and 3 self-explaining options.
I also wouldn't call tests a non-essential part.
You're right, I shouldn't have called it non-essential. However, I hope you can see that someone who just wants to install it (and not run tests or benchmarks) may want to cut down those 5 minutes.
it is easy to manually copy libtdjson.so
It's also very brittle: This would mean that we ignore td's build system, and "guess" what the install target needs, and do all that manually, even though there is already an obvious place where definitions like this should go: td's CMakeLists.txt.
Effectively I would need to check every now and then whether our "guess" is still "correct enough", which would be a non-automatable process.
You can try ninja
What? Is that a serious comment? How does this solve anything?
@BenWiederhake
build setup complication
I don't really see how this is a complication.
If it is not properly documented, then almost noone will use it, so it is useless. If it is properly documented, then it is a complication, because everyone will need to read about it.
should be irrelevant for ordinary usages
It saves 5 minutes on Travis every time we need to rebuild. (Also because Travis only goes to
-j 2.) For the cost of 5 additional lines, and 3 self-explaining options.
Unless you are going to test TDLib build or run TDLib tests, there is no reason to rebuild TDLib every time. It is much simpler and efficient to create a Docker container with prebuilt TDLib or use https://docs.travis-ci.com/user/caching/#caching-directories-bundler-dependencies.
I thought a few times about conditional building of some TDLib parts, but all the time when this was an acceptable solution, there was a simple way to achieve better result.
If it is not properly documented, then almost noone will use it, so it is useless. If it is properly documented, then it is a complication, because everyone will need to read about it.
CMake options have built-in documentation, having like 8 extra lines in the CMake file to reduce every build by 15% seems a pretty good deal to me...
Also, not everyone needs to know about them, just those who look through the list of options if they want to customize the build.
The CMake file already has a bunch of options so I'm not sure why adding 3 more to speed up compilation adds complexity if I must be honest
@mbasaglia If you know that you can see CMake options, you should also know that you can build a single target with CMake.
Also, not everyone needs to know about them, just those who look through the list of options if they want to customize the build.
That's exactly the reason why they will be useless. 1% of developers will do that, but even these developers should be able to customize CMakeLists.txt themselves in no time.
Customizing CMakeLists.txt in CI is not viable, also I would need to know which targets are needed...
This significantly speeds up build times for projects which just want to use tdlib as-is.
On a single core this should cut about 5 minutes; on 4 cores it cuts a bit more than a minute.
All this for no measurable cost.
Without skipping:
$ cmake -DCMAKE_BUILD_TYPE=Release -DTD_SKIP_BENCHMARK=OFF -DTD_SKIP_TEST=OFF -DTD_SKIP_TG_CLI=OFF .. ( ... regular output ... ) $ time /usr/bin/time make install -j4 DESTDIR=../../td_destdir ( ... regular output ... ) 1414.69user 55.66system 7:01.60elapsed 348%CPU (0avgtext+0avgdata 2752144maxresident)k 0inputs+1210120outputs (0major+21767636minor)pagefaults 0swaps real 7m1,604s user 23m34,699s sys 0m55,665sWith skipping:
$ cmake -DCMAKE_BUILD_TYPE=Release -DTD_SKIP_BENCHMARK=ON -DTD_SKIP_TEST=ON -DTD_SKIP_TG_CLI=ON .. ( ... regular output ... ) $ time /usr/bin/time make install -j4 DESTDIR=../../td_destdir ( ... regular output ... ) 1118.16user 42.62system 5:56.22elapsed 325%CPU (0avgtext+0avgdata 2751196maxresident)k 0inputs+891912outputs (0major+17310062minor)pagefaults 0swaps real 5m56,230s user 18m38,163s sys 0m42,629s
https://t.me/sumisaliii
@eugenebas As explained in https://github.com/tdlib/td/pull/1127#issuecomment-654501793 and https://github.com/tdlib/td/pull/1127#issuecomment-655186949, if you need these options, you are doing something wrong. What is you use case?
It would be nice to have benchmark and test projects DISABLED by default and parallel building ENABLED! It's 2022 and people use multicore systems mostly.
@andrew-phi Enabled multicore build by default would lead only to failed build by default. If you are sure that you have enough RAM for multicore build, the you are free to pass corresponding argument to build system.
@levlam if it would fail, it would fail faster than waiting many hours just to get really, REALLY frustrated (I waited ~3h for 70%, 4cores/6GB). I'm not familiar with cmake, but cmake --build . --target install -- -j 4 seems like a hack to me. Or at least a proper way of doing so should be mentioned / added to build.html as an option.
4cores/6GB). I'm not familiar with cmake, but cmake --build . --target install -- -j 4
This is exactly the command, which must not be advertised. GCC needs about 4GB per core, so the mentioned command will trigger OOM, killing random processes on your server. It have no chance to succeed with 6GB RAM.
