shaka-packager
shaka-packager copied to clipboard
shaka-project
Switch to CMake and git submodules
The use of our ancient chromium build infrastructure has been increasingly problematic. We use old versions of chromium components so that they are compatible with gyp (gyp has been removed upstream), but this means also an older version of depot_tools, and lack of support for newer OSes, SDKs, and Linux distros.
We should replace gyp with CMake, gclient with submodules, and depot_tools should go away completely. Dependencies (like protobuf, etc) should be updated to the latest versions, and if possible, system versions should be used instead of building from source.
I'm not sure what order these should be done in, or how much work they will be. I'll experiment with it a bit to figure that out, but I may or may not be the one to finish the work. Volunteers are welcome!
Oh, and of course, some of the old Python build scripts in our old forks of Chromium repos will only run with Python 2, which is not even installed by default any more in many places.
In a pure python3-environment, such as my workstation at the office, I can't even run gyp any more. So replacing gyp is a high priority in this list.
Replacing gclient with submodules is relatively easy and straightforward. See https://github.com/joeyparrish/shaka-packager/commit/f4d560ed924c54c7ec97389d19bd5b5d25f02b9e Some of those deps could be dropped by the time this work is complete (such as gyp, packager/build).
@joeyparrish I managed to fix some problems with generating CMakeLists.txt from gyp. It was discussed some time ago at https://github.com/shaka-project/shaka-packager/pull/774 I wished to make a PR to fix issues with generation of CMakeLists.txt but I failed when discovered that some tools are part of Chromium ecosystem and is very hard to fix because those tools were from ancient revision of Chromium tools.
But still - I can tell how to fix locally some files in order to generate CMakeLists.txt.
And by the way - I use Python3 with Shaka gyp and it works (AFAIK I had to fix only about two places regarding py2->py3)
@zdanek, thank you! You shouldn't need to make permanent changes to gyp. You could make temporary changes, generate CMakeLists.txt from gyp, then clean up CMakeLists.txt by hand and commit it. The goal is to get rid of gyp, so it should be fine to hack it up as needed if it is easier than writing CMakeLists.txt completely from scratch.
@joeyparrish I got it.
I was using generated CMakeLists.txt for CLion to edit files and have includes working properly. It was OK but I never compiled Shaka with make. I always used command line and ninja like everybody. Now I tried to builld it and it failed.
There are a few problems:
- it's very very messy
- it doesn't compile
- release is quite different than debug - but we need both (merged into one CMakeLists.txt)
- it's flat - I mean, that every directory / library / third party lib has it's own section put into a single CMakeLists.txt - it should be divided into smaller files, probably for directories / modules
Biggest problem is that it does not compile. That means, some paths are messed up and some includes are bad defined so inside the source files includes doesn't work. If there's no other volunteer I might spend some evenings trying to clean it up and/or rethink better approach. Maybe it will need more people to help to split it into sub-works to speed it up and not to be so boring and arduous task :)
I will try to dig into it tomorrow. First task is to build Shaka using cmake.
I see. Thank you for the update!
It seems there isn't a better solution right now than manually creating the build files for cmake. I agree that doing it one library/dependency at a time is probably best.
I have begun the process of writing CMake files by hand for each library.
As part of this work, ~C++14~ C++17 will now be required to build Shaka Packager. This will reduce our dependencies, and we will rely more on the standard library.
@joeyparrish regarding to write a new CMake files by somebody from the community - I was thinking about it as I wrote 3 months ago but as you currently see it's quite absorbing and in my opinion must be done by somebody who is backed by bigger company that pays for the time spent. I wouldn't have enough time to do it. The time that I need to spend on my current job. The transition is hard. I'm glad that you decided to move it forward. Thank you on behalf of all developers depending on Shaka Project :)
Well, I only get to spend one day a week on it, so it will take a long time. Thankfully, @kqyang has offered to do code reviews as I make progress.
I'll start merging partial build system work to a branch in stages. If anyone is interested, they can follow along here. I'll tag this issue with each PR, and you can play with the branch I'm creating and provide feedback.
So far, I've got 4 internal libraries building, and I've ripped out Chromium build tools, Chromium base, and gflags. I've updated curl, boringssl, gmock, and gtest. I've added abseil and glog.
Those 4 internal libraries build, including their tests, but I'm stuck on crashing tests in the file library.
Although local problems (crashing tests) this is good news. Getting rid of Chromium build tools is my dream since 1.5 years :) Great decisions. I will take a look at your branch. Thank you.
Everything we have so far in PR #1072 is building and passing all tests. If anyone wants to help write CMakeLists for additional libraries once that PR lands, I would be very happy to have the help! I think we could easily divide up the work by folder. Please let me know if you are interested!
Joey, I had some comments to your work on your branch but I could not write it as there's no PR. Just your branch. I tried to find your email but no luck😁
Great work. I will try to help if I will have some spare time.
pon., 25 lip 2022, 20:29 użytkownik Joey Parrish @.***> napisał:
Everything we have so far in PR #1072 https://github.com/shaka-project/shaka-packager/pull/1072 is building and passing all tests. If anyone wants to help write CMakeLists for additional libraries once that PR lands, I would be very happy to have the help! I think we could easily divide up the work by folder. Please let me know if you are interested!
— Reply to this email directly, view it on GitHub https://github.com/shaka-project/shaka-packager/issues/1047#issuecomment-1194454417, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADLSJUN3JKW553ICMJLQ4TVV3MKFANCNFSM5QK5UV3A . You are receiving this because you were mentioned.Message ID: @.***>
The PR is #1072, targeting the cmake branch of this main repo. Feel free to leave comments on the PR if you have feedback.
We plan to stage everything on the cmake branch, then merge that branch into main when the entire project has been ported.
If you want to discuss contributions to the CMake porting effort, let's do it here rather than by email. My inbox is insane, and I tend to lose email. Plus, as this is a public project, others might benefit from a public discussion. Thanks!
If you would like to contribute to the porting effort, please reply! I get one day per week for this effort, so if I do it all by myself, it may take me months. Let's get it done together!
I have created a spreadsheet to track what is ready to port and help assign particular modules to people as we go. If you volunteer here, I will give you edit access to the sheet and add you to a chat room for porting discussions.
We currently have 3 unassigned modules with no dependencies, so you can start right away!
I'm going to help with this, but will check how much time I can get sponsored for vs losing my weekend :-)
I can help too.. can definitely spend time on this in the weekend!!
You've both been added to the spreadsheet and chat. I'll post more detailed instructions in the chat. @kevleyski, when you accept the chat invite, you should be able to see the history of what I'm about to post. If not, let me know there.
I'd love to help! Should be able to spend a hack day or two on this at least :)
@joeyparrish I could fix one small module just for sport ;) I've changed my work so I have no sponsor. I will do it my private time so I ask for small module for a good start. Any suggestions?
@Lunkers, @zdanek, please email me ([email protected]) and I will add you to a Google Sheet and Google Chat group to coordinate. So far, nobody has claimed any modules or started work, and there are three modules ready to be ported. When I finish media/base, many more start opening up. Thanks!
If there is an example module done that shows the general pattern we want to follow I think it's going to be a lot easier to jump in on the other modules.
Yes, definitely. If you look at the tracking spreadsheet, you'll see 5 modules are already completed: status, tools, third_party, version, file. (Those are all paths within the packager/ folder.) You can also look at the history in the cmake branch to see the diffs of the changes landing there.
@joeyparrish @zdanek fyi, I started working on the CMake port as well, but haven't got write permission in the spreadsheet. Ported media/codecs. I just need to clean it up and get the CLA done to open it for review :)
Thanks for the update. I used your codecs port as a base for my media/crypto. I did a port but one test fails. Today it's to late to fix it (I'm in CET). Point me as a reviewer I'd like to see your code when it's finished.
The cmake branch doesn't build for me currently on OS X, because sprintf is marked as deprecated, the build is treating this warning as an error, and the cURL check for snprintf which it should use instead if detected is broken in the version we are pulling in (and every released version as of now).
https://github.com/curl/curl/pull/10155
This is the curl pull request that is necessary to get it to build. So the options I see are to update the curl submodule to latest head (until the next release is cut), or drop -Werror,-Wdeprecated-declarations to allow the deprecated sprintf to be used for now.
Hello everyone, Thanks for the update. I want to know what's the progress now !
We're getting very close to merging cmake back into main. Just a few more PRs, we think.