blade-build
blade-build copied to clipboard
chen3feng
external dependency supported in blade
Shall we support this feature, since we may have some external libs which is also build by blade? bazel did it in this way https://docs.bazel.build/versions/master/external.html
This is more like a C++ source package repository mechanism compared to maven in java and I've seen this in bazel when I tried to make Tensorflow built in blade(painful!).
The most important thing to consider is the directory layout which is very simple in blade: a whole big directory tree starting from BLADE_ROOT. If we fetch external dependency package, we're able to place them into an external directory(//external or some folder in configuration) or download into somewhere else and then make a symlink within //external to point to the real location. Furthermore, external dependency may have its own dependencies which is transitive dependency of our projects.
The download and directory organization mentioned above are easy to implement. However, another thing to think about is dependency conflict. For instance, we want to introduce tensorflow which have plenty of dependencies one of which is protobuf-3.7.x. Let's say our projects are still using protobuf-3.4.1 or even protobuf-2.6.1, how can we resolve this? In maven, dependency management specify the rules for that. In C++, library conflict triggers undefined behavior and maybe depends on the linker script.
Alternatively, it's possible to rely on some functionality provided by scm, such as git submodules, to aggregate several separate projects. The members in the team should carefully select the external dependency modules and add/remove them in the scm configuration to avoid conflict.
@WindLeeWT Yes, all you mentioned are the actually issues we faced in development. Dependency management would be another topic, for now, we can just put its lib and include in some folder to import it directly and avoid to compile it by blade. We also need to very careful to handle with the version conflict problem.
In Blade based projects, the problem for now is : sometime, we need to integrate two projects together, to make it as one (blade is great for this). Another time, we may extract common part from several projects and make it as a common dependency (like IDL, GRPC), and every project need to depend on it. Ok, lets say those projects are maintained by different team, maybe compiled in different environment. And we need find a way to make the common dependency in a same version.
As I mentioned in very beginning, I hope dependency feature could be introduced to resolve it.
Maybe you have already know, both blade and bazel are based on the design of Google's in-house build tool blaze, which works under the nono-repo trunk-based-development mode.
It is easy to keep the mono-repo mode in the one big team with strict library management, but in most company, it is very hard, and It becomes impossible in the open internet environment.
So, bazel try to resolve this problem by introducing the external dependency mechanism. but as I know, there are still many problems in this mechanism.
It is surely a problem. I'm still working on it, try to design a better solution.
@WindLeeWT Yes, all you mentioned are the actually issues we faced in development. Dependency management would be another topic, for now, we can just put its lib and include in some folder to import it directly and avoid to compile it by blade. We also need to very careful to handle with the version conflict problem.
In Blade based projects, the problem for now is : sometime, we need to integrate two projects together, to make it as one (blade is great for this). Another time, we may extract common part from several projects and make it as a common dependency (like IDL, GRPC), and every project need to depend on it. Ok, lets say those projects are maintained by different team, maybe compiled in different environment. And we need find a way to make the common dependency in a same version.
As I mentioned in very beginning, I hope dependency feature could be introduced to resolve it.
We also have this problem. For the time being, we use submodule to deal with it, but this feeling is very unreliable
