Rule to build a fully static library

[asimshankar]

trafficstars

Among the multiple "release" outputs of my project, I'd like to include a single static library, which packages the objects of all dependencies into it. However, from what I can tell, bazel cannot build such targets.

The following target produces a single .so containing all symbols from //product/foo and //product/bar.:

cc_binary(
  name = "libproduct.so",
  linkshared = 1,
  deps = [
     "//product/foo",
     "//product/bar",
  ],
)

I believe there is no way to generate a similar static library (libproduct.a). Is that something that will be of interest to add?

(#492 seems somewhat related, but not exactly)

[steven-johnson]

+1; my project has similar hard requirement and similar problem.

[hermione521]

Is this what you are looking for? https://www.bazel.io/versions/master/docs/be/c-cpp.html#cc_binary.linkstatic

[asimshankar]

@hermione521 : I don't think so.

If I try to create the .a using a cc_binary rule like so:

cc_binary(
  name = "libproduct.a",
  linkstatic = 1,
  deps = [
    "//product/foo",
    "//product/bar",
  ],
)

Then the output isn't an object archive.

And my understanding of linkstatic=1 in cc_library rules is that it means that the symbols in the library will be statically linked into the binary that depends on it, not that the library itself will include symbols of its transitive dependencies.

[hermione521]

Ah, I'm sorry. Let's ask @lberki

[lberki]

No, we don't support that at the moment :( We were considering implementing something like that with @mhlopko , but we have other priorities at the moment.

A workaround would be to use a genrule to package the static library like this:

cc_library(name="a")
cc_library(name="b")
genrule(name="g", srcs=[":a", ":b"], cmd="$(AR) $(locations :a) $(locations :b)")

I took a bit of artistic freedom since $(locations) will return both a static and a dynamic library and that's not exactly how ar should be called, but it should work as long as you don't need symbols from transitive dependencies.

[asimshankar]

Thanks, though I do want to include symbols from transitive dependencies (for example, building a static library for tensorflow (today, there is a shared library))

[steven-johnson]

This is a pretty major problem for us, too. I'll try the genrule workaround suggested above, but it makes me mighty nervous.

[ulfjack]

cc_binary with .so is currently special, but cc_binary doesn't handle .a. One of the downsides of special-casing the specific filename is that it's not cross-platform compatible. One option would be to introduce a new cc_export rule with a type={DYNAMIC,STATIC} attribute that links the transitive closure into a dynamic or static library respectively.

I'm a bit concerned about the increased likelihood of accidental ODR violations, which requires a globally consistent partition into cc_export rules. That's not a problem per se, except when different people want different partitions. Internally, we have everything in a single repository, and we actually have requests to support multiple different overlapping partitioning schemes. Externally, you usually don't have a choice. Whatever upstream decides is what you get.

[lberki]

@ulfjack: there are a bunch of options. One is a cc_export rule you suggested, another would be an output group on regular cc_binary rules that would put all the .o files in its transitive closure into a single .a .

I was thinking that the partitioning should be the responsibility of whoever is writing the cc_binary rule(s). I'm not particularly worried about coordination problems if there are multiple such rules with multiple authors because people should know what symbols a library they use contains and library authors should know what symbols they want to export. Do you think that's too optimistic?

[ulfjack]

I think it's very easy to mess up the dependencies. Consider for example a workspace that has two cc_library and two cc_export rules, with each of the cc_export rules depending on exactly one library. Now an owner of one of the lower-level libraries isn't aware of the cc_export going on in some completely different part of the workspace, and adds a dependency on the other library. Everything keeps working just fine, except that the application blows up at runtime, because it contains two copies of some symbols. They may not be exported symbols, so neither the static linker nor the dynamic linker would be able to check.

[lberki]

I understand that. But who is in a better situation to sort these things out if not the owner of the top-level rule?

[ulfjack]

I'm not saying that they aren't. But I think it's a problem that it's a runtime error rather than a compile-time error. If we can make it a compile-time error, then I think that may be an acceptable solution for now. If the burden on the top-level owners is too large, or if it causes users to create duplicate (multiplicate) rule hierarchies, then we can still see if we can come up with a better solution. We certainly need to document it carefully because it's easy to shoot yourself in the foot.

I can think of two ways of making it a compile-time error. 1. Use constraints to annotate every rule that goes into a specific cc_export rule. 2. Look for all cc_export rules in the transitive closure of a cc_binary or cc_test and check that their corresponding transitive closures are non-overlapping.

2 seems like it could be expensive; 1 would be optional.

I'd like to keep the option open of statically linking everything even if there are intermediate cc_export rules, so maybe cc_export isn't a good name for that. Maybe we should call it cc_package or cc_transitive_library. Then at the cc_binary level we can decide whether we cut the binary into individual .so's at the cc_transitive_library level or whether to link everything statically after all.

[ulfjack]

Consider that many debian packages come with both .so and .a which allows developers to decide after the fact whether to link a specific library statically or dynamically.

[steven-johnson]

Related to my previous comment: I have put together a utility script for myself that does the necessary hackery with with the .a files to produce a statically-linked output for my purposes; unfortunately, this introduced me to https://github.com/bazelbuild/bazel/issues/1740, in which a cc_library() has different outputs depending on compilation mode (in -c opt it also outputs a .pic.a library).

[ahyangyi]

@steven-johnson I am the creator of issue #1740. Yes, I also have some homebaked rule that builds a fully static shared library (and with extra functionalities such as linking with a version script).

Currently I use a hack: if a cc_library outputs at least some .pic.a library, I use them, otherwise I use all .a library. It somehow worked in all combinations of settings I encounter, so I do recommend you to do something else if you need it to work immediately.

However if we get better support from Official Bazel I can simplify things. You know, I am the sole maintainer of our Bazel workspace, since it contains too much cryptic workarounds that work around cryptic workarounds ...

[steven-johnson]

if a cc_library outputs at least some .pic.a library, I use them, otherwise I use all .a library

Thanks, I'll give that a try. On the whole, Bazel is great, but the number of workarounds necessary for various outside-of-Google necessities of life is still disappointing.

[asimshankar]

@lberki @ulfjack : Just wanted to check in on this. Would you guys have the bandwidth to address this soon?

[lberki]

@mhlopko is OOO for a long while, so I don't think we can get to this soon-ish :(

[hlopko]

So I'm back :) We have no design ready for this (and other linking related requests), but I'll try to push this forward. We really have to polish our linking story. Sadly, I expect it will take a few weeks to fully implement considering my current load.

[lababidi]

@mhlopko how can we help? where should we begin?

[hlopko]

I don't think there is anything you could do right now. I'm discussing the design internally and soon I'll publish a design doc to bazel-discuss. Then I'll start implementing.

[ulfjack]

How about publishing the doc now, and having the discussion on bazel-discuss?

[hlopko]

Finally! :) Please take a look and comment on the transitive libraries design doc: https://docs.google.com/document/d/1-tv0_79zGyBoDmaP_pYWaBVUwHUteLpAs90_rUl-VY8/edit?usp=sharing

[lababidi]

@mhlopko Thank you for sharing the doc! What are the next steps?

[hlopko]

We still discuss this a lot. I'll add one more approach to the design doc this week that would make the possibility of introducing diamond "by accident", and therefore possibility of changes to distant dependencies breaking the top level build, smaller. More points still need investigation:

• dead code elimination
• symbol visibility specification (as entry points)
• exposed header specification

I'm flying to New York next week and will discuss this with the rest of the team. I really hope to have a final design ready by the end of the next week.

[steven-johnson]

Eager to hear how this turns out -- it's one of the last sticking points for my project.

[hlopko]

Good news, work on transitive libraries has started :)

[annemenini]

Any update on the rule for building static libraries with Bazel? I would like to use tensorflow as a static library in my project (which is not built with Bazel) as it is the rule for all its dependencies.

[snnn]

Hi @annemenini https://github.com/bazelbuild/bazel/issues/4078#issuecomment-345620850

[hlopko]

Let me repeat it here: transitive libraries have been deprioritized in favor of C++ sandwich. We will post an update for transtive libraries later in Q1/2018.