conan-io
[question] object library support
What is your question?
While it's relatively uncommon, sometimes object libraries are invaluable. It's unclear to me how to properly package an object library with Conan, however.
Unlike with normal static libraries, there is no single output that can be easily named. Instead there may be a large directory tree with many object files. It seems like package_info() has to walk a directory tree looking for object files.
When the package is created normally and packaged in its own directory, this seems doable. E.g. object files can be installed into an obj/ subdirectory and all files within that directory can be treated as object files and added to cpp_info.objects.
When the package is used in editable mode, as ever, things seem a lot more complicated. When using CMake for example, object files are placed in fairly unpredictable locations and with unpredictable names. E.g. a source file named foo.cpp of an object library named bar may correspond to $BUILD_DIR/bar.dir/foo.obj or $BUILD_DIR/bar.dir/src/foo.cpp.obj. A file listing all object files could be generated as part of the build, but even so, it seems difficult to make use of it in the Conan recipe.
Another question is which package_type should be used. There is no object-library package type. Some aspects of static-library are probably fine, but it's unclear if it is the right choice.
Have you read the CONTRIBUTING guide?
- [ ] I've read the CONTRIBUTING guide
Hi @vasama
Thanks for your question.
When the package is used in editable mode, as ever, things seem a lot more complicated. When using CMake for example, object files are placed in fairly unpredictable locations and with unpredictable names.
Have you tried a os.walk or a glob? I think Python has built-in utilities that can achieve this in just a couple of lines of code, even if .obj files are in different locations, but they will always be inside the self.build_folder. I wouldn't say really unpredictable, the build system knows exactly where they will be, and this will also be fully deterministic, they will always be in the same place. I'd recommend to use some of those tools to find the .obj files in multiple directories if necessary.
Another question is which package_type should be used. There is no object-library package type. Some aspects of static-library are probably fine, but it's unclear if it is the right choice.
Yes, I think from the linkage point of view they will be identical to static-library, so that package_type can be used.
self.build_folder cannot be accessed in package_info().
Also the file extension is platform dependent. And it's possible that the build creates temporary objects that are somehow modified to create the final object files that should be used by the consumer. I don't think it's a good idea to just take all the files within the build directory that look like they might be object files. This would also break if the package contains multiple components.
self.build_folder cannot be accessed in package_info().
When we are talking about editable packages, the definition of the package happens in layout() method, not in package_info() method. The self.build_folder is defined in the layout() method.
Also the file extension is platform dependent. And it's possible that the build creates temporary objects that are somehow modified to create the final object files that should be used by the consumer. I don't think it's a good idea to just take all the files within the build directory that look like they might be object files. This would also break if the package contains multiple components.
Still, if we want to make objects available to consumer packages when the package is editable mode, it is necessary to define those objects. The only one that knows about that is the recipe and the recipe author. If some objects have to be excluded, then a filter can be applied over the result of glob, or exclude them in the os.walk iteration.
If the package contains multiple components, those components can also be defined while in editable in the self.cpp.source.components and self.cpp.build.components in the same way they can be defined in the self.cpp_info when it is a final package in package_info().
When the package is used in editable mode, as ever, things seem a lot more complicated. When using CMake for example, object files are placed in fairly unpredictable locations and with unpredictable names.
Yes, I understand it might not be obvious, but it is what it is, there is little else that could be done. Editable packages need to know where are the things that will be used by the consumers. When CPS does support build-trees, this will not be necessary, but so far it is not a provided feature, so in the meantime it is necessary to specify these things in recipes with user logic.
Hi @vasama
Any further question or comment here? Thanks for the feedback.
Right, sorry.
Still, if we want to make objects available to consumer packages when the package is editable mode, it is necessary to define those objects. The only one that knows about that is the recipe and the recipe author.
I feel like this is a recurring theme. My CMake build script is able to provide all of this information. I can emit a file listing all the relevant objects in the build directory. I just need to be able to make use of that information in the Conan recipe. It's simple enough when packaging, but how to do it in editable mode has been unclear to me. Is it possible to do this in the layout method?
I feel like this is a recurring theme. My CMake build script is able to provide all of this information. I can emit a file listing all the relevant objects in the build directory. I just need to be able to make use of that information in the Conan recipe. It's simple enough when packaging, but how to do it in editable mode has been unclear to me. Is it possible to do this in the layout method?
It might be possible with some provisions. Take into account that the layout() method executes before the build, for example to determine where are the sources needed for the build. Then, it is not possible to use a file that is the result of the actual build as input for the layout() method execution. Likewise, that file will not be there in all the scenarios where the build is not happening, for example when downloading a package with a precompiled binary that is not building.
So to be able to do something like that, the layout() method should have provisions to fallback to something correct when the file is not present, and it will be necessary to do a build first, in a separate command, before that build result can be used by other packages when in editable mode.
Like I said, this seems like a recurring theme. I've made multiple issues where I've had problems with using build artifacts in editable mode. To me that tells that there's something missing in the editable package support. Maybe there should be a new editable_package_info() method added with access to the build directory, invoked in addition to the regular package_info(), or some other means of achieving this should be provided.
Like I said, this seems like a recurring theme. I've made multiple issues where I've had problems with using build artifacts in editable mode. To me that tells that there's something missing in the editable package support. Maybe there should be a new editable_package_info() method added with access to the build directory, invoked in addition to the regular package_info(), or some other means of achieving this should be provided.
Not necessary to invent a new mechanism, but what is necessary is the CPS getting more traction and build systems being able to output CPS files over the build tree, not only the install tree. This is something that we have been working in the past and collaborating with other players such as Kitware/CMake, they are already aware of this need and might try to provide them in the future.