[question] How can I not export sources when creating a package and publishing it

[midhun1998]

What is your question?

Hi @memsharded , I'm currently using Conan 1.64 and I would like to understand how can I not export the sources when creating a package?

Background and Requirement

Currently, We have a CI pipeline that leverages workspace-specific CONAN_USER_HOME and this is passed on as an artifact to other steps. We noticed that after conan export-pkg the local conan cache size increases significantly due to sources and object files being copied to the local cache. We would like to prevent building from sources in our packages due to 2 main reasons:

• As these pipelines are self-service and developers would need a specific account and setting to get the distributed build working that basically means that they can't build from source even if they download the sources and recipe.
• Few projects are very confidential and publishing the sources is a risk and security threat.

Here are the steps done in our CI pipeline to build and publish the package to Artifactory:

1. conan source
2. conan install --profile CUSTOM_PROFILE
3. conan build
4. conan export-pkg
5. conan test
6. conan upload

I believe as part of conan export-pkg the sources are also being exported. I tried adding build_policy=never. I also don't have export_sources() function defined explicitly. I read through the docs and couldn't figure out how I could prevent the sources from being exported.

Your help is appreciated! 😇

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @midhun1998

Thanks for your question.

We'd need to know more about the conanfile.py, can you please share it? Most likely it contains some exports = or exports_sources = ... attributes that are copying all the local files into the recipe, or your package() method contains too broad patterns.

Note that the local flow (source + build + export-pkg) is more a development flow, but not that intended for CI. The reproducibility will be lower and the possibility of doing more efficient CI is lost too.

Also, the recommended approach for having recipes that can build for authorized users and avoid code access for non-authorized users is https://docs.conan.io/en/1.64/reference/conanfile/tools/scm/git.html#example-implementing-the-scm-feature. Using the "scm" approach, the recipe captures the "url+commit" of the source, so authorized users can actually build new binaries from source when they want, while not authorized ones will only be able to use binaries, without code access.

[midhun1998]

Thanks for quick reply, @memsharded !

We'd need to know more about the conanfile.py, can you please share it? Most likely it contains some exports = or exports_sources = ... attributes that are copying all the local files into the recipe, or your package() method contains too broad patterns.

I double-checked and I don't see export* attributes. I have added recipe for reference.

Note that the local flow (source + build + export-pkg) is more a development flow, but not that intended for CI. The reproducibility will be lower and the possibility of doing more efficient CI is lost too.

So, Does this mean the right flow is to run conan create ? If not, Could you please guide me? One of the many reasons why we chose the source, install, build, etc process is to run custom regression such as smoke test and full regression after conan build and conan export-pkg. Let me know if we can still achieve this with conan create?

Also, the recommended approach for having recipes that can build for authorized users and avoid code access for non-authorized users is https://docs.conan.io/en/1.64/reference/conanfile/tools/scm/git.html#example-implementing-the-scm-feature. Using the "scm" approach, the recipe captures the "url+commit" of the source, so authorized users can actually build new binaries from source when they want, while not authorized ones will only be able to use binaries, without code access.

Thanks for pointing me in the right direction! 🙂 All our packages have sources and recipes in the same repository. Should we still stick the above approach?

Here is a redacted version of conanfile.py. I have removed specifics of product and commands other than that its the same.

class Xyz(ConanFile):
    name = "xyz"
    build_dir = "src"
    def set_version(self):
        if get_env("CONAN_PACKAGE_VERSION") is not None:
            self.version = get_env("CONAN_PACKAGE_VERSION")
        else:
            self.version = "2.2.1"

    def build_requirements(self):
        self.build_requires("curl/202203.0.0@ci/prod")
        ...
        # ---- More build_requires statements here ----       

    def build(self):
                self.run("CUSTOM BUILD COMMAND")
                
    def package(self):
       # ---- Many self.copy instances which are copying specific files. Generic ones are below ---- #
        for file in [
            "src/common/a.h",
            "src/b.h",
           ...
        ]:
            self.copy(
                os.path.basename(file),
                dst=f"include/{self.name}",
                src=os.path.dirname(file),
                keep_path=False,
            )
	  self.copy("README.md", dst=".", src="src/", keep_path=False)
          self.copy(
	          "*.a",
	          dst=f"lib/{self.settings.target_arch}",
	          src="src/object_root/lib-" + str(self.settings.target_arch),
	          keep_path=False,
          )
          self.copy(
	          "*.so",
	          dst=f"lib/{self.settings.target_arch}",
	          src="src/object_root/pscb/lib-" + str(self.settings.target_arch),
	          keep_path=False,
          )
          self.copy(
	          "*",
	          dst=f"bin/{self.settings.target_arch}",
	          src="src/lib/module-x/install/" + str(self.settings.target_arch),
	          keep_path=True,
          )
          self.copy(
	          "product.bom",
	          dst=f"bin/bom",
	          src="src/lib/module.y",
	          keep_path=False,
          )

    def package_id(self):
        del self.info.settings.compiler.libcxx
        del self.info.settings.compiler.version
        del self.info.settings.compiler.cppstd

    def package_info(self):
        self.cpp_info.libs += [
            file.replace(
                f"{self.package_folder}/lib/{self.settings.target_arch}/lib", ""
            )
            .replace(".a", "")
            .replace(".lib", "")
            for file in glob.glob(
                f"{self.package_folder}/lib/{self.settings.target_arch}/lib*.a"
            )
        ]
        self.cpp_info.libs += [
            file.replace(
                f"{self.package_folder}/lib/{self.settings.target_arch}/lib", ""
            )
            .replace(".a", "")
            .replace(".lib", "")
            for file in glob.glob(
                f"{self.package_folder}/lib/{self.settings.target_arch}/lib*.lib"
            )
        ]
        self.cpp_info.libdirs = [f"lib/{self.settings.target_arch}"]
        self.cpp_info.libs = [lib for lib in self.cpp_info.libs]

To give more clarity, Here is the log of conan export-pkg:

$ conan export-pkg --force -bf _build -sf _build -if _build . xyz/2.2.1@midhun/dev
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'url'. It is recommended to add it as attribute
Exporting package recipe
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'license'. It is recommended to add it as attribute
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'description'. It is recommended to add it as attribute
xyz/2.2.1@midhunn/dev: Calling export_sources()
xyz/2.2.1@midhunn/dev export_sources() method: Copied 13 '.depends' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 11 '.hier' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 120 '.make' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 45 '.c' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 441 '.h' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 39 files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 8 '.js' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 2 '.json' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 2 '.html' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 1 '.yml' file
xyz/2.2.1@midhunn/dev export_sources() method: Copied 1 '.md' file
xyz/2.2.1@midhunn/dev export_sources() method: Copied 28 '.a' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 47 '.so' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 1 '.zip' file
xyz/2.2.1@midhunn/dev export_sources() method: Copied 84 '.1' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 3 '.txt' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 12 '.map' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 21 '.py' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 137 '.cc' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 18 '.png' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 23 '.test' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 5 '.sh' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 197 '.script' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 6 '.cpp' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 8 '.m4' files
...
xyz/2.2.1@midhunn/dev: A new conanfile.py version was exported
xyz/2.2.1@midhunn/dev: Folder: /workspace/midhunn/_work/.conan/data/xyz/2.2.1/midhun/dev/export
xyz/2.2.1@midhunn/dev: Exported revision: 948921590483f70de2de5280d1f9360e
ERROR: No package matching 'xyz' pattern found.
Packaging to ff4bbb08e95d07da66d2fed7d63999710a61ad13
xyz/2.2.1@midhunn/dev: Generating the package
xyz/2.2.1@midhunn/dev: Package folder /workspace/midhunn/_work/.conan/data/xyz/2.2.1/midhunn/dev/package/ff4bbb08e95d07da66d2fed7d63999710a61ad13
xyz/2.2.1@midhunn/dev: Calling package()
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.common' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.depends' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.md' file
xyz/2.2.1@midhunn/dev package(): Packaged 9 '.h' files
xyz/2.2.1@midhunn/dev package(): Packaged 4 '.a' files
xyz/2.2.1@midhunn/dev package(): Packaged 47 '.so' files
xyz/2.2.1@midhunn/dev package(): Packaged 4 files
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.5' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.0' file
xyz/2.2.1@midhunn/dev package(): Packaged 3 '.rules' files
xyz/2.2.1@midhunn/dev package(): Packaged 3 '.conf' files
xyz/2.2.1@midhunn/dev package(): Packaged 84 '.1' files
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.html' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.txt' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.sbom' file
xyz/2.2.1@midhunn/dev: Package 'ff4bbb08e95d07da66d2fed7d63999710a61ad13' created
xyz/2.2.1@midhunn/dev: Created package revision 3e03ff821e0e927edbb9f89c4ae024a9

[memsharded]

So, Does this mean the right flow is to run conan create ? If not, Could you please guide me?

Yes, conan create would typically be easier and more robust and work more consistently across developer and CI machines.

One of the many reasons why we chose the source, install, build, etc process is to run custom regression such as smoke test and full regression after conan build and conan export-pkg. Let me know if we can still achieve this with conan create?

You can add the specific tests in the build() method, and control if they execute or not via user confs, for example. That way a simple conan create . -c user.myteam:tests_integration=True for example would be doing that automatically.

All our packages have sources and recipes in the same repository. Should we still stick the above approach?

Yes, the "scm" approach is for packages that the recipe and the sources are in the same repository, still don't want to export the sources into the recipe for confidentiality/privacy.

self.build_requires("curl/202203.0.0@ci/prod")

Just to make sure, this is only valid if you are using curl executable only and only at build time, but not using the libcurl library, not the headers, not linking with the library. If you are doing the second case you need to start using the 2 profiles approach with --profile:build=xxx and it will break. But this is how Conan 2 works, and it is extremely recommended to prioritize the upgrade to Conan 2, it was released more than 1 year ago and it is the supported version. Same with using the legacy self.copy(), you need to start using the recommended copy(self, ...) alternative.

xyz/2.2.1@midhunn/dev export_sources() method: Copied 13 '.depends' files

There is some detail missing in your recipe. The logs clearly indicate that your recipe has an export_sources() method. Can you please double check?

[memsharded]

Hi @midhun1998

Any further feedback here?

[memsharded]

I am closing this ticket as staled, but please re-open or create new tickets when you have further feedback or follow ups from the above. Thanks!

[midhun1998]

Hi @memsharded ,

Apologies for the delayed reply. I mistakenly overlooked this issue. I investigated the conan recipe again today and found no 'export' or 'export_sources' methods but interestingly there was 'export_sources' method defined in the conan recipe of a module defined under 'python_requires'. The 'export_sources' gets called from the conan recipe of the module defined under 'python_requires' when 'python_requires_extend' is used. I removed the extension and it did not export the sources.

Just wanted to document this incase someone stumbles on the same issue in the future.

Thank you for the pointers @memsharded . 😄

[memsharded]

Apologies for the delayed reply. I mistakenly overlooked this issue.

No problem, thanks for your feedback!!

I investigated the conan recipe again today and found no 'export' or 'export_sources' methods but interestingly there was 'export_sources' method defined in the conan recipe of a module defined under 'python_requires'. The 'export_sources' gets called from the conan recipe of the module defined under 'python_requires' when 'python_requires_extend' is used. I removed the extension and it did not export the sources.

Great investigation. Indeed, the python_requires is a great way to reuse code, but as every abstraction it has a cognitive overhead cost, and it can hide some behavior and issues that is not easily visible.

Thanks for your feedback again, and don't hesitate to create new tickets for any further question.