[bug] symlinks are not copied to source_dir and build_dir by conan

[jhzhangsnps]

Environment Details (include every applicable attribute)

• Operating System+version: centOS 7.3
• Compiler+version: gcc 9.2.0
• Conan version: 1.50.0
• Python version: 3.9

Steps to reproduce (Include if Applicable)

conan create . app/0.1@user/testing

Logs (Executed commands with output) (Include/Attach if Applicable)

under my source dir, there are symlinks as below.

ls -l snps/include/ common -> ../../src/include/

but under the build dir and source dir created by conan, the symlink doesn't exist. ls -l testing/source/snps/include/ testing/build/998e9d6a4cb0ecb04912bff2830f2efad0dc6521/snps/include/ testing/build/998e9d6a4cb0ecb04912bff2830f2efad0dc6521/snps/include/: total 0

testing/source/snps/include/: total 0

[memsharded]

Hi @jhzhangsnps

You need to specify the conanfile.py that you are using. The modern approach to handle symlinks is:

• Use the copy() tool, but not the self.copy() legacy method
• User is responsible to "prepare" or manage symlinks before they are copied. There are tools in from conan.tools.files to help with this.

[jhzhangsnps]

@memsharded thanks for the quick reply. I One more question, which function might affects the result of build_dir and source_dir, where I can use this copy function in my conanfilepy, it only has 4 functions which are build(), package(), package_info(), export_sources() those links are used in build

[memsharded]

It seems it is the export_sources() method that you need to update to use the copy() (and whatever other management of symlinks that you would like to do)

[jhzhangsnps]

I have done that in export_sources() already. it works well in the directory export_source but not in build and source testing/build/998e9d6a4cb0ecb04912bff2830f2efad0dc6521/snps/include/: total 0

testing/export_source/snps/include/: total 0 common -> ../../src/include/

testing/source/snps/include/: total 0

please share how source/build dir are created when using 'conan create'

[memsharded]

I see. Internally, there are some copies of files from the exports folders to the source folder in the cache. The copy from source->build folder can be completely avoided using the no_copy_source = True attribute in your conanfile recipe.

The export->source folder copy is managing symlinks. I have tried a bit, and it is also covered by tests, so apparently it should work. What we would need is a minimalistic but complete conanfile.py recipe, and also the exact steps to reproduce the environment and the problem, like ln -s ...... + ln -s .... + conan create ....., etc.

[jhzhangsnps]

thanks for looking into it. here are the steps to reproduce it.

create the module by copy 'conan/examples/features/makefiles/hellolib'

cd hellolib/src/ mkdir include cd include ln -s ../hello.h

In hellolib/conanfile.py remove exports_sources = "src/" add def export_sources(self): self.copy("src/", symlinks=True);

conan create . user/testing

.conan/data/hello/0.1/user/testing/build/de9c231f84c85def9df09875e1785a1319fa8cb6/src/include:

.conan/data/hello/0.1/user/testing/export_source/src/include: hello.h@

.conan/data/hello/0.1/user/testing/source/src/include:

[memsharded]

This is what I meant:

• You shouldn't be using self.copy("src/", symlinks=True) in your export_sources() method anymore, that has been already removed from Conan 2.0
• You need to use the new copy(self, ...) method, check https://docs.conan.io/en/latest/reference/conanfile/tools/files/basic.html
• If need some help to "prepare" symlinks, you could use tools from https://docs.conan.io/en/latest/reference/conanfile/tools/files/symlinks.html

[jhzhangsnps]

self.copy("src/", symlinks=True) works properly so that the export_source dir is correct

.conan/data/hello/0.1/user/testing/export_source/src/include: hello.h@

but the source folder is empty .conan/data/hello/0.1/user/testing/source/src/include:

BTY, I tried to use copy() in export_sources but failed with the following error

hello/0.1@user/testing: Calling export_sources() ERROR: hello/0.1@user/testing: Error in export_sources() method, line 15 copy(self, "*", src="./src", dst=self.source_folder); TypeError: expected str, bytes or os.PathLike object, not NoneType

[jhzhangsnps]

it works after I implement the export_sources as below.

def export_sources(self): copy(self, "*", src="./src", dst=os.path.join(self.export_sources_folder,"src"));

[memsharded]

self.copy("src/", symlinks=True) works properly so that the export_source dir is correct

Yes, at the moment it might work. But it has already been removed from Conan 2.0 (already in beta), so better not use it anymore.

ERROR: hello/0.1@user/testing: Error in export_sources() method, line 15 copy(self, "*", src="./src", dst=self.source_folder); TypeError: expected str, bytes or os.PathLike object, not NoneType

Yes, the source_folder is not defined in this method. If you read https://docs.conan.io/en/latest/reference/conanfile/methods.html#export-sources, it will tell you that you need to use self.export_sources_folder folder in this method.

[jhzhangsnps]

thanks for the support.

[memsharded]

thanks for the support.

Glad to help. This last one we were basically writing the response simultaneously :)

[jhzhangsnps]

sorry, that I made a mistake. it still doesn't work. I tried again with a clean cache.

as before, the export_sources dir has the symlink

but there is no link under source_folder and build_folder. ls /u/jhzhang/.conan/data/hello/0.1/user/testing/export_source/src/include/ hello.h@

ls: cannot access /u/jhzhang/.conan/data/hello/0.1/user/testing/source/src/include/hello.h: No such file or directory

the copy function works in the export_sources function but my problem is in the source_folder and then build_folder

[jhzhangsnps]

I have debugged it. it is really a bug. --Call--

python/venv/lib/python3.9/site-packages/conans/util/files.py(406)link_to_rel() -> def link_to_rel(pointer_src): (Pdb) n python/venv/lib/python3.9/site-packages/conans/util/files.py(407)link_to_rel() -> linkto = os.readlink(pointer_src) (Pdb) n python/venv/lib/python3.9/site-packages/conans/util/files.py(408)link_to_rel() -> if not os.path.isabs(linkto): (Pdb) p linkto '../hello.h' (Pdb) n python/venv/lib/python3.9/site-packages/conans/util/files.py(409)link_to_rel() -> linkto = os.path.join(os.path.dirname(pointer_src), linkto) (Pdb) n python/venv/lib/python3.9/site-packages/conans/util/files.py(412)link_to_rel() -> out_of_source = os.path.relpath(linkto, os.path.realpath(src)).startswith(".") (Pdb) p linkto '/u/jhzhang/.conan/data/hello/0.1/user/testing/export_source/src/include/../hello.h' (Pdb) n python/venv/lib/python3.9/site-packages/conans/util/files.py(413)link_to_rel() -> if out_of_source: (Pdb) p out_of_source True (Pdb) p pointer_src '/u/jhzhang/.conan/data/hello/0.1/user/testing/export_source/src/include/hello.h' (Pdb) p linkto '/u/jhzhang/.conan/data/hello/0.1/user/testing/export_source/src/include/../hello.h' (Pdb) p src '/u/jhzhang/.conan/data/hello/0.1/user/testing/export_source'

[jhzhangsnps]

it think the link /u/jhzhang/.conan/data/hello/0.1/user/testing/export_source/src/include/../hello.h is out of source. So skip the link creation

[jhzhangsnps]

I have figured out the root cause. since I need a big conan cache, so .conan under my home is a symlink.

when execute the following line, need call os.path.realpath for both linkto and src

python/venv/lib/python3.9/site-packages/conans/util/files.py(412)link_to_rel() -> out_of_source = os.path.relpath(linkto, os.path.realpath(src)).startswith(".")

[jhzhangsnps]

@memsharded James, do you agree to deliver a fix? I have applied a local patch which works properly

[memsharded]

We will have a look, but it is difficult to guarantee a fix, at least in 1.X, because the risk of breaking is high, so it might be targeted to 2.0 instead.

If so .conan under my home is a symlink. is the cause, the recommended way is to define CONAN_USER_HOME, not to define a symlink. Have you tried that approach instead? Please try and let us know. .

[jhzhangsnps]

I will try to use CONAN_USER_HOME as a fix.

I am ok with a fix in 2.0

thank you

[czoido]

Hi @jhzhangsnps,

Did using CONAN_USER_HOME solve your problem? I don't see clearly that this is a bug, as I can't reproduce it. Would it be possible for you to provide a minimal reproducible example (maybe starting from a template like using something like conan new mylib/1.0 -m=cmake_lib) so I can try to reproduce? Thanks a lot!

[jhzhangsnps]

it works with ENV setting CONAN_USER_HOME.

I have provided a simple example to reproduce it in this thread. please look at the message above

[czoido]

it works with ENV setting CONAN_USER_HOME.

I have provided a simple example to reproduce it in this thread. please look at the message above

Hi @jhzhangsnps, I'm very glad that it worked. Yes, I tried to reproduce using the information in the thread but unless I'm missing something It did not fail for me. Providing something step-by-step is always easier to reproduce this kind of behavior. If that works for you now, and considering that we can't reproduce, l will close the issue but please feel free to reopen if you have more questions. Thanks!

[jhzhangsnps]

if your .conan is a symlink, it can be reproduced always I don't think this ticket should be closed

[czoido]

if your .conan is a symlink, it can be reproduced always I don't think this ticket should be closed

Hi @jhzhangsnps,

Thanks for the feedback. Defining the Conan user home as a symlink is something that should not be done, if you want to set the location of the cache to other place than the user home, CONAN_USER_HOME is the correct way to do it. If that's the root of the problem I think we can't consider this as a bug, but a result of undefined behavior derived from setting that as a symlink. I will reopen the ticket and I think that we could explicitly say in the documentation that this is not a good practice to avoid others in the future to have the same problem. WDYT?

[jhzhangsnps]

@czoido, even if the .conan is not a symlink, the problem can happen if any dir in the path is a symlink.

and i don't understand the inconsistency to call realpath for the second parameter but not the first one. python/venv/lib/python3.9/site-packages/conans/util/files.py(412)link_to_rel() -> out_of_source = os.path.relpath(linkto, os.path.realpath(src)).startswith(".