[bug]conan source fails when layout depends on self.settings.os

[rainbowgoblin]

Describe the bug

I have a recipe that was written in house to build a 3rd party tool that uses OS-dependent build systems (CMake on Linux, Visual Studio on Windows, and XCode on macOS). The layout method looks like this:

  def layout(self):
    if self.settings.os == "Linux":
      cmake_layout(self, src_folder="src")
    elif self.settings.os == "Windows":
      vs_layout(self, src_folder="src")
    elif self.settings.os == "Macos":
      basic_layout(self, src_folder="src")
    else:
      raise ConanException(f"Unknown os: {self.settings.os}")

Building this package with conan create works fine, but we often use conan to build packages stepwise, i.e.,

conan source ...
conan install ...
conan build ...
conan export-pkg ...

This fails at conan source, with that exception from the layout() method:

ERROR: conanfile.py (foo/x.y.z): Error in layout() method, line 83
	raise ConanException(f"Unknown os: {self.settings.os}")
	ConanException: Unknown os: None

There's nothing in the documentation of layout() that suggests that OS-specific layout shouldn't be allowed, and the documentation does mention using settings and options to configure the layout. I can't find any way to pass settings into the conan source. I have a default host profile set in my global.conf, which does specify the os setting, so apparently this is ignored. I also tried using conan profile detect to set a default profile, but the os setting was still ignored.

How to reproduce it

1. Create a recipe with OS-specific layout and raise an exception if self.settings.os is invalid (see the example in the description).
2. Run conan source using this recipe. Result: an exception is raised.

[memsharded]

Hi @rainbowgoblin

Thanks for your report.

I think this is expected, by design, let me try to clarify:

• The source() method must be invariant, as well as the conan source command. That is, no matter what are the inputs, settings, etc, it must always resolve to the exact same source. Because the source folder is reused for all different builds and variants, it cannot be conditioned to any inputs.
• This is the reason why the conan source doesn't have any settings, options or profile arguments. Also the source() method will raise an exception if any of .settings, .options is being used.
• On the other hand, the layout() method can encode the variability in the "build" tree, what are the build folders, the library folders, the binary folders, the headers folders, on the build tree. This changes for different variants, different OSs, compilers, etc.
• When the conan source command executes, the source() method needs only the definition of the self.folders.source. This is not a problem for the cmake_layout, because it declares it unconditionally

For your case, that you have such a variability, it seems that the solution would be:

 def layout(self):
     self.folders.source = "src" # for all platforms, same source folder
    if self.settings.os == "Linux":
      cmake_layout(self, src_folder="src")
    elif self.settings.os == "Windows":
      vs_layout(self, src_folder="src")
    elif self.settings.os == "Macos":
      basic_layout(self, src_folder="src")

That way the conan source shouldn't fail, and the different layouts will be selected to customize the build folders when the build really happens.

Please let me know if this helps.

[rainbowgoblin]

@memsharded I thought your solution would work, but it turned out I made a mistake in my layout code. vs_layout() doesn't take a src_folder parameter. So if I do something like this:

def layout(self):
     self.folders.source = "src" # for all platforms, same source folder
    if self.settings.os == "Linux":
      cmake_layout(self, src_folder="src")
    elif self.settings.os == "Windows":
      vs_layout(self)
    elif self.settings.os == "Macos":
      basic_layout(self, src_folder="src")

This fails if I run conan source/conan install/conan build because at the time conan source runs the sources will be put in a different place from where build expects them to be.

I can do some kind of hacky workaround with platform, e.g.

if platform.system() != "Windows":
  self.folders.source = "src"
  ...

But this seems gross. Setting the source location after calling vs_layout() also works, e.g.

    ...
    elif self.settings.os == "Windows":
      vs_layout(self)
      self.folders.source = "src"
   ...

Which is probably what I want to do anyway, since it's actually not very nice that conan source dumps sources directly into the recipe folder otherwise. Note that vs_layout() sets folders.source to the value of folders.subproject if it's valid, so this wouldn't work in all cases, but it works in this one.

I'm not sure if there's a better solution.

[memsharded]

Hi @rainbowgoblin

Thanks for your feedback.

This fails if I run conan source/conan install/conan build because at the time conan source runs the sources will be put in a different place from where build expects them to be.

I can do some kind of hacky workaround with platform, e.g.

But this seems gross.

If you mean doing the if platform.system() != "Windows": in the source() method, this is not hacky at all in the sense that it is expected and possible, if the author of the recipe knows what they are doing, and accepts that responsibility. The self.settings.xxx are not allowed in the source() method for the reasons above, and raising an error is good as it let the users learn about this, but if they want to accept that responsibility. then it is good.

Setting the source location after calling vs_layout() also works, e.g.

That makes sense. The vs_layout() contain code such as:

def vs_layout(conanfile):
    ...
    subproject = conanfile.folders.subproject
    conanfile.folders.source = subproject or "."
    conanfile.folders.generators = os.path.join(subproject, "conan") if subproject else "conan"
    conanfile.folders.build = subproject or "."

So setting the folders.source after it to override the defined value, achieves what you are trying. Also setting the conanfile.folders.subproject before the call to vs_layout() which might be a bit cleaner as it puts the generators and build folder under it too.

And the vs_layout() didn't implement a src_folder argument yet, because in practice we found the VS projects layout very "fixed", with the solution always on the root and not necessary. But if you think the vs_layout(..., src_folder) argument could make sense for that case, I think it is possible to add it.

Maybe check first if the self.folders.subproject = "src" before the call to vs_layout(self) works nicely, and if not, we can consider the addition of the argument?

[memsharded]

Maybe check first if the self.folders.subproject = "src" before the call to vs_layout(self) works nicely, and if not, we can consider the addition of the argument?

Hi! Did you manage to give this a try?

[memsharded]

Hi @rainbowgoblin

Any further question or comment here? Otherwise, marking the ticket as staled, it will be closed soon if there aren't any follow ups. Thanks for the feedback!

[rainbowgoblin]

@memsharded sorry, I forgot to reply earlier.

I think it would be helpful to take a src_folder arg in vs_layout, but I get that stepwise building is probably already fairly niche, particularly for projects that use different layouts on different platforms.

I do think that the documentation should mention (somewhere) that OS-specific layout code might be problematic for running conan source.

I'm happy for this to be closed though.

[memsharded]

I have tried to add an argument vs_layout(..., src_folder=), but I have encountered a few issues trying to make things work. Because once I move the sources (.sln, .vcxproj, .cpp) to that src_folder, then all the other folders, the "build" folder, the "generators" folder, have to be moved there too, under the src_folder to be consistent and to work correctly. So it doesn't really look like a src_folder anymore, I mean semantically, but the correct semantics would be the:

def layout(self):
     .,..
     self.folders.subproject = "src"
     vs_layout(self)

which moves everything, the whole solution, project, generated files, everything, to the "src" subfolder.

But I might still be missing something. By:

but I get that stepwise building is probably

Do you mean that you have projects in which the build folders are not below the folder containing the "sln" and "vcxproj", but sibling to it? Where do you put the Conan generators in that case?

[rainbowgoblin]

Do you mean that you have projects in which the build folders are not below the folder containing the "sln" and "vcxproj", but sibling to it? Where do you put the Conan generators in that case?

OK, so, prior to the work I was doing when I logged this as a bug, the layout and source methods in my recipe looked like this, with no src_folder specified:

def layout(self):
    if self.settings.os  ==  "Linux":
        cmake_layout(self)
    elif self.settings.os == "Windows":
        vs_layout(self)
    elif self.settings.os == "Macos":
        basic_layout(self)

def source(self):
    git = Git(self)
    git.fetch_commit(repo, commit=f"releases/{self.version}")

On Windows if I run conan source..., conan install..., conan build..., the sources from the repo are dumped into the same location as the conanfile.py recipe, with the vcxproj and sln files sibling to the recipe. There's a src folder that's also there, along with the solution build folder and the generators, so it's something like this:

someLibrary
├── conanfile.py
├── someLibrary.vcxproj
├── someLibrary.sln
├── src
│   └── ...
├── someLibrary
│   └── x64
│       └── Release MDLL
│           └── ...
├── conan
│   └── ...
├── test_package

I tried making the following changes to layout and source methods:

def layout(self):
    self.folders.source = "src"
    ...
    elif self.settings.os == "Windows":
        self.folders.subproject = "src"
        vs_layout(self)
    ...

def source(self):
    git = Git(self, folder=self.source_folder)
    ...

The result is there's a folder called src alongside conanfile.py that contains the sources, etc, from the repo, as well as the build folder and generators (I think this is what you were saying). This is probably more or less what I want, even if it's not quite right semantically!

Which I suppose brings me to why I wanted to mess with adding a src_folder arg in the first place.

1. When I build this way, it's really gross to clean up after. I can't just rm -rf * without deleting my conanfile, etc.
2. If I need to try building more than once I have to clean up or I'll get a warning from git because the repo is already cloned in my recipe folder.
3. For this project in particular, there's another conanfile that comes from the git repo, which we don't use! Fortunately it's a conanfile.txt recipe, whereas we use a Python script recipe, so it's just a matter of passing the name of the correct recipe to conan, e.g. conan source ./conanfile.py... But this is still annoying.

All three of these problems are resolved by using self.folders.subproject = "src" in any case.

[memsharded]

Thanks very much for the feedback, sorry it took a while to follow up.

All three of these problems are resolved by using self.folders.subproject = "src" in any case.

Then, even if a possible new argument could make some sense, if there is already a way to achieve this, we typically prefer to follow the python zen and leave just 1 way to achieve a thing if possible. So I suggest to use the self.folders.subproject for this issue. And in that case, I think we could consider this ticket solved and close it? You can create new tickets for any further question or issue. Many thanks!