Errno::E2BIG: Arg list too long

[beckerzito]

Hey guys,

Recently I've been working in a C project that has a long folder structure. With that, in my first compilation attempt, I got the error "Errno::E2BIG: Arg list too long" due to the big include argument list generated for the project.

My question is: Have you already faced it? How did you manage to solve that?

I was thinking in some kind of tool to generate the include arguments and then pass to the GCC (compiler that I'm using). But it would require a new process on the building system to generate the includes file.

What do you think?

[beckerzito]

@mvandervoord help me out here! Is there any solution already done for that?

[Letme]

Maybe not related, but I wonder how linux does it... The link list is probably really long, so there might be some trick to avoid that.

Are you using windows? Is that error message from Ruby?

[beckerzito]

Maybe not related, but I wonder how linux does it... The link list is probably really long, so there might be some trick to avoid that.

Are you using windows? Is that error message from Ruby?

Yes I'm using windows. I'm no really sure, but I'm guessing that this is an issue from the command prompt. It happens right after the ruby calls the entire command with gcc.exe and its arguments, including the very long include directories.

[Letme]

Can you count number of arguments you have, and we can maybe get a limit?

[mvandervoord]

I think it's less about the number of arguments and more about total line length? (which is 8191 characters on Windows)

[beckerzito]

Yes, the point is that since the start of the command "gcc.exe" until the end of it with "-DTEST -DGNU_COMPILER", the complete command has almost 37.815 characters.

So, I see that this can be a pretty common issue to the tool. It's not too hard to find big projects on the industries and other applications, that could easily overflow the 8191 digits.

Any thoughts on how to solve that? From my side, I'm trying to modify the ceedling in other to not pass the include directories through the command line, but by the environment variable 'CPATH' that could be a possibility to the GCC, but it doesn't seem like a global solution for other compilers.

Let me know your comments!

[mvandervoord]

is that what most of the characters are in your setup? include directories?

[beckerzito]

Yes, the biggest part of the characters comes from the include directories.

[beckerzito]

Any suggestion on that guys?

[dpostorivo]

I've hit this before on linux when having globs in paths with a lot of directories. My gut is that we can give a smarter error for this, but it would take some edge testing. If you can trim down your include paths to be more specific it would take care of itself, but I agree ceedling should be a bit smarter and give a better error message.

[MichaelBMiner]

I have the same issue trying to use ceedling for an embedded linux project. All of the includes generate a line that is 200K characters long.

Have you found a solution for this? Is there a way to keep my includes and run ceedling?

[serjche]

Hi, I'm in a similar situation, which I didn't account for when establishing ceedling for a big project. Additionally to what has been said in this issue, I observed that the whole list of all the values defined in the :paths: variable in the project.yml file, are always included in all the operations with gcc, regardless if they pertain to the test or not.

For example, let's say that we have two test files in different folders

• driver1/unit_test/test_driver1.c
• driver2/unit_test/test_driver2.c

And a file structure such as:

.
├── driver1
│   ├── inc
│   ├── src
│   └── unit_test
├── driver2
│   ├── inc
│   ├── src
│   └── unit_test

Where the inc and src folders contain the .h and .c files, as you might expect.

Let's say that both drivers are completely independent of each other, they don't requiere headers from each other.

with a project.yml file such as

:paths:
  :tests:
    - driver1/unit_test
    - driver2/unit_test
  :source:
    - driver1/src
    - driver2/src
  :include:
    - driver1/inc
    - driver2/inc

When running ceedling verbosity[4] test:all I found that for each test ran, the preprocess/compilation/linker steps all have the whole list of includes (omitting other gcc options): gcc.exe ... -I"driver1/inc" -I"driver1/src" -I"driver1/unit_test" -I"driver2/inc" -I"driver2/src" -I"driver2/unit_test" ... test_driver1.c

Even though test_driver1 does not need all the includes from driver2, but it does anyway.

This of course, does not affect in small projects, but on bigger ones, as some of you folks have mentioned, provoking the Errno::E2BIG error.

I was trying the CPATH approach by @beckerzito, but had no success so far.

Have any of you reached a solution or a workaround to this predicament?

This issue seems to be more related to the way Ceedling organizes all the include paths in every operation it makes, so I think it is a more fundamental problem that we might think at first.

If there is no known workaround, do you guys have any proposal or idea on how we could implement a way in which only certain paths are included exclusively for certain tests.

I would really appreciate your help. I'll keep trying to find a work around in the meantime.

Thank you!

[SilverBulletEngineering]

I suggest you make two separate projects (yml files). It's easy to switch between projects.

[serjche]

Hi @SilverBulletEngineering, thank you for your suggestion, unfortunately given the structure of my project it is not so convenient, as in my previous example, there are not only two drivers, but at least a couple dozen, so it is not practical to have one project for each one.

[serjche]

Successful workaround found!

On that note I did manage to find a workaround, that involved the CPATH method suggested by @beckerzito, but this involved manual modifications to the ceedling code.

The first Errno::E2BIG error I got was during the preprocessor stage. I tested creating a CPATH environment variable from the project.yml file, which called an external powershell script that displayed all of the needed paths in the project.

I also removed all the paths from the project.yml file, except for the test folders.

This didn't solve the issue completely, it removed the Errno::E2BIG error, but another error occurred:

ERROR: Found no file 'header.h' in search paths.
rake aborted!

(The 'header.h' is just for example purposes)

Which indicated that the preprocessor stage did actually run successfully, and the next error was in another stage.

As mentioned before, the core issue is that all the include paths from the project.yaml list are embedded into the external commands, and following around the code, I found that all of them are saved into a constant array: COLLECTION_PATHS_TEST_SUPPORT_SOURCE_INCLUDE_VENDOR, which in the default configuration is in the defaults.rb file, which configures the different tasks for running the compiler and other external programs.

The line 18 in the following image has the include options:

This and other tasks had the same line, so the solution was to just comment them:

And instead of the first approach of adding CPATH trough the project.yml file, I added this line to the file configurator_setup.rb in the function build_constants_and_accessors (line 57):

Which would create the CPATH environment variable with all the necessary paths that the compiler, linker, preprocessor and other tools might need.

Which of course I know it's a HORRIBLE hack, but after testing that it did work! The Errno::E2BIG was removed and I could run all my tests on my project with many files and long paths

This might not work for other compilers other than GCC, and I'm also not sure if this should be part or not of the code, and how a more structured solution might be integrated into it.

That depends on the other maintainers, what do you guys think?, should this be only a workaround or should this be honed down to a more structured solution, for example, having the option to add the includes to CPATH via .yml configuration.

And if this is not integrated into the Ceedling project, that's fine, I just hope anyone that finds themselves in this predicament can find this comment to apply the workaround.

Thank you all!

Might be worth trying again the experiment with the Linux Kernel in https://github.com/ThrowTheSwitch/Ceedling/issues/480 ;)

[bmcgrath-mp]

I am trying to use ceedling to build and run units tests for an esp32 project. The esp-idf/components/ directory has 2651 directories, 15354 files some of which are include files needed to build or at least be mocked.

I also get the "Errno::E2BIG: Arg list too long" if I try to include the esp-idf/components directory, and this is when building on the throwtheswitch docker image of ubuntu.

I tried to narrow it down a bit (see code snippet below) but still end up with "no such file or directory" type errors. Has anyone successfully solved this issue or can point me to an example of building unit tests for esp32 with ceedling? I don't need to use the esp-idf toolchain to build for target within ceedling, my plan is to keep unit tests on the host via ceedling and target builds via esp-idf. I don't want to use espressif's half baked linux example either. Thanks!

if it helps this is what my project.yml is turning into

    - +:/esp-idf/*/include/**
    # the below shouldn't be necessary but "ceedling files:include" output shows they are omitted if not included explicitly...
    - +:/esp-idf/esp_system/include/*
    - +:/esp-idf/esp_common/include/*
    - +:/esp-idf/soc/**

[jnz86]

Just happened to me too. I added some files to the project and must have just been on the line previously.

Any recommended fix for Windows?

[jnz86]

I'm disappointed this still isn't fixed. It seems like it's probably pretty common for anyone using larger projects in windows with it's long paths.

Trying the work around. But the silence on the issues here is really a bummer.

[Letme]

This is not a Ceedling problem - this is Windows problem.

You have some workarounds on top, but I think best advice is: switch to Linux. Much more developer friendly.

[serjche]

@Letme, really? that's not very helpful on your part.

The error itself is from GCC itself, not windows, although the windows build of GCC might be more susceptible to it, some folks on this very thread said that this has happened also on linux, and on the following issue it is reported for linux as well: https://github.com/ThrowTheSwitch/Ceedling/issues/480

The main reason this happens is the way ceedling takes every single path in the .yaml configuration and appends it to every single compilation step for every single file, which may be unnecessary for most of the test files.

It is a fundamental issue on how ceedling works, but a proper solution might be quite cumbersome to produce, to only include the necessary -I paths on each compilation step (I'm not really an expert on the project, take that with a grain of salt).

@jnz86 were you able to solve the issue with the workaround?

[jnz86]

Yes and No.

I used the exact workaround and still had the error, but before double checking anything. I realized that Ceedling was compiling very slowly, so I removed the all inclusive source line from my project. I much more narrowly include files now, its faster and I'm no longer at the limit.

However, I agree with you that this isn't really a solution and if I keep using this I'll hit the limit again. I've kicked the can down the road a little bit.

The workaround may be flawless and I messed up somewhere else. I have at least two installs of Ruby and both have Ceedling.