mesonbuild
Add support for the TASKING compiler family, and its MIL linking feature
This PR adds support for the TASKING compiler family, and its MIL linking feature as mentioned in #11866.
Can you squash your commits where appropriate? Git history isn't making sense.
Here are the commits I envision:
- Add tricore to CPU families
- Add TASKING compiler support
Should there be any other reason for more commits?
The MIL thing is a bit problematic as it is unique to this compiler and thus makes it a notable maintenance burden (as we don't have access to said compiler at all). All other compilers that I know of do this by outputting plain .o files with extra data in them (this is how link-time optimization works in e.g. Clang et al).
Is there any data on how effective it is compared to plain unity builds? That is, if people could get the same performance using builtin unity support rather than this very toolchain specific thingy, would we need to support MIL at all?
Unfortunately, I do not have any performance data. At first I didn't want to implement the MIL linking for the very reason you mention, but I was unable to make unity build work for the project I was converting to Meson due to some conditional compilation pattern used in almost all files, which pull in a specified part generated headers. With MIL linking, this is not a problem, files are compiled in an identical manner until this MIL intermediate language part.
Not sure how to approach the fact that the compiler is not available for use, the best that I could think of was writing a dummy CLI emulation of the compiler, so some tests could run at the very least.
Thanks for the TASKING Compiler support PR, I successfully used it with TASKING v6.3r1 to build firmware for Infineon TriCore TC375. The output hex file was flashed and it runs successfully on target.
Hi, Is it possible to share sample files or configurations required to set up a build directory for Tasking Tricore please? Thank you.
The cross_build.txt file I am using is attached. cross_build.txt
I ran command :
meson setup --cross-file cross_build.ini builddir
But I am getting errors :
Source dir: D:\work\buildsystem\meson_crossbuild
Build dir: D:\work\buildsystem\meson_crossbuild\builddir
Build type: cross build
Project name: tc_hello
Project version: 1.0
Activating VS 17.8.4
C compiler for the host machine: d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\cctc.exe (cctc 6.3.1.19041558)
Traceback (most recent call last):
File "c:\Python310\lib\site-packages\meson-1.2.99-py3.10.egg\mesonbuild\mesonmain.py", line 194, in run
return options.run_func(options)
... snip ...
File "c:\Python310\lib\site-packages\meson-1.2.99-py3.10.egg\mesonbuild\interpreter\interpreter.py", line 1549, in add_languages_for
mlog.bold(' '.join(comp.linker.get_exelist())), comp.linker.id, comp.linker.version)
File "c:\Python310\lib\site-packages\meson-1.2.99-py3.10.egg\mesonbuild\linkers\linkers.py", line 160, in get_exelist
return self.exelist.copy()
AttributeError: 'NoneType' object has no attribute 'copy'
meson.build:1:0: ERROR: Unhandled python exception
This is a Meson bug and should be reported!
One other question, I don't understand why I am getting :
Activating VS 17.8.4
Is there something I missed in my configurations ? I am not expecting my Visual Studio installation to be used in this build.
Thank you.
This should work, however, I did not implement support for C++. Not sure why it wants VS, for good measure I would pass in --wipe when configuring.
[constants]
tools_folder = 'd:/Bitbucket/Tools_Inv_HMC_IDM'
compiler_path = tools_folder / 'Compiler'
[built-in options]
c_args = ['-D__CPU__=tc37x', '-D__CPU_TC37X__','-D_TASKING_C_TRICORE_','-g','--align=4','--iso=99']
[project options]
[binaries]
c = compiler_path / 'ctc' / 'bin' / 'cctc.exe'
#cpp = compiler_path / 'ctc' / 'bin' / 'cctc.exe' I did not implement support for C++, it would be cptc.exe
ld = compiler_path / 'ctc' / 'bin' / 'cctc.exe'
ar = compiler_path / 'ctc' / 'bin' / 'artc.exe'
as = compiler_path / 'ctc' / 'bin' / 'cctc.exe'
exe_wrapper = 'meson_exe_wrapper.sh'
[properties]
[host_machine]
system = 'none'
#subsystem = ''
kernel = 'none'
cpu_family = 'tricore'
cpu = 'TC377'
endian = 'little'
Hi @gerioldman,
Thank you for your reply. I tried the settings you gave but it did not help. I am still getting the same error.
I will continue to look into this as my team is trying to switch to meson/ninja. We are on a gmake based build system currently. I think the VS 17.8.4 may be suspicious. I do have VS (Visual Studio) and VS build Tools installed but they shouldn't be used or detected in this case.
update : I ported meson and my project to a computer without a VS (Visual Studio) installation, and I got the following :
WARNING: Failed to activate VS environment: Could not find C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe
But I am still getting the same error as in the previous post.
regards, TseTsong, Teo
Hi @tsetsong
Are you sure you are running this PR meson and not a previous version of it ?
Try running it as:
<PR checkout location>/meson.py setup --cross-file cross_build.ini builddir
Hi @tsetsong Are you sure you are running this PR meson and not a previous version of it ? Try running it as:
<PR checkout location>/meson.py setup --cross-file cross_build.ini builddir
Thank you @afahmy-ANGENG for your reply. I should have attached details of the version of the repo I am using.
I work on a Windows 10 machine.
Here are repository details and how I get the error :
D:\work\mesontasking>git log -1
commit 4f44bbea6d81b059cd5d00fc3b976d5f034d0ba4 (HEAD -> TaskingCCompiler, origin/TaskingCCompiler)
Author: Krisztián Gergő <[email protected]>
Date: Fri Dec 22 20:30:04 2023 +0100
Update mesonbuild/backend/ninjabackend.py
Co-authored-by: Charles Brunet <[email protected]>
D:\work\mesontasking>git branch
* TaskingCCompiler
master
D:\work\mesontasking>git remote
origin
D:\work\mesontasking>git remote -v
origin https://github.com/gerioldman/meson.git (fetch)
origin https://github.com/gerioldman/meson.git (push)
And I built and installed my meson like this :
cd d:\work\mesontasking
python setup.py build
python setup.py install
My python installation is at
C:\Program Files\Python311
D:\work\mesontasking>python --version
Python 3.11.6
My Ninja is at : c:\Ninja
According to your instructions, I have done the following from my project folder :
d:\work\mesontasking>meson.py setup --cross-file cross_build.txt --wipe builddir
The output is the same :
The Meson build system
Version: 1.2.99
Source dir: D:\work\meson_crossbuild
Build dir: D:\work\meson_crossbuild\builddir
Build type: cross build
Project name: tc_hello
Project version: 1.0
WARNING: Failed to activate VS environment: Could not find C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe
C compiler for the host machine: d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\cctc.exe (cctc 6.3.1.19041558)
Traceback (most recent call last):
File "D:\work\mesontasking\mesonbuild\mesonmain.py", line 194, in run
return options.run_func(options)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\msetup.py", line 356, in run
app.generate()
File "D:\work\mesontasking\mesonbuild\msetup.py", line 179, in generate
return self._generate(env, capture, vslite_ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\msetup.py", line 201, in _generate
intr = interpreter.Interpreter(b, user_defined_options=user_defined_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\interpreter\interpreter.py", line 331, in __init__
self.parse_project()
File "D:\work\mesontasking\mesonbuild\interpreterbase\interpreterbase.py", line 132, in parse_project
self.evaluate_codeblock(self.ast, end=1)
File "D:\work\mesontasking\mesonbuild\interpreterbase\interpreterbase.py", line 188, in evaluate_codeblock
raise e
File "D:\work\mesontasking\mesonbuild\interpreterbase\interpreterbase.py", line 180, in evaluate_codeblock
self.evaluate_statement(cur)
File "D:\work\mesontasking\mesonbuild\interpreterbase\interpreterbase.py", line 194, in evaluate_statement
return self.function_call(cur)
^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\interpreterbase\interpreterbase.py", line 520, in function_call
res = func(node, func_args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\interpreterbase\decorators.py", line 260, in wrapper
return f(*nargs, **wrapped_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\interpreterbase\decorators.py", line 579, in wrapper
return f(*wrapped_args, **wrapped_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\interpreter\interpreter.py", line 1308, in func_project
self.add_languages(proj_langs, True, MachineChoice.HOST)
File "D:\work\mesontasking\mesonbuild\interpreter\interpreter.py", line 1476, in add_languages
success = self.add_languages_for(args, required, for_machine)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\interpreter\interpreter.py", line 1549, in add_languages_for
mlog.bold(' '.join(comp.linker.get_exelist())), comp.linker.id, comp.linker.version)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\work\mesontasking\mesonbuild\linkers\linkers.py", line 160, in get_exelist
return self.exelist.copy()
^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'copy'
meson.build:1:0: ERROR: Unhandled python exception
Hi all,
Update :
The line :
WARNING: Failed to activate VS environment: ...
or
Activating VS 17.8.4
Will not show if you have gcc installed. The section of meson code responsible for this is in Interpreter.py :
.. snip ...
if not force:
if shutil.which('cc'):
return False
if shutil.which('gcc'):
return False
if shutil.which('clang'):
return False
if shutil.which('clang-cl'):
return False
It is possible to follow the instructions to install MinGW here : https://dev.to/gamegods3/how-to-install-gcc-in-windows-10-the-easier-way-422j and include the path to gcc.exe in PATH.
This will allow the shutil.which('gcc') to be True, and the function will return with 'False'.
However, this does not solve the original problem of the generation crash. I am still looking into this.
Hi,
Update :
I am now able to generate my build folder.
An additional modification is needed in my cross_build.txt file.
The
ld = compiler_path / 'ctc' / 'bin' / 'cctc.exe'
entry needs to be specified as :
c_ld = compiler_path / 'ctc' / 'bin' / 'cctc.exe'
I will proceed with compilation and porting and will report any other findings here as well.
Here is the stdout :
d:\work\buildsystem\meson_crossbuild>meson.exe setup --cross-file cross_build.txt --wipe builddir
The Meson build system
Version: 1.2.99
Source dir: D:\work\buildsystem\meson_crossbuild
Build dir: D:\work\buildsystem\meson_crossbuild\builddir
Build type: cross build
Project name: tc_hello
Project version: 1.0
WARNING: Failed to activate VS environment: Could not find C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe
C compiler for the host machine: d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\cctc.exe (cctc 6.3.1.19041558)
C linker for the host machine: d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\cctc.exe ltc 6.3.1.19041558
Compiler for language c for the build machine not found.
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: tricore
Host machine cpu: TC377
Target machine cpu family: tricore
Target machine cpu: TC377
Build targets in project: 1
tc_hello 1.0
User defined options
Cross files: cross_build.txt
Found ninja-1.11.1 at c:\ninja\ninja.EXE
WARNING: Cross file does not specify strip binary, result will not be stripped.
WARNING: Cross file does not specify strip binary, result will not be stripped.
I have also attached the build setup log (meson-log.txt file for reference. meson-log.txt
Cross_build file : cross_build.txt
Command ran :
meson.exe setup --cross-file cross_build.txt --wipe builddir
I notice in the meson-log.txt file that the archiver is used instead of the linker when detecting linker :
Detecting linker via: `d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\artc.exe --version` -> 0
stderr:
TASKING VX-toolset for TriCore: ELF archiver v6.3r1 Build 19041558
Copyright 2002-2019 TASKING BV
How come ?
I notice in the meson-log.txt file that the archiver is used instead of the linker when detecting linker :
Detecting linker via: `d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\artc.exe --version` -> 0 stderr: TASKING VX-toolset for TriCore: ELF archiver v6.3r1 Build 19041558 Copyright 2002-2019 TASKING BV
How come ?
That looks like a Meson bug, we have the same message for both the archiver and dynamic linker... I'll send a patch for that.
edit: https://github.com/mesonbuild/meson/pull/12784
On the .o / .mil thing, multiple .mil files form one .o file and then the one .o file gets linked into a single binary?
If so, then if you squint at it, it basically resembles more standard compilers where a .mil is effectively a .o and then creation of the .o and final outputs are two halves of the exact same link rule.
Also worth noting is that it's not exactly unheard of for the compiled objects to not be "traditional objects", because LTO bytecode isn't traditional objects and not all forms of it have a traditional objects glued to the side either.
(The fixups should be committed via git commit --amend and force pushed.)
On the .o / .mil thing, multiple .mil files form one .o file and then the one .o file gets linked into a single binary?
Yes, source files are compiled to .mil files and form one .o file, then linked into a single binary. However, this initial linking is where .mil files form a single .o file is only required when static linking. If you pass '--mil-link' to the compiler and pass in .mil files, this stage can be omitted. An additional thing to note is that source files written in assembly cannot be turned into .mil files, as the assembler's input is the MIL format, as far as I understand the compiler, so for those files, the regular compilation command has to be used.
All in all, the file flow could look something like this:
Hi !
Thank you for the changes. I am working now with
commit 29526ffbaece51f7e0db4f2ea4594da89e761bd8 (HEAD -> TaskingCCompiler, origin/TaskingCCompiler)
And if I do not specify "c_ld" in my cross-build.txt file, there is now an error informing me about it. Thank you.
I tried compiling and linking using the generated ninja build scripts today and found the following :
- all my .c files compiles
- elf output has been generated
- link error is reported even though link completed
FAILED: tc_hello.elf
"d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\cctc.exe" -o tc_hello.elf tc_hello.elf.p/sources_source_hello.c.o tc_hello.elf.p/sources_source_cstart.c.o tc_hello.elf.p/sources_source_cstart_tc1.c.o tc_hello.elf.p/sources_source_cstart_tc2.c.o tc_hello.elf.p/sources_source_sync_on_halt.c.o "" "-Ctc27xb" "--lsl-core=vtc" "-t" "../tc_helloc.lsl" "-Wl-OtxycL" "--map-file" "-Wl-mcrfiklSmNOduQ" "--no-warnings" "-Wl--error-limit=42" "-g" "--fp-model=clfznrST" "--c++=14" "--pass-linker=--output=tc_hello.s19:srec"
cctc E205: don't know what to do with
ninja: build stopped: subcommand failed.
On closer examination, my link command looks like this :
[6/6] "d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\cctc.exe" -o tc_hello.elf tc_hello.elf.p/sources_source_hello.c.o tc_hello.elf.p/sources_source_cstart.c.o tc_hello.elf.p/sources_source_cstart_tc1.c.o tc_hello.elf.p/sources_source_cstart_tc2.c.o tc_hello.elf.p/sources_source_sync_on_halt.c.o "" "-Ctc27xb" "--lsl-core=vtc" "-t" "../tc_helloc.lsl" "-Wl-OtxycL" "--map-file" "-Wl-mcrfiklSmNOduQ" "--no-warnings" "-Wl--error-limit=42" "-g" "--fp-model=clfznrST" "--c++=14" "--pass-linker=--output=tc_hello.s19:srec"
Notice that there is an empty argument "" just before the option "-Crc27xb" ? It seems this is causing the link error. The problem is also detectable in build.ninja file :
build tc_hello.elf: c_LINKER tc_hello.elf.p/sources_source_hello.c.o tc_hello.elf.p/sources_source_cstart.c.o tc_hello.elf.p/sources_source_cstart_tc1.c.o tc_hello.elf.p/sources_source_cstart_tc2.c.o tc_hello.elf.p/sources_source_sync_on_halt.c.o
LINK_ARGS = "" "-Ctc27xb" "--lsl-core=vtc" "-t" "../tc_helloc.lsl" "-Wl-OtxycL" "--map-file" "-Wl-mcrfiklSmNOduQ" "--no-warnings" "-Wl--error-limit=42" "-g" "--fp-model=clfznrST" "--c++=14" "--pass-linker=--output=tc_hello.s19:srec"
I tried to locate the source of the problem in the meson code, and looked at "tasking.py" :
def get_tasking_mil_link_args(self, option_enabled: bool) -> T.List[str]:
return ['--mil-link'] if option_enabled else ['']
I removed the '' in "else ['']" :
def get_tasking_mil_link_args(self, option_enabled: bool) -> T.List[str]:
return ['--mil-link'] if option_enabled else []
Can someone review and evaluate if this change is ok ? My linking completed without errors after the above change, but I am not sure if problems will be caused else where, so I need at least one more pair of expert eyes here.
Thank you.
Hi,
I have one other question.
The tasking linker produces an elf file by default.
I have attached my meson.build file for reference.
I had to specify the project name as "tc_hello.elf" and have this passed as an argument into executable(), in order to have the elf file named correctly. Otherwise, "tc_hello" will be produced (but it's an elf file).
Is there anyway I can get the produced elf file to be named correctly ?
Thanks!
@tsetsong Thanks for the feedback, that was a strong typo, I have corrected it. I also added that by default, the files should have .elf suffix. There is a name_suffix option where you can specify it as well.
Hi again,
I am now trying to build libraries with the tasking archiver and am encountering two problems :
- The build system assumes a .so extension (e.g. mylib.so), In tasking, it be ".a"
- The linker is called instead of the archiver. In this case, we need artc.exe to be called but cctc.exe is called instead.
I am attaching my meson.build file and the relevant declarations in the file is :
mylib_sources=[
'libs/test_lib/source/lib_test_main.c'
]
mylib_headers= include_directories('libs/test_lib/header')
library('mylib', mylib_sources,include_directories:mylib_headers)
The library cannot be produced but the single .o file used for the library has been built.
Here's the error message I get :
[7/8] "d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\cctc.exe" -o libmylib.so libmylib.so.p/libs_test_lib_source_lib_test_main.c.o "-Ctc27xb" "--lsl-core=vtc" "-t" "../tc_helloc.lsl" "-Wl-OtxycL" "--map-file" "-Wl-mcrfiklSmNOduQ" "--no-warnings" "-Wl--error-limit=42" "-g" "--fp-model=clfznrST" "--c++=14" "--pass-linker=--output=tc_hello.s19:srec"
FAILED: libmylib.so
"d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\cctc.exe" -o libmylib.so libmylib.so.p/libs_test_lib_source_lib_test_main.c.o "-Ctc27xb" "--lsl-core=vtc" "-t" "../tc_helloc.lsl" "-Wl-OtxycL" "--map-file" "-Wl-mcrfiklSmNOduQ" "--no-warnings" "-Wl--error-limit=42" "-g" "--fp-model=clfznrST" "--c++=14" "--pass-linker=--output=tc_hello.s19:srec"
ltc E106: unresolved external: main - (cstart.o)
ltc F019: unrecoverable error: fatal link error
I also attach my cross-build file for reference :
My full meson.build is also attached :
I am using this version of meson :
D:\work\mesontasking>git log -1
commit a10504f70a922907abf7b9381c3714897679d00e (HEAD -> TaskingCCompiler, origin/TaskingCCompiler)
Author: gerioldman <[email protected]>
Date: Sat Sep 23 00:33:39 2023 +0200
Add TASKING compiler support
Commands used :
d:\work\mesontasking\meson.py setup --cross-file cross_build.txt builddir --wipe
meson compile -v --ninja-args="-dkeepdepfile"
I'll be looking into meson code to see what I can find.
Thanks again for the support !
The library function builds shared libraries by default it seems, try using the static_library function, or set the default to static.
Hi @gerioldman,
The library function builds shared libraries by default it seems, try using the static_library function, or set the default to static.
Thanks! This allows me to build the library with .a file output.
modified meson.build file attached. meson.build.txt
[3/8] "d:/Bitbucket/Tools_Inv_HMC_IDM\Compiler\ctc\bin\artc.exe" "-r" -n libmylib.a libmylib.a.p/libs_test_lib_source_lib_test_main.c.o
Will proceed with additional work from here.
Hi,
I am now testing the command below :
meson compile --clean -v
I notice that the .src files (assembly files) output from the compiler stage is not cleaned. The .o's and the .dep's are deleted, leaving the .src files still in the folder, so I think the clean is partial.
Is there anyway to get these files deleted as well ?
I will also try to look into the code to see if I can find something.
Thanks !
Are you using "--keep-temporary-files" arg? Ninja doesn't "know" about these files, as it only knows about the object and dep files, there is no command that generates .src files in the build file. In any case, it shouldn't cause problems if they aren't cleaned, incremental compilation should be unaffected.
mesonbuild/compilers/c.py:833:9: error: "init" of "CCompiler" gets multiple values for keyword argument "linker"
This seems to be the same as the CodeQL error above.
We try really hard to avoid adding new options without a good reason (because otherwise you have 1000+ options in the blink of an eye). Since this is "sort of" a link time optimization, would it be possible to make the code use the b_lto option instead of its own?