Having trouble in building with cross-compiler

[jongwonjlee]

Hi, I appreciate your hard work! I'm now trying to create BLIS static library for a target microcontroller (NXP MPC5748G, having 32bit PowerPC architecture on which code built by GCC is mounted) through the corresponding cross compiler with its specialized C standard library called EWL, but I'm in trouble while building the code.

Prerequisite:

• having the corresponding GCC cross-compiler (powerpc-eabivle-gcc) for the target device on the host machine in the directory of `${HOME}/NXP/S32DS_Power_v2.1/S32DS/build_tools/powerpc-eabivle-4_9/bin/'
• having the corresponding standard C libraries (EWL) for the target device on the host machine in the directory of `${HOME}/NXP/S32DS_Power_v2.1/S32DS/build_tools/e200_ewl2/'

What I tried was:

• Go into config/generic/ and populate make_defs.mk with details / flags on the target cross compiler. (I attach the make_defs.mk file I used.)
• Run command ./configure CC=${HOME}/NXP/S32DS_Power_v2.1/S32DS/build_tools/powerpc-eabivle-4_9/bin/powerpc-eabivle-gcc --disable-shared --disable-threading --disable-system --enable-verbose-make generic to genrate Makefile.
• As make -j encounters "Cannot determine operating system" error, I go into include/generic/blis.h file, comment two blocks causing the error, and explicitly define #define BLIS_OS_GNU 1 (I attach the modified header file I used.)
• Run make -j, but it shows warning statements pertaining to using fprintf. (I attach the build logs (containing standard statements and warnings separately) as well.)

Anyway, it ends up creating the static library file libblis.a. But when I try to link it with the BLIS tutorial code, it throws similar errors as mentioned above; this is not so surprising as the library file has been built 'incomplete'.

As the system root (where the standard C libraries are located) has been set properly to be ${HOME}/NXP/S32DS_Power_v2.1/S32DS/build_tools/e200_ewl2/' and stdio` is in there, I believe that it wouldn't cause such issues but it didn't. This problem has eaten up to much time and effort so far. It would be really appreciated if anyone knows a silver bullet to get around such an issue. Otherwise, it would be also great if somebody can suggest a possible direction I should take. Thanks in advance!

files mentioned above

files.zip

Checklist (with documentation of solutions):

• [ ] Define appropriate OS macro.
  • Maybe BLIS_OS_NONE?
• [x] Select cross-compiler
  • Specify CC during configure.
• [ ] system_root/flags for EWL
• [ ] Platform-specific flags (in config directory)
• [ ] Platform-specific flags (CFLAGS at configure time)
• [x] fprintf
  • Needs platform-specific macro (which one(s)?)
• [ ] Linker problems

[hominhquan]

@jongwonjlee I'm also interested in the portability of BLIS on embedded platforms. Can you share the error of But when I try to link it with the BLIS tutorial code, it throws similar errors as mentioned above; this is not so surprising as the library file has been built 'incomplete'. ? I did not see any error log in your zip file.

Then, I believe "Cannot determine operating system" is a preprocessing error (https://github.com/flame/blis/blob/master/frame/include/bli_system.h#L94), while you mentioned a link issue ?

[jongwonjlee]

@hominhquan It states as follows when linked to the tutorial code:

10:22:53 **** Incremental Build of configuration Debug for project hello_Z4_0 ****
make -j16 all 
Building file: ../src/main.c
Invoking: Standard S32DS C Compiler
powerpc-eabivle-gcc "@src/MPC57xx__Interrupt_Init.args" -MMD -MP -MF"src/main.d" -MT"src/main.o" -o "src/main.o" "../src/main.c"
Finished building: ../src/main.c
 
Building target: hello_Z4_0.elf
Invoking: Standard S32DS C Linker
powerpc-eabivle-gcc -o "hello_Z4_0.elf" "@hello_Z4_0.args"    
/home/jlee/NXP/S32DS_Power_v2.1/S32DS/build_tools/powerpc-eabivle-4_9/bin/../lib/gcc/powerpc-eabivle/4.9.4/../../../../powerpc-eabivle/bin/ld: hello_Z4_0.elf section `.text' will not fit in region `m_text'
/home/jlee/NXP/S32DS_Power_v2.1/S32DS/build_tools/powerpc-eabivle-4_9/bin/../lib/gcc/powerpc-eabivle/4.9.4/../../../../powerpc-eabivle/bin/ld: region `m_text' overflowed by 216234 bytes
/home/jlee/workspace/S32DS.Power.2.1/hello/hello_Z4_0/lib/libblis.a(bli_error.o): In function `bli_print_msg':
/home/jlee/workspace/blis/frame/base/bli_error.c:122: undefined reference to `fprintf'
/home/jlee/workspace/blis/frame/base/bli_error.c:123: undefined reference to `fprintf'
/home/jlee/workspace/S32DS.Power.2.1/hello/hello_Z4_0/lib/libblis.a(bli_util_oapi_ba.o): In function `bli_fprintm':
/home/jlee/workspace/blis/frame/util/bli_util_oapi.c:643: undefined reference to `fprintf'
/home/jlee/workspace/blis/frame/util/bli_util_oapi.c:643: undefined reference to `fprintf'
/home/jlee/workspace/blis/frame/util/bli_util_oapi.c:643: undefined reference to `fprintf'
/home/jlee/workspace/S32DS.Power.2.1/hello/hello_Z4_0/lib/libblis.a(bli_util_oapi_ba.o):/home/jlee/workspace/blis/frame/util/bli_util_oapi.c:643: more undefined references to `fprintf' follow
collect2: error: ld returned 1 exit status
make: *** [makefile:37: hello_Z4_0.elf] Error 1

10:22:56 Build Finished (took 2s.668ms)

For the second question, yes it seems to be a preprocessing error when making the BLIS library file. Doesn't it have to do with configuring for the target processor?

[hominhquan]

On the linking issue, it looks like your compiler toolchain does not provide any fprintf implementation.

On the preprocessing error "Cannot determine operating system", yes, one who wants to port BLIS on a new architecture (as well as new OS) needs adding associated macros in bli_system.h. There are also other things to do, like:

• adding a custom implementation (if any) of bli_clock(), mapped on OS-provided timer
• check for Pthread/OpenMP support.
• For performance, tuning blocksizes accordingly to HW cache sizes (L1, L2, L3). See https://www.cs.utexas.edu/users/flame/pubs/TOMS-BLIS-Analytical.pdf
• For more performance, writing micro-kernels. See https://github.com/flame/blis/blob/master/docs/KernelsHowTo.md
• ... optimization never ends

Good luck.

[jongwonjlee]

@hominhquan Oh, the target processor's OS (FreeRTOS) is classified into none of them in the list that BLIS is supposed to support. Does it mean that I can't use BLIS for the target environment?

[hominhquan]

BLIS has been written as much portable as possible. Any compiler supporting C99 can compile BLIS mostly out-of-the-box. Any new architecture not figuring in the OS list can still be cross-compiled after some code modification (points mentioned above). That can be some more tricky, but is not impossible either.

Then you can create PR to contribute to BLIS. I think @fgvanzee, @rvdg and @devinamatthews will be more than happy to see BLIS ported on new architectures.

[devinamatthews]

@jongwonjlee I've added a checklist to the Issue description to try and break down the different sub-issues, then we can work down the list one at a time.

[devinamatthews]

Re BLIS_OS_<X>, there are three options:

1. Add a new OS macro, with appropriate checks for predefined macro(s). As @hominhquan suggests, there may be a number of other places in the code which would need updates.
2. If one of the other BLIS_OS_<X> macros does what your platform needs:
   1. Supply -DBLIS_OS_WHATEVER during configuration: ./configure CFLAGS="-DBLIS_OS_WHATEVER ..." ..., and modify bli_system.h to detect a predefined OS macro.
   2. Add new checks for predefined macros in bli_system.h so that your platform gets detected as the "correct" OS.

[devinamatthews]

Re fprintf: worst case you can write a simple fprintf yourself which just calls vprintf (and if vprintf isn't available, maybe define fprintf as a variadic macro).

[devinamatthews]

FYI, it definitely seems that you'll need to create your own configuration, whether or not you eventually want to write optimized microkernels or not. Please see here for instructions.

[jongwonjlee]

Re BLIS_OS_<X>, there are three options:

1. Add a new OS macro, with appropriate checks for predefined macro(s). As @hominhquan suggests, there may be a number of other places in the code which would need updates.

2. If one of the other BLIS_OS_<X> macros does what your platform needs:

   1. Supply -DBLIS_OS_WHATEVER during configuration: ./configure CFLAGS="-DBLIS_OS_WHATEVER ..." ..., and modify bli_system.h to detect a predefined OS macro.
   2. Add new checks for predefined macros in bli_system.h so that your platform gets detected as the "correct" OS.

OS macros except Windows and OSX cause no problem while running make after configuring. (When it comes to enforcing either Windows or OSX, they throw errors while including header files of std c libraries, such as #include <windows.h> or #include <mach/mach_time.h>). Thus, from now on, I enforce the OS macro to be __gnu_hurd__ unless otherwise mentioned.

$ ./configure CC=${HOME}/NXP/S32DS_Power_v2.1/S32DS/build_tools/powerpc-eabivle-4_9/bin/powerpc-eabivle-gcc CFLAGS=-D"__gnu_hurd__" --disable-shared --disable-threading --disable-system --enable-verbose-make generic

[jongwonjlee]

@devinamatthews Thanks for your comments. I am stilling with fprintf implicit declaration error and cannot figure out how to resolve this issue.

Why do you think the error occurs though the EWL library supports fprintf (as mentioned here)? What would be the best solution to sort this out?

FYI, I attached the log file while doing make.

[devinamatthews]

@jongwonjlee please attach the file include/generic/blis.h. You might also try specifically looking at the EWL headers to verify that fprintf is in there (maybe it needs a macro defined to be accessible?).

[devinamatthews]

BTW, can you also send the output of powerpc-eabivle-gcc -dM -E - < /dev/null?

[devinamatthews]

@jongwonjlee shouldn't you have to explicitly link some library or libraries with -l for EWL?

[jongwonjlee]

@jongwonjlee please attach the file include/generic/blis.h. You might also try specifically looking at the EWL headers to verify that fprintf is in there (maybe it needs a macro defined to be accessible?).

@devinamatthews Thanks for your suggestion. Enforcing the macro required for EWL's fprintf resolved the issue, like below:

./configure CC=${HOME}/NXP/S32DS_Power_v2.1/S32DS/build_tools/powerpc-eabivle-4_9/bin/powerpc-eabivle-gcc CFLAGS="-D__gnu_hurd__ -D_EWL_OS_DISK_FILE_SUPPORT" --disable-shared --disable-threading --disable-system --enable-verbose-make generic

BTW, it faced warnings as stated below. Do you think this may result in a significant problem later on? (I attach the log while executing make.)

frame/base/noopt/bli_dlamch.c: In function 'bli_dlamch':
frame/base/noopt/bli_dlamch.c:1060:2: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]

[jongwonjlee]

@jongwonjlee please attach the file include/generic/blis.h. You might also try specifically looking at the EWL headers to verify that fprintf is in there (maybe it needs a macro defined to be accessible?).

@devinamatthews FYI, I attach the resultant zip file by running the configuration causing no error while making.

./configure CC=${HOME}/NXP/S32DS_Power_v2.1/S32DS/build_tools/powerpc-eabivle-4_9/bin/powerpc-eabivle-gcc CFLAGS="-D__gnu_hurd__ -D_EWL_OS_DISK_FILE_SUPPORT" --disable-shared --disable-threading --disable-system --enable-verbose-make generic

[jongwonjlee]

BTW, can you also send the output of powerpc-eabivle-gcc -dM -E - < /dev/null?

@devinamatthews For another FYI, I also attach macros for the cross compiler I'm using.

[devinamatthews]

BTW, it faced warnings as stated below. Do you think this may result in a significant problem later on? (I attach the log while executing make.)

frame/base/noopt/bli_dlamch.c: In function 'bli_dlamch':
frame/base/noopt/bli_dlamch.c:1060:2: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]

This is fine.

[devinamatthews]

There's not really any pre-defined macro in that list that gives an idea what OS it is/will be running on. @fgvanzee for builds with --disable-system maybe bli_system.h should instead define something like BLIS_OS_NONE?

[devinamatthews]

@jongwonjlee re the flags for EWL, based on the GCC docs I think all you need is -nostdinc -Ipath/to/EWL/include -Ipath/to/EWL/include/<extra_dirs>. These should probably be specified at configure-time.

[fgvanzee]

There's not really any pre-defined macro in that list that gives an idea what OS it is/will be running on. @fgvanzee for builds with --disable-system maybe bli_system.h should instead define something like BLIS_OS_NONE?

If you think this will help, we can do this.

[devinamatthews]

@jongwonjlee my understanding is that you initially got the error "Cannot determine operating system"? @fgvanzee if so then yes this is needed for @jongwonjlee's configuration to work properly without masquerading as HURD which seem dangerous.

[fgvanzee]

@devinamatthews Take a look at my modifications to bli_system.h. Do you see anything wrong with it? Three of our four AppVeyor builds are failing now.

[devinamatthews]

Are you sure BLIS_ENABLE_SYSTEM is getting defined? It looks like not.

[fgvanzee]

Oh, maybe I need to re-express the conditional in terms of BLIS_DISABLE_SYSTEM. I'll check.

[fgvanzee]

So, turns out that's fine. This is from bli_config.h.in:

#if @enable_system@
#define BLIS_ENABLE_SYSTEM
#else
#define BLIS_DISABLE_SYSTEM
#endif

[fgvanzee]

BTW, I've temporarily disabled the changes in 8e0c425, so we're back to passing on AppVeyor.

[jongwonjlee]

Re fprintf: worst case you can write a simple fprintf yourself which just calls vprintf (and if vprintf isn't available, maybe define fprintf as a variadic macro).

@devinamatthews That was the case. As the libc.a library file of the cross-toolchain I'm using doesn't have fprintf symbol as below, I explicitly copied the fprintf.c file from the EWL library and pasted it to the cross-compiling project file I'm working on and now it succeeds to be built.

$ readelf -s libc99.a | grep "fprintf"
File: libc99.a(fprintf.o)
     1: 00000000     0 FILE    LOCAL  DEFAULT  ABS fprintf.c
File: libc99.a(fprintf_s.o)
     1: 00000000     0 FILE    LOCAL  DEFAULT  ABS fprintf_s.c
File: libc99.a(vfprintf.o)
     1: 00000000     0 FILE    LOCAL  DEFAULT  ABS vfprintf.c
    51: 00000000    70 FUNC    GLOBAL DEFAULT   34 vfprintf
File: libc99.a(vfprintf_s.o)
     1: 00000000     0 FILE    LOCAL  DEFAULT  ABS vfprintf_s.c
    51: 00000000    96 FUNC    GLOBAL DEFAULT   34 vfprintf_s

However, the problem is, when I try to go through the tutorial codes (i.e. examples in blis/examples/) with the target microcontroller, it seems like it is stuck with merely generating a very simple matrix. For instance, when I go through tutorials in blis/examples/tapi/00level1v.c, the MCU debugger can no longer proceed at bli_dsetv functions. See the pictures of debugging console with breakpoints below;

My question is, is this a strong red flag that hampers me from move forward? In other words, I am just wondering if this will require a lot of additional trials errors. What is your opinion on this? Otherwise, it is also welcomed if you can point out any suspicious points to be investigated.