cpython icon indicating copy to clipboard operation
cpython copied to clipboard
verified publisher icon python

Unable to build to WASI

Open thesuhas opened this issue 1 year ago • 4 comments

Bug report

Bug description:

Hi, I've been trying to follow the build instructions from here to build to wasi. I get the following error when running python3 Tools/wasm/wasi.py build --quiet -- --config-cache --with-pydebug: image

I see that the clang from wasi-sdk was correctly picked up but it's unable to use it? This is the relevant data from config.log:

configure:4708: result: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
configure:4830: checking for gcc
configure:4862: result: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/share/wasi-sysroot
configure:5215: checking for C compiler version
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/share/wasi-sysroot --version >&5
./configure: line 5226: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang: cannot execute binary file
configure:5235: $? = 126
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/share/wasi-sysroot -v >&5
./configure: line 5226: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang: cannot execute binary file
configure:5235: $? = 126
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/share/wasi-sysroot -V >&5
./configure: line 5226: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang: cannot execute binary file
configure:5235: $? = 126
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/share/wasi-sysroot -qversion >&5
./configure: line 5226: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang: cannot execute binary file
configure:5235: $? = 126
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/share/wasi-sysroot -version >&5
./configure: line 5226: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang: cannot execute binary file
configure:5235: $? = 126
configure:5255: checking whether the C compiler works
configure:5277: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/share/wasi-sysroot    conftest.c  >&5
./configure: line 5279: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk/wasi-sdk-22.0/bin/clang: cannot execute binary file
configure:5281: $? = 126
configure:5321: result: no
configure: failed program was:
| /* confdefs.h */
| #define _NETBSD_SOURCE 1
| #define __BSD_VISIBLE 1
| #define _DARWIN_C_SOURCE 1
| #define _PYTHONFRAMEWORK ""
| /* end confdefs.h.  */
| 
| int
| main (void)
| {
| 
|   ;
|   return 0;
| }
configure:5326: error: in `/Users/suhasthalanki/Documents/CMU/WASM/cpython':
configure:5328: error: C compiler cannot create executables
See `config.log' for more details

CPython versions tested on:

3.11

Operating systems tested on:

macOS

thesuhas avatar May 16 '24 18:05 thesuhas

Based on the log it looks like the issue is this:

clang: cannot execute binary file

Perhaps the wrong wasi-sdk was downloaded? Regardless the compiler executable doesn't work on your machine.

Please make sure you download the macOS variant when setting up wasi-sdk image

emmatyping avatar May 20 '24 12:05 emmatyping

Thanks! I just used the steps mentioned in the README file and that did not mention any release, I just pulled the repo and built it. Now I get the following error on downloading the Mac OS specific version: image Logs:

configure:4830: checking for gcc
configure:4862: result: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot
configure:5215: checking for C compiler version
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot --version >&5
clang version 18.1.2 (https://github.com/llvm/llvm-project 26a1d6601d727a96f4301d0d8647b5a42760ae0c)
Target: wasm32-unknown-wasi
Thread model: posix
InstalledDir: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin
configure:5235: $? = 0
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot -v >&5
clang version 18.1.2 (https://github.com/llvm/llvm-project 26a1d6601d727a96f4301d0d8647b5a42760ae0c)
Target: wasm32-unknown-wasi
Thread model: posix
InstalledDir: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin
configure:5235: $? = 0
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot -V >&5
clang: error: argument to '-V' is missing (expected 1 value)
clang: error: no input files
configure:5235: $? = 1
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot -qversion >&5
clang: error: unknown argument '-qversion'; did you mean '--version'?
clang: error: no input files
configure:5235: $? = 1
configure:5224: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot -version >&5
clang: error: unknown argument '-version'; did you mean '--version'?
clang: error: no input files
configure:5235: $? = 1
configure:5255: checking whether the C compiler works
configure:5277: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot    conftest.c  >&5
configure:5281: $? = 0
configure:5331: result: yes
configure:5334: checking for C compiler default output file name
configure:5336: result: a.out
configure:5342: checking for suffix of executables
configure:5349: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot -o conftest    conftest.c  >&5
configure:5353: $? = 0
configure:5376: result: 
configure:5398: checking whether we are cross compiling
configure:5406: /Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/bin/clang --sysroot=/Users/suhasthalanki/Documents/CMU/WASM/wasi-sdk-22.0/share/wasi-sysroot -o conftest    conftest.c  >&5
configure:5410: $? = 0
configure:5417: ./conftest
./configure: line 5419: ./conftest: cannot execute binary file
configure:5421: $? = 126
configure:5428: error: in `/Users/suhasthalanki/Documents/CMU/WASM/cpython':
configure:5430: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details

Fails in the first command mentioned here: image

thesuhas avatar May 20 '24 15:05 thesuhas

Oh I see, you're not supposed to use the wasi-sdk for the build Python. Did you perhaps run something else before running the below?

python3 Tools/wasm/wasi.py build --quiet -- --config-cache --with-pydebug

If not, then its probably a bug in the script.

Regardless, you should be able to build things correctly if you remove --config-cache.

emmatyping avatar May 20 '24 18:05 emmatyping

Oh I see, you're not supposed to use the wasi-sdk for the build Python. Did you perhaps run something else before running the below?

What do you mean by not supposed to use wasi-sdk? I thought that was mentioned in the documentation. I did not run anything before that.

I now pass the first part but it fails the next part image

thesuhas avatar May 20 '24 22:05 thesuhas

To be clear, the build requires two stages. First, we build a Python for your machine, which is required to set up a second Python build for WASM.

Your issue was you were using the WASI-SDK for the first build, when it should only be used for the second. Can you upload the log mentioned in your screenshot?

emmatyping avatar May 21 '24 14:05 emmatyping

I understand that, and I believe python3 Tools/wasm/wasi.py build --quiet -- --config-cache --with-pydebug does both the builds, as per the documentation. The logs:

gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Programs/python.o ../../Programs/python.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/token.o ../../Parser/token.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/pegen.o ../../Parser/pegen.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/pegen_errors.o ../../Parser/pegen_errors.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/action_helpers.o ../../Parser/action_helpers.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/parser.o ../../Parser/parser.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/string_parser.o ../../Parser/string_parser.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/peg_api.o ../../Parser/peg_api.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/lexer/buffer.o ../../Parser/lexer/buffer.c
gcc -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Parser/lexer/lexer.o ../../Parser/lexer/lexer.c
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/peg_api.c:1:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/pegen_errors.c:1:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/token.c:3:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/pegen.c:1:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/pegen.h:4,
                 from ../../Parser/parser.c:2:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/lexer/buffer.c:1:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Programs/python.c:3:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/string_parser.c:3:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/lexer/lexer.c:1:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/assert.h:42,
                 from ../../Include/Python.h:19,
                 from ../../Parser/action_helpers.c:1:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:30: error: missing ')' after "__has_attribute"
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                              ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include/sys/cdefs.h:554:31: error:  ':' without preceding '?'
  554 | #if __has_cpp_attribute(clang::unsafe_buffer_usage)
      |                               ^
make: *** [Programs/python.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [Parser/token.o] Error 1
make: *** [Parser/string_parser.o] Error 1
make: *** [Parser/pegen_errors.o] Error 1
make: *** [Parser/peg_api.o] Error 1
make: *** [Parser/lexer/buffer.o] Error 1
make: *** [Parser/lexer/lexer.o] Error 1
make: *** [Parser/pegen.o] Error 1
make: *** [Parser/action_helpers.o] Error 1
make: *** [Parser/parser.o] Error 1

thesuhas avatar May 21 '24 14:05 thesuhas

[the script] does both the builds, as per the documentation.

Yep, but it was incorrectly using the incorrect CC variable, probably because your configure cache was incorrect AIUI.

I'm sorry but I'm not familiar with building on macOS, I think the script may be making some error but it's not clear to me what is going wrong now.

emmatyping avatar May 21 '24 18:05 emmatyping

I now pass the first part but it fails the next part

Unfortunately you left out the logs. I wouldn't build w/ --quiet until you get a successful build. I also wouldn't include anything past -- until you succeed.

I understand that, and I believe python3 Tools/wasm/wasi.py build --quiet -- --config-cache --with-pydebug does both the builds, as per the documentation.

It does, but it looks like your build got in a weird place.

Also note you are using an unsupported version of WASI SDK; only SDK 21 has been verified to work (see the instructions you linked to which says how to check what the latest version has been verified to work).

The logs:

That's an issue of the compiler pulling in C++ code from macOS and that isn't WASI-specific.

I'm sorry but I'm not familiar with building on macOS, I think the script may be making some error but it's not clear to me what is going wrong now.

FYI I developed the script initially on macOS, so it at least used to work w/o issue.

brettcannon avatar Jun 11 '24 20:06 brettcannon

@thesuhas I'm going to assume this isn't an issue anymore as we haven't heard back from you. If you're still having problems, then please include more logging and use WASI SDK 21.

brettcannon avatar Jul 08 '24 18:07 brettcannon