Sherpa-Onnx xcFramework lacks support for Native visionOS Development

[carlosmbe]

It would be really nice if Sherpa-Onnx could support native VisionOS Development with SwiftUI. At the moment, the current Sherpa-Onnx.xcframework does not contain the binaries to build a native app with the VisionOS SDK. However, it does let you build an iPad app that runs on VisionOS.

I've tried to modify the source code to implement this.

I was able to update the OnnxRuntime.xcframework by using this branch of OnnxRuntime and adding the change suggested at the very bottom of this discussion.

However, when running building Sherpa-Onnx for VisionOS, I ran into some challenges on the C side that are outside my scope. Specifically with the building.

Moreover, I had to update the toolchain to support VisionOS. This can be found in ./toolchains/ios.toolchain.cmake an updated version should be at toolchains repo here

Afterwards, I modified the build-ios.sh script to build for both the VisionOS Device and Simulator. My script kept failing at the linking stage.

Hopefully this context helps implement visionOS support later.

[csukuangfj]

Sherpa-Onnx.xcframework does not contain the binaries to build a native app with the VisionOS SDK

We do provide vision os onnxruntime libs. Please see https://github.com/csukuangfj/onnxruntime-libs/releases/download/v1.20.2/onnxruntime-vision_os-xros_arm64-1.20.2.zip

It should be straightforward to update https://github.com/k2-fsa/sherpa-onnx/blob/c9d6859df7922279dbfeae6dfb36354c41919175/build-ios.sh#L17 to support vision os.

---

Please describe in detail the difficulties you have.

---

We don't have a mac with apple chips so we cannot test sherpa-onnx with vision os support.

[carlosmbe]

Hello! Thanks for the response. I truly appreciate it.

I tried running my build script with the runtime you shared but still face the same issue. Undefined symbols for architecture arm64: .... .... ld: symbol(s) not found for architecture arm64 clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I've refactored the build script to be as close to the original and still generate the error. It's in this gist

I'm using the following Toolchain

If there's any information I can share, I'd be more than happy to help.

This is a more detailed version of the error

  "_OrtGetApiBase", referenced from:
      ___cxx_global_var_init in sherpa-onnx-offline-speaker-diarization.cc.o
      ___cxx_global_var_init in libsherpa-onnx-core.a[70](online-stream.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[71](online-transducer-decoder.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[84](onnx-utils.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[89](provider.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[91](session.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[101](transpose.cc.o)
      ...
  "_OrtSessionOptionsAppendExecutionProvider_CoreML", referenced from:
      sherpa_onnx::GetSessionOptionsImpl(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, sherpa_onnx::ProviderConfig const*) in libsherpa-onnx-core.a[91](session.cc.o)
ld: symbol(s) not found for architecture arm64
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [bin/sherpa-onnx-offline-speaker-diarization.app/sherpa-onnx-offline-speaker-diarization] Error 1
make[1]: *** [sherpa-onnx/csrc/CMakeFiles/sherpa-onnx-offline-speaker-diarization.dir/all] Error 2
ld: warning: duplicate -rpath '@loader_path' ignored
ld: warning: ignoring duplicate libraries: '-lm'
Undefined symbols for architecture arm64:
  "_OrtGetApiBase", referenced from:
      ___cxx_global_var_init in sherpa-onnx-keyword-spotter.cc.o
      ___cxx_global_var_init in libsherpa-onnx-core.a[4](cat.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[11](hypothesis.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[12](keyword-spotter-impl.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[13](keyword-spotter.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[54](online-conformer-transducer-model.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[61](online-lstm-transducer-model.cc.o)
      ...
  "_OrtSessionOptionsAppendExecutionProvider_CoreML", referenced from:
      sherpa_onnx::GetSessionOptionsImpl(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, sherpa_onnx::ProviderConfig const*) in libsherpa-onnx-core.a[91](session.cc.o)
ld: symbol(s) not found for architecture arm64
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [bin/sherpa-onnx-keyword-spotter.app/sherpa-onnx-keyword-spotter] Error 1
make[1]: *** [sherpa-onnx/csrc/CMakeFiles/sherpa-onnx-keyword-spotter.dir/all] Error 2
Undefined symbols for architecture arm64:
  "_OrtGetApiBase", referenced from:
      ___cxx_global_var_init in sherpa-onnx.cc.o
      ___cxx_global_var_init in libsherpa-onnx-core.a[4](cat.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[11](hypothesis.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[54](online-conformer-transducer-model.cc.o)
      ___cxx_global_var_init.7 in libsherpa-onnx-core.a[56](online-ctc-fst-decoder.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[57](online-ctc-greedy-search-decoder.cc.o)
      ___cxx_global_var_init in libsherpa-onnx-core.a[58](online-ctc-model.cc.o)
      ...
  "_OrtSessionOptionsAppendExecutionProvider_CoreML", referenced from:
      sherpa_onnx::GetSessionOptionsImpl(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, sherpa_onnx::ProviderConfig const*) in libsherpa-onnx-core.a[91](session.cc.o)
ld: symbol(s) not found for architecture arm64
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [bin/sherpa-onnx.app/sherpa-onnx] Error 1
make[1]: *** [sherpa-onnx/csrc/CMakeFiles/sherpa-onnx.dir/all] Error 2
2 warnings generated ...

[csukuangfj]

This is a more detailed version of the error

Do you get the error within xcode or with build-ios.sh?

[carlosmbe]

With build-ios.sh

[csukuangfj]

I suggest that you try the toolchain file from https://github.com/leetal/ios-cmake/blob/master/ios.toolchain.cmake

Remember to pass

-DPLATFORM=VISIONOS

or

-DPLATFORM=SIMULATOR_VISIONOS

to cmake.

Please delete the folder build-ios before you re-try.

[carlosmbe]

Alas, I already tried that but it still results in the same error. This is how build-ios.sh looks now https://gist.github.com/carlosmbe/9803658ff7f575a4916e56df5cbfa1e9

cmake \
  -DBUILD_PIPER_PHONMIZE_EXE=OFF \
  -DBUILD_PIPER_PHONMIZE_TESTS=OFF \
  -DBUILD_ESPEAK_NG_EXE=OFF \
  -DBUILD_ESPEAK_NG_TESTS=OFF \
  -S .. \
  -DCMAKE_TOOLCHAIN_FILE=./toolchains/ios.toolchain.cmake \
  -DPLATFORM=SIMULATOR_VISIONOS \
  -DENABLE_BITCODE=0 \
  -DENABLE_ARC=1 \
  -DENABLE_VISIBILITY=0 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=./install \
  -DBUILD_SHARED_LIBS=OFF \
  -DSHERPA_ONNX_ENABLE_PYTHON=OFF \
  -DSHERPA_ONNX_ENABLE_TESTS=OFF \
  -DSHERPA_ONNX_ENABLE_CHECK=OFF \
  -DSHERPA_ONNX_ENABLE_PORTAUDIO=OFF \
  -DSHERPA_ONNX_ENABLE_JNI=OFF \
  -DSHERPA_ONNX_ENABLE_C_API=ON \
  -DSHERPA_ONNX_ENABLE_WEBSOCKET=OFF \
  -DDEPLOYMENT_TARGET=1.0 \
  -B build/simulator_arm64

cmake --build build/simulator_arm64 -j 14 --verbose

[csukuangfj]

From https://gist.github.com/carlosmbe/9803658ff7f575a4916e56df5cbfa1e9#file-build-vision-sh-L18

Is /onnxruntime-vision_os-xros_arm64-${onnxruntime_version} not for simulator?

But you are using

-DPLATFORM=SIMULATOR_VISIONOS

Have you tried

-DPLATFORM=VISIONOS

Remember to delete build-ios before you re-try.

[csukuangfj]

Please see https://github.com/k2-fsa/sherpa-onnx/pull/1957

I appreciate if you can finish it and create a new PR so that future users can build sherpa-onnx for vision os directly.

[carlosmbe]

Thanks for the branch. I've spent some time on it and I've identified that the linker issue above was caused by DBUILD_SHARED_LIBS=OFF so your change fixes that issue.

However, I tried building a xcFramework but got the following error:

error: unable to find any architecture information in the binary at '.../sherpa-onnx/build-vision-os/build/vision_os_arm64/lib/libsherpa-onnx.a': Unknown header: 0xb17c0de

Based on my research, this error code 0xb17c0de appears to mean "bitcode" and it appears that building for Release on visionOS/iOS/macOS results in bitcode being enabled automatically.

https://github.com/jrouwe/JoltPhysics/issues/1261#issue-2527917780 https://developer.apple.com/forums/thread/714324

Some have suggested disabling LTO should fix the issue. I tried adding the cmake flags to turn it off but that didn't work.

I was able to workaround this by building for Debug instead of Release. So I'm able to build the framework. Theoretically, this shouldn't be an issue as some claim that bitcode was depreciated in Xcode 14.

Xcode seems to have a few issues with the built framework though so I'm working on debugging that and will post an update on this Issue. I suspect that it's due to how I'm handling the .dylib files.

Here's another thread that I've read regarding the bitcode issue https://github.com/facebook/facebook-ios-sdk/issues/1582#issuecomment-741731531

[carlosmbe]

Working build-vision-os.sh for visionOS Devices can be found on this fork https://github.com/carlosmbe/sherpa-onnx-vision-build/tree/vision-os

Here is the gist of the .sh file https://gist.github.com/carlosmbe/3328dc72ec0cc32ce9fd8ec779d543e8

[csukuangfj]

Thanks! By the way, libcargs can be deleted. You don't need it.

[carlosmbe]

Thanks for the info and all of the help!

I've also added a script for building the simulator version of the script. A few things to note for using this vision build:

1. It requires a plist to be in the framework
2. Since this is a debug build of the framework, you'll have to go to Targets->Your Project -> Build Phases ->Link Binary With Library and set the framework's status to optional. Stack overflow I referenced -> Link

https://github.com/carlosmbe/sherpa-onnx-vision-build/blob/vision-os/build-vision-os-sim.sh

Edit: Also embed and sign the frameworks