"Undefined symbols for architecture arm64" when linking a converted framework

[zachgrayio]

Noticed there's a good amount of missing symbols after converting a few arm64 device binaries over and attempting to use them on the M1s - some examples coming from opencv:

  "cv::g_8x32fTab", referenced from:
      _cvRawDataToScalar in libopencv_arm64.a(array.o)
      cv::hal::cpu_baseline::mul8u(unsigned char const*, unsigned long, unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int, double const*) in libopencv_arm64.a(arithm.dispatch.o)
      cv::hal::cpu_baseline::div8u(unsigned char const*, unsigned long, unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int, double const*) in libopencv_arm64.a(arithm.dispatch.o)
      cv::hal::cpu_baseline::addWeighted8u(unsigned char const*, unsigned long, unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int, double const*) in libopencv_arm64.a(arithm.dispatch.o)
      cv::hal::cpu_baseline::recip8u(unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int, double const*) in libopencv_arm64.a(arithm.dispatch.o)
  "cv::g_Saturate8u", referenced from:
      void cv::reduceR_<unsigned char, unsigned char, cv::OpMax<unsigned char> >(cv::Mat const&, cv::Mat&) in libopencv_arm64.a(matrix_operations.o)
      void cv::reduceR_<unsigned char, unsigned char, cv::OpMin<unsigned char> >(cv::Mat const&, cv::Mat&) in libopencv_arm64.a(matrix_operations.o)
      void cv::reduceC_<unsigned char, unsigned char, cv::OpMax<unsigned char> >(cv::Mat const&, cv::Mat&) in libopencv_arm64.a(matrix_operations.o)
      void cv::reduceC_<unsigned char, unsigned char, cv::OpMin<unsigned char> >(cv::Mat const&, cv::Mat&) in libopencv_arm64.a(matrix_operations.o)
      cv::hal::cpu_baseline::add8u(unsigned char const*, unsigned long, unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int) in libopencv_arm64.a(arithm.dispatch.o)
      cv::hal::cpu_baseline::sub8u(unsigned char const*, unsigned long, unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int) in libopencv_arm64.a(arithm.dispatch.o)
      cv::hal::cpu_baseline::min8u(unsigned char const*, unsigned long, unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int) in libopencv_arm64.a(arithm.dispatch.o)
      ...

[zachgrayio]

I guess maybe this is somewhat unsurprising when diffing the contents of the x86_64 vs arm64 slices (opencv), which I included in full below for the sake of discussion.

The missing files here for the most part are expected as they're mostly AVX instruction set obj files, etc; of course they're not in the arm64 slice. I guess my question is really -- in cases like this, what's the expected behavior? and are the missing symbols here related to the difference in obj or not? 🤔 I haven't worked that out yet.

Here's the diff:

 __.SYMDEF
-accum.avx.o
-accum.avx2.o
 accum.dispatch.o
 accum.o
-accum.sse4_1.o
 algorithm.o
 alloc.o
 approx.o
-arithm.avx2.o
 arithm.dispatch.o
 arithm.o
-arithm.sse4_1.o
 array.o
 batch_distance.o
 bilateral_filter.o
 bindings_utils.o
 blend.o
 box_filter.o
 canny.o
 channels.o
 check.o
 clahe.o
 color.o
 color_hsv.o
 color_lab.o
 color_rgb.o
 color_yuv.o
 colormap.o
 command_line_parser.o
 conjugate_gradient.o
 connectedcomponents.o
 contours.o
 convert.o
 convert_c.o
 convert_scale.o
 convhull.o
 copy.o
-corner.avx.o
 corner.o
 cornersubpix.o
 count_non_zero.o
 cuda_gpu_mat.o
 cuda_host_mem.o
 cuda_info.o
 cuda_stream.o
 datafile.o
 datastructs.o
 demosaicing.o
 deriv.o
 directx.o
 distransform.o
 downhill_simplex.o
 drawing.o
 dxt.o
 emd.o
 featureselect.o
 filesystem.o
-filter.avx2.o
 filter.o
 floodfill.o
 gabor.o
 generalized_hough.o
 geometry.o
 gl_core_3_1.o
 glob.o
 grabcut.o
 hal_internal.o
 hershey_fonts.o
 histogram.o
 hough.o
-imgwarp.avx2.o
 imgwarp.o
-imgwarp.sse4_1.o
 intersection.o
 kmeans.o
 lapack.o
 lda.o
 linefit.o
 logger.o
 lpsolver.o
 lsd.o
 lut.o
 main.o
 matchcontours.o
 mathfuncs.o
-mathfuncs_core.avx.o
-mathfuncs_core.avx2.o
 mathfuncs_core.dispatch.o
 matmul.o
 matrix.o
 matrix_c.o
 matrix_decomp.o
 matrix_expressions.o
 matrix_iterator.o
 matrix_operations.o
 matrix_sparse.o
 matrix_wrap.o
 mean.o
 median_blur.o
 merge.o
 min_enclosing_triangle.o
 minmax.o
 moments.o
 morph.o
 norm.o
 ocl.o
 opencl_clamdblas.o
 opencl_clamdfft.o
 opencl_core.o
 opencl_kernels_core.o
 opencl_kernels_imgproc.o
 opengl.o
 out.o
 ovx.o
 parallel.o
 parallel_impl.o
 pca.o
 persistence.o
 persistence_base64.o
 persistence_c.o
 persistence_cpp.o
 persistence_json.o
 persistence_types.o
 persistence_xml.o
 persistence_yml.o
 phasecorr.o
 pyramids.o
 rand.o
-resize.avx2.o
 resize.o
-resize.sse4_1.o
 rotcalipers.o
 samplers.o
 samples.o
 segmentation.o
 shapedescr.o
 smooth.o
 softfloat.o
 spatialgradient.o
 split.o
-stat.avx2.o
 stat.dispatch.o
-stat.sse4_2.o
 stat_c.o
 stl.o
 subdivision2d.o
 sum.o
 sumpixels.o
 system.o
 tables.o
 templmatch.o
 thresh.o
 trace.o
 types.o
 umatrix.o
-undistort.avx2.o
 undistort.o
 utils.o
 va_intel.o

[zachgrayio]

Missing symbols in opencv are defined here it seems:

https://github.com/opencv/opencv/blob/17234f82d025e3bbfbf611089637e5aa2038e7b8/modules/core/src/tables.cpp https://github.com/opencv/opencv/blob/8b4fa2605e1155bbef0d906bb1f272ec06d7796e/modules/core/src/precomp.hpp

[zachgrayio]

I'm reopening this--thought I'd worked it out but I haven't.

what im trying to do:

• lipo -thin arm64 libopencv.a
• ar x it
• for file in *.o; do arm64-to-sim ...
• ar crv the converted files
• lipo -create the final result
• links in OK without mismatch arch warnings but always missing objects.

FWIW I'm moving in another direction here (no good reason to really be converting this one) but maybe it's worth figuring out if .a's are a valid use case.

maybe it's safe to chalk this up to incompatibility with CC archives produced by opencv's build tools and close?

[hendych]

I'm also experiencing this issue. This time I tried to convert Realm framework. The realm framework used prebuilt librealmcore-ios.a as it's core library. The issue seems when we ar x the .a library, some symbols will be missing.

[zachgrayio]

@hendych what symbols are missing in your case? because another issue I know a few users of this hack have seen is missing swift std lib, and theres a specific way we managed to work past that.

I'll try to PR to the readme sometime on that. its a somewhat separate issue than this one

[bogo]

Sorry to have gone MIA on this!

I am not entirely surprised by some libraries having missing symbols – it's entirely possible they were compiled out entirely through usage of TARGET_IPHONE_SIMULATOR macro or other build-time conditionals. For OpenCV that would make a lot of sense, since it's partially developed by Intel, likely leading to x86-specific implementations of some algorithms. (the fact the missing objects refer to AVX2 and SSE4, i.e. extended instruction sets in modern x86, would confirm this)

If that's the case, there is not a whole lot that could be done here. One thing I would try is to break up the x86 and ARM Simulator frameworks into two separate ones, where the Simulator ARM one is using the original Native ARM headers. Might make ld and LLVM happier, if the headers match up sufficiently. If you point me to the specific Realm framework version, I'm happy to take a look!

[bogo]

@zachgrayio - if you can share, would be super interested to hear how you solve the Swift standard library linking! Is this related to the usage of TAPIs/swiftmodules in any way? Would love to learn more about this.

[zachgrayio]

@bogo for sure. do you want to come on our podcast to talk about all this stuff?? was thinking it'd be a cool episode anyways lol. https://podcasts.apple.com/ca/podcast/the-bazel-show/id1544005904 :laughing:

[hendych]

@bogo in realm, this symbol should've exist in both simulator or device. The missing symbol was supposed to be make_in_realm_history something. This were supposed to not only exist in specific CPU arch only. Sorry the log was missing since I was testing it 3 days ago using Xcode.

What is happening?

What I found is that, when I unarchive using ar x this specific symbol were in history.o object when I checked using nm but after unarchive, the symbol is not in history.o rather in __.SYMDEF file. When I run ar crv again, the symbol is missing.

What I understand from man ar is that when we create an archive using ar the symbol table gets re-writed based on the *.o objects.

I was trying to convert Realm version 3.21.0.

Workaround

Luckily, Realm version 3.21.0 support SPM. Using SPM, the 3rd party is downloaded with pure source code (instead of cocoapods, it's still using prebuilt core librealmcore.a).

After the source code is compiled, SPM will produce Realm.o and RealmCore.o in DerivedData. Then I created Realm static library from it using ar

[tapthaker]

Hey @hendych, We saw this happening as well but it is because the ar utility cannot extract files correctly if there are more than 2 object entries with the same name. As a solution, you can use this script that which handles this edge case: https://github.com/tapthaker/extract-archive

I tried this is Realm and clearly history.o is repeated twice:

tapanth@tapanth-JK90543C2F ios % ar -t /tmp/realm_arm64 | grep -i history
history.o
client_history_impl.o
object_id_history_state.o
history.o

[bogo]

Ooh, duplicate names in ar is an interesting edge case I never considered. I should probably amend the articles to mention that, hah.

[tarbaiev-smg]

Another workaround is searching for the missing symbols on GitHub, then building them for arm64 simulator and merging them together using libtool. This way I successfully patched MLKit, which had missing symbols from the abseil library and some more string escaping functions copied across multiple Google repos. Though I had to perform some adjustments, such as renaming namespaces using -Dabsl=MLKITx_absl in Other C Flags as well as writing a couple of missing adapter functions and changing int to to long.