xed-sys icon indicating copy to clipboard operation
xed-sys copied to clipboard

enable enc2

Open flier opened this issue 1 year ago • 4 comments

add feature to enable enc2 encoder, and disable it by default because it is huge size

flier avatar Jan 09 '24 03:01 flier

~Thanks for the PR! It actually prompted me to do a bit of a rethink of how the build scripts in this crate work over in #66. As a part of that I ended up adding the enc2 feature as well.~

~I'll be closing this now but I'll be making a release with the enc2 feature right after :)~

See the comment below.

Phantomical avatar Jan 16 '24 10:01 Phantomical

So, as it turns out, the two libraries that are produced by the xed build system when you build with --enc2 have some duplicate symbols between them. If you try to link both you end up getting a bunch of linker errors.

I'm not entirely sure what the solution for this is. As is, I've reopened the PR and updated it to work against the current master. The real solution for this looks like it should be somehow linking both within the xed build system, but that seems like it would be hard to do. I'm going to leave this open for now in case someone wants to figure out a solution.

Detailed link error message
   Compiling xed-sys v0.4.0 (/home/swlynch/projects/xed-sys)
error: linking with `cc` failed: exit status: 1
  |
  = note: <command line elided>
  = note: ld.lld: error: duplicate symbol: enc_evex_vindex_xmm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_vindex_xmm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x47D) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_vindex_ymm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_vindex_ymm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x4EE) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_vindex_zmm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_vindex_zmm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x55F) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_vex_vindex_xmm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_vex_vindex_xmm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x5D0) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_vex_vindex_ymm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_vex_vindex_ymm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x626) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_vvvv_kreg
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_vvvv_kreg) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x67C) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_modrm_reg_kreg
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_modrm_reg_kreg) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x6CC) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_modrm_rm_kreg
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_modrm_rm_kreg) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x708) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_vex_vvvv_kreg
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_vex_vvvv_kreg) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x744) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_modrm_reg_kreg
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_modrm_reg_kreg) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x782) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_modrm_rm_kreg
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_modrm_rm_kreg) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x7D0) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_kmask
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_kmask) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x81E) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_vvvv_reg_xmm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_vvvv_reg_xmm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x85A) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_modrm_reg_xmm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_modrm_reg_xmm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x8B2) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_modrm_rm_xmm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_modrm_rm_xmm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x923) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_vvvv_reg_ymm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_vvvv_reg_ymm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x994) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_modrm_reg_ymm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_modrm_reg_ymm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0x9EC) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_modrm_rm_ymm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_modrm_rm_ymm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0xA5D) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_vvvv_reg_zmm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_vvvv_reg_zmm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0xACE) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: duplicate symbol: enc_evex_modrm_reg_zmm
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(enc_evex_modrm_reg_zmm) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m32-a32.a
          >>> defined at xed-encode-direct.c
          >>>            xed-encode-direct.o:(.text+0xB26) in archive /home/swlynch/projects/xed-sys/target/debug/build/xed-sys-efad09ad303ae8e4/out/install/lib/libxed-enc2-m64-a64.a
          
          ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
          collect2: error: ld returned 1 exit status

Phantomical avatar Jan 16 '24 10:01 Phantomical

So, as it turns out, the two libraries that are produced by the xed build system when you build with --enc2 have some duplicate symbols between them. If you try to link both you end up getting a bunch of linker errors.

Maybe we could split it to enc2-m32 and enc2-m64 features?

I'm not entirely sure what the solution for this is. As is, I've reopened the PR and updated it to work against the current master. The real solution for this looks like it should be somehow linking both within the xed build system, but that seems like it would be hard to do. I'm going to leave this open for now in case someone wants to figure out a solution.

Detailed link error message

agree, enc2 is too large for a prebuilt code

flier avatar Feb 02 '24 13:02 flier

I have played around with this PR a bit and I think I've got it to mostly work.

Maybe we could split it to enc2-m32 and enc2-m64 features?

Normally I would be rather hesitant about this because these features are not additive. If two of your dependencies end up depending on both enc2 features in xed-sys then that leaves you as an indirect user of xed-sys stuck with no other options. This is generally not a footgun I want to have in a library.

With that said, it turns out that if you link both of the enc2 libraries as shared libraries then linking works just fine! This means that there is a way out for dependencies - even if it is a little bit inconvenient. xed-sys isn't used enough for this to be a big issue so I'm ok with this as-is.


Now the one last thing I need you to do before I would be confident in merging this is writing some tests that use the xed-m32-a32 and xed-m64-a64 features. These don't have to be complicated - I just want to verify that linking in these libraries work and that calls go to the right place.

Phantomical avatar Feb 03 '24 08:02 Phantomical