use flag -march=native may cause trouble
So I got 2 different instances, both CPU supports AVX2, I was expecting the shared library built from one instance can be used in another, however that's not the case. I was able to narrow the issue down to the -march=native flag: When replace it with -mavx2 -mpclmul it works perfectly in both instances. Just a note for anyone who may got same issue.
It is definitively the case that if you use the -march=native flag, then the resulting binary is not portable to other hardware. The flags you propose (-mavx2 -mpclmul) are better but note that if the hardware does not support AVX2, then you will likely get a crash. That's arguably a minor concern in 2022 since most x64 processors support AVX2.