How to build pytorch text with system third_party libraries?

[jiapei100]

❓ Questions and Help

Description

Three packages are under pytorch text third_party. However, I personally prefer using system installed packages,

• libre2-dev
• libdouble-conversion-dev
• libsentencepiece-dev

In addition, isn't there a CMakeLists.txt for pytorch text??

Cheers

[mthrok]

In addition, isn't there a CMakeLists.txt for pytorch text??

Not at the moment. @Nayef211 and I are looking into this.

using system installed packages,

Once re-organizing the build process / system, introducing USE_SYSTEM_XXX options seem to be a straightforward and reasonable solution.

PyTorch core has this, which is compliant with Debian according to https://github.com/pytorch/pytorch/issues/14699.

https://github.com/pytorch/pytorch/blob/b3e3eb9935b65dd4cfbf20f23264f5a51c7bb791/CMakeLists.txt#L378-L414

[VRehnberg]

@mthrok @Nayef211 Any progress on this?

I started to look into it myself but first attempt didn't succeed so I had to drop it for now due to time constraints.

[Nayef211]

@mthrok just double checking if this be accomplished by adding the USE_SYSTEM_LIBS variable to CMAKE and then skipping the submodule build process similar to what PyTorch core does here: https://github.com/pytorch/pytorch/blob/baa06790f835b40190b49c26a8c88531a3da2e87/setup.py#L350-L351

[VRehnberg]

Torchdata has also implemented USE_SYSTEM_LIBS if you want a second example. And based on these https://github.com/easybuilders/easybuild-easyconfigs/pull/17545/files I could make something for 0.13.1, but not as clean as what's used in torchdata or PyTorch.

[Nayef211]

Torchdata has also implemented USE_SYSTEM_LIBS if you want a second example. And based on these https://github.com/easybuilders/easybuild-easyconfigs/pull/17545/files I could make something for 0.13.1, but not as clean as what's used in torchdata or PyTorch.

Sure @VRehnberg, if you want to take a stab at it, I'd be happy to provide a review and/or loop in folks that have more expertise with the build system!

[VRehnberg]

Sure VRehnberg, if you want to take a stab at it, I'd be happy to provide a review and/or loop in folks that have more expertise with the build system!

Sorry @Nayef211, I was perhaps a bit unclear with that could. I meant that I did make something for 0.13.1, and while I could turn it into a PR if you'd like it is probably not what you'd like. As it is now it is all or nothing when it comes to using system libs and as in https://github.com/easybuilders/easybuild-easyconfigs/pull/17545/files it assumes some specific paths for some libraries.

I won't have time to work on this for now, but might return to it in the future if I find the time.