Meta-issue: ICU4X as a low-level dependency

[sffc]

ICU4X aims to be lightweight and portable, and it achieves those goals after all is said and done, but during the build process, ICU4X is fairly hefty when it comes to linker size, compile times, source code size, and number of crates.

This was observed downstream in https://github.com/servo/rust-url/issues/939 (#5120, #5121), as well as elsewhere.

There are low-hanging things we can do to improve this as well as other things that might have tradeoffs that we need to weigh.

CC @Manishearth @hsivonen