Maybe we can remove xformers

[Jannchie]

The recommended version of PyTorch for installation is 2.1.0, so it appears that pipe.enable_xformers_memory_efficient_attention() is redundant. Based on my tests, there is no significant difference in speed whether using xformers or not.

This is because PyTorch has already implemented an efficient algorithm: https://huggingface.co/docs/diffusers/optimization/torch2.0.