Implementation ideas for equivalent replacement from xformers to pytorch

[tzayuan]

❓ Questions and Help

Independently implementing the equivalent replacement from xformers to pytorch would be a great engineering practice for greatly improving the underlying capabilities of network design. May I ask if you can provide me with some systematic routes for implementing equivalent alternatives of various operator modules to PyTorch implementation (such as xformers.ops.fmha.BlockDiagonalMask.from_seqlens and etc)? Because some modules do not provide equivalent replacement templates in the documentation, how can independent exploration and debug of equivalent implementation solutions be efficient? Looking forward to your suggestions, such as what knowledge to learn and what related papers or code to read for traceability. Thank you.