xformers assertion errors (5-10x greater error than allowed)

[Thomas-MMJ]

Describe the bug

When running the pytest suite, almost all xformers tests are all failing due to assertion errors,

Reproduction

pytest -k "xformers" tests

Logs

`pytest -k "xformers" tests
`


tests/models/test_models_unet_2d.py .                                                                                                                [  7%]
tests/pipelines/altdiffusion/test_alt_diffusion.py F                                                                                                 [  7%]
tests/test_pipelines_common.py:476 AltDiffusionPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.0008659363 not less th…
tests/pipelines/dance_diffusion/test_dance_diffusion.py .                                                                                            [ 21%]
tests/pipelines/ddim/test_ddim.py F                                                                                                                  [ 21%]
tests/test_pipelines_common.py:476 DDIMPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.0005880296 not less than 0.000…
tests/pipelines/paint_by_example/test_paint_by_example.py F                                                                                          [ 28%]
tests/test_pipelines_common.py:476 PaintByExamplePipelineFastTests.test_xformers_attention_forward_pass - AttributeError: 'NoneType' object has …
tests/pipelines/repaint/test_repaint.py F                                                                                                            [ 35%]
tests/test_pipelines_common.py:476 RepaintPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.00038033724 not less than 0…
tests/pipelines/stable_diffusion/test_cycle_diffusion.py F                                                                                           [ 42%]
tests/test_pipelines_common.py:476 CycleDiffusionPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.00062417984 not less…
tests/pipelines/stable_diffusion/test_stable_diffusion.py F                                                                                          [ 50%]
tests/test_pipelines_common.py:476 StableDiffusionPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.00097203255 not les…
tests/pipelines/stable_diffusion/test_stable_diffusion_image_variation.py F                                                                          [ 57%]
tests/test_pipelines_common.py:476 StableDiffusionImageVariationPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.00064…
tests/pipelines/stable_diffusion/test_stable_diffusion_img2img.py F                                                                                  [ 64%]
tests/test_pipelines_common.py:476 StableDiffusionImg2ImgPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.0039129257 n…
tests/pipelines/stable_diffusion/test_stable_diffusion_inpaint.py F                                                                                  [ 71%]
tests/test_pipelines_common.py:476 StableDiffusionInpaintPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.0008302927 n…
tests/pipelines/stable_diffusion_2/test_stable_diffusion.py F                                                                                        [ 78%]
tests/test_pipelines_common.py:476 StableDiffusion2PipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.0007764101 not les…
tests/pipelines/stable_diffusion_2/test_stable_diffusion_depth.py F                                                                                  [ 85%]
tests/test_pipelines_common.py:476 StableDiffusionDepth2ImgPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.0046414137…
tests/pipelines/stable_diffusion_2/test_stable_diffusion_inpaint.py F                                                                                [ 92%]
tests/test_pipelines_common.py:476 StableDiffusion2InpaintPipelineFastTests.test_xformers_attention_forward_pass - AssertionError: 0.0006713867 …    [100%]

self = <tests.pipelines.altdiffusion.test_alt_diffusion.AltDiffusionPipelineFastTests testMethod=test_xformers_attention_forward_pass>

    @unittest.skipIf(
        torch_device != "cuda" or not is_xformers_available(),
        reason="XFormers attention is only available with CUDA and `xformers` installed",
    )
    def test_xformers_attention_forward_pass(self):
        if not self.test_xformers_attention:
            return

        components = self.get_dummy_components()
        pipe = self.pipeline_class(**components)
        pipe.to(torch_device)
        pipe.set_progress_bar_config(disable=None)

        inputs = self.get_dummy_inputs(torch_device)
        output_without_offload = pipe(**inputs)[0]

        pipe.enable_xformers_memory_efficient_attention()
        inputs = self.get_dummy_inputs(torch_device)
        output_with_offload = pipe(**inputs)[0]

        max_diff = np.abs(output_with_offload - output_without_offload).max()
>       self.assertLess(max_diff, 1e-4, "XFormers attention should not affect the inference results")
E       AssertionError: 0.0008659363 not less than 0.0001 : XFormers attention should not affect the inference results

All of the errors are substantially the same with different amounts of assertion error (from 3x - 10x the allowed amount)

note that the xfromers test suite, all relevant tests appear to pass.

### System Info

• diffusers version: 0.11.0.dev0
• Platform: Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
• Python version: 3.9.13
• PyTorch version (GPU?): 1.13.1 (True)
• Huggingface_hub version: 0.10.1
• Transformers version: 4.26.0.dev0
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

[patrickvonplaten]

Hey @Thomas-MMJ,

Note that xformers uses a different attention mechanism so on certain hardware differences in the order of:

 AssertionError: 0.0008659363 not less than 0.0001

are expected.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.