Test failures

[samuela]

I'm observing a number of test suite failures:

    "test_coverage_bernoulli_cpu_float32"
    "test_coverage_column_stack_cpu_float32"
    "test_coverage_diagflat_cpu_float32"
    "test_coverage_flatten_cpu_float32"
    "test_coverage_linalg_lu_factor_cpu_float32"
    "test_coverage_linalg_lu_factor_ex_cpu_float32"
    "test_coverage_multinomial_cpu_float32"
    "test_coverage_nn_functional_dropout2d_cpu_float32"
    "test_coverage_nn_functional_feature_alpha_dropout_with_train_cpu_float32"
    "test_coverage_nn_functional_feature_alpha_dropout_without_train_cpu_float32"
    "test_coverage_nn_functional_kl_div_cpu_float32"
    "test_coverage_normal_cpu_float32"
    "test_coverage_normal_number_mean_cpu_float32"
    "test_coverage_pca_lowrank_cpu_float32"
    "test_coverage_round_decimals_0_cpu_float32"
    "test_coverage_round_decimals_3_cpu_float32"
    "test_coverage_round_decimals_neg_3_cpu_float32"
    "test_coverage_scatter_reduce_cpu_float32"
    "test_coverage_svd_lowrank_cpu_float32"

all fail with errors of the form

_______ TestFuncTorchLaggingOpDbCPU.test_coverage_bernoulli_cpu_float32 ________

self = <test_functorch_lagging_op_db.TestFuncTorchLaggingOpDbCPU testMethod=test_coverage_bernoulli_cpu_float32>
device = 'cpu', dtype = torch.float32
op = <torch.testing._internal.common_methods_invocations.OpInfo object at 0x7ffee1c80700>

    @ops(op_db, allowed_dtypes=(torch.float,))
    def test_coverage(self, device, dtype, op):
        if in_functorch_lagging_op_db(op):
            return
>       raise RuntimeError(
            f"{(op.name, op.variant_test_name)} is in PyTorch's OpInfo db ",
            "but is not in functorch's OpInfo db. Please regenerate ",
            "test/functorch_lagging_op_db.py and add the new tests to ",
            "denylists if necessary.")
E       RuntimeError: ("('bernoulli', '') is in PyTorch's OpInfo db ", "but is not in functorch's OpInfo db. Please regenerate ", 'test/functorch_lagging_op_db.py and add the new tests to ', 'denylists if necessary.')

test/test_functorch_lagging_op_db.py:27: RuntimeError

And

    "test_functorch_lagging_op_db_has_opinfos_cpu"

fails with

___ TestFuncTorchLaggingOpDbCPU.test_functorch_lagging_op_db_has_opinfos_cpu ___

self = <test_functorch_lagging_op_db.TestFuncTorchLaggingOpDbCPU testMethod=test_functorch_lagging_op_db_has_opinfos_cpu>
device = 'cpu'

    def test_functorch_lagging_op_db_has_opinfos(self, device):
>       self.assertEqual(len(functorch_lagging_op_db), len(op_db))
E       AssertionError: Scalars are not equal!
E       
E       Absolute difference: 19
E       Relative difference: 0.03525046382189239

test/test_functorch_lagging_op_db.py:21: AssertionError

There are a handful of other errors regarding PyTorch not being compiled with LLVM support, but I suspect that those are due to my particular PyTorch package.

What do I need to do to get the test suite running successfully?

[vfdev-5]

@samuela Do you have latest nightly pytorch version ? I'd say you can skip coverage tests unless it is important for your work...

[zou3519]

Those test are actually expected to fail, they are there to remind us that we need to sync our testing database with PyTorch's testing database :/. I should make this clearer somewhere and/or just remove these from the test directory

[samuela]

Thanks for clarifying @vfdev-5 and @zou3519! I'll just disable the failing tests for now.

@samuela Do you have latest nightly pytorch version ? I'd say you can skip coverage tests unless it is important for your work...

I'm using pytorch 1.11.0. I'm packaging functorch for Nixpkgs atm (https://github.com/NixOS/nixpkgs/pull/174248), and we don't have pytorch nightly packaged in Nixpkgs for better or worse. Is LLVM support only in nightly?

[vfdev-5]

I'm using pytorch 1.11.0. I'm packaging functorch for Nixpkgs atm (https://github.com/NixOS/nixpkgs/pull/174248), and we don't have pytorch nightly packaged in Nixpkgs for better or worse. Is LLVM support only in nightly?

@samuela I do not know exactly how built pytorch binaries with respect to LLVM. Maybe, here you can find more info: https://github.com/pytorch/builder/blob/main/wheel/build_wheel.sh

I was checking if LLVM is enabled in 1.11 or nightly binaries and seems like it is enabled:

import torch
print(torch.__version__, torch._C._llvm_enabled())
# ('1.11.0', True)
# ('1.12.0.dev20220511+cu113', True)

By the way, functorch <-> pytorch compatibility is that : v0.1.1 <==> 1.11.0 and functorch source <==> pytorch nightly

[samuela]

Thanks for explaining @vfdev-5! I'm also building pytorch from source, so I'll have to dig into that some more to figure out what's going on...