functorch
functorch copied to clipboard
Test failures
I'm observing a number of test suite failures:
"test_coverage_bernoulli_cpu_float32"
"test_coverage_column_stack_cpu_float32"
"test_coverage_diagflat_cpu_float32"
"test_coverage_flatten_cpu_float32"
"test_coverage_linalg_lu_factor_cpu_float32"
"test_coverage_linalg_lu_factor_ex_cpu_float32"
"test_coverage_multinomial_cpu_float32"
"test_coverage_nn_functional_dropout2d_cpu_float32"
"test_coverage_nn_functional_feature_alpha_dropout_with_train_cpu_float32"
"test_coverage_nn_functional_feature_alpha_dropout_without_train_cpu_float32"
"test_coverage_nn_functional_kl_div_cpu_float32"
"test_coverage_normal_cpu_float32"
"test_coverage_normal_number_mean_cpu_float32"
"test_coverage_pca_lowrank_cpu_float32"
"test_coverage_round_decimals_0_cpu_float32"
"test_coverage_round_decimals_3_cpu_float32"
"test_coverage_round_decimals_neg_3_cpu_float32"
"test_coverage_scatter_reduce_cpu_float32"
"test_coverage_svd_lowrank_cpu_float32"
all fail with errors of the form
_______ TestFuncTorchLaggingOpDbCPU.test_coverage_bernoulli_cpu_float32 ________
self = <test_functorch_lagging_op_db.TestFuncTorchLaggingOpDbCPU testMethod=test_coverage_bernoulli_cpu_float32>
device = 'cpu', dtype = torch.float32
op = <torch.testing._internal.common_methods_invocations.OpInfo object at 0x7ffee1c80700>
@ops(op_db, allowed_dtypes=(torch.float,))
def test_coverage(self, device, dtype, op):
if in_functorch_lagging_op_db(op):
return
> raise RuntimeError(
f"{(op.name, op.variant_test_name)} is in PyTorch's OpInfo db ",
"but is not in functorch's OpInfo db. Please regenerate ",
"test/functorch_lagging_op_db.py and add the new tests to ",
"denylists if necessary.")
E RuntimeError: ("('bernoulli', '') is in PyTorch's OpInfo db ", "but is not in functorch's OpInfo db. Please regenerate ", 'test/functorch_lagging_op_db.py and add the new tests to ', 'denylists if necessary.')
test/test_functorch_lagging_op_db.py:27: RuntimeError
And
"test_functorch_lagging_op_db_has_opinfos_cpu"
fails with
___ TestFuncTorchLaggingOpDbCPU.test_functorch_lagging_op_db_has_opinfos_cpu ___
self = <test_functorch_lagging_op_db.TestFuncTorchLaggingOpDbCPU testMethod=test_functorch_lagging_op_db_has_opinfos_cpu>
device = 'cpu'
def test_functorch_lagging_op_db_has_opinfos(self, device):
> self.assertEqual(len(functorch_lagging_op_db), len(op_db))
E AssertionError: Scalars are not equal!
E
E Absolute difference: 19
E Relative difference: 0.03525046382189239
test/test_functorch_lagging_op_db.py:21: AssertionError
There are a handful of other errors regarding PyTorch not being compiled with LLVM support, but I suspect that those are due to my particular PyTorch package.
What do I need to do to get the test suite running successfully?
@samuela Do you have latest nightly pytorch version ? I'd say you can skip coverage tests unless it is important for your work...
Those test are actually expected to fail, they are there to remind us that we need to sync our testing database with PyTorch's testing database :/. I should make this clearer somewhere and/or just remove these from the test directory
Thanks for clarifying @vfdev-5 and @zou3519! I'll just disable the failing tests for now.
@samuela Do you have latest nightly pytorch version ? I'd say you can skip coverage tests unless it is important for your work...
I'm using pytorch 1.11.0. I'm packaging functorch for Nixpkgs atm (https://github.com/NixOS/nixpkgs/pull/174248), and we don't have pytorch nightly packaged in Nixpkgs for better or worse. Is LLVM support only in nightly?
I'm using pytorch 1.11.0. I'm packaging functorch for Nixpkgs atm (https://github.com/NixOS/nixpkgs/pull/174248), and we don't have pytorch nightly packaged in Nixpkgs for better or worse. Is LLVM support only in nightly?
@samuela I do not know exactly how built pytorch binaries with respect to LLVM. Maybe, here you can find more info: https://github.com/pytorch/builder/blob/main/wheel/build_wheel.sh
I was checking if LLVM is enabled in 1.11 or nightly binaries and seems like it is enabled:
import torch
print(torch.__version__, torch._C._llvm_enabled())
# ('1.11.0', True)
# ('1.12.0.dev20220511+cu113', True)
By the way, functorch <-> pytorch compatibility is that : v0.1.1 <==> 1.11.0 and functorch source <==> pytorch nightly
Thanks for explaining @vfdev-5! I'm also building pytorch from source, so I'll have to dig into that some more to figure out what's going on...