ipex fails for Adan from package pytorch_optimizer

[ldv1]

Describe the bug

Hi,

ipex does not work at least for Adan from the package pytorch_optimizer.

Here is a toy example:

import torch
import torch.nn as nn
import numpy as np
import intel_extension_for_pytorch as ipex
from pytorch_optimizer import Adan

input_size = 1
output_size = 1

# hyper-parameters
num_epochs = 1
learning_rate = 0.001

# toy dataset
x = np.random.randn(10, input_size).astype(np.float32)
y = np.random.randn(10, output_size).astype(np.float32)

# linear regression model
model = nn.Linear(input_size, output_size)

# loss and optimizer
criterion = nn.MSELoss()
optimizer = Adan(model.parameters(), lr=learning_rate)  

# ipex
model, optimizer = ipex.optimize(model, optimizer=optimizer)

# train the model
for epoch in range(num_epochs):
    inputs = torch.from_numpy(x)
    targets = torch.from_numpy(y)
    
    # forward pass
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    
    # backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Result:

AttributeError: 'Adan' object has no attribute 'use_gc'

Versions

PyTorch version: 2.0.1+cpu PyTorch CXX11 ABI: No IPEX version: 2.0.100+cpu IPEX commit: 6a341a3 Build type: Release

OS: openSUSE Leap 15.5 (x86_64) GCC version: (SUSE Linux) 7.5.0 Clang version: N/A IGC version: N/A CMake version: version 3.20.4 Libc version: glibc-2.31

Python version: 3.11.4 (main, Jul 06 2023, 16:27:46) [GCC] (64-bit runtime) Python platform: Linux-5.14.21-150500.55.19-default-x86_64-with-glibc2.31 Is XPU available: False DPCPP runtime version: N/A MKL version: N/A GPU models and configuration:

Intel OpenCL ICD version: N/A Level Zero version: N/A

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: GenuineIntel Model name: Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz CPU family: 6 Model: 60 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 Stepping: 3 CPU max MHz: 3400.0000 CPU min MHz: 800.0000 BogoMIPS: 4788.84 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d Virtualization: VT-x L1d cache: 128 KiB (4 instances) L1i cache: 128 KiB (4 instances) L2 cache: 1 MiB (4 instances) L3 cache: 6 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-7 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Unknown: No mitigations Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Mitigation; Microcode Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] intel-extension-for-pytorch==2.0.100 [pip3] numpy==1.25.2 [pip3] pytorch_optimizer==2.11.2 [pip3] torch==2.0.1+cpu [pip3] torch-cluster==1.6.1+pt20cpu [pip3] torch-geometric==2.3.1 [pip3] torch-scatter==2.1.1+pt20cpu [pip3] torch-sparse==0.6.17+pt20cpu [pip3] torch-spline-conv==1.2.2+pt20cpu [pip3] torchaudio==2.0.2+cpu [pip3] torchvision==0.15.2+cpu [pip3] vector-quantize-pytorch==1.7.1 [conda] N/A

[kminhta]

Thanks for reporting. Have you noticed this issue with other optimizers from that package? Custom optimizers may behave unexpected, but we will look into this issue

[jgong5]

cc @zhuhaozhe

[ldv1]

I haven't tested any other optimizer from this package, but I know for sure that many others have this attribute use_gc (short for use gradient centralization).

[zhuhaozhe]

Hi, @ldv1, thanks for reporting. May you try

model, optimizer = ipex.optimize(model, optimizer=optimizer, inplace=True)

I suspect the non-inplace optimize triggers a deepcopy and the deepcopy does not keep the use_gc method.

[ldv1]

Thanks for the help.

Yesterday I installed PyTorch 2.1 and it seems that ipex is not PyTorch 2.1 ready, since I got

Traceback (most recent call last):

  import intel_extension_for_pytorch as ipex
 File "/usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/__init__.py", line 11, in <module>
   from .cpu import _cpu_isa
 File "/usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/cpu/__init__.py", line 1, in <module>
   from . import runtime
 File "/usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/cpu/runtime/__init__.py", line 3, in <module>
   from .multi_stream import MultiStreamModule, get_default_num_streams, \
 File "/usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/cpu/runtime/multi_stream.py", line 4, in <module>
   import intel_extension_for_pytorch._C as core
ImportError: /usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so: undefined symbol: _ZN3c1010TensorTypeC1ENS_8optionalINS_10ScalarTypeEEENS1_INS_6DeviceEEERKNS_13SymbolicShapeERKNS_12VaryingShapeINS_6StrideEEENS1_IbEESE_

I will see what I can do. Maybe got back to PyTorch 2.0.1 for the sake of testing your suggestion.

[kminhta]

Following up, @ldv1. The suggestion works on my end. IPEX 2.1 should be released soon. Please check again by end of the week.

Alternatively, it should work with torch+ipex 2.0.1 if version is not a limiting factor for you

[mustarfighter]

Thanks for the help.

Yesterday I installed PyTorch 2.1 and it seems that ipex is not PyTorch 2.1 ready, since I got

Traceback (most recent call last):

  import intel_extension_for_pytorch as ipex
 File "/usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/__init__.py", line 11, in <module>
   from .cpu import _cpu_isa
 File "/usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/cpu/__init__.py", line 1, in <module>
   from . import runtime
 File "/usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/cpu/runtime/__init__.py", line 3, in <module>
   from .multi_stream import MultiStreamModule, get_default_num_streams, \
 File "/usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/cpu/runtime/multi_stream.py", line 4, in <module>
   import intel_extension_for_pytorch._C as core
ImportError: /usr/lib64/python3.11/site-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so: undefined symbol: _ZN3c1010TensorTypeC1ENS_8optionalINS_10ScalarTypeEEENS1_INS_6DeviceEEERKNS_13SymbolicShapeERKNS_12VaryingShapeINS_6StrideEEENS1_IbEESE_

I will see what I can do. Maybe got back to PyTorch 2.0.1 for the sake of testing your suggestion.

i get the same import error :

---

ImportError Traceback (most recent call last) Cell In[3], line 1 ----> 1 import intel_extension_for_pytorch as ipex 2 import torch 3 from diffusers import StableDiffusionPipeline

File /usr/local/lib/python3.9/dist-packages/intel_extension_for_pytorch/init.py:11 9 from ._version import version 10 from .utils import _custom_fx_tracer ---> 11 from .cpu import _cpu_isa 12 _cpu_isa.check_minimal_isa_support() 14 torch_version = ''

File /usr/local/lib/python3.9/dist-packages/intel_extension_for_pytorch/cpu/init.py:1 ----> 1 from . import runtime 2 from . import autocast 3 from . import auto_ipex

File /usr/local/lib/python3.9/dist-packages/intel_extension_for_pytorch/cpu/runtime/init.py:3 1 from .task import Task 2 from .cpupool import pin, CPUPool, is_runtime_ext_enabled ----> 3 from .multi_stream import MultiStreamModule, get_default_num_streams,
4 MultiStreamModuleHint, _MultiStreamBenchmarkModule 5 from .runtime_utils import get_core_list_of_node_id

File /usr/local/lib/python3.9/dist-packages/intel_extension_for_pytorch/cpu/runtime/multi_stream.py:4 2 import torch.nn as nn 3 from typing import Union, Optional ----> 4 import intel_extension_for_pytorch._C as core 5 from .cpupool import CPUPool 6 from .task import Task

ImportError: /usr/local/lib/python3.9/dist-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so: undefined symbol: ZN3c1010TensorTypeC1ENS_8optionalINS_10ScalarTypeEEENS1_INS_6DeviceEEERKNS_13SymbolicShapeERKNS_12VaryingShapeINS_6StrideEEENS1_IbEESE

[zhuhaozhe]

Usually, this error is got by version not matched ( you install a new pytorch and no not re-install IPEX):

ImportError: /usr/local/lib/python3.9/dist-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so: undefined symbol: ZN3c1010TensorTypeC1ENS_8optionalINS_10ScalarTypeEEENS1_INS_6DeviceEEERKNS_13SymbolicShapeERKNS_12VaryingShapeINS_6StrideEEENS1_IbEESE

You may try uninstall torch / ipex and install them. Or you can wait a few days since we are going to release IPEX 2.1

[dbalabka]

@zhuhaozhe seems that installing IPEX 2.1 does not help

[kminhta]

@zhuhaozhe seems that installing IPEX 2.1 does not help

Did you try setting: model, optimizer = ipex.optimize(model, optimizer=optimizer, inplace=True) ?

[ldv1]

I confirm that my code (see above) works fine with pytorch and ipex 2.1 if I set model, optimizer = ipex.optimize(model, optimizer=optimizer, inplace=True) Thanks to @zhuhaozhe !