Trying to convert torch.multinomial to PTE model

[vikasbalaga]

🐛 Describe the bug

Hi,

I have started working with Executorch and I am trying to convert sample models with different NN operators and converting them to .pte models.

I am currently trying to convert "torch.multinomial()" function to a .pte model using following python script :

import torch
from torch.export import export
from executorch.exir import to_edge

# Start with a PyTorch model that adds two input tensors (matrices)
class multinomial(torch.nn.Module):
  def __init__(self, num_samples=1, replacement=False):
      super().__init__()
      self.num_samples = num_samples
      self.replacement = replacement

  def forward(self, probs: torch.Tensor):
      """
      Args:
          probs (Tensor): A 1D or 2D tensor of probabilities. Each row should sum to 1 if 2D.

      Returns:
          Tensor: Sampled indices from the multinomial distribution.
      """
      if probs.dim() == 1:
          return torch.multinomial(probs, self.num_samples, self.replacement)
      elif probs.dim() == 2:
          return torch.stack([
              torch.multinomial(row, self.num_samples, self.replacement)
              for row in probs
          ])
      else:
          raise ValueError("Input must be 1D or 2D tensor of probabilities.")

# 1. torch.export: Defines the program with the ATen operator set.
aten_dialect = export(multinomial(), (torch.ones(1),))

# 2. to_edge: Make optimizations for Edge devices
edge_program = to_edge(aten_dialect)

# 3. to_executorch: Convert the graph to an ExecuTorch program
executorch_program = edge_program.to_executorch()

# 4. Save the compiled .pte program
with open("multinomial.pte", "wb") as file:
    file.write(executorch_program.buffer)

But I am facing the below errors :

Operator torch._ops.aten.multinomial.default is not in Core ATen opset (https://pytorch.org/docs/stable/torch.compiler_ir.html#core-aten-ir)."
There are a few things to try:
1. You can proceed with `to_edge(compile_config=EdgeCompileConfig(_core_aten_ops_exception_list=[torch.ops.aten.multinomial.default]))`.
   Please make sure that the backend(s) you are planning to lower to is able to handle aten.multinomial.default, or you have a corresponding kernel linked to your runtime.

2. Sometimes inference and training gives slightly different op set. Try adding `with torch.no_grad():` context manager if you are export for inference only.

3. If the error persists after 2, this is likely caused by torch.export() + core ATen decomposition producing unexpected operators for your model.
   If you believe this operator should be included into core ATen opset, please create an issue in https://github.com/pytorch/pytorch/issues and add `module: core aten` tag.

Now, I know that multinomial() is not in Core ATen Ops list but so is Silu() and in the latter case, it is decomposed into sigmoid() and mul().

So, shouldn't it be the same case for multinomial also?

Versions

Collecting environment information... PyTorch version: 2.6.0+cpu Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Clang version: Could not collect CMake version: version 4.0.0 Libc version: glibc-2.31

Python version: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0] (64-bit runtime) Python platform: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.31 Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 39 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 140 Model name: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz Stepping: 1 CPU MHz: 2419.200 BogoMIPS: 4838.40 Virtualization: VT-x Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 192 KiB L1i cache: 128 KiB L2 cache: 5 MiB L3 cache: 8 MiB Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Mitigation; Enhanced IBRS Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear flush_l1d arch_capabilities

Versions of relevant libraries: [pip3] executorch==0.5.0 [pip3] numpy==2.0.0 [pip3] torch==2.6.0+cpu [pip3] torchao==0.8.0+gitebc43034 [pip3] torchaudio==2.6.0 [pip3] torchsr==1.0.4 [pip3] torchvision==0.21.0+cpu [conda] executorch 0.5.0 pypi_0 pypi [conda] numpy 2.0.0 pypi_0 pypi [conda] torch 2.6.0+cpu pypi_0 pypi [conda] torchao 0.8.0+gitebc43034 pypi_0 pypi [conda] torchaudio 2.6.0 pypi_0 pypi [conda] torchsr 1.0.4 pypi_0 pypi [conda] torchvision 0.21.0+cpu pypi_0 pypi

cc @larryliu0820 @manuelcandales

[vikasbalaga]

Update:

2. Sometimes inference and training gives slightly different op set. Try adding with torch.no_grad(): context manager if you are export for inference only.

I have tried this, but the issue still persists

[GregoryComer]

It looks like we are missing an operator implementation for this op. @manuelcandales can we add this to our op requests? Alternatively, we could request a decomp in export.

[vikasbalaga]

@GregoryComer, @manuelcandales, thanks for your quick response.

I have another question similar to the previous OP conversion. I am trying to convert a torch.nn.Identity() into *.pte and I could see that it doesn't involve any OPs. So, how will this Identity layer work in ExecuTorch environment?