HGB

I'm trying to setup pytorch and zluda to run cuda on amd gpus, but to no avail ``` (base) r4-0@r40-desktop:~/pytorch$ LD_LIBRARY_PATH="$HOME/zluda:$LD_LIBRARY_PATH" python3 Python 3.12.3 | packaged by Anaconda, Inc. |...

Please Add Support For Triton Flash Attention Inference

1

Please add ```config.attn_config['attn_impl'] = 'triton'``` for Triton Flash Attention Inference ``` import torch from PIL import Image from transformers import AutoModel, AutoConfig, CLIPImageProcessor # Define the model name model_name =...

Anyone having this problem? ``` ___2____Input tokenized shape: torch.Size([1, 57]) /var/lib/jenkins/pytorch/aten/src/ATen/native/hip/Indexing.hip:1294: indexSelectLargeIndex: Device-side assertion `srcIndex < srcSelectDimSize' failed. ``` ``` if self.use_video: print("___1____Video shape:", video.size()) video = self.visual_encoder(video) # B...

Do you plan to quantize the model/ reduce the size of the ShareCaptioner-Video model?

1

As said in the title, the model size right now is about 17.2GB, it would be nice if there was a significantly smaller model with slight degrade in performance

No reduction in VRAM usage

5

I tried running the following code, with just having the ```ridger/MMfreeLM-1.3B```` model initialized: ``` root@r4-0:~/matmulfreellm# python >>> import os >>> os.environ["TOKENIZERS_PARALLELISM"] = "false" >>> import mmfreelm >>> from transformers import...

Cannot move tensors to cpu when in a xmp spawn process

3

## 🐛 Bug ``` all_frames = torch.cat(all_frames, dim=0).cpu().numpy() RuntimeError: Bad StatusOr access: INTERNAL: during context [pre-optimization]: RET_CHECK failure (third_party/tensorflow/compiler/xla/service/hlo_verifier.cc:402) replica_count == 1 || n == replica_count In kCrossReplica mode, replica...

How to run XLA with CPU offloaded models

12

## ❓ Questions and Help How do you run models that are offloaded to the CPU, Trying to work with ```enable_sequential_cpu_offload``` or ```enable_model_cpu_offload```, when running ```torch_xla.sync()/xm.mark_step() ```, the graph seems...

Support For TPU/XLA Devices

3

### Feature Idea Support For TPU/XLA Devices ### Existing Solutions _No response_ ### Other _No response_

This PR adds support for TPU/XLA devices in ComfyUI. users can enable by adding the command line arg `--xla` - Update README about support for TPU/XLA devices. - Detects TPU/XLA...