SHARK
SHARK copied to clipboard
nod-ai
facebook_convnext_tiny_224 fails during iree-compile for cuda, vulkan
E iree.compiler.tools.binaries.CompilerToolError: Error invoking IREE compiler tool iree-compile
E Diagnostics:
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py:1097:0: error: failed to materialize conversion for result #0 of operation 'hal.interface.binding.subspan' that remained live after conversion
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:65:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py:317:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py:969:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:96:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py:1097:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:65:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py:557:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py:969:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:96:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py:1097:0: note: see existing live user here:
E linalg.generic {indexing_maps = [affine_map<() -> ()>], iterator_types = []} outs(%5 : memref<f32>) attrs = {__internal_linalg_transform__ = "__workgroup_tiling__", lowering_config = #iree_codegen.lowering_config<tile_sizes = [[]]>} {
E ^bb0(%arg0: f32 loc(fused["Mean:", callsite("tf_conv_next_for_image_classification/convnext/layernorm/moments/mean@__inference_forward_887657"("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py":1097:0) at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":65:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py":317:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py":969:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":96:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py":1097:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":65:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py":557:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py":969:0 at "/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":96:0)))))))))])):
E %584 = arith.divf %arg0, %cst : f32 loc(fused["Mean:", callsite("tf_conv_next_for_image_classification/convnext/layernorm/moments/mean@__inference_forward_887657"("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py":1097:0) at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":65:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py":317:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py":969:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":96:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py":1097:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":65:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py":557:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py":969:0 at "/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":96:0)))))))))])
E linalg.yield %584 : f32 loc(fused["Mean:", callsite("tf_conv_next_for_image_classification/convnext/layernorm/moments/mean@__inference_forward_887657"("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py":1097:0) at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":65:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py":317:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py":969:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":96:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py":1097:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":65:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py":557:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py":969:0 at "/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":96:0)))))))))])
E } loc(fused["Mean:", callsite("tf_conv_next_for_image_classification/convnext/layernorm/moments/mean@__inference_forward_887657"("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py":1097:0) at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":65:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py":317:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py":969:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":96:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py":1097:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":65:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py":557:0 at callsite("/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py":969:0 at "/data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py":96:0)))))))))])
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py:1097:0: error: failed to run translation of source executable to target executable for backend #hal.executable.target<"vulkan", "vulkan-spirv-fb", {spv.target_env = #spv.target_env<#spv.vce<v1.6, [Shader, Float64, Float16, Int64, Int16, Int8, StorageBuffer16BitAccess, StorageUniform16, StoragePushConstant16, StorageBuffer8BitAccess, UniformAndStorageBuffer8BitAccess, StoragePushConstant8, GroupNonUniform, GroupNonUniformVote, GroupNonUniformArithmetic, GroupNonUniformBallot, GroupNonUniformShuffle, GroupNonUniformShuffleRelative, GroupNonUniformClustered, GroupNonUniformQuad, VariablePointers, VariablePointersStorageBuffer, CooperativeMatrixNV], [SPV_KHR_16bit_storage, SPV_KHR_8bit_storage, SPV_KHR_storage_buffer_storage_class, SPV_KHR_variable_pointers, SPV_NV_cooperative_matrix]>, NVIDIA:DiscreteGPU, #spv.resource_limits<max_compute_shared_memory_size = 49152, max_compute_workgroup_invocations = 1024, max_compute_workgroup_size = [1024, 1024, 64], cooperative_matrix_properties_nv = [#spv.coop_matrix_props<m_size = 8, n_size = 8, k_size = 32, a_type = i8, b_type = i8, c_type = i32, result_type = i32, scope = <Subgroup>>, #spv.coop_matrix_props<m_size = 16, n_size = 16, k_size = 16, a_type = f16, b_type = f16, c_type = f16, result_type = f16, scope = <Subgroup>>, #spv.coop_matrix_props<m_size = 16, n_size = 16, k_size = 16, a_type = f16, b_type = f16, c_type = f32, result_type = f32, scope = <Subgroup>>]>>}>
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:65:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py:317:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py:969:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:96:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py:1097:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:65:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py:557:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py:969:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:96:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py:1097:0: error: failed to serialize executables
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:65:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py:317:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py:969:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:96:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/engine/base_layer.py:1097:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:65:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/models/convnext/modeling_tf_convnext.py:557:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/transformers/modeling_tf_utils.py:969:0: note: called from
E /data/anush/actions-runner/_work/SHARK/SHARK/shark.venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py:96:0: note: called from
E compilation failed
E
E
E Invoked with:
E iree-compile /home/ean/SHARK/shark.venv/lib/python3.10/site-packages/iree/compiler/tools/../_mlir_libs/iree-compile - --iree-input-type=mhlo --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=vulkan --iree-llvm-embedded-linker-path=/home/ean/SHARK/shark.venv/lib/python3.10/site-packages/iree/compiler/tools/../_mlir_libs/iree-lld --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --iree-llvm-target-cpu-features=host --iree-mhlo-demote-i64-to-i32=false --iree-flow-demote-i64-to-i32 -iree-vulkan-target-triple=ampere-rtx3080-linux --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64
E
E Need more information? Set IREE_SAVE_TEMPS=/some/dir in your environment to save all artifacts and reproducers.
shark.venv/lib/python3.10/site-packages/iree/compiler/tools/binaries.py:196: CompilerToolError
This occurs for all cases except CPU.
