No such file or directory: executorch/exir/_serialize/program.fbs and scalar_type.fbs

[adonnini]

Hi I installed executorch from the main branch.

When I ran code to produce and store a lowered mode, execution failed with the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/program.fbs'

I checked older versions of executorch installed from main, e.g. on 3/18/2024, program.fbs was in executorch/exir/_serialize/

Please find the complete traCeback log below.

Did I make a mistake?

Please let me know if you need additional information and need me to do anything.

Thanks

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 460, in <module>
    f.write(lowered_module.to_executorch().buffer)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/program/_program.py", line 871, in to_executorch
    return ExecutorchProgramManager(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/program/_program.py", line 925, in __init__
    self._pte_data: Cord = _serialize_pte_binary(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/_program.py", line 424, in serialize_pte_binary
    result: _FlatbufferResult = _program_json_to_flatbuffer(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/_flatbuffer.py", line 283, in _program_json_to_flatbuffer
    schema_info = _prepare_schema(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/_flatbuffer.py", line 156, in _prepare_schema
    schemas = _ResourceFiles([program_schema] + deps)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/_flatbuffer.py", line 113, in __init__
    self._files[name] = importlib.resources.read_binary(__package__, name)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/importlib/resources.py", line 88, in read_binary
    with open_binary(package, resource) as fp:
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/importlib/resources.py", line 46, in open_binary
    return reader.open_resource(resource)
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/importlib/abc.py", line 433, in open_resource
    return self.files().joinpath(resource).open('rb')
  File "/home/adonnini1/anaconda3/envs/executorch/lib/python3.10/pathlib.py", line 1117, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/program.fbs'

[orionr]

cc @guangy10 @dbort @byjlw @mergennachin

[adonnini]

Btw, Copying program.fbs and scalar_type.fbs in exir/_serializer solved my problem. I know it's not a solution but it tells me that's it's probably not a big deal to resolve.

[kirklandsign]

Try unset PYTHONPATH?

I ran into this as well. If I set PYTHONPATH to executorch root directory, it first searches for executorch/exir/_serializer before it searches for the pip package for executorch. We only copy program.fbs and scalar_type.fbs to pip package, not executorch source package.

[kirklandsign]

That seems to be introduced in https://github.com/pytorch/executorch/pull/2499. @dbort Maybe there can be compatibility concern like this?

[dbort]

Yes, the problem here seems to be that python is executing the .py files under your executorch directory instead of using the files installed by the pip package.

The old pip installation step incorrectly copied the .fbs files into the source tree; the new pip installation builds the package hermetically without modifying the source tree.

@adonnini what command are you running when this happens? Also, are you running that command from a directory that contains the executorch repo/directory?

[adonnini]

@dbort I follow the instructions in https://pytorch.org/executorch/stable/getting-started-setup.html (running ./install_requirements.sh) in the executorch directory. I have done this many times both installing the v0.1.0 and from the main branch. This was the first time this issue happened. I will be doing this again around the middle of next week. Thanks

[dbort]

@adonnini That is the correct way to install the pip package. But the problems you're seeing are probably related to how you're running the failing program.

Earlier you said

When I ran code to produce and store a lowered mode, execution failed with the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/_serialize/program.fbs'

What command/code are you running when this error happens? What is your current directory when you run this command?

[adonnini]

@dbort , sorry I misunderstood your question. Please find below the code I run to produce the lowered model, which is the code I executed when the problem occurred. Please let me know if you need anything else. Thanks

CODE TO LOWER MODEL

#Delegating to a Backend - Lowering the Whole Module
#---------------------------------------------------
# Export and lower the module to Edge Dialect
pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (enc_input, dec_input, dec_source_mask, dec_target_mask))
aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask), strict=False)
edge_program: EdgeProgramManager = to_edge(aten_dialect)
to_be_lowered_module = edge_program.exported_program()

from executorch.exir.backend.backend_api import LoweredBackendModule, to_backend

# Lower the module
lowered_module = edge_program.to_backend(XnnpackPartitioner())

print(" - train_minimum - Lowering the Whole Module - lowered_module - ", lowered_module)

# Serialize and save it to a file
save_path = save_path = "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/models/tpt_delegate.pte"
with open(save_path, "wb") as f:
    f.write(lowered_module.to_executorch().buffer)

[dbort]

Thanks for that, @adonnini.

When you run this command, what is your current directory? Does that directory contain a directory named executorch, which contains the ExecuTorch repo?

[adonnini]

I run the code from a directory which contains the executorch directory. It contains the Executorch repo (including exir) Does this answer your questions?

[metascroy]

@adonnini can you try running your code from a directory that does not contain the cloned executorch repo?

[adonnini]

@metascroy, sorry for the delay in responding. I ran the code as you suggested. The No such file or directory error did not occur. Please let me know if I need to do anything else

[dbort]

I'm glad that this worked. It is a pain that things behave differently based on the contents of the current directory; that's something that we hope to fix later this year.

[tusharxoxoxo]

https://pytorch.org/executorch/stable/getting-started-setup.html

i think documentation need to be updated, with the new code by @adonnini

[dbort]

@tusharxoxoxo are you talking about the block of lowering code in https://github.com/pytorch/executorch/issues/2910#issuecomment-2053865994? That code uses XNNPACK, which isn't appropriate for the top-level Getting Started doc; to keep things simple, we don't do any delegation in that doc.

https://pytorch.org/executorch/stable/native-delegates-executorch-xnnpack-delegate.html might be a better place, if you think that @adonnini's code is a better example. https://github.com/pytorch/executorch/blob/main/docs/source/native-delegates-executorch-xnnpack-delegate.md is the source for that page if anyone would like to send out a PR

[tusharxoxoxo]

I am feeling very mentally exhausted, can u please help me this one

https://pytorch.org/executorch/stable/tutorial-xnnpack-delegate-lowering.html i followed this tutorial, step by step and was finally able to execute this I had to move scaler_type.fbs and program.fbs from executorch/schema/ to executorch/exir/_serialize/ while executing the given commands given in that article and also had to move .pte file in and out of the executorch folder during some part of execution cause for reason ../ was not working or i was not able to figure it out (skill issues intensified)

After burning hours at this, i started following the instruction given in this article https://pytorch.org/executorch/stable/demo-apps-android.html

-> Luckily Build the CMake target for the library with XNNPACK backend was smooth sailing but finally things started to break apart again, when i started executing the commands for Build the Android extension and this is where i am stuck now

cmake extension/android \
  -DCMAKE_TOOLCHAIN_FILE=/Users/blouse_man/Library/Android/sdk/ndk/26.1.10909125/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a \
  -DCMAKE_INSTALL_PREFIX=cmake-android-out \
  -Bcmake-android-out/extension/android

and this is the error i am getting

-- ANDROID_PLATFORM not set. Defaulting to minimum supported version
21.
CMake Warning (dev) at CMakeLists.txt:24 (find_package):
  Policy CMP0144 is not set: find_package uses upper-case <PACKAGENAME>_ROOT
  variables.  Run "cmake --help-policy CMP0144" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  CMake variable EXECUTORCH_ROOT is set to:

    /Users/blouse_man/Downloads/coding/mljs/porsch/executorch/extension/android/../..

  For compatibility, find_package is ignoring the variable, but code in a
  .cmake module might still use it.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Error at CMakeLists.txt:24 (find_package):
  Could not find a package configuration file provided by "executorch" with
  any of the following names:

    executorchConfig.cmake
    executorch-config.cmake

  Add the installation prefix of "executorch" to CMAKE_PREFIX_PATH or set
  "executorch_DIR" to a directory containing one of the above files.  If
  "executorch" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!  

i tried to suppress this issue using

if (POLICY CMP0144)
  cmake_policy(SET CMP0144 NEW)
endif()

as this was shown in this StackOverflow question https://stackoverflow.com/questions/56586348/how-to-handle-with-policy-cmp0074-is-not-set-warning-while-using-cmake-to-buil

i also tried to add Wno-dev to the cmake command, but it still now working

cmake extension/android \
  -DCMAKE_TOOLCHAIN_FILE=/Users/blouse_man/Library/Android/sdk/ndk/26.1.10909125/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a \
  -DCMAKE_INSTALL_PREFIX=cmake-android-out \
  -Bcmake-android-out/extension/android
  -Wno-dev

fzf
Downloads/coding/mljs/porsch/executorch/build/executorch-config.cmake

executorch-config.cmake is present in the above folder, i don't know why is it erroring

I fzf my whole system but i couldn't find any file named executorchConfig.cmake in my system

cmake extension/android \
  -DCMAKE_TOOLCHAIN_FILE=/Users/blouse_man/Library/Android/sdk/ndk/26.1.10909125/build/cmake/android.toolchain.cmake \
  -DANDROID_ABI=arm64-v8a \
  -DCMAKE_INSTALL_PREFIX=cmake-android-out \
  -Bcmake-android-out/extension/android \
  -Dexecutorch_DIR=/Users/blouse_man/Downloads/coding/mljs/porsch/executorch/build \
  -Wno-dev

I changed the command to show the path of executorch-config.cmake but still getting is same error

-- ANDROID_PLATFORM not set. Defaulting to minimum supported version
21.
CMake Error at CMakeLists.txt:24 (find_package):
Could not find a package configuration file provided by "executorch" with
any of the following names:

  executorchConfig.cmake
  executorch-config.cmake

Add the installation prefix of "executorch" to CMAKE_PREFIX_PATH or set
"executorch_DIR" to a directory containing one of the above files.  If
"executorch" provides a separate development package or SDK, be sure it has
been installed.


-- Configuring incomplete, errors occurred!

woh surprisingly cmake --help-policy CMP0144" for policy detail error has stopped showing only the issue of executorchConfig.cmake is remaining ig

[dbort]

@tusharxoxoxo Thanks for all of the details, and I'm sorry you're running into all of these problems. But it sounds like you have issues unrelated to "No such file or directory: executorch/exir/_serialize/program.fbs and scalar_type.fbs"

Please create a new issue to track the problems you're seeing so we can help you there. Commenting on this closed issue is tough for us to triage.