4K4D icon indicating copy to clipboard operation
4K4D copied to clipboard

An error occurred(Segmentation fault) while loading the model

Open GuaGod opened this issue 1 year ago • 17 comments

9354794a67a2db0b8f54ca1d53c881d

GuaGod avatar Jan 08 '24 15:01 GuaGod

Hi @GuaGod , looks like the model has already been successfully loaded since the log you highlighted should be printed after model loading. The segmentation fault is most likely OpenGL-related so I might need your assistance on checking out your environment:

  1. If on Linux, could you try running glxgears and check whether the window is displayed correctly
  2. Try running evc-gui, which should open a black window for you with some controls, and check whether that works
  3. Check OpenGl version by glxinfo -B to see if any error is shown and check PyOpenGL installation, which should be of version 3.1.7 as of today. If we're on 3.1.0, update using pip install -U pyopengl

dendenxu avatar Jan 08 '24 16:01 dendenxu

  1. The window is displayed correctly after glxgears. image
  2. An error occured after evc-gui image
  3. image image

GuaGod avatar Jan 09 '24 06:01 GuaGod

That's strange 😶‍🌫️ . Your OpenGL setup looks fine. Could you try whether the off-screen rendering works?

evc -t test -c configs/projects/realtime4dv/rendering/4k4d_sport1.yaml,configs/specs/eval.yaml

Another thing to try is to disable the compatibiliy profile window hint by commenting these two lines here: image

dendenxu avatar Jan 09 '24 07:01 dendenxu

Hello, and thank you for your excellent work.

In my situation, off-screen rendering works fine, but I encounter an error when attempting rendering with a GUI. I followed your suggestion to comment out two lines, but I am still experiencing issues running in GUI mode.

Here are the details of my environment: OS: Ubuntu 22.04 LTS GPU: NVIDIA GeForce RTX 3090 Driver Version: 545.23.08 CUDA Version: 11.8"

Package                   Version            Editable project location
------------------------- ------------------ ------------------------------------------
absl-py                   2.0.0
addict                    2.4.0
ansi2html                 1.9.1
antlr4-python3-runtime    4.9.3
anyio                     4.2.0
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asttokens                 2.4.1
async-lru                 2.0.4
attrs                     23.2.0
autopep8                  2.0.4
Babel                     2.14.0
beautifulsoup4            4.12.2
black                     23.12.1
bleach                    6.1.0
blessed                   1.20.0
blinker                   1.7.0
cachetools                5.3.2
calmsize                  0.1.3
certifi                   2022.12.7
cffi                      1.16.0
charset-normalizer        2.1.1
chumpy                    0.70
clang-format              17.0.6
click                     8.1.7
cloudpickle               3.0.0
cmake                     3.25.0
comm                      0.2.1
commentjson               0.9.0
ConfigArgParse            1.7
contourpy                 1.2.0
cuda-python               12.3.0
cycler                    0.12.1
dash                      2.14.2
dash-core-components      2.0.0
dash-html-components      2.0.0
dash-table                5.0.0
debugpy                   1.8.0
decorator                 5.1.1
defusedxml                0.7.1
detectron2                0.6                /media/ym/linux-data/_src/_ref/detectron2
diff-gauss                1.0.5
docstring-parser          0.15
dotmap                    1.3.30
easyvolcap                0.0.0              /media/ym/linux-data/_src/_test/EasyVolcap
einops                    0.7.0
et-xmlfile                1.1.0
exceptiongroup            1.2.0
executing                 2.0.1
fast-autocomplete         0.9.0
fastjsonschema            2.19.1
filelock                  3.9.0
Flask                     3.0.0
flatbuffers               23.5.26
fonttools                 4.47.0
fqdn                      1.5.1
func-timeout              4.3.5
fvcore                    0.1.5.post20221221
glfw                      2.6.4
google-auth               2.26.1
google-auth-oauthlib      1.2.0
gpustat                   1.2.dev19+gf4cd639
grpcio                    1.60.0
h5py                      3.10.0
human-body-prior          2.2.2.0
hydra-core                1.3.2
idna                      3.4
imageio                   2.33.1
imgui-bundle              1.2.1
importlib-metadata        7.0.1
iopath                    0.1.9
ipdb                      0.13.13
ipykernel                 6.28.0
ipython                   8.20.0
ipywidgets                8.1.1
isoduration               20.11.0
itsdangerous              2.1.2
jedi                      0.19.1
Jinja2                    3.1.2
joblib                    1.3.2
json5                     0.9.14
jsonpointer               2.4
jsonschema                4.20.0
jsonschema-specifications 2023.12.1
jupyter                   1.0.0
jupyter_client            8.6.0
jupyter-console           6.6.3
jupyter_core              5.7.1
jupyter-events            0.9.0
jupyter-lsp               2.2.1
jupyter_server            2.12.3
jupyter_server_terminals  0.5.1
jupyterlab                4.0.10
jupyterlab_pygments       0.3.0
jupyterlab_server         2.25.2
jupyterlab-widgets        3.0.9
kiwisolver                1.4.5
kornia                    0.7.1
lark-parser               0.7.8
lazy_loader               0.3
lit                       15.0.7
lpips                     0.1.4
Markdown                  3.5.2
markdown-it-py            3.0.0
MarkupSafe                2.1.3
matplotlib                3.8.2
matplotlib-inline         0.1.6
mdurl                     0.1.2
mediapipe                 0.10.9
memory-tempfile           2.2.3
mistune                   3.0.2
mpmath                    1.3.0
msgpack                   1.0.7
munch                     4.0.0
mypy-extensions           1.0.0
nbclient                  0.9.0
nbconvert                 7.14.0
nbformat                  5.9.2
nest-asyncio              1.5.8
networkx                  3.0
ninja                     1.11.1.1
notebook                  7.0.6
notebook_shim             0.2.3
numpy                     1.24.1
nvdiffrast                0.3.1              /media/ym/linux-data/_src/_ref/nvdiffrast
nvidia-ml-py              12.535.133
oauthlib                  3.2.2
omegaconf                 2.3.0
open3d                    0.18.0
opencv-contrib-python     4.9.0.80
opencv-python             4.9.0.80
openpyxl                  3.1.2
overrides                 7.4.0
packaging                 23.2
pandas                    2.1.4
pandocfilters             1.5.0
parso                     0.8.3
pathspec                  0.12.1
pdbr                      0.8.8
pexpect                   4.9.0
Pillow                    9.5.0
pip                       23.3.1
platformdirs              4.1.0
plotly                    5.18.0
plyfile                   1.0.3
portalocker               2.8.2
prometheus-client         0.19.0
prompt-toolkit            3.0.43
protobuf                  3.20.3
psutil                    5.9.7
ptyprocess                0.7.0
pure-eval                 0.2.2
pyasn1                    0.5.1
pyasn1-modules            0.3.0
pycocotools               2.0.7
pycodestyle               2.11.1
pycparser                 2.21
pyglfw                    0.2.2
PyGLM                     2.7.1
Pygments                  2.17.2
PyMCubes                  0.1.4
pymeshlab                 2023.12
pyntcloud                 0.3.1
PyOpenGL                  3.1.7
pyparsing                 3.1.1
pyperclip                 1.8.2
pyquaternion              0.9.9
python-dateutil           2.8.2
python-json-logger        2.0.7
pytorch-memlab            0.3.0
pytorch-msssim            1.0.0
pytorch3d                 0.7.5              /media/ym/linux-data/_src/_ref/pytorch3d
PyTurboJPEG               1.7.3
pytz                      2023.3.post1
PyYAML                    6.0.1
pyzmq                     25.1.2
qtconsole                 5.5.1
QtPy                      2.4.1
referencing               0.32.1
requests                  2.31.0
requests-oauthlib         1.3.1
retrying                  1.3.4
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rich                      13.7.0
rpds-py                   0.16.2
rsa                       4.9
ruamel.yaml               0.18.5
ruamel.yaml.clib          0.2.8
scikit-image              0.22.0
scikit-learn              1.3.2
scipy                     1.11.4
Send2Trash                1.8.2
setuptools                68.2.2
shtab                     1.6.5
simple-knn                0.0.0
six                       1.16.0
smplx                     0.1.28
sniffio                   1.3.0
sounddevice               0.4.6
soupsieve                 2.5
stack-data                0.6.3
sympy                     1.12
tabulate                  0.9.0
tenacity                  8.2.3
tensorboard               2.15.1
tensorboard-data-server   0.7.2
tensorboardX              2.6.2.2
termcolor                 2.4.0
terminado                 0.18.0
threadpoolctl             3.2.0
tifffile                  2023.12.9
timg                      1.1.6
tinycss2                  1.2.1
tinycudann                1.6
tomli                     2.0.1
torch                     2.0.1+cu118
torch-scatter             2.1.2
torch-tb-profiler         0.4.3
torchaudio                2.0.2+cu118
torchdiffeq               0.2.3
torchmcubes               0.1.0
torchvision               0.15.2+cu118
tornado                   6.4
tqdm                      4.66.1
traitlets                 5.14.1
trimesh                   4.0.8
triton                    2.0.0
types-python-dateutil     2.8.19.20240106
typing_extensions         4.4.0
tyro                      0.6.3
tzdata                    2023.4
ujson                     5.9.0
uri-template              1.3.0
urllib3                   1.26.13
wcwidth                   0.2.13
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.7.0
Werkzeug                  3.0.1
wheel                     0.41.2
widgetsnbextension        4.0.9
xatlas                    0.0.8
yacs                      0.1.8
yapf                      0.40.2
zipp                      3.17.0

YongmoonPark avatar Jan 12 '24 02:01 YongmoonPark

Hi @YongmoonPark thanks for trying out our code! Does the command also throw a segmentation fault in your case?

dendenxu avatar Jan 13 '24 05:01 dendenxu

Hey guys @GuaGod @YongmoonPark , could you try replacing the evc command with python -q -X faulthandler easyvolcap/scripts/main.py to let the program print more useful information for the segmentation fault?

For example, try running python -q -X faulthandler easyvolcap/scripts/main.py -t gui -c configs/specs/gui.yaml.

dendenxu avatar Jan 15 '24 00:01 dendenxu

Thank you, @dendenxu, for your prompt and kind response.

In my case, off-screen rendering works fine, but when I attempt rendering with a GUI, I encounter the same error. I tried following your instructions to comment out two lines, but I'm still facing an error when running in GUI mode. Here is the error message I received:

When executing the command: python -q -X faulthandler easyvolcap/scripts/main.py -t gui -c configs/specs/gui.yaml I encountered the following error:

2024-01-15 09:32:27.317839 __main__ -> preflight: Starting experiment: base, command: gui main.py:80
2024-01-15 09:32:27.318702 easyvolcap.utils.console_utils -> inner: Runtime exception: 'batch_size' console_utils.py:341
AttributeError: 'batch_size'
*** 'batch_size'
> /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/base_utils.py(101)__getitem__()
whole error message
(easyvolcap) ym@ym-linux:.../_test/EasyVolcap$ python -q -X faulthandler easyvolcap/scripts/main.py -t gui -c configs/specs/gui.yaml.
2024-01-15 09:32:27.317839 __main__ -> preflight: Starting experiment: base, command: gui                                                  main.py:80
2024-01-15 09:32:27.318702 easyvolcap.utils.console_utils -> inner: Runtime exception: 'batch_size'                              console_utils.py:341
╭─────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/base_utils.py:99 in __getitem__                                              │
│                                                                                                                                          │
│ ❱  99 │   │   │   return dict.__getitem__(self, key)                                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'batch_size'

During handling of the above exception, another exception occurred:

╭─────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/console_utils.py:338 in inner                                                │
│                                                                                                                                          │
│ ❱ 338 │   │   │   return func(*args, **kwargs)                                                                                           │
│                                                                                                                                          │
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:56 in inner                                                     │
│                                                                                                                                          │
│ ❱  56 │   │   return call_from_cfg(func, cfg)                                                                                            │
│                                                                                                                                          │
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:47 in call_from_cfg                                             │
│                                                                                                                                          │
│ ❱  47 │   return func(**call_args)                                                                                                       │
│                                                                                                                                          │
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/scripts/main.py:158 in test                                                        │
│                                                                                                                                          │
│ ❱ 158 │   val_dataloader: "VolumetricVideoDataloader" = DATALOADERS.build(val_dataloader_cfg)  # reuse the validataion                   │
│                                                                                                                                          │
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:300 in build                                                    │
│                                                                                                                                          │
│ ❱ 300 │   │   return self.build_func(*args, **kwargs, registry=self)                                                                     │
│                                                                                                                                          │
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:131 in build_from_cfg                                           │
│                                                                                                                                          │
│ ❱ 131 │   return call_from_cfg(obj_cls, args)                                                                                            │
│                                                                                                                                          │
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/engine/registry.py:47 in call_from_cfg                                             │
│                                                                                                                                          │
│ ❱  47 │   return func(**call_args)                                                                                                       │
│                                                                                                                                          │
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/dataloaders/volumetric_video_dataloader.py:93 in __init__                          │
│                                                                                                                                          │
│ ❱  93 │   │   if batch_sampler_cfg.batch_size == -1: batch_sampler_cfg.batch_size = len(dataset)                                         │
│                                                                                                                                          │
│ /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/base_utils.py:101 in __getitem__                                             │
│                                                                                                                                          │
│ ❱ 101 │   │   │   raise AttributeError(e)                                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'batch_size'
*** 'batch_size'
> /media/ym/linux-data/_src/_test/EasyVolcap/easyvolcap/utils/base_utils.py(101)__getitem__()
     99             return dict.__getitem__(self, key)
    100         except KeyError as e:
--> 101             raise AttributeError(e)
    102     # MARK: Might encounter exception in newer version of pytorch
    103     # Traceback (most recent call last):

(Pdbr) 

[1]+  Stopped                 python -q -X faulthandler easyvolcap/scripts/main.py -t gui -c configs/specs/gui.yaml.

The process stopped with the error: AttributeError: 'batch_size'. Any guidance you could provide would be greatly appreciated.

YongmoonPark avatar Jan 15 '24 00:01 YongmoonPark

Thanks for the feedback @YongmoonPark ! I could not reproduce the batch_size error on my end. That error usually means we are in the wrong directory or could not find the config file the user specified (yes I need to make the errors more human readable immediately :] ). Are you on the latest commit?

dendenxu avatar Jan 15 '24 01:01 dendenxu

Thank you for your prompt response, @dendenxu . I am indeed using the latest commit. I'll take your advice and recheck everything, including reinstalling my environment, to ensure

YongmoonPark avatar Jan 15 '24 01:01 YongmoonPark

@YongmoonPark Ah, no need for reintalling the environment. But you could try reinstalling 4K4D itself by executing pip install -e . inside the directory.

dendenxu avatar Jan 15 '24 01:01 dendenxu

Could you check whether this file configs/specs/gui.yaml exists in your working directory? The first line of log should be like this, where the experiment name is changed to gui instead of base:

2024-01-15 09:36:28.746102 __main__ -> preflight: Starting experiment: gui, command: gui

dendenxu avatar Jan 15 '24 01:01 dendenxu

I have verified the location of configs/specs/gui.yaml. However, I still encounter the following error when executing:

2024-01-15 11:42:30.025936 __main__ -> preflight: Starting experiment: base, command: gui                              main.py:80
2024-01-15 11:42:30.026796 easyvolcap.utils.console_utils -> inner: Runtime exception: 'batch_size'  

It seems that the issue might be occurring because the configs/specs/gui.yaml file is not being read and executed properly.

Thank you. Your advice has been helpful in narrowing down the cause of the problem.

YongmoonPark avatar Jan 15 '24 02:01 YongmoonPark

image Glad to know that we're making progress. The next reasonable step seems to be putting a breakpoint here and examine the content of args.config and os.getcwd(). To add a breakpoint, simply add the line breakpoint()

dendenxu avatar Jan 15 '24 03:01 dendenxu

@GuaGod @YongmoonPark Curious to ask, are you running on WSL2 instead of native Windows/Ubuntu? Related issue.

dendenxu avatar Jan 20 '24 07:01 dendenxu

@GuaGod @YongmoonPark Hi folks! Sorry to bother again, but I pushed an update that made the GUI more compatible and make errors messages easier to understand (including segmentation faults). Could you try reinstalling easyvolcap (just run pip install -e . inside the repo) and rerun the erronous commands above and see whether the errors go away?

dendenxu avatar Jan 20 '24 10:01 dendenxu

@dendenxu Thank you. I am using native Ubuntu (22.04 LTS). I have been trying to trace the cause of a Segmentation fault error. I am still in the process of tracing it, but it seems like imgui_bundle was incompletely installed when set up through requirements.txt.

I have also confirmed the same error occurs when running the simple sample code of imgui_bundle below:

from imgui_bundle import imgui, immapp
imgui.create_context()

def gui():
    imgui.text("Hello, world!")

immapp.run(
    gui_function=gui,  # The Gui function to run
    window_title="Hello!",  # the window title
    window_size_auto=True,  # Auto size the application window given its widgets
    # Uncomment the next line to restore window position and size from previous run
    # window_restore_previous_geometry=True
)

YongmoonPark avatar Jan 23 '24 00:01 YongmoonPark

Glad to know we're narrowing down the search. Maybe you could also try starting up python with -q -X faulthandler to debug the exact cause of the segfault like this:

python -q -X faulthandler -c "from imgui_bundle import imgui, immapp; imgui.create_context(); immapp.run(gui_function=lambda: imgui.text('Hello, world.'));"

dendenxu avatar Jan 23 '24 03:01 dendenxu