Observation shows black images/screen

[hanniakku]

Hi, When I run the evaluation python main.py --split val --eval 1 --load pretrained_models/sem_exp.pth -v 1the output visualization shows black screen for observations and the semantic map does not show any objects. [See reference image below]

I am using python 3.7.0, pytorch==1.9.0, torchvision==0.10.0, torchaudio==0.9.0, cudatoolkit=11.1. My data seems to be in place: The .glb and .navmesh files are under ~/Object-Goal-Navigation/data/scene_datasets/gibson_semantic/ The .glb.json.gz files are under ~/Object-Goal-Navigation/data/datasets/objectnav/gibson/v1.1/{split}/content/ The .gz and .pbz2 files are under ~/Object-Goal-Navigation/data/datasets/objectnav/gibson/v1.1/{split}/

Running the python test.py --agent random -n1 --num_eval_episodes 1 --auto_gpu_config 0 script shows Test successfully completed

Following is the console output for running python main.py --split val --eval 1 --load pretrained_models/sem_exp.pth -v 1 `Auto GPU config: Number of processes: 1 Number of processes on GPU 0: 1 Number of processes per GPU: 0 Dumping at ./tmp//models/exp1/ Namespace(agent='sem_exp', alpha=0.99, auto_gpu_config=1, camera_height=0.88, cat_pred_threshold=5.0, clip_param=0.2, collision_threshold=0.2, cuda=True, du_scale=1, dump_location='./tmp/', entropy_coef=0.001, env_frame_height=480, env_frame_width=640, eps=1e-05, eval=1, exp_name='exp1', exp_pred_threshold=1.0, floor_thr=50, frame_height=120, frame_width=160, gamma=0.99, global_downscaling=2, global_hidden_size=256, hfov=79.0, intrinsic_rew_coeff=0.02, load='pretrained_models/sem_exp.pth', log_interval=10, lr=2.5e-05, map_pred_threshold=1.0, map_resolution=5, map_size_cm=2400, max_d=100.0, max_depth=5.0, max_episode_length=500, max_grad_norm=0.5, min_d=1.5, min_depth=0.5, no_cuda=False, num_eval_episodes=200, num_global_steps=20, num_local_steps=25, num_mini_batch=1, num_processes=1, num_processes_on_first_gpu=1, num_processes_per_gpu=0, num_sem_categories=16, num_train_episodes=10000, num_training_frames=10000000, ppo_epoch=4, print_images=0, reward_coeff=0.1, save_interval=1, save_periodic=500000, seed=1, sem_gpu_id=-1, sem_pred_prob_thr=0.9, sim_gpu_id=1, split='val', success_dist=1.0, task_config='tasks/objectnav_gibson.yaml', tau=0.95, total_num_scenes=5, turn_angle=30, use_gae=False, use_recurrent_global=0, value_loss_coef=0.5, version='v1.1', vision_range=100, visualize=1) Scenes per thread: Thread 0: ['Collierville.glb', 'Corozal.glb', 'Darden.glb', 'Markleeville.glb', 'Wiconisco.glb'] 2022-07-12 22:17:44,765 Initializing dataset PointNav-v1 2022-07-12 22:17:44,766 initializing sim Sim-v0 WARNING: Logging before InitGoogleLogging() is written to STDERR I0712 22:17:44.778574 252548 Simulator.cpp:96] Loading navmesh from data/scene_datasets/gibson_semantic//Collierville.navmesh I0712 22:17:44.778640 252548 Simulator.cpp:98] Loaded. I0712 22:17:44.778653 252548 SceneGraph.h:92] Created DrawableGroup: Renderer: Mesa Intel(R) UHD Graphics (TGL GT1) by Intel OpenGL version: 4.6 (Core Profile) Mesa 21.2.6 Using optional features: GL_ARB_ES2_compatibility GL_ARB_direct_state_access GL_ARB_get_texture_sub_image GL_ARB_invalidate_subdata GL_ARB_multi_bind GL_ARB_robustness GL_ARB_separate_shader_objects GL_ARB_texture_filter_anisotropic GL_ARB_texture_storage GL_ARB_texture_storage_multisample GL_ARB_vertex_array_object GL_KHR_debug Using driver workarounds: no-layout-qualifiers-on-old-glsl mesa-implementation-color-read-format-dsa-explicit-binding mesa-forward-compatible-line-width-range I0712 22:17:44.809857 252548 ResourceManager.cpp:1380] Importing Basis files as ASTC 4x4 I0712 22:17:46.590839 252548 Simulator.cpp:167] Loading house from data/scene_datasets/gibson_semantic//Collierville.scn I0712 22:17:46.590855 252548 Simulator.cpp:188] Not loading semantic mesh I0712 22:17:46.591197 252548 simulator.py:146] Loaded navmesh data/scene_datasets/gibson_semantic//Collierville.navmesh I0712 22:17:46.591398 252548 simulator.py:158] Recomputing navmesh for agent's height 0.88 and radius 0.18. I0712 22:17:46.602424 252548 PathFinder.cpp:375] Building navmesh with 108x202 cells I0712 22:17:46.676290 252548 PathFinder.cpp:643] Created navmesh with 246 vertices 126 polygons I0712 22:17:46.676309 252548 Simulator.cpp:609] reconstruct navmesh successful 2022-07-12 22:17:46,678 Initializing task ObjectNav-v1 /home/arhanni/anaconda3/envs/ogn/lib/python3.7/site-packages/torchvision/transforms/transforms.py:281: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum. "Argument interpolation should be of type InterpolationMode instead of int. " [07/12 22:17:46 detectron2]: Arguments: Namespace(confidence_threshold=0.9, config_file='configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml', input=['input1.jpeg'], opts=['MODEL.WEIGHTS', 'detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl', 'MODEL.DEVICE', 'cuda:0'], output=None, video_input=None, webcam=False) Changing scene: 0/data/scene_datasets/gibson_semantic//Collierville.glb Loading episodes from: data/datasets/objectnav/gibson/v1.1/val/content/Collierville_episodes.json.gz /home/arhanni/anaconda3/envs/ogn/lib/python3.7/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at /opt/conda/conda-bld/pytorch_1623448265233/work/aten/src/ATen/native/BinaryOps.cpp:467.) return torch.floor_divide(self, other) /home/arhanni/anaconda3/envs/ogn/lib/python3.7/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /opt/conda/conda-bld/pytorch_1623448265233/work/c10/core/TensorImpl.h:1156.) return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode) Loading model pretrained_models/sem_exp.pth /home/arhanni/anaconda3/envs/ogn/lib/python3.7/site-packages/torch/nn/functional.py:4044: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details. "Default grid_sample and affine_grid behavior has changed " /home/arhanni/anaconda3/envs/ogn/lib/python3.7/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /opt/conda/conda-bld/pytorch_1623448265233/work/c10/core/TensorImpl.h:1156.) return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode) QObject::moveToThread: Current thread (0x557aa5d2f580) is not the object's thread (0x557b53203880). Cannot move to target thread (0x557aa5d2f580)

QObject::moveToThread: Current thread (0x557aa5d2f580) is not the object's thread (0x557b53203880). Cannot move to target thread (0x557aa5d2f580)

QObject::moveToThread: Current thread (0x557aa5d2f580) is not the object's thread (0x557b53203880). Cannot move to target thread (0x557aa5d2f580) ......

QObject::moveToThread: Current thread (0x557aa5d2f580) is not the object's thread (0x557b53203880). Cannot move to target thread (0x557aa5d2f580)

Time: 00d 00h 00m 00s, num timesteps 0, FPS 0, Rewards: Losses: Time: 00d 00h 00m 01s, num timesteps 10, FPS 8, Rewards: Losses: Time: 00d 00h 00m 02s, num timesteps 20, FPS 9, Rewards: Losses: Time: 00d 00h 00m 03s, num timesteps 30, FPS 9, Rewards: Losses: Time: 00d 00h 00m 04s, num timesteps 40, FPS 9, Rewards: Losses: ........... `

Following is what I have tried so far and have been unsuccessful. I found the following issue reported on habitat-sim https://github.com/facebookresearch/habitat-sim/issues/1080. I downloaded the 3DSceneGraph_tiny.zip and ran the tools/gen_gibson_semantics.sh script as specified here: https://github.com/facebookresearch/habitat-sim/blob/main/DATASETS.md. This script generated .ids and .scn files which I copied to ~/Object-Goal-Navigation/data/scene_datasets/gibson_semantic/ but the visualization still shows a blank screen. The gen_gibson_semantics.sh script gave the following error in generating _semantic.ply files Allensville wrote 2557798 bytes wrote 27718 bytes ESP_CHECK failed: esp::logging::LoggingContext: No current logging context. tools/gen_gibson_semantics.sh: line 17: 237742 Aborted "${TOOLS_DIR}"/../build/utils/datatool/Datatool create_gibson_semantic_mesh "${OBJ_PATH}"/"${scene}"/mesh.obj "${OUT_PATH}"/"${scene}".ids "${OUT_PATH}"/"${scene}"_semantic.ply because it couldn't find the mesh.obj files. The gibson_habitat_trainval.zip does not contain the .obj files.

I need the RGB observations to extract the semantic predictions and build my project on top of this. Any support would be appreciated!

[devendrachaplot]

The code does not require the 3DSceneGraph to be built as it does not require access to ground-truth semantics.

Could you try saving the images obtained from habitat directly here: https://github.com/devendrachaplot/Object-Goal-Navigation/blob/master/envs/habitat/objectgoal_env.py#L380

[hanniakku]

I figured out why that was happening. I was running this code on a PC with RTX 3080 GPU. I believe there is a Cudatoolkit compatibility issue for this GPU. Even the docker installation had the same issue. However, I got through by using a docker installation on a PC with RTX 1080 GPU.

Thank you for the response! But I cannot try that as I'm no longer working on that project. Please feel free to close the issue.