Grasp Vizualization: Evaluate pixel relative offset at every pixel with single gripper pose offset model

[ahundt]

We need to be able to generate a 3D visualization of many poses and the predicted grasp success values to determine if the results look reasonable.

Here is the model file to load for visualization: 2018-01-20-06-41-24_grasp_model_weights-delta_depth_sin_cos_3-grasp_model_levine_2016-dataset_062_b_063_072_a_082_b_102-epoch-014-val_loss-0.641-val_acc-0.655.h5.zip

Here is the updated scene file: 2018-01-20-0630-kukaRemoteApiCommandServerExample.ttt.zip

TODO:

• [x] show matplotlib rendering of predictions

• [ ] make boundaries of crop clear

• [ ] fix prediction visualization to use correct depth image cropping and prediction cropping

• [ ] overlay real image with predictions

• [ ] actually show color on predictions & point cloud in V-REP

• [ ] make sure offset of point cloud is correct

• [ ] don't overwrite main colored time step point cloud

• [ ] make pixels configurable from remote api so they can appear different from scene pixels

• [ ] Add colored point cloud to V-REP based on grasp success value at a depth

• [ ] Plot the heat map in V-REP point cloud

• [ ] Do the same for pixel wise training

• [ ] Generate current to final pose transform for every pixel in resized clear view + current image at a fixed depth offset

• [ ] Create loop that generates end-effector relative poses at every pixel and evaluates

• [ ] Create an input vector with all of this data to run through the prediction algorithm, and call predict() on each

  • See eval() in grasp_train.py

• [ ] low priority: Use tf.image.resize_images for a ResizeMethod.NEAREST_NEIGHBOR nearest neighbor resize of the depth image after applying the median filter.

V-REP code steps:

• [ ] Enable setting the color of dummies
• [ ] Incorporate function that maps 0-1 values to colors
• [ ] Add rescaled image display that shows the heat map at each pixel
• [ ] set color and thickness of lines being drawn so we can show surface relative portion of transform

Remember, we will need to get from the full sized images to the small images and back!

Bonus features that would help, but are not required:

• [ ] v-rep gui checkbox to show/hide point clouds & labels with one click, plus to display the confidence

TensorBoard steps:

• [ ] Enable gradient visualization
• [ ] Enable image visualization
• [ ] Enable picture visualization

Gradient visualization:

• [ ] Consider using keras-viz to perform grad-cam visualization of attention

[ahundt]

This visualization enhancement was suggested by @cpaxton, could you add your thoughts on this issue description?

[ahundt]

Initial work in https://github.com/cpaxton/costar_plan/pull/429, this issue is still in progress.