Deck.gl assertion failed uncaught exception is crashing the UI when specifying a customLayer

[jruddell]

Hi, I spent a chunk of time digging into what was happening. For some context I'm rendering DeepMap elements into the UI via the customLayers prop on the LogViewer. I followed the documentation here to add a new layer (i.e. new GeoJSONLayer). However, I noticed that sometimes a really obscure and seemingly random bug happens, where deck.gl asserts that a layer doesn't have any state when it does. That layer happens to be the customLayer that I added.

When debugging, I noticed that deck.gl is trying to loop through layers and initialize them, in the middle of a ROSBag playback. All of the xvis layers needed to be initialized ( as in I inspected them all while debugging and they didn't have a state ), however not the GeoJSONLayer, because its added separately (I believe).

What should I do to fix this?

[jruddell]

So, did a lot more debugging... ROS's messaging is socket based, which is non-deterministic. Sometimes messages would come out of order (this is why it was seemingly random), if a message with an older timestamp is passed, the layers reinitialize.

First fix is on my side to make sure the timestamp is a future time from the last message. However, the bug on the streetscape side prevents me from replaying a ROSbag. If I try to replay the bag then the same issue happens where the GeoJSONLayer is already initialized but it tries to initialize it again. Thoughts?

[twojtasz]

@jruddell Thanks for digging into this and great find on the out of order root cause.

Are you connecting directly to a ROS instance or are you using the @xviz/ros package? Any details on this that I could get to repro would be very useful. (roslibjs or other?)

I would have thought that if a message comes in out of order (but the timestamp is still valid) it should just be inserted into our data store at the appropriate spot. Are you running in a live mode?

@Pessimistress do you know why we would be reinitializing if a message comes in out-of-order or with an older timestamp? It feels like if something is invalidating the layers due to the timing issue, but this is propogated/done with custom layers we would need some event to keep the custom layer insync?

[jruddell]

@twojtasz Yes, it is running in live mode. My assumption is that the layers are invalidated because of timing, but i haven't actually validated that because debugging compiled source code is quite annoying haha. Seems like whatever is doing this invalidation is not taking into account the customLayers. They definitely are registered though in the sense that deck.gl is trying to reinitialize them.

We have our own custom Protobuffs that are published from ROS modules. The renderer server for xvis takes each message in the ROS ecosystem and translates that to xvis... then sends / publishes on the socket for the UI to render (ROS isn't / wasn't supported at the time of adoption). The conversion and stuff is def not the cause (lots of validation and testing to be sure). But if i trigger a message that is older to send in the live system that is what crashes the UI. I can try and explain more but need to make sure I can lol :)

[jruddell]

@twojtasz Any update on this? Its a frequent bug that requires a restart of the system, would like to see what we can do to get it fixed :) Thanks!

[twojtasz]

I will try to get to investigate out of order data, we don't have many test cases for this right now.

[Portur]

I got this bug with the following, everytime, This happens when toggling the same layer or removing one and adding another

             // create layers
        layers = {
            current: {
                active: true,
                data: geoJsonLayer(old_data, 'current')
            },
            new: {
                active: true,
                data: geoJsonLayer(new_data, 'new')
            },
            points: {
                active: true,
                data: pointsLayer(pointData)
            }
        };

               // create overlay
        overlay.setProps({
            layers: [
                layers.current.data,
                layers.new.data,
                layers.points.data,
            ]
        });

//============
              // this is called by UI 
function toggle (layer_name) {

               // change layer state "active"
    layers[layer_name].active = !layers[layer_name].active;

              // get "active" layers
    let active_layers = Object.values(layers).filter(l => l.active).map(l => l.data);

              // set google maps overlay to newly added layers
    overlay.setProps({
        layers: active_layers
    });
}

[ryanjduffy]

I encountered a similar error so I'm sharing my workaround in case it is useful to others.

My analysis of the issue is that the streetscape layers are recreated frequently but the layers provided via customLayers are not. As a result, the LogViewer logic (or something downstream in deck.gl) tries to initialize the layers assuming they are new which works for the built-in layers but fails for custom layers.

In my case, this seems to throw when seeking to a point in the timeline in which the data doesn't exist yet. When the data comes in, the layers are created and initialized and the error throws.

To work around this, I added a ready check that subscribes to log updates. If at any point I can't retrieve the current frame, I clear the ready flag until I can and then recreate the layer. Seems to do the trick.

Here's the bulk of the hook I'm using:

function useMapLayer (geoJsonUrl?: string, log?: XVIZLoader) {
  const [layerData, setLayerData] = useState();
  const [ready, setReady] = useState(false);

  useEffect(() => {
    if (!log) return;

    function checkReady () {
      const frame = log?.getCurrentFrame();
      setReady(frame != null);
    }

    log.subscribe(checkReady);
    return () => log.unsubscribe(checkReady);
  }, [log, setReady]);

  useEffect(() => {
    if (!geoJsonUrl) return;

    fetch(geoJsonUrl)
      .then(res => res.json())
      .then(setLayerData);
  }, [geoJsonUrl, setLayerData]);

  return useMemo(() => ready && layerData && createLayer(layerData), [ready, layerData]);
}