Moving a point around an annotation returns inconsistent data.

[hoxbro]

When moving a point around an annotation, the returned data is no longer valid:

https://user-images.githubusercontent.com/19758978/179739253-1882e858-07fd-4b37-ba7b-33d0bef8dfd5.mp4

The output in the DataFrame seems to not be converted to numpy arrays:

import base64

output = "Xq2uksFTY8EAAADgPFNjwQAAAMAmU2PBAAAAAMtSY8E="
decoded = np.frombuffer(base64.decodebytes(output.encode("ascii")))
decoded

To reproduce, run https://pyviz-dev.github.io/geoviews/user_guide/Annotators.html locally and move around a point

First noticed in #583

Edit: Seem to only be a problem when 4 points are present.