Datashader with HoverLayer

[dwervin]

I tried to add HoverLayer to a datashader image and the hover data initially appears, but when I zoom in, the hover functionality doesn't seem to work.

I was using the nyc_taxi demo notebook as the basis, but with some modifications to plot my own data. It successfully plots the data and interactive zooming works, but it seems something is not quite right with the hover layer after zooming.

##Example: import numpy as np from functools import partial from datashader.bokeh_ext import HoverLayer

x_range, y_range = ((-10975134.269,-10512411.001), (4414961.940,4896987.468))

def create_image90(x_range, y_range, w=plot_width, h=plot_height): cvs = ds.Canvas(plot_width=w, plot_height=h, x_range=x_range, y_range=y_range) agg = cvs.points(df, 'X_CORD', 'Y_CORD', ds.count()) img = tf.shade(agg.where(agg>np.percentile(agg,90)), cmap=inferno, how='eq_hist') return tf.dynspread(img, threshold=0.3, max_px=4)

p = base_plot() p.add_tile(STAMEN_TERRAIN) export(create_image(*NYC),"NYCT_90th")

hover_layer = HoverLayer(agg=cvs.points(df, 'X_CORD', 'Y_CORD', ds.count()), extent=(-10975134.269,4414961.940,-10512411.001,4896987.468), field_name='Average Call Count',how='sum') p.renderers.append(hover_layer.renderer) p.add_tools(hover_layer.tool) InteractiveImage(p, create_image90)

[jbednar]

Sorry -- I don't think the HoverLayer is currently working, either in the previous release or on master. Something changed in Bokeh that we have yet to track down, and we've been focusing instead on making hover work for Bokeh in HoloViews (as shown here).

[dwervin]

So does the hover in Holoviews work with datashader then?

If I make the rectangle small enough, will the numbers adjust as you zoom in as the pixels are updated?

[jbednar]

Yes and yes; see that link (or the holoviews_datashader notebook in the current datashader repo). It won't work on the website, because it requires a live Python server, but it should work fine if you re-run the notebook yourself. You'll need the master version of datashader, or at least the latest dev release (conda install -c bokeh/label/dev datashader; conda install -c ioam holoviews bokeh).

[dwervin]

I installed the latest releases per your recommendation and tried to run the holoviews_datashader notebook, but it throws the following error. I think it errors on datashade()

decimate(points) + datashade(points) + datashade(paths)

WARNING:root:dynamic_operation: Exception raised in callable 'dynamic_operation' of type 'function'. Invoked as dynamic_operation(height=400, scale=1.0, width=400, x_range=None, y_range=None)

ValueError Traceback (most recent call last) C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\formatters.py in call(self, obj) 305 pass 306 else: --> 307 return printer(obj) 308 # Finally look for special method names 309 method = get_real_method(obj, self.print_method)

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in pprint_display(obj) 257 if not ip.display_formatter.formatters['text/plain'].pprint: 258 return None --> 259 return display(obj, raw=True) 260 261

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in display(obj, raw, **kwargs) 243 elif isinstance(obj, (HoloMap, DynamicMap)): 244 with option_state(obj): --> 245 html = map_display(obj) 246 else: 247 return repr(obj) if raw else IPython.display.display(obj, **kwargs)

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in wrapped(element) 129 try: 130 html = fn(element, --> 131 max_frames=OutputSettings.options['max_frames']) 132 133 # Only want to add to the archive for one display hook...

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in map_display(vmap, max_frames) 198 return None 199 --> 200 return render(vmap) 201 202

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in render(obj, **kwargs) 59 if renderer.fig == 'pdf': 60 renderer = renderer.instance(fig='png') ---> 61 return renderer.html(obj, **kwargs) 62 63

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\plotting\renderer.py in html(self, obj, fmt, css, comm, **kwargs) 253 code to initialize a Comm, if the plot supplies one. 254 """ --> 255 plot, fmt = self._validate(obj, fmt) 256 figdata, _ = self(plot, fmt, **kwargs) 257 if css is None: css = self.css

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\plotting\renderer.py in _validate(self, obj, fmt) 189 if isinstance(obj, tuple(self.widgets.values())): 190 return obj, 'html' --> 191 plot = self.get_plot(obj, renderer=self) 192 193 fig_formats = self.mode_formats['fig'][self.mode]

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\plotting\bokeh\renderer.py in get_plot(self_or_cls, obj, doc, renderer) 110 combining the bokeh model with another plot. 111 """ --> 112 plot = super(BokehRenderer, self_or_cls).get_plot(obj, renderer) 113 if doc is not None: 114 plot.document = doc

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\plotting\renderer.py in get_plot(self_or_cls, obj, renderer) 164 """ 165 # Initialize DynamicMaps with first data item --> 166 initialize_dynamic(obj) 167 168 if not isinstance(obj, Plot) and not displayable(obj):

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\plotting\util.py in initialize_dynamic(obj) 174 continue 175 if not len(dmap): --> 176 dmap[dmap._initial_key()] 177 178

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\core\spaces.py in getitem(self, key) 1008 # Not a cross product and nothing cached so compute element. 1009 if cache is not None: return cache -> 1010 val = self._execute_callback(*tuple_key) 1011 if data_slice: 1012 val = self._dataslice(val, data_slice)

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\core\spaces.py in _execute_callback(self, *args) 844 845 with dynamicmap_memoization(self.callback, self.streams): --> 846 retval = self.callback(*args, **kwargs) 847 return self._style(retval) 848

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\core\spaces.py in call(self, *args, **kwargs) 518 519 try: --> 520 ret = self.callable(*args, **kwargs) 521 except: 522 posstr = ', '.join(['%r' % el for el in self.args]) if self.args else ''

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\util_init_.py in dynamic_operation(*key, **kwargs) 341 self.p.kwargs.update(kwargs) 342 obj = map_obj[key] if isinstance(map_obj, HoloMap) else map_obj --> 343 return self._process(obj, key) 344 else: 345 def dynamic_operation(*key, **kwargs):

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\util_init_.py in _process(self, element, key) 327 kwargs = {k: v for k, v in self.p.kwargs.items() 328 if k in self.p.operation.params()} --> 329 return self.p.operation.process_element(element, key, **kwargs) 330 else: 331 return self.p.operation(element, **self.p.kwargs)

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\core\operation.py in process_element(self, element, key, **params) 119 """ 120 self.p = param.ParamOverrides(self, params) --> 121 return self._process(element, key) 122 123

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\operation\datashader.py in _process(self, element, key) 466 467 def _process(self, element, key=None): --> 468 agg = aggregate._process(self, element, key) 469 shaded = shade._process(self, agg, key) 470 return shaded

C:\ProgramData\Anaconda3\lib\site-packages\holoviews\operation\datashader.py in _process(self, element, key) 320 datatype=['xarray'], vdims=vdims) 321 --> 322 agg = getattr(cvs, glyph)(data, x, y, self.p.aggregator) 323 if agg.ndim == 2: 324 # Replacing x and y coordinates to avoid numerical precision issues

C:\ProgramData\Anaconda3\lib\site-packages\datashader\core.py in points(self, source, x, y, agg) 174 if agg is None: 175 agg = count() --> 176 return bypixel(source, self, Point(x, y), agg) 177 178 def line(self, source, x, y, agg=None):

C:\ProgramData\Anaconda3\lib\site-packages\datashader\core.py in bypixel(source, canvas, glyph, agg) 374 dshape = dshape_from_dask(source) 375 else: --> 376 raise ValueError("source must be a pandas or dask DataFrame") 377 schema = dshape.measure 378 glyph.validate(schema)

ValueError: source must be a pandas or dask DataFrame

Out[30]: :DynamicMap []

[philippjfr]

@dwervin It looks like you got the datashader dev release, either downgrade to datashader 0.5.0 or upgrade to holoviews 1.9dev1 with:

conda install -c ioam/label/dev holoviews

[dwervin]

I'll give that a try, but I ran into a problem with Conda. Thinking that I might need to revert back to a previous revision after installing the dev version of datashader, I tried to rollback to a previous version. Unfortunately, conda deleted itself and now I need to figure out if there is an easy way to get back to the actual revision I specified and restore Conda. It seems that this has happened to a lot of folks recently and the only resolution folks are talking about is a reinstall of Anaconda. Does that sound correct?

[dwervin]

I got my environment back up and running and I was able to run through the holoviews_datashader example notebook, but I don't follow how to make this work with my map data. I was using datashader similar to the examples in the nyc_taxi notebook with a map background.

[jbednar]

Ah, yes. The geographic extensions for HoloViews (projections, map tile support, etc.) are shipped as a separate project GeoViews, because they bring in additional dependencies that are often difficult to install. Once it's installed, it's easy to add a map tile layer, as illustrated in this notebook. If your datapoints aren't already in Web Mercator coordinates, you'll need to reproject them, e.g. using datashader.utils.lnglat_to_meters(). The GeoViews website needs a lot of updating, and for now a better intro is probably this blog post.

[dwervin]

Looking at datashade under holoviews which might work better for what I want to do, but now I'm not certain how to accomplish some of the conditional colorization like you show in the nyc_taxi notebook with bokeh and datashader. Is there a way to do the colorization by hour, percentile, or comparison (picks>drops) with holoviews/datashade?

[jbednar]

Sure. The main reason that datashader is overdue for a proper release is that I haven't had time to set up all the examples of how to do that sort of thing. Too many projects, too little time!

Briefly, the HoloViews interface to datashader publishes three main functions: hv.datashade(), hv.aggregate(), and hv.shade(). hv.datashade() is just a convenient one-step shorthand for hv.shade(aggregate()), and for this more specialized work you'll often need to use the separate functions. hv.shade() is roughly equivalent to tf.shade, while hv.aggregate() combines ds.Canvas() with one of the various aggregating functions (ds.Canvas.points(), ds.Canvas.lines(), etc. , returning an xarray on which you can do arithmetic just as those Canvas methods do.

Colorizing by hour is probably very simple, using either hv.shade() or hv.datashade(); it's just a matter of making sure the information on each category is there in the aggregation (as already illustrated in the holoviews_datashader notebook for various colored gaussians), and then choosing the right color key for the output (I believe the argument is called "cmap"?). Colorizing by percentile should also only need changing the colormap, which again can be passed as an argument to either hv.shade() or hv.datashade.

The picks>drops comparison will probably take more work, and is why I haven't yet reworked that notebook. It should be simple enough to do precisely what's done already; if you take .data from the output of hv.aggregate() for the pickups and dropoffs separately you should have an xarray, which you can do arithmetic on the same as without holoviews, and then you can presumably use hv.shade(hv.Image(agg)) to render it as an image. Unfortunately, doing that will lose the ability to get dynamic image updating as you zoom in, which is one of the main reasons to use the holoviews approach. So I'll need to figure out an easy way to recommend for people to supply arbitrary code to be applied in a dynamic operator like that. And that's what I haven't had time to investigate recently, due to other projects.

Hopefully that's helpful and not too confusing!

[dwervin]

Yes, that was helpful.

I appreciate your time. Thanks.