On applying datashade to a scatter plot, scatter points don't show up in the graph

[hegde-anish]

ALL software version info

python 3.7.7 bokeh 2.1.1 datashader 0.11.1 holoviews 1.13.4 jupyterlab 2.2.9

Description

I'm trying to create a dashboard with scatter plots. The dataframe I'm using has around 5 million rows hence I'm applying dynspread and datashade on hv.Scatter. Some of the columns in this dataframe have a constant value through out except for some NaNs. Whenever that is the case, the graph output is empty. Only after a zoom or pan operation the points show up.

Complete, minimal, self-contained example code that reproduces the issue

import pandas as pd
import numpy as np
import datashader as ds
import holoviews as hv
from holoviews.operation.datashader import datashade, dynspread
hv.extension('bokeh')

dates = pd.date_range(start="2010-01-01", end="2020-01-01", freq='1min')
data = pd.DataFrame()
data['timestamp'] = dates
data['variable1'] = 5
data['variable2'] = 8
data.iloc[::3,1] = np.nan
data.iloc[::7,2] = np.nan

point_df1 = data[['timestamp', 'variable1']].dropna().head(1)
point_df2 = data[['timestamp', 'variable2']].dropna().head(1)

curve1 = hv.Scatter((data['timestamp'], data['variable1']),label='variable1')
curve2 = hv.Scatter((data['timestamp'], data['variable2']),label='variable2')

point1 = hv.Scatter((point_df1['timestamp'], point_df1['variable1']),label='variable1').opts(color='blue')
point2 = hv.Scatter((point_df2['timestamp'], point_df2['variable2']),label='variable2').opts(color='red')

curve_datashade1 = dynspread(datashade(curve1, cmap=['blue'],aggregator=ds.count()), threshold=0.8, max_px=80, shape='circle').opts(width=800, show_grid=True)
curve_datashade2 = dynspread(datashade(curve2, cmap=['red'],aggregator=ds.count()), threshold=0.8, max_px=80, shape='circle').opts(width=800, show_grid=True)

graph1 = point1*curve_datashade1
graph2 = point2*curve_datashade2
graph1*graph2

Screenshots or screencasts of the bug in action

This is the output I see

This is the expected output

[brl0]

For fun, I tried the example provided above and I get the same results with recently updated versions: holoviews 1.14.3 bokeh 2.3.1 datashader 0.12.1 python 3.8.8 jupyterlab 3.0.14

If I make reference to the datashader objects in subsequent cells, the original plot is immediately updated with that object. I also tried passing precompute=True without effect.

curve_datashade1 = dynspread(datashade(curve1, cmap=['blue'],aggregator=ds.count(), precompute=True), threshold=0.8, max_px=80, shape='circle').opts(width=800, show_grid=True)
curve_datashade2 = dynspread(datashade(curve2, cmap=['red'],aggregator=ds.count(), precompute=True), threshold=0.8, max_px=80, shape='circle').opts(width=800, show_grid=True)

[hoxbro]

I could get it to work by setting ylim in opts:

curve_datashade1 = dynspread(datashade(curve1, cmap=['blue'], aggregator=ds.count())).opts(width=800, show_grid=True, ylim=(2, 10))
curve_datashade2 = dynspread(datashade(curve2, cmap=['red'], aggregator=ds.count())).opts(width=800, show_grid=True, ylim=(2, 10))

This could be related to a max, min on initialization, and then something like nanmax, nanmin used when updating, but this is just a guess.

[hegde-anish]

Even using ylim, I still see the same output (With just 2 dots in the beginning) If I individually plot curve_datashade1 curve_datashade2 in different cells then I see all the points in the graph. But when I multiply the 2 so that I can overlay one on the other, the graph output has no points

[hoxbro]

Try updating holoviews and datashader to the latest version.

[hegde-anish]

Yes, that resolved the issue. bokeh=2.3.1 holoviews=1.14.3 datashader=0.12.1

Thanks

[hoxbro]

After looking at it a bit more, I think there is a bug as I would expect the following to work, but it returns an empty plot:

import numpy as np
import holoviews as hv
from holoviews.operation.datashader import datashade, dynspread
hv.extension('bokeh')

x = np.arange(10000)
y1 = np.full(x.size, 5.0)
y2 = np.full(x.size, 8.0)

curve1 = hv.Scatter((x, y1), label='variable1')
curve2 = hv.Scatter((x, y2), label='variable2')

curve_datashade1 = dynspread(datashade(curve1))
curve_datashade2 = dynspread(datashade(curve2))

curve_datashade1.opts(width=800)
curve_datashade2.opts(width=800)

(curve_datashade1 * curve_datashade2).opts(ylim=(4, 9))

I can get it to work by either removing the label from both curves like:

curve1 = hv.Scatter((x, y1))  #, label='variable1')
curve2 = hv.Scatter((x, y2))  #, label='variable2')

or comment out one of the opts of the curve_datashade like:

curve_datashade1.opts(width=800)
# curve_datashade2.opts(width=800)

Which return this output:

Both of these does not return a thick line as you get by zoom or panning around:

[hoxbro]

I just checked if this is still happening and it is. So I still think this should be reopened.

Another thing I noticed is that if you click on the legend and scroll the figure freaks out:

Software version

holoviews.__version__ = '1.14.6'
datashader.__version__ = '0.13.0'
bokeh.__version__ = '2.4.2'

Console Log

Uncaught (in promise) TypeError: this.global_alpha is undefined
    _render http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:607
    render http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:492
    _render http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:488
    render http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:346
    _paint_levels http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:659
    _actual_paint http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:659
    paint http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:659
    repaint http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:659
    throttled_paint http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:659
    l http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:660
    throttle http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:660
    throttle http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:660
    schedule_paint http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:659
    request_paint http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:659
    request_paint http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:346
    request_render http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:346
    e http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:488
    s http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:347
    emit http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:320
    emit http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:320
    _setv http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:319
    setv http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:319
    set http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:319
    on_hit http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:527
    __trigger http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:569
    _trigger http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:569
    _tap http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:569
    _configure_hammerjs http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:569
    emit http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    emit http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    tryEmit http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    _timer http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    T http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    setTimeout handler*l http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    process http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    recognize http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    recognize http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    k http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    handler http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    T http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    handler http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    domHandler http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    I http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    f http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    I http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    init http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    q http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    J http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    pt http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    Wt http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    Ft http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:544
    g http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:569
    initialize http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:568
    c http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:418
    build_view http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:418
    lazy_initialize http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:659
    c http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:418
    build_views http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:418
    build_child_views http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:636
    lazy_initialize http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:636
    c http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:418
    build_view http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:418
    f http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:724
    add_document_standalone http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:724
    embed_items_notebook http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:731
    embed_document http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:5
    <anonymous> http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:8
    <anonymous> http://localhost:8888/lab/tree/Untitled.ipynb line 2 > injectedScript:24
    attachWidget http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    insertWidget http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    _insertOutput http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    onModelChanged http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    m http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    l http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    emit http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    _onListChanged http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    m http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    l http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    emit http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    push http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    _add http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    add http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    _onIOPub http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    _handleIOPub http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2
    handleMsg http://localhost:8888/static/lab/jlab_core.075245c92eb560303156.js?v=075245c92eb560303156:2

[jbednar]

Thanks for the report! @mattpap, the "freaks out" version is something that used to happen in Bokeh years ago, but I haven't seen in recent versions for quite some time. I don't remember what the cause was, but this is definitely the same symptom!

[JoachimSchaeffer]

It looks like there's a bug in datashade when all values are the same. data['variable1'] = 5 data['variable2'] = 8

adding some Gaussian noise seems to yield better results....