Downsample with RangeTool source sometimes ignores set height

[droumis]

ALL software version info

(this library, plus any other relevant software, e.g. bokeh, python, notebook, OS, browser, etc)

Description of expected behavior and the observed behavior

Sometimes when just the source plot of a RangeToolLink is downsample=True, the set height for the source plot is ignored.

In the video below, after setting the target plot downsample=False, sometimes when the cell is executed, the height for the source plot is changes.

Complete, minimal, self-contained example code that reproduces the issue

import pandas as pd
import numpy as np
import hvplot.pandas
from holoviews.plotting.links import RangeToolLink

n_samples = 1000
data = np.random.randn(n_samples)

df = pd.DataFrame({'time': np.arange(len(data)), 'value': data})
plot = df.hvplot(x="time", y="value", height=400, downsample=False)
minimap = df.hvplot(x="time", y="value", height=150, downsample=True)
link = RangeToolLink(minimap, plot, axes=["x", "y"])

(plot + minimap).opts(shared_axes=False).cols(1)

Stack traceback and/or browser JavaScript console output

Screenshots or screencasts of the bug in action

https://github.com/holoviz/hvplot/assets/6613202/bf71119b-bd5a-4b91-a406-c9e92b9d633b

• [ ] I may be interested in making a pull request to address this