Largest Canvas size allowed?

[apiszcz]

I am interested in advice concerning creating and operating on canvas sizes. (beyond 100,000 x 100,000 ) It appears there are some internal limits though I'm unsure if it is dask or datashader? Using 4096x4096 works, going to 8192 fails. Thank you.

(Pdb) cvs=ds.Canvas(plot_width=8192,plot_height=8192)
(Pdb) agg=cvs.points(df, 'lon','lat', ds.count('epoch'))
*** MemoryError

I am attaching the stack trace. note.txt

[jbednar]

You must have a much better monitor than I do!

In any case, the stack trace indicates a memory error, so maybe your other running processes or data structures have depleted your memory? My machine doesn't have enough memory to test 100Kx100K, but I just tested the nyc_taxi dataset on my MacBook Pro laptop with 16GB RAM using a 20Kx20K aggregate, and that worked fine (completing in 25s).

I'd be interested to hear what your use case for such large aggregate arrays would be. Usually we assume that the aggregate array will be much smaller than the original columnar dataset, so we haven't much optimized the memory usage of the aggregate.

[apiszcz]

It is memory, I am moving to a larger system and do not have any issues. If you recall I requested creation of large geotifs in the past. I spent the time and implemented it. Datashader density is 'really' working taking quite a while on the aggregate ds.count. However generally things are functioning.

I am seeing this warning message on occasion (from DASK) I think

C:\g\prj\lib\dask\dataframe\core.py:3776: UserWarning: Insufficient elements for `head`. 5 elements requested, only 0 elements available. Try passing larger `npartitions` to `head`.
  warnings.warn(msg.format(n, len(r)))

[jbednar]

I've seen that too, presumably in cases where it's generating a nearly empty image (e.g. when zooming in very far). I haven't tracked it down yet; we should presumably be catching that warning wherever it's generated, because it's not something worth warning about for an interactive application. Let me know if you figure out where we are calling 'head'. Presumably it's somehow indirectly being called, as I don't see head() anywhere in the datashader source code.

[apiszcz]

Will review this when time permits. Thanks.

On Tue, Jul 18, 2017 at 5:58 PM, James A. Bednar [email protected] wrote:

I've seen that too, presumably in cases where it's generating a nearly empty image (e.g. when zooming in very far). I haven't tracked it down yet; we should presumably be catching that warning wherever it's generated, because it's not something worth warning about for an interactive application. Let me know if you figure out where we are calling 'head'. Presumably it's somehow indirectly being called, as I don't see head() anywhere in the datashader source code.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bokeh/datashader/issues/409#issuecomment-316210471, or mute the thread https://github.com/notifications/unsubscribe-auth/ABXVTXx_QvMO6IZBdbsMgLRJ6B1ePEdIks5sPSqGgaJpZM4OYT-S .