MemoryError encountered when analyzing the long chromosome

[yuanhelianyi]

Hi Alex,

Great tool. I'm trying to run moddotplot static on the one chromosome, which length is 496Mb. When I run the following:

nohup moddotplot static \
           -w 2000 \
           -f chr.fasta \
           --dpi 600 \
>moddotplot.out 2>&1 &

I get this error:

Saved self-identity matrix as a paired-end bed file to chr.bedpe

Traceback (most recent call last):
  File "/data6/bin/miniconda3/envs/moddotplot/bin/moddotplot", line 3, in <module>
    from moddotplot.__main__ import main
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/moddotplot/__main__.py", line 11, in <module>
    sys.exit(main())
             ~~~~^^
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/moddotplot/moddotplot.py", line 1010, in main
    create_plots(
    ~~~~~~~~~~~~^
        sdf=[bed],
        ^^^^^^^^^^
    ...<18 lines>...
        annotation=args.bed,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/moddotplot/static_plots.py", line 1706, in create_plots
    ggsave(
    ~~~~~~^
        full_plot,
        ^^^^^^^^^^
    ...<5 lines>...
        verbose=False,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/plotnine/ggplot.py", line 644, in save
    sv = self.save_helper(
        filename=filename,
    ...<8 lines>...
        **kwargs,
    )
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/plotnine/ggplot.py", line 593, in save_helper
    figure = self.draw(show=False)
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/plotnine/ggplot.py", line 233, in draw
    self._draw_layers()
    ~~~~~~~~~~~~~~~~~^^
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/plotnine/ggplot.py", line 399, in _draw_layers
    self.layers.draw(self.layout, self.coordinates)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/plotnine/layer.py", line 463, in draw
    l.draw(layout, coord)
    ~~~~~~^^^^^^^^^^^^^^^
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/plotnine/layer.py", line 367, in draw
    self.geom.draw_layer(self.data, layout, coord, **params)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/plotnine/geoms/geom.py", line 295, in draw_layer
    self.draw_panel(pdata, panel_params, coord, ax, **params)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/plotnine/geoms/geom_raster.py", line 172, in draw_panel
    X = np.ones((nrow, ncol, 4))
  File "/data6/bin/miniconda3/envs/moddotplot/lib/python3.13/site-packages/numpy/_core/numeric.py", line 233, in ones
    a = empty(shape, dtype, order, device=device)
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 1.79 TiB for an array with shape (248009, 248009, 4) and data type float64

Maybe you've encountered this before and have a solution?

Best, Jing

[alexsweeten]

Hi Jing @yuanhelianyi ,

numpy is unable to output a matrix this large (1.8 terabytes!!) in memory. The window parameter you set is simply too small. We set the default w = genome_size/1000, which would be ~500000, for a chromosome this large.

I'll add some warning text about this in a future update.