yt icon indicating copy to clipboard operation
yt copied to clipboard

Slices and projections do not use cached data

Open cphyc opened this issue 5 months ago • 0 comments

In the following example, the gas density will be read twice from disk: once to get the data and a second one to make the slice, disregarding the fact it's already been loaded!

import yt
ds = yt.load_sample("output_00080")
ds.index

yt.set_log_level(10)  # to see the IO footprint
ad = ds.all_data()
yt.mylog.debug("---------> Now calling `get_data`")
ad.get_data([("gas", "density")])

yt.mylog.debug("---------> Now making slice")

p = yt.SlicePlot(ds, "x", ("gas", "density"), data_source=ad)

Please take a look at the logs below. The data are located in 16 files, and you can see that there are 16 + 5 file reads (the lines with Filing Density with ...), where we could have done 16 reads only.

yt : [DEBUG    ] 2025-08-04 10:48:49,293 ---------> Now calling `get_data`
yt : [DEBUG    ] 2025-08-04 10:48:49,298 Identified domain 1
yt : [DEBUG    ] 2025-08-04 10:48:49,298 Identified domain 2
yt : [DEBUG    ] 2025-08-04 10:48:49,298 Identified domain 3
yt : [DEBUG    ] 2025-08-04 10:48:49,298 Identified domain 4
yt : [DEBUG    ] 2025-08-04 10:48:49,298 Identified domain 5
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 6
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 7
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 8
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 9
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 10
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 11
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 12
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 13
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 14
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 15
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Identified domain 16
yt : [INFO     ] 2025-08-04 10:48:49,299 Identified    16/   16 intersecting domains (   16 through hilbert key indexing)
yt : [DEBUG    ] 2025-08-04 10:48:49,299 Appending object to info_00080 (type: <class 'yt.frontends.ramses.data_structures.RAMSESDomainSubset'>)
yt : [DEBUG    ] 2025-08-04 10:48:49,301 Reading domain AMR    1 (2.918e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,328 Filling Density with 191131 (3.669e-03 7.354e+01) (191131 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,329 Reading domain AMR    2 (2.988e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,350 Filling Density with 171092 (3.239e-03 2.886e+04) (171092 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,350 Reading domain AMR    3 (2.016e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,364 Filling Density with 107241 (4.180e-03 3.994e+05) (107241 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,364 Reading domain AMR    4 (3.336e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,386 Filling Density with 169647 (4.435e-03 2.359e+03) (169647 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,387 Reading domain AMR    5 (3.306e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,410 Filling Density with 170041 (2.411e-03 4.159e+02) (170041 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,410 Reading domain AMR    6 (1.098e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,418 Filling Density with 59762 (1.791e-02 4.311e+05) (59762 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,418 Reading domain AMR    7 (4.443e+03, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,421 Filling Density with 21172 (1.136e-01 1.411e+06) (21172 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,422 Reading domain AMR    8 (5.090e+03, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,425 Filling Density with 19811 (2.669e-02 1.301e+05) (19811 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,426 Reading domain AMR    9 (5.292e+03, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,430 Filling Density with 21265 (1.727e-02 2.087e+06) (21265 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,430 Reading domain AMR   10 (8.980e+03, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,436 Filling Density with 40259 (1.125e-02 6.737e+05) (40259 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,437 Reading domain AMR   11 (1.016e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,444 Filling Density with 47476 (5.135e-03 1.107e+06) (47476 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,444 Reading domain AMR   12 (1.693e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,456 Filling Density with 90649 (4.600e-03 3.387e+05) (90649 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,457 Reading domain AMR   13 (3.025e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,478 Filling Density with 172297 (3.510e-03 7.974e+04) (172297 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,479 Reading domain AMR   14 (2.509e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,496 Filling Density with 128416 (3.794e-03 7.317e+05) (128416 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,497 Reading domain AMR   15 (2.508e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,514 Filling Density with 134167 (5.324e-03 4.083e+04) (134167 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,514 Reading domain AMR   16 (3.532e+04, 0.000e+00)
yt : [DEBUG    ] 2025-08-04 10:48:49,541 Filling Density with 205029 (4.979e-03 5.993e+01) (205029 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,548 ---------> Now making slice
yt : [DEBUG    ] 2025-08-04 10:48:49,549 Appending object to info_00080 (type: <class 'yt.data_objects.selection_objects.slices.YTSlice'>)
yt : [INFO     ] 2025-08-04 10:48:49,619 Identified 5 intersecting domains
yt : [DEBUG    ] 2025-08-04 10:48:49,619 Appending object to info_00080 (type: <class 'yt.frontends.ramses.data_structures.RAMSESDomainSubset'>)
yt : [DEBUG    ] 2025-08-04 10:48:49,631 Filling Density with 1024 (8.770e-03 3.248e+00) (1024 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,643 Filling Density with 256 (8.233e-03 4.111e-01) (256 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,654 Filling Density with 256 (1.203e-02 2.777e+00) (256 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,664 Filling Density with 256 (1.095e-02 5.452e-01) (256 zones)
yt : [DEBUG    ] 2025-08-04 10:48:49,677 Filling Density with 2304 (7.664e-03 2.233e+01) (2304 zones)
yt : [INFO     ] 2025-08-04 10:48:49,678 xlim = 0.000000 1.000000
yt : [INFO     ] 2025-08-04 10:48:49,678 ylim = 0.000000 1.000000
yt : [INFO     ] 2025-08-04 10:48:49,680 xlim = 0.000000 1.000000
yt : [INFO     ] 2025-08-04 10:48:49,680 ylim = 0.000000 1.000000
yt : [INFO     ] 2025-08-04 10:48:49,680 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800

cphyc avatar Aug 04 '25 08:08 cphyc