Look into memory usage when changing time states.

[aowen87]

Here is the e-mail exchange related to this issue with a Chombo data set. We should really get to the bottom of this. We have this come up occasionally and never investigate. Hi Robert, The scalable rendering request is most likely crashing because VisIt has run out of memory. Most likely, there is an increase in memory usage each time you move to a new time state. There is a data structure that stores neighbor information for ghost zone generation that takes up a particularly large amount of memory that is calculated and replicated across all processors. One outcome of this is that as you increase the number of cores the memory usage for that data structure doesnt decrease. There is an option to use an alternate data structure that takes up less memory. The larger default data structure is only used by the AMR Dual Grid And Stitch Cells operator, which you are probably not using, so using the smaller data structure is probably ok. You can enable this by adding the following to the command line. disableghostsfort-intersections This might not get rid of the issue completely, but should make the crash less frequent. I will submit a ticket to look at memory usage over time and have someone investigate this. Are you using the movie generating wizard or are you using a script? If you are using a script, you could manually close the compute engine every 20 time steps to avoid the crash, this isnt optimal, but better than crashing and automatically restarting. If you are using the movie wizard, a possible option would be to add an option to automatically restart after N time steps. Eric From: Robert Marskar via visitusers <visituserselist.ornl.gov> Sent: Wednesday, September 19, 2018 11:51 AMTo: visituserselist.ornl.govCc: Robert Marskar [email protected]Subject: [visitusers] Scalable Render request failed (VisItException) Hi, I'm trying to generate a sequence of frames from a Chombo database with VisIt 2.13.0. The frames contain one boundary plot and one volume plot. Occasionally, the rendering fails with the message VisIt: Error Scalable Render Request Failed (VisItException)viewer: Obtained null data reader for rendered image for engine login1-2.fram.sigma2.no The files are 50150GB with 5 AMR levels but I get this error message even with a low number of ray samples, no lighting/smoothing, and rendering only on the coarsest AMR level. I've tried anything between 32 and 2048 cores, and this always happens after e.g. 20100 frames generated. Incidentally, I do not have this problem if the time step is fixed, and I the frames with e.g. different camera angles. Also, the job step occasionally resumes, the frame then renders successfully, and the process repeats. This is nevertheless a time (and CPU) consuming problem that interrupts my workflow and mood. Cheers/Robert

-----------------------REDMINE MIGRATION----------------------- This ticket was migrated from Redmine. As such, not all information was able to be captured in the transition. Below is a complete record of the original redmine ticket.

Ticket number: 3230 Status: New Project: VisIt Tracker: Bug Priority: High Subject: Look into memory usage when changing time states. Assigned to: - Category: - Target version: - Author: Eric Brugger Start: 09/20/2018 Due date: % Done: 0% Estimated time: Created: 09/20/2018 11:47 am Updated: 09/20/2018 12:34 pm Likelihood: 3 - Occasional Severity: 4 - Crash / Wrong Results Found in version: 2.12.3 Impact: Expected Use: OS: All Support Group: Any Description: Here is the e-mail exchange related to this issue with a Chombo data set. We should really get to the bottom of this. We have this come up occasionally and never investigate. Hi Robert, The scalable rendering request is most likely crashing because VisIt has run out of memory. Most likely, there is an increase in memory usage each time you move to a new time state. There is a data structure that stores neighbor information for ghost zone generation that takes up a particularly large amount of memory that is calculated and replicated across all processors. One outcome of this is that as you increase the number of cores the memory usage for that data structure doesnt decrease. There is an option to use an alternate data structure that takes up less memory. The larger default data structure is only used by the AMR Dual Grid And Stitch Cells operator, which you are probably not using, so using the smaller data structure is probably ok. You can enable this by adding the following to the command line. disableghostsfort-intersections This might not get rid of the issue completely, but should make the crash less frequent. I will submit a ticket to look at memory usage over time and have someone investigate this. Are you using the movie generating wizard or are you using a script? If you are using a script, you could manually close the compute engine every 20 time steps to avoid the crash, this isnt optimal, but better than crashing and automatically restarting. If you are using the movie wizard, a possible option would be to add an option to automatically restart after N time steps. Eric From: Robert Marskar via visitusers <visituserselist.ornl.gov> Sent: Wednesday, September 19, 2018 11:51 AMTo: visituserselist.ornl.govCc: Robert Marskar [email protected]Subject: [visitusers] Scalable Render request failed (VisItException) Hi, I'm trying to generate a sequence of frames from a Chombo database with VisIt 2.13.0. The frames contain one boundary plot and one volume plot. Occasionally, the rendering fails with the message VisIt: Error Scalable Render Request Failed (VisItException)viewer: Obtained null data reader for rendered image for engine login1-2.fram.sigma2.no The files are 50150GB with 5 AMR levels but I get this error message even with a low number of ray samples, no lighting/smoothing, and rendering only on the coarsest AMR level. I've tried anything between 32 and 2048 cores, and this always happens after e.g. 20100 frames generated. Incidentally, I do not have this problem if the time step is fixed, and I the frames with e.g. different camera angles. Also, the job step occasionally resumes, the frame then renders successfully, and the process repeats. This is nevertheless a time (and CPU) consuming problem that interrupts my workflow and mood. Cheers/Robert

Comments: Here are some more email exchanges:Hi Robert,Explicitly opening and closing the files would avoid this issue, since closing a file will release the cache items associated with the file. When you open a group of files and change the time slider, VisIt is keeping some data structures around as it changes time state. So, from that stand point there are memory overheads associated with opening a group of files. I am surprised that clearing the cache isnt helping. Ill add this information to the ticket.Well investigate with some of our own data sets and if we cant replicate, well get back to you for logs.EricFrom: Robert Marskar <Robert.Marskarsintef.no> Sent: Thursday, September 20, 2018 9:21 AMTo: VisIt software users community <visituserselist.ornl.gov>Cc: Brugger, Eric [email protected]Subject: RE: Scalable Render request failed (VisItException)Hi Eric,I ran this through a script; I didn't try closing the engine but I did try clearing the cache, which did not work.Incidentally, I had no crashes if I opened the files onebyone instead of together at similar core counts. That is, instead of TimeSliderSetState, I did a OpenDatabase/CloseDatabase for each file. This seemed to work fine. Are there memory overheads associated with opening a group of files that I should be particularly aware of?Let me know if you want debug logs or anything.Cheers/Robert