modflow6 icon indicating copy to clipboard operation
modflow6 copied to clipboard

feat(memorymanager) Replace the Memorylist used in the MemoryManager by a MemoryHasTable

Open Manangka opened this issue 10 months ago • 2 comments

In this commit the container used in the MemoryList has been replaced by a HashTable Tests have been performed on the new data container and show that a HashTable out performs when it comes to initializing an simulation as well as performing value lookup. The results can be found below.

The test simulation contains 105743 variables. A worst case scenario is tested where we try to obtain the list variable in the list. Each simulation has been tested with a debug and release version of ModFlow. The values displayed below are the average values after running each test case 5 times

List (Debug) HashTable (Debug) List (Release) HashTable (Release)
Elapsed init time, avg (s) 492,1802948 155,4768408 328,6092749 94,18164263
Elapsed get time, avg (s) 0,044770432 0,00027034 0,025319958 0,00016108

This PR is still in draft waiting for https://github.com/MODFLOW-USGS/modflow6/pull/1765 to be merged

Manangka avatar Apr 03 '24 14:04 Manangka

Hello @Manangka, @mjr-deltares - some weeks ago I mentioned that I had a version of a FloPy script that attempted to generate a version of the "Barends" model that was a series of interconnected 1D models (setup in a manner similar to what's shown in the image).
multi-model

Overall, there were 2,002 models: 1 GWF and 1 GWE model representative of the saturated zone, and 1,000 GWF and 1,000 GWE 1-dimensional, vertically oriented, models representative of the "overburden". I eventually abandoned this approach in favor of a DISU approach; however, I'm attaching the script here in case it may be useful for your purposes. The large number of models within the simulation caused both FloPy and MF6 to run excessively slow. I never got to the point of running the model successfully (i.e., convergence), but as I recall the script would fill-out the simulation directory with the necessary model input (took over an hour as I recall) and then MF6 would attempt to run it, but just reading in the input took over an hour, as best I can recall.
test_gwe_thermal_bleed_2.txt

emorway-usgs avatar Apr 10 '24 14:04 emorway-usgs

Hello @Manangka, @mjr-deltares - some weeks ago I mentioned that I had a version of a FloPy script that attempted to generate a version of the "Barends" model that was a series of interconnected 1D models (setup in a manner similar to what's shown in the image). multi-model

Overall, there were 2,002 models: 1 GWF and 1 GWE model representative of the saturated zone, and 1,000 GWF and 1,000 GWE 1-dimensional, vertically oriented, models representative of the "overburden". I eventually abandoned this approach in favor of a DISU approach; however, I'm attaching the script here in case it may be useful for your purposes. The large number of models within the simulation caused both FloPy and MF6 to run excessively slow. I never got to the point of running the model successfully (i.e., convergence), but as I recall the script would fill-out the simulation directory with the necessary model input (took over an hour as I recall) and then MF6 would attempt to run it, but just reading in the input took over an hour, as best I can recall. test_gwe_thermal_bleed_2.txt

@emorway-usgs Thank you for the example. I used it to compare this branch against the develop branch and the performance difference is quite impressive. I first used your script to produce the the simulation results then I measured how long it took MODFLOW to run it before encountering an error:

branch container time
develop list 1h33m41s
MemoryManagerHashTable hashtable 5m30s

Manangka avatar May 08 '24 09:05 Manangka

@mjr-deltares @langevin As discussed on the call I renamed the MemoryList to MemoryStore. I will create a separate PR in which I will cleanup/rename all references to memorylist. The reason for a separate PR is because there are over 180 references to a memorylist and I don't want to pollute this PR.

Further we also talked about throwing a warning when an item is added to the MemoryStore with a name and path that has already been added (duplicate item). I thought I didn't add it but I was wrong. I added it to the HastTable class

Manangka avatar Jun 14 '24 09:06 Manangka

This is great, @Manangka. I'd like to get this in. There is an unrelated failure (I think), which I triggered to rerun. When it passes, I'll push the button to bring this in. Thanks! Really nice work.

langevin-usgs avatar Jun 14 '24 11:06 langevin-usgs