modflow6
modflow6 copied to clipboard
feat(memorymanager) Replace the Memorylist used in the MemoryManager by a MemoryHasTable
In this commit the container used in the MemoryList has been replaced by a HashTable Tests have been performed on the new data container and show that a HashTable out performs when it comes to initializing an simulation as well as performing value lookup. The results can be found below.
The test simulation contains 105743 variables. A worst case scenario is tested where we try to obtain the list variable in the list. Each simulation has been tested with a debug and release version of ModFlow. The values displayed below are the average values after running each test case 5 times
List (Debug) | HashTable (Debug) | List (Release) | HashTable (Release) | |
---|---|---|---|---|
Elapsed init time, avg (s) | 492,1802948 | 155,4768408 | 328,6092749 | 94,18164263 |
Elapsed get time, avg (s) | 0,044770432 | 0,00027034 | 0,025319958 | 0,00016108 |
This PR is still in draft waiting for https://github.com/MODFLOW-USGS/modflow6/pull/1765 to be merged
Hello @Manangka, @mjr-deltares - some weeks ago I mentioned that I had a version of a FloPy script that attempted to generate a version of the "Barends" model that was a series of interconnected 1D models (setup in a manner similar to what's shown in the image).
Overall, there were 2,002 models: 1 GWF and 1 GWE model representative of the saturated zone, and 1,000 GWF and 1,000 GWE 1-dimensional, vertically oriented, models representative of the "overburden". I eventually abandoned this approach in favor of a DISU approach; however, I'm attaching the script here in case it may be useful for your purposes. The large number of models within the simulation caused both FloPy and MF6 to run excessively slow. I never got to the point of running the model successfully (i.e., convergence), but as I recall the script would fill-out the simulation directory with the necessary model input (took over an hour as I recall) and then MF6 would attempt to run it, but just reading in the input took over an hour, as best I can recall.
test_gwe_thermal_bleed_2.txt
Hello @Manangka, @mjr-deltares - some weeks ago I mentioned that I had a version of a FloPy script that attempted to generate a version of the "Barends" model that was a series of interconnected 1D models (setup in a manner similar to what's shown in the image).
Overall, there were 2,002 models: 1 GWF and 1 GWE model representative of the saturated zone, and 1,000 GWF and 1,000 GWE 1-dimensional, vertically oriented, models representative of the "overburden". I eventually abandoned this approach in favor of a DISU approach; however, I'm attaching the script here in case it may be useful for your purposes. The large number of models within the simulation caused both FloPy and MF6 to run excessively slow. I never got to the point of running the model successfully (i.e., convergence), but as I recall the script would fill-out the simulation directory with the necessary model input (took over an hour as I recall) and then MF6 would attempt to run it, but just reading in the input took over an hour, as best I can recall. test_gwe_thermal_bleed_2.txt
@emorway-usgs Thank you for the example. I used it to compare this branch against the develop branch and the performance difference is quite impressive. I first used your script to produce the the simulation results then I measured how long it took MODFLOW to run it before encountering an error:
branch | container | time |
---|---|---|
develop | list | 1h33m41s |
MemoryManagerHashTable | hashtable | 5m30s |
@mjr-deltares @langevin As discussed on the call I renamed the MemoryList to MemoryStore. I will create a separate PR in which I will cleanup/rename all references to memorylist. The reason for a separate PR is because there are over 180 references to a memorylist and I don't want to pollute this PR.
Further we also talked about throwing a warning when an item is added to the MemoryStore with a name and path that has already been added (duplicate item). I thought I didn't add it but I was wrong. I added it to the HastTable class
This is great, @Manangka. I'd like to get this in. There is an unrelated failure (I think), which I triggered to rerun. When it passes, I'll push the button to bring this in. Thanks! Really nice work.