terragrunt
terragrunt copied to clipboard
gruntwork-io
Extremely poor performance in v0.55.5+
Describe the bug
It seems that the performance since v0.55.6 is very bad. A simple terragrunt apply without caches (that is with .terragrunt-cache and .terraform.lock.hcl recursively removed) is taking ages.
What took about a minute on v0.55.5 didn't even finish after 10 minutes on v0.55.6.
I then downgraded back to v0.55.5 and it worked again. It seems like it is getting caught in some kind of cyclic redundancy as it keeps looping over the same modules over and over again ("Evaluated X locals (remaining XX)")
Versions
- Terragrunt version: v0.55.6
- Terraform version: 1.7.4
- Environment details Ubuntu 23.10
Additional context
- Cause could be the changes done in #2957 to fix #2955
- Other users are reporting performance issues as well in #2806
Renovate planned an upgrade of Terragrunt today after I downgraded it yesterday.
One of the plan jobs that is impacted by this change took 25 minutes (!) to execute where it would normally take 1~2 minutes at most.
v0.55.5 has its own problems since it causes a "no variable named dependency" in some jobs so I guess we'll have to downgrade that to v0.55.3.
I have seen similar behaviors with "noticeable" slowdowns across various different releases. For the moment I am back using version 0.50.14 because after that version I begin to notice slowdowns.
My experience is that the slow down is far worse in the examples directory of our infrastructure-modules repo, where there is a long pause before it does any output. I’m on 55.1 now due to an earlier problem with dependencies that started in 55.4. I’ll try going back to 50.14.
Hi, will be helpful to get feedback on changes from https://github.com/gruntwork-io/terragrunt/pull/2983 to see if performance is improved for real users
Hi @denis256, I noticed the same performance issue as @syphernl and was also able to pin it down to the release of v0.55.6.
I did some "benchmarking" across different versions of Terragrunt by running terragrunt run-all init --terragrunt-parallelism 1 (no plan or apply) in a project with 35 modules and the performance issues do not seem to be solved until the latest version v0.56.5.
I created a plot of the timespans between every module that shows a massive increase of runtime for the final modules (looks like it's increasing exponentially).
The plot shows that for versions below 0.55.6 (I tested 0.55.0, 0.55.5) the "runtime" per module is mostly below 30 seconds. Starting from 0.55.6 and above the final module takes over 30min to initialize. Note that the y-axis is set to log-scale and the time next to the version (in the legend) is the real output of the time command.
Regarding the cause of the exponential slowdown, @syphernl already made a good guess
It seems like it is getting caught in some kind of cyclic redundancy as it keeps looping over the same modules over and over again ("Evaluated X locals (remaining XX)")
Either way, according to my analysis the bug should be somewhere between https://github.com/gruntwork-io/terragrunt/compare/v0.55.5...v0.55.6.
Let me know if there is anything else I can help with :slightly_smiling_face:. Thanks in advance for digging into this and fixing the bug!
P.S. To collect the time deltas, I used the timestamp of the log message Downloading Terraform configurations ... which seems to be issued once per module, e.g.
time=2024-04-12T08:59:32+02:00 level=info msg=Downloading Terraform configurations from ... into ... prefix=[<absolute-path-to-module]
So I can confirm that I still experience huge downgrade in performance of terragrunt after v0.55.6
We have a pretty big project that deploys mainly AWS resources. run-all plan takes usually around ~20 minutes
but after upgrading terraform to anything higher or equal than v0.55.6 it can take even over 1h!
here is screenshot from our gitlab ci - passed jobs were using v0.55.5, job that failed was using v0.57.2 and it failed only because gitlab job timed-out at 1h... the same code 20min vs 60min+
I also noticed when working locally that terragrunt run-all * commands are starting to hang after processing x modules more and more, so that terragrunt is not producing any logs for some time and after n seconds/minutes it starts working on next module again and the time of delay is increasing...
Not sure how valuable this info is, but definitely I would reopen this issue
Don't have much to add unfortunately, but we are seeing the same issue after upgrading from v0.52 to any version v0.53 and above. Seems to be init causing the issue. run-all init has gone from taking a few minutes to taking over 30 minutes. Terraform version is 1.5.7 and we are setting both TERRAGRUNT_DOWNLOAD and TF_PLUGIN_CACHE_DIR. The majority of our modules are stored in the same repository and sourced through a relative path.
One of my components is extremely slow in v0.58.1.
It appears to be looping constantly (I kept seeing it evaluating same locals blocks). With debug mode it showed that it went up to DEBU[0142]
Even running terragrunt force-unlock xxxxxxxxx while in this directory takes 2~3 minutes to execute from start to the confirmation prompt.
In between it appears to be hanging with no debug output (terraform or terragrunt) for a long time, this happens a few times during execution.
v0.55.3 shows the confirmation prompt of terragrunt force-unlock within 2 seconds.
With v0.55.3 a terragrunt plan (even with debug mode of both terraform + terragrunt enabled) on this module completes really fast:
terragrunt plan 8,47s user 0,89s system 107% cpu 8,704 total
Echoing that I am seeing the same poor performance in 0.58.1. In earlier versions (e.g.: 0.53.6) I was completing a terragrunt run-all plan in 3 minutes and 32 seconds. Now, with the same underlying code and just a switch to 0.58.1, I see a terragrunt run-all plan completion in 19 minutes and 40 seconds. I also see the terragrunt PID landing at 722% peak CPU.
It seems to be proportional to the amount of dependencies - put another way, my earlier modules with 0 or 1 dependencies are still reasonably quick, but as you reach the end of a stack where I have 5-6 dependencies, the slowness reflects into a much larger slowdown.
I'm seeing poor performance on 0.58.2
I was able to mitigate this slowness completely (on par with 0.55.5 speeds) by removing the block at https://github.com/gruntwork-io/terragrunt/blob/master/config/config_partial.go#L296-L303. Keep in mind that dependencies of parent modules will display the "unknown variable" error, relevant in #2955 (though terragrunt will still exit with 0)
In order to maintain quick plans with modules that have many ancestor dependencies, maybe recursing into dependencies can be moved to a command line option?
It seems to be re-evaluating the same sets of locals over and over for each dependency.
I ran this commands on one of our more problematic projects with terragrunt 0.55.4 and 0.58.2
terragrunt plan --terragrunt-log-level debug 2>&1 | grep -o 'Found locals.*prefix=.*' | sort | uniq -c | sort -n
All of our projects include a single terragrunt.hcl file at the root of the repo which uses read_terragrunt_config(find_in_parent_folders()) to read additional variables from various places around the repo.
Forgive the redactions, but it's pretty clear that 0.58.2 is processing the same sets of locals from the same files over and over much more than 0.55.4 did.
Version 0.55.4
1 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
1 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
1 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
1 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
1 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
1 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
3 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
4 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
5 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
6 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
6 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
7 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
7 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
7 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
7 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
7 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
7 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
7 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
7 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
8 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
9 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
12 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
12 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
12 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
12 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
14 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
20 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
20 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
20 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
27 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
27 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
38 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
44 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
47 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
69 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
167 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
173 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
192 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
199 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
321 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Root terragrunt.hcl
Version 0.58.2
12 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
14 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
14 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
14 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
14 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
16 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
20 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
20 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
20 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
24 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
24 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
24 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
28 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
28 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
32 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
32 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
32 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
36 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
36 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
40 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
40 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
46 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
48 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
58 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
60 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
62 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
65 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
80 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
81 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
81 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
86 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
88 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
105 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
112 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
135 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
136 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
144 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
160 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
164 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
180 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
200 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
208 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
218 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
218 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
270 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
289 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
289 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
304 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
329 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
417 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
432 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
433 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
477 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
609 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
690 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
956 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
1360 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
1388 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
1757 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
1886 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Direct or indirect dependency]
1913 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
2777 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
3170 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
3412 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
3461 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
5279 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
5888 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
8055 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
8773 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: File referenced by root terragrunt.hcl]
14421 Found locals block: evaluating the expressions. prefix=[REDACTED PATH: Root terragrunt.hcl]
It's also worth noting that running this plan on 0.55.4 took 20 seconds but running it on 0.58.2 took 47 minutes.
We've attempted to upgrade from 0.51.7 to 0.57.5. In 0.51.7, run-all apply ran for 3min 36s, in 0.57.5 it took 12mins 37s.
We have a project with 13 modules, every module has a terragrunt.hcl file which uses read_terragrunt_config(find_in_parent_folders("terragrunt.hcl")) and includes a common.hcl config with some additional configurations.
Confirming @jescholl, Terragrunt reevaluates the same locals for each dependency.
- 0.51.7: 228 locals evaluations
- 0.57.5: 2645 locals evaluations
We would have loved to upgrade to a newer version as there are some features we really need, but not with such a drop in performance.
we are experiencing the same issues, one small fix we found was disabling expose = true when its not needed (most deploys are boilerplate copy pasted for the include root) and that cuts down on the reprocessing of the root locals for each dependency where its not needed. of course its not universal and just a bandaide. Since this seems to of been introduced around the time of exporting the inputs too maybe its related?
Hi @denis256, are there any updates on this issue?
We're experiencing the same problem: our deployment pipeline with version 0.56.0 takes over 1.5 hours, whereas it only took 10 minutes with version 0.55.0.
We've been following this ticket for quite some time, and it seems like there's been no recent activity on the issue. The issue in the roadmap is still marked as Done. Perhaps it could be moved back to draw more attention to it?
Thank you for your help!
Ya. We are leaving behind terragrunt because of this. Our workflow is sooo slow now. A simple local apply takes more than 15 minutes to get started.
Hey folks!
Sorry for the negative experience regarding performance in Terragrunt. Definitely not what we want out of the tool.
We'd like to have more information on the scenarios in which you're experiencing performance degradation, in addition to the upgrade. Information like the general number of files involved in the repository, and what they're doing can help us track down places where optimizations can take place.
Please refer to the analysis done here to give some ideas as to how those performance bottlenecks can be uncovered.
Note that in the issue linked above, we discuss how certain issues like re-evaluating locals repeatedly has to be done by default in order to ensure the correctness of Terragrunt configuration parsing (see this for example), but that we're investigating workarounds to reduce the impact of those restrictions on performance.
Thank you for your cooperation and patience as we work through these issues. Terragrunt's performance improvements are going to depend on collaboration with the community to find solutions that work for everyone.
Thanks for your response, @yhakbar!
As mentioned in my comment above, I figured that v0.55.6 introduced changes that drastically increase the time for initializing modules (especially the ones in the final groups). I also tried it again with v0.63.2 but canceled terragrunt run-all init after 5 hours of runtime as it did not finish.
According to my analysis the bug should be somewhere between https://github.com/gruntwork-io/terragrunt/compare/v0.55.5...v0.55.6.
Before everyone starts to file issues custom to their Terragrunt setup I would very much appreciate it if you could confirm that v0.55.6 did not introduce unexpected behavior or bugs related to this.
Thank you very much! :slightly_smiling_face:
@RafaelWO ,
Sharing when you started experiencing performance degradation can help us find out when something was introduced that impacted performance, but not necessarily what is impacting performance for your use-case.
As far as we can tell right now, that release addressed some necessary changes that did introduce more work by Terragrunt, which can reduce performance, but nothing that is obviously a bug in logic. If you could demonstrate that Terragrunt is doing extra work that it deterministically doesn't have to do in a particular scenario, that can guide implementations for fixes that increases caching, or fixes cache keys, etc.
For that reason, we're soliciting analyses of scenarios where performance is an issue, hopefully with reproducible fixtures that we can look at to see where Terragrunt can be optimized to improve performance, if it can without reducing correctness.
For everyone's awareness, I will be proposing an RFC in the near future that gives users more fine grained control over how work is memoized in Terragrunt. I think there's room for Terragrunt to give users more control over how configuration parsing is memoized so that configurations can be cached even if Terragrunt cannot determine that it's safe to do so deterministically.
I tried passing cli options: --terragrunt-fetch-dependency-output-from-state --terragrunt-use-partial-parse-config-cache and found a significant difference in the performance. Docs
Times with the options that @dhanbdrkarki1 noted. Testing in a directory with 8 modules, each of which includes a separate file that may have dependencies. Each module has at least 1 dependency and 1 include.
(control) time terragrunt run-all plan --terragrunt-ignore-external-dependencies
169.06s user 36.19s system 201% cpu 1:42.02 total
time terragrunt run-all plan --terragrunt-ignore-external-dependencies --terragrunt-fetch-dependency-output-from-state
59.83s user 8.75s system 236% cpu 28.962 total
time terragrunt run-all plan --terragrunt-ignore-external-dependencies --terragrunt-use-partial-parse-config-cache
Found a dependency cycle between modules
terragrunt --version
terragrunt version v0.64.2
Hello, in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.66.3 I added some performance improvements, will be helpful to get some feedback on the results
I would also suggest to try --terragrunt-use-partial-parse-config-cache
A small update from my side: I ran my benchmark again with Terragrunt 0.66.3 (no additional options) and got back to the runtime of ~10min, i.e. the same as before 0.55.6 :heart_eyes:
Using --terragrunt-fetch-dependency-output-from-state gives another improvement of 1 min and enabling the provider cache (obviously) speeds up the init duration as well resulting in a final runtime of ~7min.
Thanks for the improvements and fixes! I'm psyched to finally upgrade our CI/CD pipeline to the newest Terragrunt version :slightly_smiling_face: