[TwigComponent] out of memory

[choarau-craft]

Hello,

I've started to refactor a qui big application using twig components, but recently I ran into out of memories. When I'm on the breach, I tried several things to find out what could cause those. Replacing a simple twig component (without php file) by it's content fixed the memory issue when that component is called about a thousand of time in a loop. So it's seems like components are adding a lot of memory consumption when used in loops. Is that the expected behavior, is there any way to fix this ?

Thanks a lot

[WebMamba]

How many times do you loop?

[choarau-craft]

It's a double loop. In my case I'm starting to have memory issue when displaying a table with > 100 rows and about 10 Columns. Without components I think I was able to display about 400 rows with the same amount of memory. If I disable the profiler I can double the amount of data displayed i, both cases.

I wanted to use components as macros replacements as it's a lot cleaner and I really like how components are working. In my small components. In each a single cell I can have up to 6-7 components used. Components might not have been designed to be used this way ? I wasn't expected that much over memory consumption :sweat:

[smnandre]

What type of component are you using ? Anonymous ? Twig ? Or Live ?

Could you share a bit of the component code, maybe we can suggest you some ideas to lower the memory.

Is this also the case in production ? The profiler can cause a lot of memory usage in dev

[choarau-craft]

Thanks for reactivity.

I'm using both anonymous and non anonymous components. But no live components

Here is an example when using an anonymous component to display a cell. If I use the code below I have a memory exception:

{% for k, place in episodeCollection.places %}
    <twig:Episodes:ProductionTable:Partials:Td :episodeClass="episodeClass"
    :place="place"
    :workflowPlaces="workflowPlaces"
    :workflowsPlacesLoop="workflowsPlacesLoop"/>
{% endfor %}

But if I include the code directly it's working fine with same amount of memory. I didn't even have to change sub components usage, but I assume this could result in even bigger memory usage improvements.

{% for k, place in episodeCollection.places %}
    {% include 'components/Episodes/ProductionTable/Partials/Td.html.twig' %}
{% endfor %}

The component itself set a bunch of variables then call other components. I have just commented the props part to make include work. Could have set the variables in a php code associated to the twig, but I didn't want to create extra files for simple logics

{#{% props#}
{#	episodeClass,#}
{#	place,#}
{#	workflowPlaces,#}
{#	workflowsPlacesLoop, %}#}

{% apply spaceless %}
	{% set workflowStart = (place == (workflowPlaces | first)) %}
	{% set lastPlace = (workflowPlaces | last) %}
	{% if lastPlace == constant('App\\Enum\\WorkflowPlacesEnum::final_package_validation_ok').value %}
		{% set lastPlace = (workflowPlaces[workflowPlaces|length -2]) %}
	{% endif %}

	{% set workflowEnd = (place == lastPlace) %}

	{% if workflowStart %}
		{{- setGlobalVariable('episode-row-workflowInProgress', true) -}}
	{% endif %}
	{% if workflowEnd %}
		{{- setGlobalVariable('episode-row-workflowInProgress', false) -}}
	{% endif %}

	{% if
		place not in workflowPlaces
		or attribute(episodeClass, place~'_type') is not defined %}
		<twig:Episodes:ProductionTable:Partials:TdEmpty :place="place"
															   :workflowInProgress="getGlobalVariable('episode-row-workflowInProgress')"/>
	{% elseif (not workflowsPlacesLoop.first) and workflowStart %}
		<twig:Episodes:ProductionTable:Partials:TdStartOfWorkflow
				:episodeClass="episodeClass"
				:place="place"
				:workflowPlaces="workflowPlaces"
				:workflowsPlacesLoop="workflowsPlacesLoop"/>
	{% elseif (not workflowsPlacesLoop.first) and workflowEnd %}
		<twig:Episodes:ProductionTable:Partials:TdEndOfWorkflow
				:episodeClass="episodeClass"
				:place="place"
				:workflowPlaces="workflowPlaces"
				:workflowsPlacesLoop="workflowsPlacesLoop"/>
	{% else %}
		<twig:Episodes:ProductionTable:Partials:TdDefault
				:episodeClass="episodeClass"
				:place="place"
				:workflowPlaces="workflowPlaces"
				:workflowsPlacesLoop="workflowsPlacesLoop"
				:workflowStart="workflowStart"
				:workflowEnd="workflowEnd"
		/>
	{% endif %}
{% endapply %}

[choarau-craft]

By replacing almost all sub components (7 out of 8) I was able to halves memory consumption (from 200M to 100M) This is a huge difference only by using includes ...

[smnandre]

Just by curiosity could you share the "overall" content of your deepest components ?

twig:Episodes:ProductionTable:Partials:TdStartOfWorkflow

twig:Episodes:ProductionTable:Partials:TdEndOfWorkflow

?

But i can already answer: you won't ever compete with includes in that configuration, as Component do a lot more things and are renderered each one in an isolated context (leading to many duplication of vars).

[choarau-craft]

Most of those components are really simple ones rendering some twig like this one:

{#{% props#}
{#	episodeClass,#}
{#	place, %}#}

<td
		data-col-name="{{ place }}"
		class="
			{{- " cell-workflow-start " -}}
			{{- " cell-workflow-parallel-start " -}}
			{{- attribute(episodeClass, place~'_realDelivery') ? ' bg-light-green ' -}}
{#            {{- attribute(episodeClass, place~'_isCurrent') ? ' current-cell ' -}} #}
		"
>
	<i class="far fa-long-arrow-right fa-4x "></i>
</td>

I do understand that components add extra cpu/memory usage, but after rendering shouldn't the memory be freed before rendering other components ?

[choarau-craft]

It's like all components stays in memory until the whole file is rendered.

[smnandre]

It's hard to say without seeing more... 😅

If you have some time to investigate a bit and make some discoveries there, i'd be really happy to ear them and see what we could improve !

[choarau-craft]

I've tried quite a lots of thinks, without much success. I'm currently replacing my most used components by includes. I've tried to stream the response but I'm still running into out of memory. Flush in twig does not seems to free components memory usage. It would be great to have a function, usable in twig template, to force memory cleanup.

By the way flush does not work inside twig component. Which seems quite logic, but it would be great to be able to make streamable components too.

If you have any idea I could test in my workspace I would be glad to try.

Thanks for you time

[smnandre]

@choarau-craft would you agree to share here (or privately if you prefer) a capture of the full twig component profiler page ? (let's say for a render with 100-ish components) ?

[choarau-craft]

Hi,

I won't have time this week, but will be available next week. But I can't share code here, but I can send some data privately

[smnandre]

But I can't share code here, but I can send some data privately

You can find me on the Symfony Slack channel, or send me an email at smn.andre AT gmail.com if you want

[fbuchlak]

I've encountered similar issue. In my case I'm using twig components as much as possible to create reusable components for things like buttons, tags, etc. In real application (in dev mode) I'm running out of memory with ~1 000 components.

@smnandre I've created simple example here: https://github.com/fbuchlak/symfony-ux-oom-demo

Rendering 100 items (~1000 twig components) there is memory peak around 40M (APP_ENV=dev/APP_DEBUG=true - this number is based only on profiler) Rendering 1000 items causes OOM (APP_ENV=dev/APP_DEBUG=true - with memory limit of 128M)

If I set APP_ENV=prod or APP_DEBUG=false in .env I can render up to 30 000 items (300 000 twig components) without an issue (use curl as browser may crash)

This seems like a problem with collecting debug information rather than components itself.

[smnandre]

There is a new configuration option to disable the TwigComponent profiler, could you test ?

twig_component:
    profiler: false

[smnandre]

Rendering 100 items (~1000 twig components) there is memory peak around 40M (APP_ENV=dev/APP_DEBUG=true - this number is based only on profiler) Rendering 1000 items causes OOM (APP_ENV=dev/APP_DEBUG=true - with memory limit of 128M)

For information i just tested locally, here are my metrics:

[smnandre]

With the profiler disabled, 10000 with no problem \o/

I guess that will help you!

[fbuchlak]

There is a new configuration option to disable the TwigComponent profiler, could you test ?

twig_component:
    profiler: false

This resolves the issue with rendering main request in real application, however profiler dies on memory anyway. I didn't test exact same scenario with includes, so I don't know if it wouldn't die anyway. I'm not sure, but it could be caused by amount of dispatched events with twig components (this timeline is only partial).

[smnandre]

If you could test a same scenario with these 3 tests (the maximum amount before it crashes) and give us time and memory used to render the page

• default (as you have right now)
• TwigComponent profiler disbled
• LiveComponent remove (just temporarely)

Sadly we won't be able to optimize much i fear as removing the events would be a BC break ...

... but i'm very concerned by this and would like to find a smart way to introduce static components (as yours seem to be)

[fbuchlak]

Sure, I will test it. Just one question. Without live components you mean removing ux-live-component package from the application?

Components I render in this scenario are neither live components or a ux icons.

I use them just like twig macros. The main motivation is to have stronger types "in templates". I don't know if this isn't just abuse of twig components, but having static ones without unnecessary events would fit my use case.

[smnandre]

Yes i meant uninstall. Even if you dont use them i'd like to see it ´ your case the cost of the LiveComponent subscribers on events

[fbuchlak]

Sorry for the delayed response @smnandre

I've tested it today (avg 10 runs with 440 twig components used)

1. twig components + live components (installed, not used) with profiler enabled Peak memory usage: 42.00 MiB Twig render time: 251 ms Twig components render time: 212 ms Twig components memory usage: 28.0 MiB

2. twig components + live components (installed, not used) with profiler disabled Peak memory usage: 24.00 MiB Twig render time: 199 ms Twig components render time: n/a Twig components memory usage: n/a

3. twig components with profiler enabled (live components not installed) Peak memory usage: 34.00 MiB Twig render time: 184 ms Twig components render time: 150 ms Twig components memory usage: 20.0 MiB

4. twig components with profiler disabled (live components not installed) Peak memory usage: 16.00 MiB Twig render time: 130 ms Twig components render time: n/a Twig components memory usage: n/a

[WebMamba]

Thanks @fbuchlak for this big investigation. We are going to use your reproducer to find some optimizations! The thing is the optimization we have in mind may lead to BC break. So we plan to release some configuration to activate or not some optimization. Since you already work on it, if you have any idea of how we can improve the performance let us know! Cheeers!

[kbond]

@smnandre does #2167 help with this?

[smnandre]

It's a start :)