filament Multi container applications are failing

Package

filament/filament

Package Version

v3.3.14

Laravel Version

11.44.7

Livewire Version

v3.6.3.

PHP Version

v8.3.0

Problem description

We believe the issue is a regression of this: 7751

We reached out to Livewire already here: 9323

I first encountered this issue when I deployed a fully developed Filament application to GCP Cloud Run (serverless) with 3 container instances. A resource index (table) page with a couple of chart widgets would break (displaying a basic Laravel 500 error modal) after the initial payload had rendered. Chrome devtools showed that some of the /livewire/update requests for the chart widgets returning a 500. The number of requests that failed varied with page reload. I had never experienced that problem in the prior two months of development on my local machine with a single Docker container running the app.

As part of my investigation, I created a fresh Laravel project, installed Filament, and put 10 simple bar chart widgets on the default dashboard. The same issue presented itself when I deployed the new project to Cloud Run in a different GCP project.

Through testing/tweaking I've encountered a variety of symptoms, including:

500 responses with no exception/stacktrace (APP_ENV=development, APP_DEBUG=true) Partial 200 responses missing various random pieces of the chart widget mark-up development.ERROR: Livewire encountered a missing root tag when trying to render a component. The following configurations have "fixed" the issue:

A revision with a single container instance A revision with multiple container instances and Cloud Run's Session Affinity feature turned on A revision with multiple container instances and a GCS bucket mounted at ./storage/framework/views so that all container instances were sharing a common view cache It was that last working configuration that really pushed me to thinking this was a Livewire load-balancer-related Laravel view/blade cache problem.

The network tab ends up looking like this, with the count and position of failed requests different on every page load: image

That's probably the best I can provide right now. I'm not familiar with Livewire's internals.

Happy to answer any questions.

UPDATE: Setting 'cache'=>false in ./config/view.php, which "disables" the view cache, also appears to "fix" the issue.

UPDATE 2: FWIW, I was able to replicate the problem in my local dev environment using Docker Compose replicas. I instantly received the "missing root tag" exception. The problem went away as soon as I mounted my application directory to the containers (effectively creating a shared view cache directory again). I'll keep digging into this. Here's the stack trace from the Docker Compose log:

Output log has been attached.

Summary: Filament / Livewire does not run in a multi container application that is being load balanced. The issue seems to be that the cache directories are not the same between each container (by design) and this makes Filament/Livewire bug out. Running Filament / Livewire in a single instance does not trigger this bug.

Expected behavior

We expect to successfully run our application on a 3 container system behind GCPs Cloud Run load balancer.

Steps to reproduce

Install and Run git clone https://github.com/peterjbassi/filamentapp.git docker compose up -d mysql -h 127.0.0.1 -u root -p -e 'create database charts'; # password is charts docker compose exec charts php artisan migrate
docker compose exec charts php artisan make:filament-user --name charts --email [email protected] --password charts ngrok http 8080 Open your browser to https://<your_domain>/admin

Username: [email protected] Password: charts

Reproduction repository (issue will be closed if this is not valid)

https://github.com/peterjbassi/filamentapp

Relevant log output

charts-3 | When rendering a Blade view, make sure it contains a root HTML tag. at /var/www/app/vendor/livewire/livewire/src/Drawer/Utils.php:20)
charts-3 | [stacktrace]
charts-3 | #0 /var/www/app/vendor/livewire/livewire/src/Mechanisms/HandleComponents/HandleComponents.php(248): Livewire\Drawer\Utils::insertAttributesIntoHtmlRoot()
charts-3 | #1 /var/www/app/vendor/livewire/livewire/src/Mechanisms/HandleComponents/HandleComponents.php(285): Livewire\Mechanisms\HandleComponents\HandleComponents->Livewire\Mechanisms\HandleComponents\{closure}()
charts-3 | #2 /var/www/app/vendor/livewire/livewire/src/Mechanisms/HandleComponents/HandleComponents.php(233): Livewire\Mechanisms\HandleComponents\HandleComponents->trackInRenderStack()
charts-3 | #3 /var/www/app/vendor/livewire/livewire/src/Mechanisms/HandleComponents/HandleComponents.php(104): Livewire\Mechanisms\HandleComponents\HandleComponents->render()
charts-3 | #4 /var/www/app/vendor/livewire/livewire/src/LivewireManager.php(102): Livewire\Mechanisms\HandleComponents\HandleComponents->update()
charts-3 | #5 /var/www/app/vendor/livewire/livewire/src/Mechanisms/HandleRequests/HandleRequests.php(94): Livewire\LivewireManager->update()
charts-3 | #6 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(47): Livewire\Mechanisms\HandleRequests\HandleRequests->handleUpdate()
charts-3 | #7 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/Route.php(266): Illuminate\Routing\ControllerDispatcher->dispatch()
charts-3 | #8 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/Route.php(212): Illuminate\Routing\Route->runController()
charts-3 | #9 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/Router.php(808): Illuminate\Routing\Route->run()
charts-3 | #10 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(170): Illuminate\Routing\Router->Illuminate\Routing\{closure}()
charts-3 | #11 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(51): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | #12 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Routing\Middleware\SubstituteBindings->handle()
charts-3 | #13 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(88): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | #14 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Foundation\Http\Middleware\VerifyCsrfToken->handle()
charts-3 | #15 /var/www/app/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/pull/16 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\View\Middleware\ShareErrorsFromSession->handle()
charts-3 | https://github.com/livewire/livewire/issues/17 /var/www/app/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/pull/18 /var/www/app/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\Session\Middleware\StartSession->handleStatefulRequest()
charts-3 | https://github.com/livewire/livewire/issues/19 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Session\Middleware\StartSession->handle()
charts-3 | https://github.com/livewire/livewire/pull/20 /var/www/app/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/21 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle()
charts-3 | https://github.com/livewire/livewire/pull/22 /var/www/app/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(75): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/23 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Cookie\Middleware\EncryptCookies->handle()
charts-3 | https://github.com/livewire/livewire/pull/24 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(127): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/pull/25 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/Router.php(807): Illuminate\Pipeline\Pipeline->then()
charts-3 | https://github.com/livewire/livewire/issues/26 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/Router.php(786): Illuminate\Routing\Router->runRouteWithinStack()
charts-3 | https://github.com/livewire/livewire/issues/27 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/Router.php(750): Illuminate\Routing\Router->runRoute()
charts-3 | https://github.com/livewire/livewire/pull/28 /var/www/app/vendor/laravel/framework/src/Illuminate/Routing/Router.php(739): Illuminate\Routing\Router->dispatchToRoute()
charts-3 | https://github.com/livewire/livewire/issues/29 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(201): Illuminate\Routing\Router->dispatch()
charts-3 | https://github.com/livewire/livewire/pull/30 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(170): Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/31 /var/www/app/vendor/livewire/livewire/src/Features/SupportDisablingBackButtonCache/DisableBackButtonCacheMiddleware.php(19): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/32 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Livewire\Features\SupportDisablingBackButtonCache\DisableBackButtonCacheMiddleware->handle()
charts-3 | https://github.com/livewire/livewire/pull/33 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(27): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/34 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull->handle()
charts-3 | https://github.com/livewire/livewire/issues/35 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(47): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/36 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Foundation\Http\Middleware\TrimStrings->handle()
charts-3 | https://github.com/livewire/livewire/issues/37 /var/www/app/vendor/laravel/framework/src/Illuminate/Http/Middleware/ValidatePostSize.php(27): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/38 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Http\Middleware\ValidatePostSize->handle()
charts-3 | https://github.com/livewire/livewire/issues/39 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(110): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/40 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Foundation\Http\Middleware\PreventRequestsDuringMaintenance->handle()
charts-3 | https://github.com/livewire/livewire/issues/41 /var/www/app/vendor/laravel/framework/src/Illuminate/Http/Middleware/HandleCors.php(49): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/pull/42 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Http\Middleware\HandleCors->handle()
charts-3 | https://github.com/livewire/livewire/issues/43 /var/www/app/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php(58): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/pull/44 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Http\Middleware\TrustProxies->handle()
charts-3 | https://github.com/livewire/livewire/issues/45 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/InvokeDeferredCallbacks.php(22): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/46 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(209): Illuminate\Foundation\Http\Middleware\InvokeDeferredCallbacks->handle()
charts-3 | https://github.com/livewire/livewire/issues/47 /var/www/app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(127): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
charts-3 | https://github.com/livewire/livewire/issues/48 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(176): Illuminate\Pipeline\Pipeline->then()
charts-3 | https://github.com/livewire/livewire/issues/49 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(145): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter()
charts-3 | https://github.com/livewire/livewire/pull/50 /var/www/app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1220): Illuminate\Foundation\Http\Kernel->handle()
charts-3 | https://github.com/livewire/livewire/issues/51 /var/www/app/public/index.php(17): Illuminate\Foundation\Application->handleRequest()
charts-3 | https://github.com/livewire/livewire/pull/52 {main}
charts-3 | "}

May 09 '25 13:05 jonkarrer

@jonkarrer can you please share why you opened this issue, is there something Filament can do? It sounds like an issue on the Livewire side so I am not sure I should keep this open

May 09 '25 15:05 danharrin

Filament's ChartWidget is the class that triggers this issue. We have spun up a multi container raw livewire application that does not reproduce this bug. However in our example Filament application the bug can be reproduced.

Here is the repo: https://github.com/peterjbassi/filamentapp

This leads me to think that Filaments implementation could be the issue. However I can admit that this may be flawed as it is difficult to debug such a layered cake.

Bottom line is that this is at least worth bringing to Filaments attention because livewire is a major dependency. We do have a repo that reproduces the bug and that should be enough to merit an investigation. My best effort was to bring this issue up with both teams and see if we could come together on a solution.

May 09 '25 15:05 jonkarrer

Hello, hope all is going well in dev world. Wanted to give some updates on this. We have been going as deep as we can to figure out this issue.

Issue Summary

We've identified that two separate cache files are being generated for the same view, potentially causing a race condition when multiple chart widgets are loaded simultaneously.

Investigation Process

Created Test Environment
- Spun up a simple docker free Filament/Livewire repository with chart widgets
- Repository: https://github.com/peterjbassi/charts

Generated Fresh View Cache

Ran php artisan view:clear && php artisan view:cache

Initial cached file path:

/**PATH /Users/jkarrer/charts/vendor/filament/widgets/resources/views/chart-widget.blade.php ENDPATH**/

Client-Side Request Test
- Made request to http://127.0.0.1:8000/dashboard
- This generated a nearly identical cache file with a different path:
```
/**PATH /Users/jkarrer/charts/vendor/filament/widgets/src/../resources/views/chart-widget.blade.php ENDPATH**/
```

Suspected Issue

The pre-generated cache is not being used on page load. When multiple chart widgets (10 in our test) make requests, each is treated as unique by Laravel/PHP, triggering on-demand cache file generation. This creates a potential race condition where some charts attempt to read from files that are being written by others.

Questions

Is this behavior by design or a bug?
Could this explain the file content issues we've been experiencing?
Are there recommended workarounds for this scenario?

May 12 '25 18:05 jonkarrer

The file contents of the 2 files are slightly different. This may be causing php to hash the differently perhaps.

On the left is the "on request" generated file and the other is the "view::cache" one.

May 12 '25 18:05 jonkarrer

Judging by the differences in your screenshots, this is probably the problem - https://github.com/livewire/livewire/pull/8068

The fix for this has not been released by Livewire yet. You can test it out by pulling a dev version of Livewire though.

May 12 '25 19:05 danharrin

Thank you @danharrin. We pulled this commit down and ran it.

This still generated 2 cache files, BUT they are identical this time around. So we still run into the above issue.

May 12 '25 19:05 jonkarrer

I don't understand why two identical cache files would cause an issue? Have you thought about deploying a separate filesystem for each container? I feel like that is more common?

May 12 '25 19:05 danharrin

The multi container issue seems to have been a red-herring given this new discovery. It made us think cache keys were wrong, but that is not the case. A race condition is our working theory right now.

I don't understand why two identical cache files would cause an issue?

The files being identical or not is not the issue. The issue is that one is pre-generated with "view::cache", and the other is on request time.

Have you thought about deploying a separate filesystem for each container?

Yes currently that is the case in our GCP project. 3 containers, 3 separate file systems, load balanced. GCPs load balancer is not round robin to our knowledge.

Since the Chart Widget is generating a new request per Chart Widget instance (lazy loading on), this presents the race condition. Here is an image of the situation. It seems as the very first request is writing the new cache file, the second request starts to read:

From this page above, I can refresh forever no problem because now the cache is written. But if I were to clear my cache via "view::clear", I can reproduce the above after a few rounds of cache clearing and page loading.

May 12 '25 20:05 jonkarrer

The files being identical or not is not the issue. The issue is that one is pre-generated with "view::cache", and the other is on request time.

I don't understand why that would be an issue either.

May 13 '25 07:05 danharrin

Generating the cache at request time in a "concurrent" php environment will cause a race condition probability.

We have this environment in our example repo by spinning up 10 php servers that are load balanced by Caddy. We also have this in our production application using GCP load balancers and 3 containers running php fpm. Laravel allows you to pre cache the views in part to prevent this I assume.

To highlight this assertion, we have made a branch that takes Filament/Livewire completely out of the picture and simply loads the Laravel welcome page across 10 iframes. We run "view:clear" to make sure the cache will be made at request time. After a few attempts of "view:clear" and "page loading", one of our iFrames will pop with an "incomplete file" error.

Here is the image:

If you would like to test this out yourself we have a branch of the above here: https://github.com/peterjbassi/charts/tree/home-iframe

So the reason "generating cache files on request in a concurrent environment" is that it creates a race condition possibility given enough traffic and improper handling of locking the cache.

Ideally we would like to be able to rely on our pre generated cache from "view:cache". This would eliminate a race condition probability. So why do these requests create cache files? Why are the duplicate cache files pointing at different file paths?

May 13 '25 13:05 jonkarrer

Hi @jonkarrer. I’d be happy to see if I can reproduce the issue and do some investigating. But can you please make the following changes to your reproduction repo:

Update to v4 beta (there’s no point in digging into this if the problem is gone in v4)
Remove ngrok (I use a different tunnelling setup and don’t want to install ngrok just for this)
Use MYSQL_DATABASE, MYSQL_USER and MYSQL_PASSWORD to set up the default db so I don’t need to create it manually
Add a seeder for the default user so I don’t need to create it manually

The reproduction repo needs to be simpler to fire up locally. It should just be clone, compose up, migrate —seed, done.

I’ve run non-Filament Livewire apps behind a load balancer and haven’t run into this problem. I’ll be spinning up multiple instances of a Filament app soon though so I’m keen to make sure this is solved.

Jun 30 '25 05:06 binaryfire

I've narrowed down the cause of this issue, and it looks like Filament has inherited this from spatie/laravel-package-tools.

I started a discussion over there since they don't appear to accept issues, but it hasn't gone anywhere. I tried to outline the exact cause, with example, screenshots, our work-around, etc, as clearly as possible.

In short though, the flaw is preventing Filament's views cached by artisan view:cache from actually being used, resulting in what are essentially duplicates being created at request time. Send enough requests to the same route at the same time and one or more of those requests ends up trying to read from a cache file that another request hasn't finished writing yet (or worse).

Jul 09 '25 02:07 peterjbassi

In short though, the flaw is preventing Filament's views cached by artisan view:cache from actually being used, resulting in what are essentially duplicates being created at request time. Send enough requests to the same route at the same time and one or more of those requests ends up trying to read from a cache file that another request hasn't finished writing yet (or worse).

Interesting. If the view cache is being missed because of the Spatie package, that's a performance issue which will affect all Filament apps (not just load balanced ones). I might dig into this today as well.

Jul 09 '25 03:07 binaryfire

Just to confirm, you aren't actually running view cache as part of your deployment script, right? That is completely broken in Livewire 3 and the team aren't planning to fix it until Livewire 4 (judging by comments on a v3 PR to fix it).

Livewire's morph markers will only get fully generated when the view cache is generated on-demand by the first request that renders it.

Are you saying that even when the view cache is generated on demand, it also doesn't get used on subsequent requests?

Jul 09 '25 05:07 danharrin

@danharrin I’ve identified the issue and am about to push a PR to the package tools repo. Will link it here when it’s up.

Jul 09 '25 05:07 binaryfire

Does this affect deployments when view cache is not run, from your experience?

Jul 09 '25 05:07 danharrin

@danharrin Here's the PR. Even if view:cache is run, a second set of cached views were being generated.

https://github.com/spatie/laravel-package-tools/pull/172

Jul 09 '25 05:07 binaryfire

Yeah understood, I am asking if the issue happens when it is NOT run too

Jul 09 '25 05:07 danharrin

I am asking because no-one who uses Livewire 3 should be using view:cache right now, its completely broken anyway

Jul 09 '25 05:07 danharrin

@danharrin Sorry I didn't read your first comment properly. No, looks like no duplicates are generated when you don't run view:cache. I didn't realise view:cache was broken with Livewire 3. Not great for apps with a lot of non-Livewire public pages.

Jul 09 '25 06:07 binaryfire

On deployment we run artisan optimize and artisan filament-optimize. artisan optimize internally runs view:cache along with a few others caching commands.

I concur that no duplicates are generated when you don't run view:cache, either directly or indirectly through optimize. To be clear though, the presence of duplicate cached views, while not ideal, isn't the ultimate problem. It's the request-time view caching that allows the race condition to occur.

What's the currently recommended method for avoiding request-time view caching with Filament?

Jul 09 '25 11:07 peterjbassi

There's no recommended method for avoiding request-time view caching with Livewire 3, so there's no method for Filament either. I wish they merged that fix PR into Livewire 3 instead of Livewire 4, since it isn't breaking existing applications.

Jul 09 '25 11:07 danharrin

I'm going to close this, as there isn't anything we can fix on the Filament side as far as I can see. Glad it has been narrowed down to dependencies.

Jul 11 '25 10:07 danharrin