🐛 Bug Report: [1.4.X - 1.5.X] Function throws MySQL server has gone away

[istornz]

👟 Reproduction steps

Sometimes, when I try to call a function from Appwrite, I got the error :

flutter: AppwriteException: general_unknown, Server Error (500)

When I try to check on Appwrite, the function was not called at all, there is no failing execution in the execution list...

I looked into docker logs appwrite right after this issue & get this strange log:

[Error] Timestamp: 2024-04-07T09:12:42+00:00
[Error] Method: POST
[Error] URL: /v1/functions/:functionId/executions
[Error] Type: PDOException
[Error] Message: MySQL server has gone away
[Error] File: @swoole-src/library/core/Database/PDOStatementProxy.php
[Error] Line: 62

I ran docker logs appwrite-mariadb and I get:

[...]
2024-04-07  2:56:25 133 [Warning] Aborted connection 133 to db: 'appwrite' user: 'user' host: '172.20.0.14' (Got timeout reading communication packets)
2024-04-07  2:57:41 134 [Warning] Aborted connection 134 to db: 'appwrite' user: 'user' host: '172.20.0.14' (Got timeout reading communication packets)
2024-04-07  3:35:37 135 [Warning] Aborted connection 135 to db: 'appwrite' user: 'user' host: '172.20.0.14' (Got timeout reading communication packets)
2024-04-07  3:36:04 128 [Warning] Aborted connection 128 to db: 'appwrite' user: 'user' host: '172.20.0.11' (Got timeout reading communication packets)
2024-04-07  3:36:04 131 [Warning] Aborted connection 131 to db: 'appwrite' user: 'user' host: '172.20.0.12' (Got timeout reading communication packets)
2024-04-07  6:00:03 129 [Warning] Aborted connection 129 to db: 'appwrite' user: 'user' host: '172.20.0.14' (Got timeout reading communication packets)

👍 Expected behavior

It should execute all functions without error

👎 Actual Behavior

Sometimes function throw error 500

🎲 Appwrite version

Version 1.4.x

💻 Operating system

Linux

🧱 Your Environment

I use an Hetzner instance

👀 Have you spent some time to check if this issue has been raised before?

• [X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• [X] I have read the Code of Conduct

[Emilien-Lambert]

The same thing happens here when I try to delete a user. version 1.4.13

2024-04-13T18:09:56.129491976Z [Error] Timestamp: 2024-04-13T18:09:56+00:00 
2024-04-13T18:09:56.129518338Z [Error] Method: DELETE 
2024-04-13T18:09:56.129522956Z [Error] URL: /v1/users/:userId 
2024-04-13T18:09:56.129537495Z [Error] Type: Utopia\Database\Exception 
2024-04-13T18:09:56.129541492Z [Error] Message: MySQL server has gone away 
2024-04-13T18:09:56.129545160Z [Error] File: /usr/src/code/vendor/utopia-php/database/src/Database/Adapter/MariaDB.php 
2024-04-13T18:09:56.129548757Z [Error] Line: 1372

[istornz]

The same thing happens here when I try to delete a user. version 1.4.13

2024-04-13T18:09:56.129491976Z [Error] Timestamp: 2024-04-13T18:09:56+00:00 
2024-04-13T18:09:56.129518338Z [Error] Method: DELETE 
2024-04-13T18:09:56.129522956Z [Error] URL: /v1/users/:userId 
2024-04-13T18:09:56.129537495Z [Error] Type: Utopia\Database\Exception 
2024-04-13T18:09:56.129541492Z [Error] Message: MySQL server has gone away 
2024-04-13T18:09:56.129545160Z [Error] File: /usr/src/code/vendor/utopia-php/database/src/Database/Adapter/MariaDB.php 
2024-04-13T18:09:56.129548757Z [Error] Line: 1372

Is it random on your side too?

[Emilien-Lambert]

The same thing happens here when I try to delete a user. version 1.4.13

2024-04-13T18:09:56.129491976Z [Error] Timestamp: 2024-04-13T18:09:56+00:00 
2024-04-13T18:09:56.129518338Z [Error] Method: DELETE 
2024-04-13T18:09:56.129522956Z [Error] URL: /v1/users/:userId 
2024-04-13T18:09:56.129537495Z [Error] Type: Utopia\Database\Exception 
2024-04-13T18:09:56.129541492Z [Error] Message: MySQL server has gone away 
2024-04-13T18:09:56.129545160Z [Error] File: /usr/src/code/vendor/utopia-php/database/src/Database/Adapter/MariaDB.php 
2024-04-13T18:09:56.129548757Z [Error] Line: 1372

Is it random on your side too?

Yes, I'd say it's pretty random, I've had the problem 2 times on different actions and each time I solved it by restarting the appwrite docker instances.

[istornz]

@Emilien-Lambert, after conducting some investigations, I believe there’s a bug related to 1.4.X statistics.

Why?

• There is a known issue with 1.4.X statistics. This feature is broken in that version and can display negative values on the UI.
• When performing actions like deleting a user or triggering a function, Appwrite attempts to modify stats (such as adding a function execution or incrementing/decrementing user counts).
• In my case, the function is a simple HTTP call and does not interact with the Appwrite database, so it’s not related to the function itself.

Appwrite 1.5.X completely overhauls the statistics system. I upgraded my instance to 1.5.4 yesterday, and so far, it appears to be much improved. I haven’t encountered this error since the upgrade (wish me lucks :p).

As a temporary workaround, consider the following:

• Disable statistics by setting _APP_USAGE_STATS to false (not tested, but it should works?).
• Upgrade to 1.5.X.

Please note that this is purely speculative on my part. 😊

Any advice of Appwrite team like @stnguyen90 or @Meldiron would be great 👍

[Emilien-Lambert]

@istornz Thank you for this valuable information. 👍

[istornz]

@Emilien-Lambert Hello, do you have tried any of theses solutions? I’m curious about finding a correct solution/workaround 👍

[Emilien-Lambert]

I updated my instance to version 1.5.4 and since then I haven't had any recurrence. However, I'll have to test it further, as I haven't had time to use appwrite much. But this is already very encouraging 👍

[istornz]

@Emilien-Lambert Very nice, exactly the same way for me 👍

[istornz]

I'm come back because it seems the issue is not fixed yet on 1.5.X ...

I'm getting the same issue (with more error details) when executing a function:

[Error] Timestamp: 2024-04-19T06:00:24+00:00
[Error] Method: POST
[Error] URL: /v1/functions/:functionId/executions
[Error] Type: PDOException
[Error] Message: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
[Error] File: @swoole/library/core/Database/PDOStatementProxy.php
[Error] Line: 44
[Error] Type: Utopia\Exception
[Error] Message: Error handler had an error: Appwrite\Extend\Exception::__construct(): Argument #3 ($code) must be of type ?int, string given, called in /usr/src/code/app/controllers/general.php on line 711
[Error] File: /usr/src/code/vendor/utopia-php/framework/src/App.php
[Error] Line: 574

EDIT: Another crash with an another function, with the same error

[Error] Timestamp: 2024-04-21T10:49:21+00:00
[Error] Method: POST
[Error] URL: /v1/functions/:functionId/executions
[Error] Type: PDOException
[Error] Message: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
[Error] File: @swoole/library/core/Database/PDOStatementProxy.php
[Error] Line: 44
[Error] Type: Utopia\Exception
[Error] Message: Error handler had an error: Appwrite\Extend\Exception::__construct(): Argument #3 ($code) must be of type ?int, string given, called in /usr/src/code/app/controllers/general.php on line 711
[Error] File: /usr/src/code/vendor/utopia-php/framework/src/App.php
[Error] Line: 574

@stnguyen90 @Meldiron @eldadfux

[istornz]

@Emilien-Lambert Can you have tested more on your side? It seems it's better on my side but had some crash again...

[Emilien-Lambert]

@istornz Yes, I've used appwrite a lot since then and I've had no problems with version 1.5.4. On the other hand, I haven't been able to upgrade my version since version 1.4.13 and my migrations have been temperamental several times.

So I opted for a radical solution. I retrieved all my documents in JSON via the SDK, then I completely redeployed an Appwrite instance via the docker-compose.yml in version 1.5.4 then I recreated all my documents via the SDK.

Since then, with a fresh installation of version 1.5.4 I no longer have this problem.

I know the solution isn't great, but it's the only way I could find in the time I had available.

[istornz]

@istornz Yes, I've used appwrite a lot since then and I've had no problems with version 1.5.4. On the other hand, I haven't been able to upgrade my version since version 1.4.13 and my migrations have been temperamental several times.

So I opted for a radical solution. I retrieved all my documents in JSON via the SDK, then I completely redeployed an Appwrite instance via the docker-compose.yml in version 1.5.4 then I recreated all my documents via the SDK.

Since then, with a fresh installation of version 1.5.4 I no longer have this problem.

I know the solution isn't great, but it's the only way I could find in the time I had available.

Okay thanks for your reply, I noticed an amelioration with 1.5.X but some random errors appears again on my side so I decided to use Firebase functions as a temporary workaround since we found the solution of this issue. The advantage of the 1.5.X, we now have a clearly error 👍

Can you check if this error appear again (if any) and tell us here to track more info about this issue 👍