appwrite icon indicating copy to clipboard operation
appwrite copied to clipboard

πŸ› Bug Report: Multiple internal curl errors has occurred within the executor! Error Number: 111.

Open JakeAi opened this issue 1 year ago β€’ 13 comments

πŸ‘Ÿ Reproduction steps

The first execution after deployment/restart. When you're testing a function, and you have to keep deploying it and running it, and constantly get the CURL error.. it's a pain in the arse. Deploy a (node?) function and execute it.

https://discord.com/channels/564160730845151244/1162099630494973982 (Wasn't actually solved) https://discord.com/channels/564160730845151244/1151439270041628763

πŸ‘ Expected behavior

It should run normally

πŸ‘Ž Actual Behavior

Multiple internal curl errors has occurred within the executor! Error Number: 111. Error Msg: Connection refused\nError Code: 500

🎲 Appwrite version

Version 1.4.x

πŸ’» Operating system

Linux

🧱 Your Environment

Appwrite: 1.4.5 Linux docker-01 5.15.0-73-generic #80-Ubuntu SMP

πŸ‘€ Have you spent some time to check if this issue has been raised before?

  • [X] I checked and didn't find similar issue

🏒 Have you read the Code of Conduct?

JakeAi avatar Oct 13 '23 11:10 JakeAi

Same issue after upgrading 1.4.5 from 1.4.4

function runs node-20 Screenshot 2023-10-14 at 10 44 25

Screenshot 2023-10-14 at 10 54 57

xuelink avatar Oct 14 '23 07:10 xuelink

I am facing same issue. Any update ?

chakshumraval avatar Nov 30 '23 18:11 chakshumraval

I am having the same problem with node functions. For me it is not only happening after deployment or restarts, but also if i haven't tested the functions for a few hours (or minutes even?) and i come back to it. The first few executions always fail with Multiple internal curl errors has occurred within the executor! Error Number: 111. Error Msg: Connection refused\nError Code: 500.

I am using appwrite with the DigitalOcean marketplace droplet (https://marketplace.digitalocean.com/apps/appwrite).

I tried to investigate yesterday and found the pattern, that whenever I call my function via REST and it fails, i will see the following when running docker logs appwrite-worker-functions:

Screenshot 2023-12-07 at 07 38 40

So this already made me optimistic about finding the underlying issue (redis?) but this morning, when trying again, i could not get this output anymore on failing REST calls. Which actually makes this even weirder.

Any help would be appreciated. The current situation is devastating because cloud functions that don't always respond properly are - of course - worth nothing :/

UPDATE (a few minutes later): I realized, that since this morning the output of docker logs appwrite-worker-functions seems to have changed for me in such a way, that it doesn't display me anymore errors when functions are failing, but in these cases i also do not get "Fetched 1 functions..." - see the following screenshot, at the end of the log there were 3 functions calls that failed, where i never go the bright green message. And then the very last one succeeded again:

Screenshot 2023-12-07 at 08 12 03

Could be that the logs changed because i upgraded the redis docker container.

maks-io avatar Dec 07 '23 06:12 maks-io

I am also encountering the same issue. Multiple internal curl errors on first time, then have to re execute for it to go away.

nikitakofman avatar Dec 09 '23 11:12 nikitakofman

Question to all the others who are having similar issues here: are you also using a very "limited set of resources" maybe? I am currently using https://marketplace.digitalocean.com/apps/appwrite and when setting it up, i chose the cheapest droplet variant they're offering (since it is for testing purposes only anyhow). I am asking, because to me it feels like some things just "go to sleep" after some time and therefore my first few function calls fail if i haven't been using them for some time. Could be that this is only the case due to limited resources maybe? Just thinking out loud, didn't have time to investigate further yet.

maks-io avatar Dec 18 '23 16:12 maks-io

Question to all the others who are having similar issues here: are you also using a very "limited set of resources" maybe? I am currently using https://marketplace.digitalocean.com/apps/appwrite and when setting it up, i chose the cheapest droplet variant they're offering (since it is for testing purposes only anyhow). I am asking, because to me it feels like some things just "go to sleep" after some time and therefore my first few function calls fail if i haven't been using them for some time. Could be that this is only the case due to limited resources maybe? Just thinking out loud, didn't have time to investigate further yet.

The thing is, it never used to happen. And now it's been present for like 6 months.

JakeAi avatar Dec 19 '23 19:12 JakeAi

Question to all the others who are having similar issues here: are you also using a very "limited set of resources" maybe? I am currently using https://marketplace.digitalocean.com/apps/appwrite and when setting it up, i chose the cheapest droplet variant they're offering (since it is for testing purposes only anyhow). I am asking, because to me it feels like some things just "go to sleep" after some time and therefore my first few function calls fail if i haven't been using them for some time. Could be that this is only the case due to limited resources maybe? Just thinking out loud, didn't have time to investigate further yet.

The thing is, it never used to happen. And now it's been present for like 6 months.

Okay that's quite a long time, I wasn't aware of that since I only recently started using appwrite.

maks-io avatar Dec 28 '23 10:12 maks-io

It's been since 6 month now. i am still facing same issue. Any update on this ?

It is stopping us to make apps live. so it's blocker for us

chakshumraval avatar Mar 04 '24 18:03 chakshumraval

It's been since 6 month now. i am still facing same issue. Any update on this ?

It is stopping us to make apps live. so it's blocker for us

It doesn't happen on the paid version, so they probably won't fix it.

JakeAi avatar Mar 07 '24 14:03 JakeAi

It's been since 6 month now. i am still facing same issue. Any update on this ? It is stopping us to make apps live. so it's blocker for us

It doesn't happen on the paid version, so they probably won't fix it.

I really hope you are wrong, but yeah... which way are you guys going, Supabase? Firebase? Anything else?

maks-io avatar Mar 07 '24 15:03 maks-io

It's been since 6 month now. i am still facing same issue. Any update on this ? It is stopping us to make apps live. so it's blocker for us

It doesn't happen on the paid version, so they probably won't fix it.

I really hope you are wrong, but yeah... which way are you guys going, Supabase? Firebase? Anything else?

Literally struggle busing through it and making snarky remarks in discord until it gets addressed.

JakeAi avatar Mar 07 '24 16:03 JakeAi

I just upgraded my DigitalOcean Appwrite instance to the new 1.5.1 version - upgrade itself was flawless but this bug is still existing.

For now i am thinking about staying with Appwrite but using a separate express app for some additional backend logic and skipping functions completely.

maks-io avatar Mar 08 '24 17:03 maks-io

I can successfully reproduce this on Cloud consistently. This seems to occur during the cold start of the functions container, and subsequent executions succeed. image

EVDOG4LIFE avatar Mar 08 '24 21:03 EVDOG4LIFE

I got the same error, and re-executing the same function immediately seems to solve the problem. It's looks more like a bug in Appwrite.

Snag_13c6a693

lawipac avatar Mar 22 '24 03:03 lawipac

Question to all the others who are having similar issues here: are you also using a very "limited set of resources" maybe? I am currently using https://marketplace.digitalocean.com/apps/appwrite and when setting it up, i chose the cheapest droplet variant they're offering (since it is for testing purposes only anyhow). I am asking, because to me it feels like some things just "go to sleep" after some time and therefore my first few function calls fail if i haven't been using them for some time. Could be that this is only the case due to limited resources maybe? Just thinking out loud, didn't have time to investigate further yet.

I'm on plenty of resources using a dedicated test bed, with version 1.5.3, the same problem still exists.

lawipac avatar Mar 23 '24 03:03 lawipac

Question to all the others who are having similar issues here: are you also using a very "limited set of resources" maybe? I am currently using https://marketplace.digitalocean.com/apps/appwrite and when setting it up, i chose the cheapest droplet variant they're offering (since it is for testing purposes only anyhow). I am asking, because to me it feels like some things just "go to sleep" after some time and therefore my first few function calls fail if i haven't been using them for some time. Could be that this is only the case due to limited resources maybe? Just thinking out loud, didn't have time to investigate further yet.

I'm on plenty of resources using a dedicated test bed, with version 1.5.3, the same problem still exists.

Yeah it doesn’t matter and shouldn’t matter. It never used to happen prior to 1.4. I’ve been using the same VM since 0.6.

JakeAi avatar Mar 23 '24 14:03 JakeAi

We're looking into this problem. Feel free to share any logs that you think might be relevant or steps to reproduce.

eldadfux avatar Apr 02 '24 06:04 eldadfux

Hi, we used to experience this using WSL and docker, we were able to fix it by switching to linux and having more memory available for appwrite executor/containers.

Inovvia avatar Apr 02 '24 07:04 Inovvia

In other words - as i assumed - the resources do matter - but i agree with @JakeAi on the other statement, that it should not matter.

maks-io avatar Apr 02 '24 07:04 maks-io

I am having the same issue and daily CRON jobs are inconsistent due to frequent fails.

LDY681 avatar May 29 '24 04:05 LDY681

We've made some improvements to functions in Cloud and 1.5.7 and the error rate for this specific error has gone down.

The reproduction case I created previously no longer throws this error so this is looking good.

If you are still experiencing this error in your self-hosted setup, please ensure that you've upgraded to 1.5.7, for Cloud users - this is currently live.

EVDOG4LIFE avatar May 29 '24 21:05 EVDOG4LIFE