Local Edge functions: event loop error after a few minutes

[ChuckJonas]

Describe the bug

It seems that after my edge function has been running locally for some time (perhaps a few minutes) I start to get these messages in the logs:

wall clock duration warning. isolate: df3c6dc8-dcc9-4348-a278-e0e1ff102fc3
wall clock duration warning. isolate: 93b745a9-cdb0-49a0-ae88-514a87391cd6
wall clock duration warning. isolate: ce00ba14-d7f7-4e27-8b6e-92191f485a81
wall clock duration warning. isolate: 39ebddfd-8b3e-4af6-a966-178f821877b6

And then eventually I see this:

event loop error: TypeError: Cannot convert undefined or null to object
    at slice (<anonymous>)
    at new DOMException (ext:deno_web/01_dom_exception.js:126:14)
    at [[[signalAbort]]] (ext:deno_web/03_abort_signal.js:83:14)
    at AbortController.abort (ext:deno_web/03_abort_signal.js:172:30)
    at file:///home/deno/main/index.ts:147:31
    at Object.action (ext:deno_web/02_timers.js:151:11)
    at handleTimerMacrotask (ext:deno_web/02_timers.js:65:10)
    at eventLoopTick (ext:core/01_core.js:189:21)

After that error happens, all my requests fail with this:

Error in worker connection: connection closed before message completed
request failed (uri: "/my-function" reason: hyper::Error(Canceled, "connection was not ready"))

The function is as simple as this:

Deno.serve(async (req) => {
  return new Response(
    JSON.stringify({ publicToken: "123" }),
    {
      headers: { "Content-Type": "application/json" },
    }
  );
});

To Reproduce Steps to reproduce the behavior:

1. Create a new function with the above code
2. run supabase functions serve
3. Call the function at least once (not sure if this is actually necessary)
4. Wait a few minutes... You should eventually see the above event loop error
5. Try to call the function again. You will get "connection was not ready" error

Expected behavior Local functions should not randomly crash after only a few minutes :)

Desktop (please complete the following information):

• OS: OSX
• Version of CLI: 1.110.1

Additional context

At least one other person has reported experiencing this problem in the supabase discord (The above report is using their description).

[ChuckJonas]

I rolled back to 1.106.1 and the issue seem to go away. I still see the wall clock duration warning. isolate: 88c59c58-1271-4712-ae7e-863ecb2449fa errors, but it no longer crashes (so maybe those aren't related to the crash)

[sweatybridge]

Transferring because it sounds like an issue with edge runtime container.

[scottweinert]

I can also verify that I'm experiencing the same issue.

[confusedmatrix]

Seeing the same issue also

[evelant]

I'm also seeing this issue, can't update supabase beyond 1.106.1 or edge fns break every few minutes.

[rbkayz]

yep. same

[d-e-h-i-o]

I'm also experiencing this issue with CLI version 1.110.1

[hyldmo]

I had this is issue too on my M2 mac until i upgraded Docker Desktop (it was very much out of date) as well as supabase. Now running without a single crash on supabase 1.112.2 and Docker Desktop 4.25.1 (128006). Still seeing the wall clock duration warning though, but no crashes.

[d-e-h-i-o]

I had this is issue too on my M2 mac until i upgraded Docker Desktop (it was very much out of date) as well as supabase. Now running without a single crash on supabase 1.112.2 and Docker Desktop 4.25.1 (128006). Still seeing the wall clock duration warning though, but no crashes.

My Docker Desktop version is 4.19.0 (106363

[scottweinert]

Thanks @hyldmo. Updating Docker Desktop and my supabase CLI resolved the crashing issue.

[rothnic]

Any comments on whether the wall clock duration reached. isolate: 11455258470253939329 logs are a problem? It kind of seems like the connection isn't being closed out or something, but the response it getting returned from the endpoint. Is there something else we should be doing to tell supabase that the request is complete?

[all-mute]

same

[SebasScript]

same issue here, i don't get an error, but all cloud functions become very slow or irresponsive after a couple requests. Seems like the workers don't exit execution. If i restart the runtime it works again for a couple of request

[CalvinWilkinson]

I as well am getting related issues.

I get wall clock log msgs like these:

wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped

I also am having an issue where I cannot use the existsSync() and Deno.readDirSync() functions. When running these, I get the following error:

runtime has escaped from the event loop unexpectedly: event loop error: TypeError: Deno.statSync is not a function
    at existsSync (https://deno.land/[email protected]/fs/exists.ts:121:23)
    at EnvService.dirDoesNotExist (file:///home/deno/functions/instructor-ui/services/env-service.ts:100:13)
    at EnvService.validateInitEnvFile (file:///home/deno/functions/instructor-ui/services/env-service.ts:65:14)
    at EnvService.loadEnv (file:///home/deno/functions/instructor-ui/services/env-service.ts:22:48)
    at file:///home/deno/functions/instructor-ui/index.ts:5:12
failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

I have tried to search for any open or closed issues on this to see if I can get some more info but have come up dry.

[nyannyacha]

hello @CalvinWilkinson 😁

This runtime provides a limited File API compared to Deno ^link, and that's a relatively recent change. From a security perspective, allowing file access can be problematic.

As for the error message... that's something that needs to be documented for self-hosted users 🙄

wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped

Do you mind if I ask what version of the runtime you're getting this message from? I remember this is something I've already fixed.

failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

This error message is a trivial runtime message caused by a script error, so you don't have to care about it, but now that I think about it, it's a bit verbose for a user 😅

---

[CalvinWilkinson]

hello @CalvinWilkinson 😁

This runtime provides a limited File API compared to Deno 1, and that's a relatively recent change. From a security perspective, allowing file access can be problematic.

As for the error message... that's something that needs to be documented for self-hosted users 🙄

wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped

Do you mind if I ask what version of the runtime you're getting this message from? I remember this is something I've already fixed.

failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

This error message is a trivial runtime message caused by a script error, so you don't have to care about it, but now that I think about it, it's a bit verbose for a user 😅

Footnotes

1. https://supabase.com/docs/guides/functions/debugging#limitations ↩

That is good to know. Thanks for the update!!

As for the runtime, here is what I have:

• Deno - v1.41.3
• Supabase - 1.145.4

I hope that is what you need. If you need anything else. let me know.

[nyannyacha]

Your version of Deno is different from the version of Deno that edge-runtime is using internally. I assume the version of Deno you are mentioning is a standalone.

The actual version of edge-runtime included in the 1.145.4 you mentioned is v1.36.1^1 The version of Deno it is using is v1.40.3^3.

It's pretty complicated, isn't it?😅 I think this needs to be documented at some point. And it looks to me like you're using a version prior to the one that fixed the bug (that should be fixed at edge-runtime v1.38.0 and above).

You can upgrade the CLI version to beta to bring edge-runtime up to date^4.

---

[CalvinWilkinson]

Your version of Deno is different from the version of Deno that edge-runtime is using internally. I assume the version of Deno you are mentioning is a standalone.

The actual version of edge-runtime included in the 1.145.4 you mentioned is v1.36.112 The version of Deno it is using is v1.40.33.

It's pretty complicated, isn't it?😅 I think this needs to be documented at some point. And it looks to me like you're using a version prior to the one that fixed the bug (that should be fixed at edge-runtime v1.38.0 and above).

You can upgrade the CLI version to beta to bring edge-runtime up to date4.

Footnotes

1. https://github.com/supabase/cli/blob/v1.145.4/internal/utils/misc.go#L37 ↩
2. https://github.com/supabase/edge-runtime/releases/tag/v1.36.1 ↩
3. https://github.com/supabase/edge-runtime/blob/v1.36.1/crates/cli/.env.build ↩
4. https://github.com/supabase/cli/releases ↩

That makes sense to me!!

Indeed I am running a separately installed version of deno due to other projects. Thanks for the update!!

[ManoloTonto1]

I am also running into the same Issue using transformers.js

import { layer_norm, pipeline } from "npm:@huggingface/transformers";

export async function createEmbeddings(content: string): Promise<number[]> {
    const pipe = await pipeline(
        "feature-extraction",
        "nomic-ai/nomic-embed-text-v1.5",
        {
            // quantized: false
        }
    );
    const tensors = await pipe(content, { pooling: "mean" });
    const matryoshka_dim = 512;
    const embeddings = layer_norm(tensors, [tensors.dims[1]])
        .slice(null, [0, matryoshka_dim])
        .normalize(2, -1);
    return embeddings.tolist() as number[];
}


Deno.serve(async (req) => {
 ....
 
     for (const row of rows) {
        const { id, [contentColumn]: content } = row;

        if (!content) {
            console.error(`No content available in column '${contentColumn}'`);
            continue;
        }

        const output = await createEmbeddings(content);
        const embedding = JSON.stringify(output);

        const { error } = await supabase
            .from(table)
            .update({
                [embeddingColumn]: embedding,
            })
            .eq("id", id);

        if (error) {
            console.error(
                `Failed to save embedding on '${table}' table with id ${id}`,
                error
            );
            continue;
        }
    }
 })