Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local Edge functions: event loop error after a few minutes #206

Open
ChuckJonas opened this issue Nov 8, 2023 · 18 comments
Open

Local Edge functions: event loop error after a few minutes #206

ChuckJonas opened this issue Nov 8, 2023 · 18 comments
Assignees

Comments

@ChuckJonas
Copy link

ChuckJonas commented Nov 8, 2023

Describe the bug

It seems that after my edge function has been running locally for some time (perhaps a few minutes) I start to get these messages in the logs:

wall clock duration warning. isolate: df3c6dc8-dcc9-4348-a278-e0e1ff102fc3
wall clock duration warning. isolate: 93b745a9-cdb0-49a0-ae88-514a87391cd6
wall clock duration warning. isolate: ce00ba14-d7f7-4e27-8b6e-92191f485a81
wall clock duration warning. isolate: 39ebddfd-8b3e-4af6-a966-178f821877b6

And then eventually I see this:

event loop error: TypeError: Cannot convert undefined or null to object
    at slice (<anonymous>)
    at new DOMException (ext:deno_web/01_dom_exception.js:126:14)
    at [[[signalAbort]]] (ext:deno_web/03_abort_signal.js:83:14)
    at AbortController.abort (ext:deno_web/03_abort_signal.js:172:30)
    at file:///home/deno/main/index.ts:147:31
    at Object.action (ext:deno_web/02_timers.js:151:11)
    at handleTimerMacrotask (ext:deno_web/02_timers.js:65:10)
    at eventLoopTick (ext:core/01_core.js:189:21)

After that error happens, all my requests fail with this:

Error in worker connection: connection closed before message completed
request failed (uri: "/my-function" reason: hyper::Error(Canceled, "connection was not ready"))

The function is as simple as this:

Deno.serve(async (req) => {
  return new Response(
    JSON.stringify({ publicToken: "123" }),
    {
      headers: { "Content-Type": "application/json" },
    }
  );
});

To Reproduce
Steps to reproduce the behavior:

  1. Create a new function with the above code
  2. run supabase functions serve
  3. Call the function at least once (not sure if this is actually necessary)
  4. Wait a few minutes... You should eventually see the above event loop error
  5. Try to call the function again. You will get "connection was not ready" error

Expected behavior
Local functions should not randomly crash after only a few minutes :)

Desktop (please complete the following information):

  • OS: OSX
  • Version of CLI: 1.110.1

Additional context

At least one other person has reported experiencing this problem in the supabase discord (The above report is using their description).

@ChuckJonas
Copy link
Author

ChuckJonas commented Nov 8, 2023

I rolled back to 1.106.1 and the issue seem to go away. I still see the wall clock duration warning. isolate: 88c59c58-1271-4712-ae7e-863ecb2449fa errors, but it no longer crashes (so maybe those aren't related to the crash)

@sweatybridge sweatybridge transferred this issue from supabase/cli Nov 9, 2023
@sweatybridge
Copy link

Transferring because it sounds like an issue with edge runtime container.

@scottweinert
Copy link

I can also verify that I'm experiencing the same issue.

@confusedmatrix
Copy link

Seeing the same issue also

@evelant
Copy link

evelant commented Nov 10, 2023

I'm also seeing this issue, can't update supabase beyond 1.106.1 or edge fns break every few minutes.

@rbkayz
Copy link

rbkayz commented Nov 15, 2023

yep. same

@d-e-h-i-o
Copy link

I'm also experiencing this issue with CLI version 1.110.1

@hyldmo
Copy link

hyldmo commented Nov 16, 2023

I had this is issue too on my M2 mac until i upgraded Docker Desktop (it was very much out of date) as well as supabase. Now running without a single crash on supabase 1.112.2 and Docker Desktop 4.25.1 (128006). Still seeing the wall clock duration warning though, but no crashes.

@d-e-h-i-o
Copy link

I had this is issue too on my M2 mac until i upgraded Docker Desktop (it was very much out of date) as well as supabase. Now running without a single crash on supabase 1.112.2 and Docker Desktop 4.25.1 (128006). Still seeing the wall clock duration warning though, but no crashes.

My Docker Desktop version is 4.19.0 (106363

@scottweinert
Copy link

Thanks @hyldmo. Updating Docker Desktop and my supabase CLI resolved the crashing issue.

@rothnic
Copy link

rothnic commented Nov 29, 2023

Any comments on whether the wall clock duration reached. isolate: 11455258470253939329 logs are a problem? It kind of seems like the connection isn't being closed out or something, but the response it getting returned from the endpoint. Is there something else we should be doing to tell supabase that the request is complete?

@andreespirela andreespirela self-assigned this Jan 18, 2024
@all-mute
Copy link

same

@SebasScript
Copy link

same issue here, i don't get an error, but all cloud functions become very slow or irresponsive after a couple requests. Seems like the workers don't exit execution. If i restart the runtime it works again for a couple of request

@CalvinWilkinson
Copy link

I as well am getting related issues.

I get wall clock log msgs like these:

wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped

I also am having an issue where I cannot use the existsSync() and Deno.readDirSync() functions.
When running these, I get the following error:

runtime has escaped from the event loop unexpectedly: event loop error: TypeError: Deno.statSync is not a function
    at existsSync (https://deno.land/[email protected]/fs/exists.ts:121:23)
    at EnvService.dirDoesNotExist (file:///home/deno/functions/instructor-ui/services/env-service.ts:100:13)
    at EnvService.validateInitEnvFile (file:///home/deno/functions/instructor-ui/services/env-service.ts:65:14)
    at EnvService.loadEnv (file:///home/deno/functions/instructor-ui/services/env-service.ts:22:48)
    at file:///home/deno/functions/instructor-ui/index.ts:5:12
failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

I have tried to search for any open or closed issues on this to see if I can get some more info but have come up dry.

@nyannyacha
Copy link
Collaborator

nyannyacha commented Mar 18, 2024

hello @CalvinWilkinson 😁

This runtime provides a limited File API compared to Deno 1, and that's a relatively recent change. From a security perspective, allowing file access can be problematic.

As for the error message... that's something that needs to be documented for self-hosted users 🙄

wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped

Do you mind if I ask what version of the runtime you're getting this message from? I remember this is something I've already fixed.

failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

This error message is a trivial runtime message caused by a script error, so you don't have to care about it, but now that I think about it, it's a bit verbose for a user 😅


Footnotes

@CalvinWilkinson
Copy link

hello @CalvinWilkinson 😁

This runtime provides a limited File API compared to Deno 1, and that's a relatively recent change. From a security perspective, allowing file access can be problematic.

As for the error message... that's something that needs to be documented for self-hosted users 🙄

wall clock duration reached. isolate: d66d7171-0472-48db-a589-a52454d89a26
isolate memory usage sender dropped
wall clock duration warning. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
wall clock duration reached. isolate: 476ff220-04b9-41ee-a5dc-7db65464616a
isolate memory usage sender dropped

Do you mind if I ask what version of the runtime you're getting this message from? I remember this is something I've already fixed.

failed to send request to user worker: connection error: Connection reset by peer (os error 104)
user worker failed to respond: connection error: Connection reset by peer (os error 104)
InvalidWorkerResponse: user worker failed to respond
    at async Promise.allSettled (index 1)
    at async UserWorker.fetch (ext:sb_user_workers/user_workers.js:79:21)
    at async Server.<anonymous> (file:///home/deno/main/index.ts:146:12)
    at async #respond (https://deno.land/[email protected]/http/server.ts:220:18) {
  name: "InvalidWorkerResponse"
}

This error message is a trivial runtime message caused by a script error, so you don't have to care about it, but now that I think about it, it's a bit verbose for a user 😅

Footnotes

  1. https://supabase.com/docs/guides/functions/debugging#limitations

That is good to know. Thanks for the update!!

As for the runtime, here is what I have:

  • Deno - v1.41.3
  • Supabase - 1.145.4

I hope that is what you need. If you need anything else. let me know.

@nyannyacha
Copy link
Collaborator

Your version of Deno is different from the version of Deno that edge-runtime is using internally.
I assume the version of Deno you are mentioning is a standalone.

The actual version of edge-runtime included in the 1.145.4 you mentioned is v1.36.112 The version of Deno it is using is v1.40.33.

It's pretty complicated, isn't it?😅 I think this needs to be documented at some point.
And it looks to me like you're using a version prior to the one that fixed the bug (that should be fixed at edge-runtime v1.38.0 and above).

You can upgrade the CLI version to beta to bring edge-runtime up to date4.


Footnotes

  1. https://github.com/supabase/cli/blob/v1.145.4/internal/utils/misc.go#L37

  2. https://github.com/supabase/edge-runtime/releases/tag/v1.36.1

  3. https://github.com/supabase/edge-runtime/blob/v1.36.1/crates/cli/.env.build

  4. https://github.com/supabase/cli/releases

@CalvinWilkinson
Copy link

Your version of Deno is different from the version of Deno that edge-runtime is using internally. I assume the version of Deno you are mentioning is a standalone.

The actual version of edge-runtime included in the 1.145.4 you mentioned is v1.36.112 The version of Deno it is using is v1.40.33.

It's pretty complicated, isn't it?😅 I think this needs to be documented at some point. And it looks to me like you're using a version prior to the one that fixed the bug (that should be fixed at edge-runtime v1.38.0 and above).

You can upgrade the CLI version to beta to bring edge-runtime up to date4.

Footnotes

  1. https://github.com/supabase/cli/blob/v1.145.4/internal/utils/misc.go#L37
  2. https://github.com/supabase/edge-runtime/releases/tag/v1.36.1
  3. https://github.com/supabase/edge-runtime/blob/v1.36.1/crates/cli/.env.build
  4. https://github.com/supabase/cli/releases

That makes sense to me!!

Indeed I am running a separately installed version of deno due to other projects.
Thanks for the update!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests