Sadjad Fouladi

Hi @drunksaint, >the time the full job took to run was almost 3 minutes. It should have taken a few seconds plus download of around 200 MB from s3 to...

Another point I missed: make sure your AWS account actually has the capacity to run 2,000 parallel Lambdas. The red number in the status bar shows the number of running...

Could you please measure the time necessary for forcing just *one* thunk on lambda? You can run `gg force` with `--no-download` to skip downloading and just time the whole command...

Hi @drunksaint -- apologies for my delayed response. >Is it possible to debug these error messages? I see the ChildProcess and 404 messages quite frequently. The `ChildProcess` error is something...

Hi @drunksaint, My best guess is that the EC2 machine runs out of memory, and then the process is killed. Could you please check if that is the case? --Sadjad

That's still seems too much... I wonder if we're hitting some limits within S3. Would you be willing to get me some timing logs from an actual run? Here are...

Hi @drunksaint, Thank you for getting the data! So, `get_dependencies` doesn't look *too* bad, but `do_cleanup` seems to be much worse (worst case is about 25s, which is unreasonable). Let's...

Hello @drunksaint, >I thought the do_cleanup numbers were in ms XD. Oops. You are right, you're right. All numbers are in milliseconds, so `do_cleanup` is not the culprit. My apologies...

Not really. I got plenty of "rate limited" errors, but those are okay. I think I saw a few ChildProcess errors, but no 404s or 503s.

Hi @drunksaint, Actually, I'm running my script from my machine at Stanford (Northern California) to Lambdas in `us-west-2` (Oregon). I'm running Ubuntu 20.04 on an eight-core, 3.2 GHz Intel Core...