windmill icon indicating copy to clipboard operation
windmill copied to clipboard

feature: support for async def main in python

Open smuun opened this issue 1 year ago • 3 comments

Not entirely sure how this should work or if it's a use case you want to support.

Problem:

  • I have a long-running script that takes 10 files as input; most of the time is spent waiting for network requests
  • the first step of that script is to run OCR on each file: send the file to a OCR service and wait until it's processed, then return the text; this takes e.g. 10 minutes
  • the OCR is in a separate script
  • I'm currently running everything in a main python script that imports the OCR script and asyncio.gather s the 10 files
  • this works fine, but:
  1. the first 10 minutes of runtime are spent waiting for the api calls to return, and the worker is blocked during that time even though it's just waiting on API calls
  2. I can't run the ocr script as a separate job, with wmill.run_script, which would e.g. let me take advantage of built-in cache features, without tying up an extra worker per file for 10 minutes

Proposal:

  • windmill scripts with async def main could run concurrently on one worker. The worker would run the eventloop.

smuun avatar Oct 25 '24 15:10 smuun

Also - if you do try to define a async def main() script, you get a "could not parse" error. Better error message would be "async main not allowed"

smuun avatar Oct 25 '24 15:10 smuun

Hi @pyranota and @rubenfiszel, any idea if this issue is in the works? We could really use this in something we'r building on your platform.

sisilmehta733 avatar Apr 04 '25 18:04 sisilmehta733

Possible duplicate of https://github.com/windmill-labs/windmill/issues/2935

insanity54 avatar Apr 19 '25 06:04 insanity54