kestra icon indicating copy to clipboard operation
kestra copied to clipboard

Published 20 hours ago • verified publisher icon kestra-io

MinIO takes a long time to create 10k files

Open loicmathieu opened this issue 4 months ago • 1 comments

Describe the issue

A recent feedback was received that, when using MinIO as internal storage, a ForEachItem that creates 10k subflow executions takes more than 10 minutes to split the items.

We need to investigate if the issue is only related to MinIO and to ForEachItem.

To reproduce:

  • Have a flow with a ForEachItem based on a files with 10k lines
  • Split on batch of size 1

Environment

  • Kestra Version: 0.19.1
  • Operating System (OS/Docker/Kubernetes):
  • Java Version (if you don't run kestra in Docker):

loicmathieu avatar Oct 08 '24 11:10 loicmathieu