luoyuliuyin
luoyuliuyin
I submitted a PR to support this feature, https://github.com/apache/airflow/pull/39249
> I believe this is a one time thing, do we need to expose this? kube config text may contain tokens. So, we need to secure them. It is indeed...
Regarding risk considerations, in our company I have already released this change to production. Our production environment is centos8 and it has been running well until now. Our airflow scale...
> > Here are some benefits: > > > The maximum scheduling delay between two nodes is reduced from 10 seconds to less than 1 second. > > System throughput...
> broker I use rabbitmq, AIRFLOW__CELERY__BROKER_URL=amqp://admin:[email protected]:5672
> > The above data is my running results in the Linux environment. If ProcessPoolExecutor is created every time, no problem will be seen at the beginning, but over time...
> What happens if you `kill -9` one (or all) of the ProcessPool processes and then try to schedule/enqueue new tasks? I'll test it later
> I am a little skeptical that the overhead for process pool creation is meaningful. Because i think the dominant performance consideration is the blocking IO sending the task to...
> We conducted a throughput benchmarking test, executing 15k DAG's comprised of simple BashOperator tasks. We measured the success rate of tasks completed within a 2-minute interval and observed a...
> > We conducted a throughput benchmarking test, executing 15k DAG's comprised of simple BashOperator tasks. We measured the success rate of tasks completed within a 2-minute interval and observed...