embulk-output-jdbc icon indicating copy to clipboard operation
embulk-output-jdbc copied to clipboard

Allow sharing of temporary tables between tasks

Open civitaspo opened this issue 4 years ago • 0 comments

Hi maintainers,

I create an issue related to https://github.com/embulk/embulk-output-jdbc/issues/299.

If we create a temporary table for each task, we will end up with a lot of temporary tables when there are many tasks. Therefore, depending on the workload, the number of temporary tables may reach the limit of the number of tables created in the destination database. From what I've read in the code, it looks like it's okay to share temporary tables between tasks if we ignore the resume feature. So, I think we should create an option recycle_temp_table or share_temp_table to allow sharing of temporary tables between tasks. How do you think?

civitaspo avatar Sep 13 '21 14:09 civitaspo