MySqlConnector
MySqlConnector copied to clipboard
MySqlBulkCopy should have a BatchSize setting
Is your feature request related to a problem? Please describe. I saw that the MySqlBulkCopy (which is great) use a const 1MB to read from IDataReader and write to target. I think it could lead to many IOs and disk stress on high load (parallel). I saw that before chunking with 1MB there was issue with network paquet size
Describe the solution you'd like I suggest to add the possibility to change this value (1MB) using a class setting BatchSize or BatchLen that would have a 1MB default value I found the const 1MB here : https://github.com/mysql-net/MySqlConnector/blob/bbdbd782e7434b765154805b1cb61d8daac68112/src/MySqlConnector/MySqlBulkCopy.cs#L381
Describe alternatives you've considered
Additional context
This limitation was introduced by https://github.com/mysql-net/MySqlConnector/commit/feea06b7a6b499dc21299cdabaf62bafbaa7b0cc to fix https://github.com/mysql-net/MySqlConnector/issues/780.
MySQL Server 8.0 increased the default value of max_allowed_packet to 64MiB (up from 4MiB in 5.7) so this constant may be much too conservative for current servers.
A larger default size with a mechanism (such as a class property) to opt out could be a good solution.