MySqlConnector icon indicating copy to clipboard operation
MySqlConnector copied to clipboard

MySqlBulkCopy should have a BatchSize setting

Open rferraton opened this issue 1 year ago • 1 comments
trafficstars

Is your feature request related to a problem? Please describe. I saw that the MySqlBulkCopy (which is great) use a const 1MB to read from IDataReader and write to target. I think it could lead to many IOs and disk stress on high load (parallel). I saw that before chunking with 1MB there was issue with network paquet size

Describe the solution you'd like I suggest to add the possibility to change this value (1MB) using a class setting BatchSize or BatchLen that would have a 1MB default value I found the const 1MB here : https://github.com/mysql-net/MySqlConnector/blob/bbdbd782e7434b765154805b1cb61d8daac68112/src/MySqlConnector/MySqlBulkCopy.cs#L381

Describe alternatives you've considered

Additional context

rferraton avatar Mar 01 '24 14:03 rferraton

This limitation was introduced by https://github.com/mysql-net/MySqlConnector/commit/feea06b7a6b499dc21299cdabaf62bafbaa7b0cc to fix https://github.com/mysql-net/MySqlConnector/issues/780.

MySQL Server 8.0 increased the default value of max_allowed_packet to 64MiB (up from 4MiB in 5.7) so this constant may be much too conservative for current servers.

A larger default size with a mechanism (such as a class property) to opt out could be a good solution.

bgrainger avatar Mar 01 '24 18:03 bgrainger