usql icon indicating copy to clipboard operation
usql copied to clipboard

bulk load for MySQL

Open murfffi opened this issue 1 year ago • 2 comments

The PR implements bulk loading for MySQL using the "LOAD DATA from io.Reader" feature of github.com/go-sql-driver/mysql - https://github.com/go-sql-driver/mysql?tab=readme-ov-file#load-data-local-infile-support . As expected, bulk loading this way is significantly faster. 1 mln. rows in the "staff" table from the test schema are inserted for 15 sec vs. 120 sec using INSERT: 8x improvement. Note that LOAD DATA INFILE LOCAL is disabled by default on MySQL 8+ servers and must be enabled using SET GLOBAL local_infile = ON beforehand. MySQL doesn't seem to have any remote bulk loading options that are enabled by default.

The PR also extends TestCopy in drivers_test.go with comparison of copied data to ensure MySQL bulk loading is safe across data types.

Testing Done: tests in drivers_test.go

murfffi avatar Dec 02 '24 16:12 murfffi

This is ready for review.

Background: I recently resumed working on #458 and while looking for a database to experiment with generic bulk copy, I saw that MySQL supports bulk load similar to Postgres. I decided to implement that first.

murfffi avatar Dec 05 '24 09:12 murfffi

The PR checks failure seems unrelated. The checks on master fail the same way. I got them to pass in https://github.com/xo/usql/pull/503 .

murfffi avatar Dec 07 '24 13:12 murfffi