MySQL server has gone away

[matellis]

I'm trying to use taps to move a database from a local mysql instance running on an Amazon EC2 instance to an RDS instance. The DB is of medium size, about 850MB with 40,000 records in the biggest table.

Unfortunately, the system is routinely failing at the start of attempting to import this biggest table. Below is the stack trace. I imagine it's something to do with memory size? Is there some parameter I need to configure? I'm running the smallest Amazon instances for both the host and the RDS node, about 1.7GB of RAM. As we are still in dev mode there's nothing much else using memory, disk, network, etc.

The node hosting the taps server is running Mysql 5.1.37 and the destination is 5.1.45-log

Thanks

Receiving schema
Schema:        100%     |=============================================================================================================================================================| Time: 00:00:08
Receiving data
10 tables, 90,309 records
actions:       100%     |=============================================================================================================================================================| Time: 00:00:00
/var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/database.rb:822:in `query': MySQL server has gone away (Mysql::Error)                                                             | ETA:  --:--:--
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/database.rb:822:in `send'
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/database.rb:822:in `log_connection_execute'
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/database.rb:890:in `rollback_transaction'
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/database.rb:631:in `_transaction'
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/database.rb:575:in `transaction'
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/connection_pool/threaded.rb:84:in `hold'
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/database.rb:535:in `synchronize'
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/database.rb:573:in `transaction'
from /var/lib/gems/1.8/gems/sequel-3.10.0/lib/sequel/dataset/convenience.rb:129:in `import'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/data_stream.rb:284:in `import_rows'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/data_stream.rb:133:in `fetch_remote'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/operation.rb:286:in `pull_data_from_table'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/operation.rb:279:in `loop'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/operation.rb:279:in `pull_data_from_table'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/operation.rb:262:in `pull_data'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/operation.rb:256:in `each'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/operation.rb:256:in `pull_data'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/operation.rb:221:in `run'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/cli.rb:164:in `clientxfer'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/cli.rb:33:in `pull'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/cli.rb:21:in `send'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/../lib/taps/cli.rb:21:in `run'
from /var/lib/gems/1.8/gems/taps-0.3.6/bin/taps:6
from /sbin/taps:19:in `load'
from /sbin/taps:19`

[firebelly]

I'm having this same problem too, and it's just with one particular table.

[firebelly]

I eventually figured out the error message that MySQL was generating (but Sequel was hiding) was "Got a packet bigger than 'max_allowed_packet' bytes".

Setting the max_allowed_packet variable to something larger in MySQL config worked. I'm not sure if this is fixed in the latest version of Sequel or not.

[ohokay]

MySQL has a checklist for you to follow if you get this error. Check it out here: http://dev.mysql.com/doc/refman/5.1/en/gone-away.html

For me, changing my local db host from 127.0.0.1 to localhost fixed the problem.

[adamgreenhall]

For me, following firebelly's solution worked.

Specifically (for OSX), I created a config file based on one of the templates by: sudo cp /usr/local/mysql-5.5.18-osx10.6-x86_64/support-files/my-huge.cnf /etc/my.cnf and then modified the max_allowed_packet line to read:

max_allowed_packet=30M