Unable to export and anonymize database

[dndbari]

Hello,

I encounter errors during the anonymized dump on a magento 1.9.4.5.

STEP 1 :

mysqldump --complete-insert -uroot -proot mydb eav_entity_type eav_attribute | ./dbanon -config=./magento1.yml map-eav > ./magento1-mapped.yml

STEP2 :

mysqldump --complete-insert -uroot -proot mydb | ./dbanon -log-level=trace -log-file=./dbanonlog.log -config=./magento1-mapped.yml | gzip > mydb.sql.gz

ERROR:

goroutine 1 [running]: github.com/mpchadwick/dbanon/src.LineProcessor.processInsert(0x16eef36, 0x9, 0xc0002f6ea0, 0x17bb220, 0xc000124188, 0xc000334580, 0xc00067c000, 0xe762d, 0xe762d, 0xe762d) /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:32 +0x1086 github.com/mpchadwick/dbanon/src.LineProcessor.ProcessLine(0x16eef36, 0x9, 0xc0002f6ea0, 0x17bb220, 0xc000124188, 0xc000334580, 0xc00067c000, 0xe762d, 0xc0005565c0, 0x35) /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:22 +0x9b main.main() /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/main.go:112 +0x706 mysqldump: Got errno 32 on write

I tried on a magento 2.3.6 and I get the same error

STEP 1 :

mysqldump -uroot -proot mydbm2 eav_entity_type eav_attribute | dbanon -config=magento2 map-eav > ./magento2-mapped.yml

STEP2 :

mysqldump --complete-insert -uroot -proot mydbm2 | ./dbanon -log-level=trace -log-file=./dbanonlog.log -config=./magento2-mapped.yml | gzip > mydb.sql.gz

ERROR:

goroutine 1 [running]: github.com/mpchadwick/dbanon/src.LineProcessor.processInsert(0xaedb67, 0x9, 0xc0002b4e70, 0xbb93c0, 0xc00000e198, 0xc0002ee930, 0xc0010b4000, 0x15bc, 0x15bc, 0x15bc) /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:32 +0x1086 github.com/mpchadwick/dbanon/src.LineProcessor.ProcessLine(0xaedb67, 0x9, 0xc0002b4e70, 0xbb93c0, 0xc00000e198, 0xc0002ee930, 0xc0010b4000, 0x15bc, 0xc00007de40, 0x3d) /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/src/processor.go:22 +0x9b main.main() /Users/maxchadwick/go/src/github.com/mpchadwick/dbanon/main.go:112 +0x706 mysqldump: Got errno 32 on write

For the M2 and the M1, I ran the command on MySQL 5.7 docker container

This error occurs when exporting the core_session table. I dumped the "core_session" table and got the same error when exporting the "rating_option_vote" table with the "remote_ip_long" column. I set the "remote_ip_long" column to null and the export now fails on the "sales_payment_transaction" table.

I think the anonymization script does not take into account serialized variables and special characters Once I empty all the problematic tables, the script completes fine but no data is anonymized.

[mpchadwick]

Would it be possible to share the contents from your mysqldump file that causes this to reproduce? Understand sharing a full dump file could definitely have some privacy concerns but if the specific line(s) where it's choking doesn't have any sensitive data I'd like to take a look.

[mpchadwick]

I just published and update (v0.6.2...https://github.com/mpchadwick/dbanon/releases) which will maybe help. Specifically this change

https://github.com/mpchadwick/dbanon/commit/c16d6e1e1cdbcbc0aef956eb35f4292adc19c4eb

It's hard for me to know without having a copy of the the INSERT statement that was being processed.

[mpchadwick]

Closing due to inactivity