pgloader
pgloader copied to clipboard
ERROR A thread failed with error: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Using docker:
sonarqube_db = mysql docker container
db_postgres = postgresql docker container
pgloader_ccl:latest = pgloader docker container build with Dockerfile.ccl
Running pgloader finished (after ~ 5 min) with ERROR
docker run --security-opt seccomp=unconfined --rm --net net_postgres --name pgloader pgloader_ccl:latest pgloader --debug mysql://root:mysqlroot4u@sonarqube_db:3306/sonar postgresql://sonar:sonar@db_postgres:5432/sonar
2019-07-09T08:37:10.002255Z LOG pgloader version "3.6.2~devel"
2019-07-09T08:37:10.017239Z LOG Migrating from #<MYSQL-CONNECTION mysql://root@sonarqube_db:3306/sonar #x3020014F257D>
2019-07-09T08:37:10.017593Z LOG Migrating into #<PGSQL-CONNECTION pgsql://sonar@db_postgres:5432/sonar #x3020014EAAFD>
2019-07-09T08:42:46.032796Z ERROR A thread failed with error: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
2019-07-09T08:42:46.036826Z ERROR Date/time: 2019-07-09-08:42An unhandled error condition has been signalled: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
(7F1C255FA200) : 0 (PRINT-CALL-HISTORY :CONTEXT NIL :PROCESS NIL :ORIGIN NIL :DETAILED-P NIL :COUNT 1152921504606846975 :START-FRAME-NUMBER 0 :STREAM #<STRING-OUTPUT-STREAM #x30201A909B7D> :PRINT-LEVEL 2 :PRINT-LENGTH 5 :PRINT-STRING-LENGTH :DEFAULT :SHOW-INTERNAL-FRAMES NIL :FORMAT :TRADITIONAL) 941
(7F1C255FA398) : 1 (PRINT-BACKTRACE-TO-STREAM #<STRING-OUTPUT-STREAM #x30201A909B7D>) 85
(7F1C255FA3C8) : 2 (PRINT-BACKTRACE #<QMYND:PARTIAL-READ #x30201A90A48D> :OUTPUT NIL :IF-EXISTS :APPEND :VERBOSE NIL) 797
(7F1C255FA460) : 3 (FUNCALL #'#<(:INTERNAL (PGLOADER.LOAD:COPY-FROM (PGLOADER.SOURCES:COPY)))> #<QMYND:PARTIAL-READ #x30201A90A48D>) 197
(7F1C255FA488) : 4 (CONDITION-HANDLER #<QMYND:PARTIAL-READ #x30201A90A48D>) 493
(7F1C255FA500) : 5 (SIGNAL #<QMYND:PARTIAL-READ #x30201A90A48D>) 989
(7F1C255FA558) : 6 (%ERROR #<QMYND:PARTIAL-READ #x30201A90A48D> NIL 17469857854640) 117
(7F1C255FA580) : 7 (READ-WHOLE-CHUNK 1000 #<MY-PACKET-STREAM 0/1000 [241]>) 317
(7F1C255FA5B0) : 8 (PREPARE-NEXT-CHUNK #<MY-PACKET-STREAM 0/1000 [241]>) 701
(7F1C255FA5E8) : 9 (FUNCALL #'#<#<STANDARD-METHOD QMYND-IMPL:MYSQL-CONNECTION-READ-PACKET (QMYND:MYSQL-BASE-CONNECTION)>> #<QMYND-IMPL::MYSQL-INET-CONNECTION #x302002EDA95D>) 133
(7F1C255FA608) : 10 (PARSE-RESULTSET-ROW-AS-TEXT-VECTOR 37 #(#S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) ...)) 45
(7F1C255FA648) : 11 (MAP-RESULTSET-ROWS #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002EDAD2F> 37 #(#S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) #S(QMYND-IMPL::COLUMN-DEFINITION-V41-PACKET :CATALOG "def" :SCHEMA "sonar" ...) ...) :AS-TEXT T :RESULT-TYPE VECTOR) 405
(7F1C255FA6B8) : 12 (SEND-COMMAND-QUERY "SELECT `id`, `name`, `description`, `enabled`, `scope`, `qualifier`, `kee`, `language`, `long_name`, `created_at`, `path`, `deprecated_kee`, `uuid`, `project_uuid`, `module_uuid`, `module_uuid_path`, `authorization_updated_at`, `root_uuid`, `copy_component_uuid`, `developer_uuid`, `uuid_path`, `b_changed`, `b_copy_component_uuid`, `b_description`, `b_enabled`, `b_language`, `b_long_name`, `b_module_uuid`, `b_module_uuid_path`, `b_name`, `b_path`, `b_qualifier`, `b_uuid_path`, `organization_uuid`, `tags`, `private`, `main_branch_project_uuid` FROM `projects`" :ROW-FN #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002EDAD2F> :AS-TEXT T :RESULT-TYPE VECTOR) 1149
(7F1C255FA738) : 13 (MYSQL-QUERY #<QMYND-IMPL::MYSQL-INET-CONNECTION #x302002EDA95D> "SELECT `id`, `name`, `description`, `enabled`, `scope`, `qualifier`, `kee`, `language`, `long_name`, `created_at`, `path`, `deprecated_kee`, `uuid`, `project_uuid`, `module_uuid`, `module_uuid_path`, `authorization_updated_at`, `root_uuid`, `copy_component_uuid`, `developer_uuid`, `uuid_path`, `b_changed`, `b_copy_component_uuid`, `b_description`, `b_enabled`, `b_language`, `b_long_name`, `b_module_uuid`, `b_module_uuid_path`, `b_name`, `b_path`, `b_qualifier`, `b_uuid_path`, `organization_uuid`, `tags`, `private`, `main_branch_project_uuid` FROM `projects`" :ROW-FN #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002EDAD2F> :AS-TEXT T :RESULT-TYPE VECTOR) 285
(7F1C255FA7C0) : 14 (%%CHECK-KEYWORDS #(2 #(:RESULT-TYPE :AS-TEXT :ROW-FN) #<METHOD-FUNCTION PGLOADER.CONNECTION:QUERY #>) 17469857854724) 229
(7F1C255FA868) : 15 (CALL-WITH-ENCODING-HANDLER #<PGLOADER.SOURCE.MYSQL:COPY-MYSQL #x3020016BB46D> "projects" #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002ED93DF>) 693
(7F1C255FA8D0) : 16 (FUNCALL #'#<#<STANDARD-METHOD PGLOADER.SOURCES:MAP-ROWS (PGLOADER.SOURCE.MYSQL:COPY-MYSQL)>> #<PGLOADER.SOURCE.MYSQL:COPY-MYSQL #x3020016BB46D> :PROCESS-ROW-FN #<COMPILED-LEXICAL-CLOSURE (:INTERNAL #) #x302002EDAD2F>) 2813
(7F1C255FA9A8) : 17 (%%CHECK-KEYWORDS #(1 #(:PROCESS-ROW-FN) #<METHOD-FUNCTION PGLOADER.SOURCES:MAP-ROWS #>) 17469857854784) 229
(7F1C255FAA20) : 18 (FUNCALL #'#<#<STANDARD-METHOD PGLOADER.LOAD:QUEUE-RAW-DATA (PGLOADER.SOURCES:COPY T T)>> #<PGLOADER.SOURCE.MYSQL:COPY-MYSQL #x3020016BB46D> #S(LPARALLEL.VECTOR-QUEUE:VECTOR-QUEUE :IMPL #S(LPARALLEL.VECTOR-QUEUE::RAW-QUEUE :DATA # :START 1382 ...) :LOCK #<RECURSIVE-LOCK "Anonymous lock" [ptr @ #x7F1C20037C40] #x3020016BBBFD> ...) 1) 725
(7F1C255FAA70) : 19 (FUNCALL #'#<(:INTERNAL #:BODY-FN64274 LPARALLEL.KERNEL::MAKE-CHANNELED-TASK)>) 245
(7F1C255FAAD8) : 20 (FUNCALL #'#<(:INTERNAL LPARALLEL.KERNEL::MAKE-CHANNELED-TASK)>) 117
(7F1C255FAB28) : 21 (EXEC-TASK/WORKER (#<COMPILED-LEXICAL-CLOSURE # #x3020016BBB3F> . :DEFAULT) #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033400] #x3020014C54AD> ...) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033480] #x3020014C53DD> ...) ...)) 197
(7F1C255FAB90) : 22 (WORKER-LOOP #<KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T #x3020014C6CAD> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033400] #x3020014C54AD> ...) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033480] #x3020014C53DD> ...) ...)) 237
(7F1C255FAC00) : 23 (%CALL-WITH-TASK-HANDLER #<COMPILED-LEXICAL-CLOSURE (:INTERNAL LPARALLEL.KERNEL::ENTER-WORKER-LOOP) #x3020014CF82F>) 789
(7F1C255FACA8) : 24 (FUNCALL #'#<(:INTERNAL LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT)>) 869
(7F1C255FAD28) : 25 (FUNCALL #<COMPILED-LEXICAL-CLOSURE (:INTERNAL LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT) #x3020014CF77F>) 149
(7F1C255FAD58) : 26 (CALL-WITH-WORKER-CONTEXT #<COMPILED-LEXICAL-CLOSURE (:INTERNAL LPARALLEL.KERNEL::ENTER-WORKER-LOOP) #x3020014CF82F> #<Compiled-function FUNCALL #x30000007FEFF> #<KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T #x3020014C6CAD> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033400] #x3020014C54AD> ...) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL # :LOCK #<# # [ptr @ #x7F1C20033480] #x3020014C53DD> ...) ...)) 341
(7F1C255FADC0) : 27 (FUNCALL #'#<(:INTERNAL LPARALLEL.KERNEL::MAKE-WORKER-THREAD)>) 157
(7F1C255FAE18) : 28 (FUNCALL #'#<(:INTERNAL BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS)>) 797
(7F1C255FAE60) : 29 (RUN-PROCESS-INITIAL-FORM #<PROCESS lparallel(8) [Active] #x3020014C4C3D> (#<COMPILED-LEXICAL-CLOSURE # #x3020014C49CF>)) 709
(7F1C255FAEF0) : 30 (FUNCALL #'#<(:INTERNAL (CCL::%PROCESS-PRESET-INTERNAL (PROCESS)))> #<PROCESS lparallel(8) [Active] #x3020014C4C3D> (#<COMPILED-LEXICAL-CLOSURE # #x3020014C49CF>)) 573
(7F1C255FAF98) : 31 (FUNCALL #'#<(:INTERNAL CCL::THREAD-MAKE-STARTUP-FUNCTION)>) 277
KABOOM!
FATAL error: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
An unhandled error condition has been signalled: MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
What I am doing here?
MySQL ERROR: Partial Read of 769 bytes, expected 1000
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
Hi, did you check your MySQL logs? did you have timeout error messages there? did you try adjusting the parameters as instructed: you can do that from the pgloader command itself.
Detail: check MySQL logs for (Got timeout writing communication packets) Hint: adjust net_read_timeout and net_write_timeout
I have a similiar problem
lock" (free)> :NOTIFY-PUSH NIL :NOTIFY-POP #<SB-THREAD:WAITQUEUE Anonymous condition variable {10173D1DB3}>) 1) [fast-method]
16: ((FLET "BODY-FN0" :IN LPARALLEL.KERNEL::MAKE-CHANNELED-TASK))
17: ((LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-CHANNELED-TASK))
18: (LPARALLEL.KERNEL::EXEC-TASK/WORKER #<unavailable argument> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1="Anonymous lock" (free)> :CVAR #<SB-THREAD:WAITQUEUE #2=Anonymous condition variable {1005A8AF93}>) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR #<SB-THREAD:WAITQUEUE #2# {1005A8AFF3}>) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR NIL) :THREAD #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}> :RUNNING-CATEGORY :DEFAULT :INDEX 1 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #3=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #3#)))
19: (LPARALLEL.KERNEL::WORKER-LOOP #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {1006AB1833}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1="Anonymous lock" (free)> :CVAR #<SB-THREAD:WAITQUEUE #2=Anonymous condition variable {1005A8AF93}>) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR #<SB-THREAD:WAITQUEUE #2# {1005A8AFF3}>) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR NIL) :THREAD #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}> :RUNNING-CATEGORY :DEFAULT :INDEX 1 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #3=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #3#)))
20: (LPARALLEL.KERNEL::%CALL-WITH-TASK-HANDLER #<unavailable argument>)
21: ((LAMBDA NIL :IN LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT))
22: (LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT #<CLOSURE (LAMBDA NIL :IN LPARALLEL.KERNEL::ENTER-WORKER-LOOP) {1006527E6B}> #<FUNCTION FUNCALL> #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {1006AB1833}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1="Anonymous lock" (free)> :CVAR #<SB-THREAD:WAITQUEUE #2=Anonymous condition variable {1005A8AF93}>) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR #<SB-THREAD:WAITQUEUE #2# {1005A8AFF3}>) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR NIL) :THREAD #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}> :RUNNING-CATEGORY :DEFAULT :INDEX 1 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #3=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #3#)))
23: ((LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-WORKER-THREAD))
24: ((LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS))
25: ((FLET SB-UNIX::BODY :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
26: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
27: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
28: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::CALL-WITH-MUTEX))
29: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE) {7FFFF4DBED4B}> #<SB-THREAD:MUTEX "thread result lock" owner: #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}>> NIL T NIL)
30: (SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE #<SB-THREAD:THREAD "lparallel" RUNNING {1006AB2263}> NIL #<CLOSURE (LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS) {1006AB220B}> NIL)
31: ("foreign function: call_into_lisp")
32: ("foreign function: new_thread_trampoline")
2020-04-18T20:30:30.698000Z INFO Stopping monitor
What I am doing here?
MySQL ERROR: Partial Read of 157 bytes, expected 240
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
This happens when I load from local MySQL to AWS RDS Aurora Postgresql. I have changed net_read_timeout and net_write_timeout to 600 but still not working.
same problem
20: (LPARALLEL.KERNEL::%CALL-WITH-TASK-HANDLER #<unavailable argument>)
21: ((LAMBDA NIL :IN LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT))
22: (LPARALLEL.KERNEL::CALL-WITH-WORKER-CONTEXT #<CLOSURE (LAMBDA NIL :IN LPARALLEL.KERNEL::ENTER-WORKER-LOOP) {10077AFDCB}> #<FUNCTION FUNCALL> #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 4 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {1007D7A513}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1="Anonymous lock" (free)> :CVAR #<SB-THREAD:WAITQUEUE #2=Anonymous condition variable {100715DB23}>) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR #<SB-THREAD:WAITQUEUE #2# {10077AFDE3}>) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK #<SB-THREAD:MUTEX #1# (free)> :CVAR NIL) :THREAD #<SB-THREAD:THREAD "lparallel" RUNNING {1007D7AF53}> :RUNNING-CATEGORY :DEFAULT :INDEX 1 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD (LPARALLEL.SPIN-QUEUE::DUMMY (#<CLOSURE #3=(LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-CHANNELED-TASK) {100728ED3B}> . :DEFAULT) (#<CLOSURE #3# {100728ED6B}> . :DEFAULT) (#<CLOSURE #3# {100728ED9B}> . :DEFAULT) (#<CLOSURE #3# {100728EDCB}FAULT) (#<CLOSURE #3# {100728EEEB}> . :DEFAULT) (#<CLOSURE #3# {100728EF1B}> . :DEFAULT) (#<CLOSURE #3# {100728EF4B}> . :DEFAULT) (#<CLOSURE #3# {100728EF7B}> . :DEFAULT) (#<CLOSURE #3# {100728EFAB}> . :DEFAULT) (#<CLOSURE #3# {100728EFDB}> . :DEFAULT) (#<CLOSURE #3# {100728F00B}> . :DEFAULT) (#<CLOSURE #3# {100728F03B}> . :DEFAULT) (#<CLOSURE #3# {100728F06B}> . :DEFAULT) (#<CLOSURE #3# {100728F09B}> . :DEFAULT) (#<CLOSURE #3# {100728F0CB}> . :DEFAULT) . #4=((#<CLOSURE #3# {100728F0FB}> . :DEFAULT))) :TAIL #4#)))23: ((LAMBDA NIL :IN LPARALLEL.KERNEL::MAKE-WORKER-THREAD))
24: ((LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS))
25: ((FLET "WITHOUT-INTERRUPTS-BODY-4" :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
26: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE))
27: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::CALL-WITH-MUTEX))
28: (SB-THREAD::CALL-WITH-MUTEX #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK :IN SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE) {7FFFEDE9ED2B}> #<SB-THREAD:MUTEX "thread result lock" owner: #<SB-THREAD:THREAD "lparallel" RUNNING {1007D7AF53}>> NIL T NIL)29: (SB-THREAD::INITIAL-THREAD-FUNCTION-TRAMPOLINE #<SB-THREAD:THREAD "lparallel" RUNNING {1007D7AF53}> NIL #<CLOSURE (LAMBDA NIL :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS) {1007D7AEFB}> (#<SB-THREAD:THREAD #1="lparallel" RUNNING {1007D7AF53}> #<SB-THREAD:THREAD #1# RUNNING {1007D7AA03}> #<SB-THREAD:THREAD #1# RUNNING {1006C70163}> #<SB-THREAD:THREAD "main thread" RUNNING {1006C3E3C3}>) NIL NIL NIL NIL)30: ("foreign function: call_into_lisp")
31: ("foreign function: new_thread_trampoline")
What I am doing here?
MySQL ERROR: Partial Read of 9 bytes, expected 49
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
I'm facing the same issue and I'm running pgloader
over a local UNIX socket connection to MySQL. The command:
pgloader --client-min-messages info migration-script.load
with migration-script.load
containing:
LOAD DATABASE
FROM mysql:/my_user:password@unix:/var/run/mysqld/mysqld.sock:/source_db
INTO postgresql://pg_user:password@pg_db_server:pg_port/target_db?sslmode=require
WITH no truncate, create tables, include drop, create indexes, reset sequences,
foreign keys, downcase identifiers, uniquify index names, workers = 8, concurrency = 2,
multiple readers per thread, rows per range = 20000
SET MySQL PARAMETERS net_read_timeout = '1800', net_write_timeout = '1800';
As you can see, I've tried increasing net_read_timeout
and net_write_timeout
to 1800
starting from 600
and I still keep getting errors like this:
2022-02-15T16:39:20.987008+05:30 ERROR A thread failed with error: MySQL ERROR: Partial Read of 76 bytes, expected 145
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
2022-02-15T16:39:20.993859+05:30 ERROR A thread failed with error: Unexpected end of file on #<FUNDAMENTAL-FILE-SOCKET-STREAM :UTF-8 (SOCKET/5) #x302014F007FD>
2022-02-15T16:39:20.994096+05:30 ERROR A thread failed with error: MySQL ERROR: Partial Read of 40 bytes, expected 153
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
2022-02-15T16:39:21.023208+05:30 ERROR Date/time: 2022-02-15-16:39!
An unhandled error condition has been signalled: MySQL ERROR: Partial Read of 40 bytes, expected 153
Detail: check MySQL logs for (Got timeout writing communication packets)
Hint: adjust net_read_timeout and net_write_timeout
I've compiled pgloader
using CCL with a DYNSIZE
set to 8192.
$ pgloader --version
pgloader version "3.6.2"
compiled with Clozure Common Lisp Version 1.12.1 (v1.12.1) LinuxX8664
I tried running originally without the workers
, concurrency
, multiple readers per thread
and rows_per_range
parameters but that kept failing with the same error too (it ran faster and broke earlier).
I'm not tool familiar with MySQL internals and was looking for a simple way to migrate my fairly large MySQL database (~207GB) to Postgres but at this point, having spent more than a week without making much headway, I'm almost ready to give up on pgloader
. I'm unsure of what parameters and settings to tune apart from retrying with a larger net_read_timeout
and net_write_timeout
as suggested in the log.
@dimitri Any help here would be greatly appreciated.
I got things to work for migrating a 50GB DB from MySQL to Postgres with the following requirements:
- Provisioned IOPS for source/destination
- c6i.4xlarge machines (16 CPUs, 32GB)
- pgloader compiled with CCL option
I did use SET MySQL PARAMETERS net_read_timeout = '1800', net_write_timeout = '1800'
as well but I noticed the EC2 machine running pgloader was capping out on CPU. I was able to get the job using the default concurrency options to complete in 10 hours
.
Still stucked in this issue.
I'm trying to move over a 1.2TB (yes, TB) database from MySQL and hitting this same error. I wonder if this will ever work.
stucked in this issue too
I was stuck with same and/or similar issues. I was not able to convert a large mariaDB database to postgreSQL and wasted days. Neither on Intel-Mac, M1-Mac nor on any docker container i was able to convert data reliably, trying out many pgloader options. Only about half of my attempts were successful, which was not enough for the big bang day with production data.
The only solution which successfully and reliably worked 100% was renting a cloud root server with Debian 11 and converting the database there. This cost a few bucks, but is no relation to the hours spent in debugging why pgloader crashes