dimitri
pgloader MySQL to PostgreSQL Conversion Heap Issues
I am attempting to migrate from MySQL to PostgreSQL via pgloader.
MySQL Version: 8.0.37 MySQL Community Server PostgreSQL Version: 15.12 pgloader Version: pgloader version "3.6.7~devel" compiled with SBCL 2.2.10-1.rhel9
Here is my pgloader command being run at the time of the error:
pgloader --verbose --debug --dynamic-space-size 18432 --with "prefetch rows = 500" mysql://
I have been continuously seeing the heap error so I increased my RAM and CPU on the VM temporarily in hopes it would allow it to process if I specified a higher value for --dynamic-space-size. I have been trying all sorts of combinations for --dynamic-space-size and prefetch rows but it continues to fail. The MySQL database being converted is only 3GB in size, what can I do to get this to complete properly?
This is the most recent error which does not look to be related to heap but I am not sure why it failed, any help is apprecited!
2025-07-29T21:38:28.140234+01:00 DATA < #("98213" "2020-05-01 11:18:29.000000" "2025-05-07 09:21:40.866597" "PRIORITY" "type/Decimal" NIL #1="t" NIL #1# "34" "5404" NIL "Priority" "normal" NIL "2020-05-01 12:33:03.000000" NIL NIL "{"global":{"distinct-count":2,"nil%":0.9989350372736954},"type":{"type/Number":{"min":225.0,"max":225.0,"avg":225.0,"sd":null,"q1":225.0,"q3":225.0}}}" "4" "NUMBER" NIL NIL "34" "0" "type/Decimal" NIL NIL #2="f" #2# #2# NIL NIL #2# "0") 2025-07-29T21:38:28.140234+01:00 DATA < #("98214" "2020-05-01 11:18:29.000000" "2025-05-07 09:21:41.054957" "ITERATE_PARAM_INDEX" "type/Decimal" NIL #1="t" NIL #1# "10" "5404" NIL "Iterate Param Index" "normal" NIL "2020-05-01 12:33:03.000000" NIL NIL "{"global":{"distinct-count":2,"nil%":0.001064962726304579},"type":{"type/Number":{"min":-1.0,"max":-1.0,"avg":-1.0,"sd":0.0,"q1":-1.0,"q3":-1.0}}}" "4" "NUMBER" NIL NIL "10" "0" "type/Decimal" NIL NIL #2="f" #2# #2# NIL NIL #2# "0") 2025-07-29T21:38:28.140234+01:00 DATA < #("98215" "2020-05-01 11:18:29.000000" "2025-05-07 09:21:41.099189" "PARENT_STEP_ID" "type/Decimal" NIL #1="t" NIL #1# "11" "5404" NIL "Parent Step ID" "normal" NIL "2020-05-01 12:33:03.000000" NIL NIL "{"global":{"distinct-count":513,"nil%":0.001064962726304579},"type":{"type/Number":{"min":-1.0,"max":1909979.0,"avg":949943.960554371,"sd":903830.5577293859,"q1":42596.51114246267,"q3":1885718.775465599}}}" "4" "NUMBER" NIL NIL "11" "0" "type/Decimal" NIL NIL #2="f" #2# #2# NIL NIL #2# "0") 2025-07-29T21:38:28.140234+01:00 INFO Stopping monitor
What I am doing here?
The value NIL is not of type PGLOADER.CATALOG:COLUMN
pgloader version "3.6.7~devel" compiled with SBCL 2.2.9.debian
I get this error. Running WITH data only, on error stop, batch rows = 1, prefetch rows = 1, truncate, quote identifiers, batch size = 8MB
Heap exhausted during allocation: 66912256 bytes available, 69131648 requested. Immobile Object Counts Gen layout fdefn symbol code Boxed Cons Raw Code SmMix Mixed LgRaw LgCode LgMix Waste% Alloc Trig Dirty GCs Mem-age 0 0 0 0 0 0 1 0 0 0 287 0 0 933 5.5 37803824 10737418 - 0 0.0000 1 0 0 0 0 1 3 14 0 0 7 8280 0 0 0.2 271683328 145479962 8295 1 0.4960 2 0 0 0 0 0 4 16 0 0 10 6101 0 0 0.3 200291584 55704810 6118 1 0.2245 3 46 0 0 66 30 21 25 1 4 38 3777 0 0 1.0 126418656 2000000 3803 0 0.4546 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0 0 2000000 0 0 0.0000 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0 0 2000000 0 0 0.0000 6 2137 36032 30989 33784 775 841 694 19 78 67 100 0 290 2.2 91790720 2000000 13 0 0.0000 Tot 2183 36032 30989 33850 806 870 749 20 82 409 18258 0 1223 0.9 727988112 [67.8% of 1073741824 max] GC control variables: GC-INHIBIT = false GC-PENDING = true STOP-FOR-GC-PENDING = false 2025-08-20T10:40:43.744452Z ERROR A SB-KERNEL::HEAP-EXHAUSTED-ERROR condition without bindings for heap statistics. (If you did not expect to see this message, please report it.
