pgloader
pgloader copied to clipboard
Cast column using without type for enum does not work
Just trying to use a lambda function for converting enums does not work, but if I specify type, it does work.
This is in contrary to what the bottom of the mysql cast documentation says: https://pgloader.readthedocs.io/en/latest/ref/mysql.html#mysql-database-casting-rules
It’s possible to augment a default cast rule (such as one that applies against ENUM data type for example) with a transformation function by omitting entirely the type parts of the casting rule, as in the following example:
column table.type using (lambda (type) (if (string= "" type) "some_value" type) )
Does work
column table.type to table_type using (lambda (type) (if (string= "" type) "some_value" type) )
-
[ ] pgloader --version
$ pgloader --version pgloader version "3.6.7~devel" compiled with SBCL 2.3.7 ```
- [ ] did you test a fresh compile from the source tree?
Its compiled from source, but not directly from master
- [ ] did you search for other similar issues?
Yes
- [ ] how can I reproduce the bug? Create a table with an enum in mysql, try to hardcode the value using a lambda. See no change.
--
-- EDIT THIS FILE TO MATCH YOUR BUG REPORT
--
load database
from {{ SRC_CONNECTION }}
into {{ DEST_CONNECTION }}
WITH include drop, create tables, no truncate, create indexes, reset sequences, foreign keys, disable triggers, prefetch rows = 500, batch rows = 500
SET MySQL PARAMETERS
net_read_timeout = '180',
net_write_timeout = '180'
CAST
type "datetime not null" when default "0000-00-00 00:00:00"
to "timestamp default '1970-01-01 00:00:00+00'" drop default,
type "timestamp not null" when default "0000-00-00 00:00:00"
to "timestamp default '1970-01-01 00:00:00+00'" drop default,
type datetime when default "0000-00-00 00:00:00"
to "timestamp default '1970-01-01 00:00:00+00'" drop default,
type timestamp when default "0000-00-00 00:00:00"
to "timestamp default '1970-01-01 00:00:00+00'" drop default,
type date when default "0000-00-00"
to "date default '1970-01-01'",
type datetime with extra on update current timestamp to "timestamp default transaction_timestamp()" drop default,
type timestamp with extra on update current timestamp to "timestamp default transaction_timestamp()" drop default,
type date to date using (lambda (ts) (if (string= ts "0000-00-00") "1970-01-01" ts)),
type datetime to timestamp using (lambda (ts) (if (string= ts "0000-00-00 00:00:00") "1970-01-01 00:00:00+00" ts)),
type timestamp to timestamp using (lambda (ts) (if (string= ts "0000-00-00 00:00:00") "1970-01-01 00:00:00+00" ts)),
type year to integer drop typemod,
column table.type using (lambda (type) (if (string= "" type) "unset" type) )
ALTER TABLE NAMES MATCHING ~/./
SET SCHEMA 'public';
- [ ] pgloader output you obtain
2023-08-25T15:50:22.015000Z LOG pgloader version "3.6.7~devel"
2023-08-25T15:50:22.179001Z LOG Migrating from #<MYSQL-CONNECTION ...
2023-08-25T15:50:22.179001Z LOG Migrating into #<PGSQL-CONNECTION ...
2023-08-25T15:50:27.546006Z ERROR Database error 22P02: invalid input value for enum public.table_type: ""
CONTEXT: COPY table, line 5005, column type: ""
2023-08-25T15:50:27.877006Z LOG report summary reset
- [ ] data that is being loaded, if relevant
I'm converting an empty string to a proper enum value.
- [ ] How the data is different from what you expected, if relevant
NA
I had problems doing similar. I ended up writing the functions to a lisp file. Then adding the --load-lisp-file functions.lisp
arguments to the pgloader command.
Example of function in lisp file:
(defun empty-string-to-unset (type)
(if (string= "" type) "unset" type))