pgloader icon indicating copy to clipboard operation
pgloader copied to clipboard

Cast column using without type for enum does not work

Open ElementalWarrior opened this issue 10 months ago • 1 comments

Just trying to use a lambda function for converting enums does not work, but if I specify type, it does work.

This is in contrary to what the bottom of the mysql cast documentation says: https://pgloader.readthedocs.io/en/latest/ref/mysql.html#mysql-database-casting-rules

It’s possible to augment a default cast rule (such as one that applies against ENUM data type for example) with a transformation function by omitting entirely the type parts of the casting rule, as in the following example:

      column table.type using (lambda (type) (if (string= "" type) "some_value" type) )

Does work

      column table.type to table_type using (lambda (type) (if (string= "" type) "some_value" type) )
  • [ ] pgloader --version

$ pgloader --version pgloader version "3.6.7~devel" compiled with SBCL 2.3.7 ```

  • [ ] did you test a fresh compile from the source tree?

Its compiled from source, but not directly from master

  • [ ] did you search for other similar issues?

Yes

  • [ ] how can I reproduce the bug? Create a table with an enum in mysql, try to hardcode the value using a lambda. See no change.
--
-- EDIT THIS FILE TO MATCH YOUR BUG REPORT
--
load database
  from {{ SRC_CONNECTION }}
  into {{ DEST_CONNECTION }}


 WITH include drop, create tables, no truncate, create indexes, reset sequences, foreign keys, disable triggers, prefetch rows = 500, batch rows = 500

  SET MySQL PARAMETERS
      net_read_timeout  = '180',
      net_write_timeout = '180'

 CAST
      type "datetime not null" when default "0000-00-00 00:00:00"
        to "timestamp default '1970-01-01 00:00:00+00'" drop default,

      type "timestamp not null" when default "0000-00-00 00:00:00"
        to "timestamp default '1970-01-01 00:00:00+00'" drop default,

      type datetime when default "0000-00-00 00:00:00"
        to "timestamp default '1970-01-01 00:00:00+00'" drop default,

      type timestamp when default "0000-00-00 00:00:00"
        to "timestamp default '1970-01-01 00:00:00+00'" drop default,

      type date when default "0000-00-00"
        to "date default '1970-01-01'",

      type datetime with extra on update current timestamp to "timestamp default transaction_timestamp()" drop default,

      type timestamp with extra on update current timestamp to "timestamp default transaction_timestamp()" drop default,

      type date to date using (lambda (ts) (if (string= ts "0000-00-00") "1970-01-01" ts)),
      type datetime to timestamp using (lambda (ts) (if (string= ts "0000-00-00 00:00:00") "1970-01-01 00:00:00+00" ts)),
      type timestamp to timestamp using (lambda (ts) (if (string= ts "0000-00-00 00:00:00") "1970-01-01 00:00:00+00" ts)),
      type year to integer drop typemod,


      column table.type using (lambda (type) (if (string= "" type) "unset" type) )

 ALTER TABLE NAMES MATCHING ~/./
  SET SCHEMA 'public';
  • [ ] pgloader output you obtain
    2023-08-25T15:50:22.015000Z LOG pgloader version "3.6.7~devel"
    2023-08-25T15:50:22.179001Z LOG Migrating from #<MYSQL-CONNECTION ...
    2023-08-25T15:50:22.179001Z LOG Migrating into #<PGSQL-CONNECTION ...
    2023-08-25T15:50:27.546006Z ERROR Database error 22P02: invalid input value for enum public.table_type: ""
    CONTEXT: COPY table, line 5005, column type: ""
    2023-08-25T15:50:27.877006Z LOG report summary reset
  • [ ] data that is being loaded, if relevant

I'm converting an empty string to a proper enum value.

  • [ ] How the data is different from what you expected, if relevant

NA

ElementalWarrior avatar Aug 25 '23 16:08 ElementalWarrior

I had problems doing similar. I ended up writing the functions to a lisp file. Then adding the --load-lisp-file functions.lisp arguments to the pgloader command.

Example of function in lisp file:

(defun empty-string-to-unset (type)
    (if (string= "" type) "unset" type))

ElementalWarrior avatar Nov 07 '23 20:11 ElementalWarrior