pgloader icon indicating copy to clipboard operation
pgloader copied to clipboard

Casting with varchar[] (array) data types from PGSQL source

Open kingwilly71 opened this issue 3 years ago • 0 comments

  • [ ] pgloader --version

    3.6.a94a0a3
    
  • [ ] did you test a fresh compile from the source tree?
    yes

  • [ ] did you search for other similar issues? yes

  • [ ] how can I reproduce the bug?


load database
  from pgsql://source/database?sslmode=prefer
  into pgsql://target/database?sslmode=prefer
  CAST type int to int drop typemod,
       type bigint to bigint drop typemod

  • [ ] pgloader output you obtain
2022-04-12T09:24:43.015000Z LOG pgloader version "3.6.a94a0a3"
2022-04-12T09:24:43.155000Z LOG Migrating from #<PGSQL-CONNECTION pgsql://source/database {1008CE5C23}>
2022-04-12T09:24:43.155000Z LOG Migrating into #<PGSQL-CONNECTION pgsql://target/database {1008CE7413}>
2022-04-12T09:24:49.131000Z ERROR Database error 42601: syntax error at or near "("
QUERY: CREATE TABLE "public"."fun_chats"
(
  "chat_id"         bigint default NULL,
  "level"           integer default 1,
  "coordinates"     geography default NULL,
  "badges"          jsonb default NULL,
  "location"        text default NULL,
  "apply_questions" boolean default false,
  "questions"       character varying[](200) default NULL,
  "admin_approval"  boolean default false,
  "restricted"      boolean default false,
  "service_id"      text default NULL
);
2022-04-12T09:24:49.145000Z FATAL Failed to create the schema, see above.
2022-04-12T09:24:49.148000Z LOG report summary reset
       table name     errors       rows      bytes      total time
-----------------  ---------  ---------  ---------  --------------
  fetch meta data          0        112                     4.577s
   Create Schemas          0          0                     0.002s
 Create SQL Types          0          0                     1.059s
    Create tables          0          0                     0.000s
-----------------  ---------  ---------  ---------  --------------
-----------------  ---------  ---------  ---------  --------------

If I run that query directly on the database it points out that error occurs at position "questions" character varying[](200) default NULL. It is complaining about the typemod maybe.

By removing the (200) part it is possible to execute query directly and the table is succesfully created.

  • [ ] data that is being loaded, if relevant

Use this query as sample data:

INSERT INTO "public"."fun_chats" ("chat_id", "level", "coordinates", "badges", "location", "apply_questions", "questions", "admin_approval", "restricted", "service_id") VALUES
(274194652951740416, 1, 'SRID=4326;POINT(121.564558 25.03746)', NULL, NULL, 't', '{why,what}', 'f', 't', NULL),
(271622218557947904, 1, 'SRID=4326;POINT(1.4 1.4)', NULL, NULL, 't', '{Hello}', 't', 'f', NULL)
  • [ ] How the data is different from what you expected, if relevant

varchar[] data type is casted to text with or without casting specified. First I thought that I forgot some casting rules for array data types, but reading the pgloader manual did not help. I found a possible relative part set-to-enum-array that can convert MySQL enum to PGSQL arrays, but this seems not applicable here. Technically array types are identified by PG with an underline prefix (Ex. for varchar[] it is represented as _varchar), while pgloader seems to complain about that underline. Is array data types supported?

kingwilly71 avatar Apr 12 '22 09:04 kingwilly71