target-postgres icon indicating copy to clipboard operation
target-postgres copied to clipboard

Comment on pre-existing table crashes target-postgres

Open laurentS opened this issue 3 years ago • 0 comments

I tried running this target (via meltano) with postgres_schema set to an existing schema which already contained tables (with data). Some of the tables had comments on them, as created by:

COMMENT ON TABLE public.mytable IS ''my custom table with important data';

When running the target, it crashed with the following stacktrace:

target-postgres | ERROR Exception writing records
target-postgres | Traceback (most recent call last):
target-postgres |   File "~/myproject/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/postgres.py", line 237, in write_batch
target-postgres |     self.setup_table_mapping_cache(cur)
target-postgres |   File "~/myproject/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/postgres.py", line 224, in setup_table_mapping_cache
target-postgres |     table_path = json.loads(raw_json).get('path', None)
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/__init__.py", line 357, in loads
target-postgres |     return _default_decoder.decode(s)
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/decoder.py", line 337, in decode
target-postgres |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/decoder.py", line 355, in raw_decode
target-postgres |     raise JSONDecodeError("Expecting value", s, err.value) from None
target-postgres | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
target-postgres | CRITICAL ('Exception writing records', JSONDecodeError('Expecting value: line 1 column 1 (char 0)'))

With some extra logging, raw_json actually contains the value of the comment mentioned above, which clearly isn't JSON. This is the line of code that breaks https://github.com/datamill-co/target-postgres/blob/9c095d91e215f932caa897a6587c6dc6278db8cf/target_postgres/postgres.py#L223

Is this a bug, or does target-postgres expect the schema to be empty when it starts?

laurentS avatar Feb 19 '21 23:02 laurentS