Broken Pipe - Symbol not found: _PQbackendPID

[catherine-jayapandian]

There is dependency mismatch between target-postgres and psycopg2 library.

• target-postgres needs psycopg2==2.7.5
• singer-target-postgres needs psycopg2==2.8.5

From web sources, I guess the postgresql driver issue (Broken Pipe - Symbol not found: _PQbackendPID) is resolved in psycopg2 version 2.9.5.

OS environment: Mac 11.7 M1 processor, hence need an updated version of target-postgres.

INFO Server Parameters: version: 8.0.30, wait_timeout: 2700, innodb_lock_wait_timeout: 2700, max_allowed_packet: 67108864, interactive_timeout: 28800
INFO Server SSL Parameters (blank means SSL is not active): [ssl_version: ], [ssl_cipher: ]
INFO Beginning sync for InnoDB table Census.census
INFO Stream census is using incremental replication
INFO Running SELECT `age`,`sex`,`pop2008`,`state`,`pop2000`,`id` FROM `Census`.`census` ORDER BY `id` ASC
Traceback (most recent call last):
  File "/Users/catherine/.virtualenvs/target-postgres/bin/target-postgres", line 5, in <module>
    from target_postgres import cli
  File "/Users/catherine/.virtualenvs/target-postgres/lib/python3.9/site-packages/target_postgres/__init__.py", line 2, in <module>
    import psycopg2
  File "/Users/catherine/.virtualenvs/target-postgres/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
**_### ImportError: dlopen(/Users/catherine/.virtualenvs/target-postgres/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): Symbol not found: _PQbackendPID_**
  Referenced from: /Users/catherine/.virtualenvs/target-postgres/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/catherine/.virtualenvs/target-postgres/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 298, "tags": {"database": "Census", "table": "census"}}
INFO METRIC: {"type": "timer", "metric": "job_duration", "value": 0.26779699325561523, "tags": {"job_type": "sync_table", "database": "Census", "table": "census", "status": "failed"}}
CRITICAL [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/Users/catherine/.virtualenvs/tap-mysql/bin/tap-mysql", line 8, in <module>
    sys.exit(main())
  File "/Users/catherine/.virtualenvs/tap-mysql/lib/python3.9/site-packages/singer/utils.py", line 229, in wrapped
    return fnc(*args, **kwargs)
  File "/Users/catherine/.virtualenvs/tap-mysql/lib/python3.9/site-packages/tap_mysql/__init__.py", line 728, in main
    do_sync(mysql_conn, args.config, args.catalog, state)
  File "/Users/catherine/.virtualenvs/tap-mysql/lib/python3.9/site-packages/tap_mysql/__init__.py", line 679, in do_sync
    sync_non_binlog_streams(mysql_conn, non_binlog_catalog, config, state)
  File "/Users/catherine/.virtualenvs/tap-mysql/lib/python3.9/site-packages/tap_mysql/__init__.py", line 654, in sync_non_binlog_streams
    do_sync_incremental(mysql_conn, catalog_entry, state, columns, optional_limit)
  File "/Users/catherine/.virtualenvs/tap-mysql/lib/python3.9/site-packages/tap_mysql/__init__.py", line 527, in do_sync_incremental
    incremental.sync_table(mysql_conn, catalog_entry, state, columns)
  File "/Users/catherine/.virtualenvs/tap-mysql/lib/python3.9/site-packages/tap_mysql/sync_strategies/incremental.py", line 75, in sync_table
    num_rows = common.sync_query(cur,
  File "/Users/catherine/.virtualenvs/tap-mysql/lib/python3.9/site-packages/tap_mysql/sync_strategies/common.py", line 219, in sync_query
    singer.write_message(record_message)
  File "/Users/catherine/.virtualenvs/tap-mysql/lib/python3.9/site-packages/singer/messages.py", line 227, in write_message
    sys.stdout.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

[laurentS]

Hi @catherine-jayapandian I'm a bit confused by your description above, I suspect there are 2 issues there:

• target-postgres on pypi is a different repository and project, if that is the package you have trouble with, please report the issue on their repo. Sadly, I'm not too sure where that repo is.
• singer-target-postgres is indeed this repository, and you might want to try using the latest master branch, instead of the pypi release. The latest commit relies on psycopg ^2.9.5 which should fix your problem. I guess we should aim to release a new version soon with these changes :)