pq: invalid byte sequence for encoding "UTF8"

[itcuihao]

mongodb to postgresql

sink postgresql uri:

"uri": "postgres://user:[email protected]:5432/tablename?sslmode=disable"

postgresql serever:

   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 tablename     | me       | UTF8     | en_US.UTF-8 | en_US.UTF-8 |

Run

 ./transporter run pipeline.js 

ERRO[0000] pq: invalid byte sequence for encoding "UTF8": 0xe1 0x45 0xcc name=sink path="source/sink" type=postgres INFO[0000] adaptor Stopping... name=source path=source type=mongodb INFO[0000] adaptor Stopped name=source path=source type=mongodb INFO[0000] closing connection... name=source path=source type=mongodb INFO[0000] Connections to 127.0.0.1:27017 closing (2 live sockets). INFO[0000] Socket 0xc420270000 to 127.0.0.1:27017: closing: Closed explicitly (abend=false) INFO[0000] Socket 0xc42010e540 to 127.0.0.1:27017: closing: Closed explicitly (abend=false) INFO[0000] connection closed... name=source path=source type=mongodb INFO[0000] adaptor Stopping... name=sink path="source/sink" type=postgres INFO[0000] adaptor Stopped name=sink path="source/sink" type=postgres INFO[0000] closing connection... name=sink path="source/sink" type=postgres INFO[0000] connection closed... name=sink path="source/sink" type=postgres INFO[0000] metrics source records: 11 path=source ts=1496820342222718498 INFO[0000] metrics source/sink records: 1 path="source/sink" ts=1496820342222720991 INFO[0000] exit map[sink:postgres source:mongodb] ts=1496820342222721543 ERRO[0010] timeout reached waiting for Out channels to clear panic: close of closed channel

[jipperinbham]

thanks for the detailed report @itcuihao!

could you also provide the schema of the table as well as some sample data from both the source mongodb and what you would expect it to be in postgresql?

[johnnason]

I hit this as well right out of the gate. For me, the cause was the ObjectId marshalling. I kind-of expected a string, and set up a varchar _id column in my sink table accordingly. Turns out, it marshalls to a bytea.

Might be worth calling this out in the postgresql adaptor page, or perhaps adding a separate page entirely on marshalling, if for nothing else than to call out the use of go's json.marshall and to refer to that.

Furthermore, sink error reporting is currently broken making for an even worse first impression trying to get things going (looking into that separately).

[knrt10]

@jipperinbham faced the same error.