transporter icon indicating copy to clipboard operation
transporter copied to clipboard

rethinkdb tailing to postgresql doesn't work

Open nurey opened this issue 6 years ago • 1 comments

Bug report

Relevant pipeline.js:

var source = rethinkdb({
  "uri": "rethink://localhost:28015/foundation",
  "timeout": "30s",
  "tail": true,
  "debug": true
})

var sink = postgres({
  "uri": "postgres://ilia@localhost:5432/client_app_development?sslmode=disable",
  "tail": true,
  "debug": true
})

t.Source("source", source, "sold_listings_mini")
  .Transform(pick({
    "fields":
      ["id", "days-on-market", "mls", "price", "sold-date", "sold-price", "postal-code"]
  }))
  .Transform(rename({
    "field_map": {
      "id": "rethink_id",
      "days-on-market": "days_on_market",
      "sold-date": "sold_date",
      "sold-price": "sold_price",
      "postal-code": "postal_code"
    }
  }))
  .Save("sink", sink, "sold_listings_mini")

System info:

  • Transporter version Built from 387bbde4c1cd18a23b9b1c8a1a53b13066521cef

  • OS MacOS 10.13.3

  • DB version(s) Postgresql 10.3 Rethinkdb 2.3.6

Reproducible Steps:

  1. Setup pipeline.js
  2. run transporter

What did you expect to happened?

Rethinkdb table should sync to postgresql table. Transporter should continue running after syncing all the documents, to tail rethinkdb.

What actually happened?

Rethinkdb table gets synced to postgresql table. Transporter exits with:

INFO[0000] boot map[source:rethinkdb sink:postgres]      ts=1521122232873691800
INFO[0000] adaptor Listening...                          name=sink path=source/sink type=postgres
INFO[0000] starting with metadata map[]                  name=source path=source type=rethinkdb
INFO[0000] adaptor Starting...                           name=source path=source type=rethinkdb
INFO[0000] starting Read func                            db=foundation
INFO[0000] skipping iteration...                         db=foundation table=sold_listings
INFO[0000] sending for iteration...                      db=foundation table=sold_listings_mini
INFO[0000] done iterating tables                         db=foundation
INFO[0000] iterating...                                  db=foundation table=sold_listings_mini
INFO[0005] metrics source records: 9798                  path=source ts=1521122237875253979
INFO[0005] metrics source/sink records: 9786             path=source/sink ts=1521122237875256758
INFO[0010] metrics source records: 19507                 path=source ts=1521122242873261125
INFO[0010] metrics source/sink records: 19495            path=source/sink ts=1521122242873262628
INFO[0014] iterating complete                            db=foundation table=sold_listings_mini
INFO[0014] Read completed                                db=foundation
INFO[0014] adaptor Start finished...                     name=source path=source type=rethinkdb
INFO[0014] adaptor Stopping...                           name=source path=source type=rethinkdb
INFO[0015] metrics source records: 28343                 path=source ts=1521122247875216475
INFO[0015] metrics source/sink records: 28343            path=source/sink ts=1521122247875223047
INFO[0015] adaptor Stopped                               name=source path=source type=rethinkdb
INFO[0015] stopping changes...                           db=foundation table=sold_listings_mini
INFO[0015] adaptor Stopping...                           name=sink path=source/sink type=postgres
INFO[0015] received stop, message buffer is empty, closing...
INFO[0015] adaptor Listen closed...                      name=sink path=source/sink type=postgres
INFO[0015] adaptor Stopped                               name=sink path=source/sink type=postgres
INFO[0015] metrics source records: 28343                 path=source ts=1521122248660582240
INFO[0015] metrics source/sink records: 28343            path=source/sink ts=1521122248660585285
INFO[0015] exit map[source:rethinkdb sink:postgres]      ts=1521122248660586053

nurey avatar Mar 15 '18 14:03 nurey

any update on this issue? i got same problem. Can someone help me?

HOTTIN avatar Apr 22 '21 08:04 HOTTIN