transporter `run` reports "connection error, no reachable servers" even though `test` says OK.

[stabenfeldt]

Directions

I´ve created a project with docker-compose. It starts a few MongoDB servers and Elasticsearch. Then it installs transporter and verifies that the sync works. I did this back in 2015, but when I tried to upgrade to today's version, I no longer get data synced into Elasticsearch.

Bug report

Relevant pipeline.js:

transporter/pipeline.js
var source = mongodb({
  uri: "mongodb://mongo1:28017/harvester-test",
});

var sink = elasticsearch({
  uri: "http://elasticsearch:9200/harvester-test",
});

t.Source("source", source).Save("sink", sink);
// I tried with both the line(s) above and below.
//t.Source("source", source, "harvester-test.entries").Save(
//  "sink",
//  sink,
//  "harvester-test.entries"
//);

System info:

• Transporter version: v1.1.0
• OS: Runs in docker image golang:1.18.3
• DB version(s)
  • Elasticsearch docker image: elasticsearch:7.17.4
  • MongoDB docker image:3.0.6

Reproducible Steps:

git clone [email protected]:stabenfeldt/elastic-mongo.git
git checkout make-it-run-again
docker compose up --detach
docker compose logs transporter

## transport test -log.level "info" pipeline.js

elastic-mongo-transporter-1  | Transporter:
elastic-mongo-transporter-1  |  - Source:         source                                   mongodb         .*
elastic-mongo-transporter-1  |   - Sink:          sink                                     elasticsearch   .*

## transport run -log.level "info" pipeline.js


elastic-mongo-transporter-1  | time="2022-06-16T10:38:08Z" level=info msg="starting with metadata map[]" name=source path=source type=mongodb
elastic-mongo-transporter-1  | time="2022-06-16T10:38:08Z" level=info msg="adaptor Starting..." name=source path=source type=mongodb
elastic-mongo-transporter-1  | time="2022-06-16T10:38:08Z" level=info msg="boot map[sink:elasticsearch source:mongodb]" ts=1655375888470716754
elastic-mongo-transporter-1  | time="2022-06-16T10:38:08Z" level=info msg="adaptor Listening..." name=sink path=source/sink type=elasticsearch
elastic-mongo-transporter-1  | time="2022-06-16T10:38:13Z" level=info msg="metrics source records: 0" path=source ts=1655375893474071756
elastic-mongo-transporter-1  | time="2022-06-16T10:38:13Z" level=info msg="metrics source/sink records: 0" path=source/sink ts=1655375893475196965
elastic-mongo-transporter-1  | time="2022-06-16T10:38:18Z" level=info msg="metrics source records: 0" path=source ts=1655375898475006634
elastic-mongo-transporter-1  | time="2022-06-16T10:38:18Z" level=info msg="metrics source/sink records: 0" path=source/sink ts=1655375898475113634
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="adaptor Stopping..." name=source path=source type=mongodb
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="adaptor Stopped" name=source path=source type=mongodb
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="adaptor Stopping..." name=sink path=source/sink type=elasticsearch
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="received stop, message buffer is empty, closing..."
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="adaptor Listen closed..." name=sink path=source/sink type=elasticsearch
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="adaptor Stopped" name=sink path=source/sink type=elasticsearch
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="closing BulkProcessor" version=5 writer=elasticsearch
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="metrics source records: 0" path=source ts=1655375899030879301
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="metrics source/sink records: 0" path=source/sink ts=1655375899030902801
elastic-mongo-transporter-1  | time="2022-06-16T10:38:19Z" level=info msg="exit map[sink:elasticsearch source:mongodb]" ts=1655375899030945134
elastic-mongo-transporter-1  | connection error, no reachable servers

What did you expect to happened?

I expected Elasticsearch to return the data that I had inserted into mongo.

What actually happened?

The transporter reported connection error, no reachable servers

[buildizer]

Have same issue and looking for any sloutions

[atomicules]

Just as status update / bit of history:

• AFAIK (I wasn't around at the beginning) Compose developed this tool to allow customers to move data between deployment types as new database types were offered

• There was quite a bit of development activity up until 2017. Then I'm guessing since Compose were being much more heavily integrated in to IBM goals changed and work stopped on the project.

• In 2018 there was a bit of resurgence of activity as Compose also used this tool (under the hood) so customers could migrate from the Classic offering that was being shutdown

• Many of the original contributors left IBM/Compose over the years

• In advance of the shutdown of Compose there was a brief spell from the end of 2021 to mid 2022 where IBM/Compose dedicated some folk to work on this project again so we could help customers migrate from Compose to IBM Cloud. That's where the 1.0 release came from. The primary focus there was on Postgresql and MySQL.

• Unfortunately internal focus has shifted again so no one is working on this as part of their job

• It does state under the 1.0 release:

  This version officially only supports the mongodb and postgresql adaptors. Support for other DBs will be added later on. Other adaptors may or may not work. You're encouraged to still use v0.5.2 for non mongo/postgres migrations.

So... from a personal point of view I am interested in this project and working on it, but it'd have to come from my spare non-work time which is severely lacking.

I guess that's a long way of saying "don't hold your breath". But also perhaps don't give up hope: It only took us seven years to add a MySQL adaptor.

[JTtime]

I got same error while working with Mongo. I just updated my Mongodb (current version 5) and problem was solved