Can we use mongo-connector to sync between MongoDb Atlas to Elasticsearch cloud stack?

[vishaleclass]

trafficstars

Hello,

I'm using MongoDB Atlas and Elastic stack (Elasticsearch cloud stack). I wanted to have sync from MongoDB to ElasticSearch. Can mongo-DB connector will work for for this? mongoDB atalas version is 4.X and Elasticsearch version in 6.x

Please suggest if you have any better options also as river is now deprecated, I think I can't use that.

Thanks

[vishaleclass]

Update01 Getting this error message

Logging to /root/mongo-connector.log.
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/site-packages/mongo_connector/util.py", line 107, in wrapped
    func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mongo_connector/connector.py", line 353, in run
    self.main_conn = self.create_authed_client()
  File "/usr/lib/python2.7/site-packages/mongo_connector/connector.py", line 337, in create_authed_client
    client["admin"].authenticate(self.auth_username, self.auth_key)
  File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 1274, in authenticate
    connect=True)
  File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 604, in _cache_credentials
    raise OperationFailure('Another user is already authenticated '
OperationFailure: Another user is already authenticated to this database. You must logout first.

[vishaleclass]

Error : "OperationFailure: Another user is already authenticated to this database. You must logout first" Above commented error got resolved after removing -a username & -a password as parameter and adding the same in MainAddress. eg. mongo-connector -m 'mongodb://:@<servername/ip_address>:/test?ssl=true&retryWrites=true' .......

[vishaleclass]

But My issue is still there, still I'm not able to connect to my elastic cloud using mongo-connector :(

[vishaleclass]

Bymistakly closes! Reopened.

[vishaleclass]

Hello, I'm new in this stuff. I have to integrate mongo-db Atlas to Elastic search. I have succefully connected to mongoDB atlas by applying @ShaneHarvey comment however not able to connect to target URL(elastic cloud).
Can you please help me with syntax you had use or can you please share brief description about it ? I'm also having same kind of issue. I'm trying to connect elasticsearch cloud but no luck. Getting error.

[root@vm02 ~]# mongo-connector -c mongo-connector.json
Traceback (most recent call last):
  File "/usr/bin/mongo-connector", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/mongo_connector/util.py", line 107, in wrapped
    func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mongo_connector/connector.py", line 1409, in main
    conf.parse_args()
  File "/usr/lib/python2.7/site-packages/mongo_connector/config.py", line 119, in parse_args
    raise errors.InvalidConfiguration(str(exc)).with_traceback(tb)
AttributeError: 'InvalidConfiguration' object has no attribute 'with_traceback'

[root@vm02 ~]# cat mongo-connector.json
{
  "mainAddress": 'mongodb://admin01:password@<fqdn>:27017,fqdb:27017,fqdn:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true/admin?ssl=true&replicaSet=name&authSource=admin',
  "verbosity": 1,
  "docManagers": [
    {
      "docManager": "elastic2_doc_manager",
      "targetURL": "https://46a7532846fd4853929245a0e7edadc8.eu-west-1.aws.found.io:9243",
      "args": {
        "clientOptions": {
           "http_auth": ["elastic", "password"],
           "use_ssl": true
        }
      }
    }
  ]
}

thanks

[johnfrades]

Hi, can you show your config on how you connect MongoDB Atlas.

Here's mine

{
    "mainAddress": "mongodb+srv://USERNAME:[email protected]",
    "verbosity": 2,
    "namespaces": {
        "proto2.users": {
            "excludeFields": ["password", "accountActivationToken"],
            "rename": "user.default"
        },
        "proto2.businesses": "supplier.default",
        "proto2.posts": "post.default",
        "proto2.products": "product.default",
        "proto2.offers": "offer.default"
    },
    "docManagers": [
        {
            "docManager": "elastic2_doc_manager",
            "targetURL": "localhost:9200",
            "autoCommitInterval": 0
        }
    ]
}

[johnfrades]

And also to answer your question, i think you cannot use Elasticsearch that has 6.x.x version and above. This library only supports Elasticsearch version 5. Currently im using 5.5.3

[jackrvaughan]

Did you ever sort this out? I believe I might be running into the same issue.

The mongo-connector seems to connect just fine to mongodb but the elastic connection doesn't seem to be working. This is the config.json that I'm using:

{
    "mainAddress": "mongodb+srv://***URL***.mongodb.net",
    "verbosity": 2,
    "authentication": {
        "adminUsername": "***USERNAME***",
        "password": "***PASSWORD***"
    },
    "ssl": {
        "sslCertificatePolicy": "required"
    },
    "docManagers": [
        {
            "docManager": "elastic2_doc_manager",
            "targetURL": "***ELASTIC_CLOUD_URL***",
            "__uniqueKey": "_id",
            "args": {
                "clientOptions": {
                    "http_auth": ["***USERNAME***", "***PASSWORD***"],
                    "use_ssl": true
                }
            }
        }
    ]
}

I can't even seem to get an error or any helpful info when I use a fake url as the "targetURL" for the "docManager".

The elastic version I'm using on elastic cloud is v5.6.16.

[AClarkie]

Any update on this?

[senthilengg]

Any update on this

[donaldfernandes]

I am using ES 7 with MongoDB Atlas. below is my config file. I believe its connecting with ES and MongoDB but i do not see any data sync'd when i look at the ES index.. Here's what is see in the connector logs . Any help here is much appreciated.

OplogThread for replica set '' is up to date with the oplog.

{
    "mainAddress": "mongodb+srv://<NAME>.mongodb.net",
    "verbosity": 3,
    "authentication": {
        "adminUsername": “username,
        "password": “password”
    },
    "namespaces": {
        “db.Colletion_name”: “db.index_name"
   },
    "ssl": {
        "sslCertificatePolicy": "required"
    },
    "docManagers": [
        {
            "docManager": "elastic2_doc_manager",
            "targetURL": "localhost:9200",
            "__uniqueKey": "_id"
        }
    ]
}

[donaldfernandes]

this worked.. I had o just delete the ops.log file and re-run the connector. This link was helpful : http://pcbbc.site.mobi/templates/mobile/facade_transcoder_iframe.php?u=%2F10gen-labs%2Fmongo-connector%2Fwiki%2FFAQ%3Fimz_s%3Dpv59sirqm5vvrq4gtet2g2eu40&lang=en