bitsail
bitsail copied to clipboard
bytedance
使用读连接器连接MongoDB，报错Caused by: com.mongodb.MongoCommandException: Command failed with error 18 (AuthenticationFailed): 'Authentication failed.' on server
What happened
报错日志为： 2023-06-12 17:56:04,399 INFO org.apache.flink.client.cli.CliFrontend [] - -------------------------------------------------------------------------------- 2023-06-12 17:56:04,401 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, bigdata-test 2023-06-12 17:56:04,401 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123 2023-06-12 17:56:04,402 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m 2023-06-12 17:56:04,402 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m 2023-06-12 17:56:04,402 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 8 2023-06-12 17:56:04,402 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 6 2023-06-12 17:56:04,402 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region 2023-06-12 17:56:04,498 WARN org.apache.flink.runtime.util.HadoopUtils [] - Could not find Hadoop configuration via any of the supported methods (Flink configuration, environment variables). 2023-06-12 17:56:04,515 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to root (auth:SIMPLE) 2023-06-12 17:56:04,533 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-2246462791333134135.conf. 2023-06-12 17:56:04,539 INFO org.apache.flink.client.cli.CliFrontend [] - Running 'run' command. 2023-06-12 17:56:04,543 INFO org.apache.flink.client.cli.CliFrontend [] - Building program from JAR file 2023-06-12 17:56:04,570 INFO org.apache.flink.client.ClientUtils [] - Starting program (detached: false) 2023-06-12 17:56:04,603 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component default loaded in clazz interface com.bytedance.bitsail.core.api.parser.ConfigurationParser. 2023-06-12 17:56:04,607 INFO com.bytedance.bitsail.core.api.parser.ConfigurationHelper [] - Config parser: default accepted the arguments. 2023-06-12 17:56:04,780 INFO com.bytedance.bitsail.core.Engine [] - BitSail configuration: { "job":{ "common":{ "dirty_records_percentage_threshold":-1, "dirty_records_count_threshold":-1, "instance_id":-2501346, "job_name":"qz_listed_financial_zichanfuzai2hive", "user_name":"bytedance_dts", "job_id":-20501346, "parallelism_chain":false, "max_dirty_records_stored_num":50 }, "reader":{ "hosts_str":"1111:18009,1112:18009,1113:18009", "password":"************", "db_name":"data_collect_prod", "auth_db_name":"data_collect_prod", "user_name":"data_collect_user", "columns":[ { "name":"ent_name", "index":0, "type":"string" }, { "name":"financial_date", "index":0, "type":"string" }, { "name":"type", "index":0, "type":"string" } ], "split_pk":"_id", "class":"com.bytedance.bitsail.connector.legacy.mongodb.source.MongoDBInputFormat", "collection_name":"qz_listed_financial_zichanfuzai" }, "writer":{ "partition":"p_date=20230612", "db_name":"cyyn_ods", "columns":[ { "name":"ent_name", "type":"string" }, { "name":"financial_date", "type":"string" }, { "name":"type", "type":"string" } ], "metastore_properties":"{"hive.metastore.uris":"thrift://bigdata-test:9083"}", "class":"com.bytedance.bitsail.connector.legacy.hive.sink.HiveOutputFormat", "table_name":"qz_listed_financial_zichanfuzai" } } } 2023-06-12 17:56:04,785 INFO com.bytedance.bitsail.base.statistics.VMInfo [] - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2023-06-12 17:56:04,786 INFO com.bytedance.bitsail.core.Engine [] - the machine info =>
osInfo: Oracle Corporation 1.8 25.341-b10
jvmInfo: Linux amd64 3.10.0-1160.el7.x86_64
cpu num: 16
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 3,950.00MB | 189.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 31.00MB | 31.00MB
PS Old Gen | 8,024.00MB | 503.00MB
Metaspace | -0.00MB | 0.00MB
2023-06-12 17:56:04,788 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component localFS loaded in clazz interface com.bytedance.bitsail.base.packages.PluginFinder. 2023-06-12 17:56:04,789 INFO com.bytedance.bitsail.base.packages.PluginStore [] - Try load plugins from base plugin dir /data/src_git/bitsail-dist/target/bitsail-dist-0.2.0-SNAPSHOT-bin/bitsail-archive-0.2.0-SNAPSHOT/./libs/connectors. 2023-06-12 17:56:04,789 INFO com.bytedance.bitsail.base.packages.PluginStore [] - Try load plugins from base plugin mapping dir /data/src_git/bitsail-dist/target/bitsail-dist-0.2.0-SNAPSHOT-bin/bitsail-archive-0.2.0-SNAPSHOT/./libs/connectors/mapping. 2023-06-12 17:56:04,921 INFO com.bytedance.bitsail.base.packages.PluginStore [] - Try load plugins from base plugin dir /data/src_git/bitsail-dist/target/bitsail-dist-0.2.0-SNAPSHOT-bin/bitsail-archive-0.2.0-SNAPSHOT/./libs/engines. 2023-06-12 17:56:04,921 INFO com.bytedance.bitsail.base.packages.PluginStore [] - Try load plugins from base plugin mapping dir /data/src_git/bitsail-dist/target/bitsail-dist-0.2.0-SNAPSHOT-bin/bitsail-archive-0.2.0-SNAPSHOT/./libs/engines/mapping. 2023-06-12 17:56:04,922 WARN com.bytedance.bitsail.base.packages.PluginStore [] - Class flink not register in mapping file. 2023-06-12 17:56:04,923 INFO com.bytedance.bitsail.base.packages.LocalFSPluginFinder [] - Plugin class loader add plugin url: file:/data/src_git/bitsail-dist/target/bitsail-dist-0.2.0-SNAPSHOT-bin/bitsail-archive-0.2.0-SNAPSHOT/./libs/engines/bitsail-engine-flink-0.2.0-SNAPSHOT.jar. 2023-06-12 17:56:04,927 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component flink loaded in clazz interface com.bytedance.bitsail.core.api.program.Program. 2023-06-12 17:56:04,927 INFO com.bytedance.bitsail.core.Engine [] - Final program: flink. 2023-06-12 17:56:05,683 INFO com.bytedance.bitsail.core.flink.bridge.program.FlinkDAGBuilderFactory [] - Reader class name is com.bytedance.bitsail.connector.legacy.mongodb.source.MongoDBInputFormat 2023-06-12 17:56:05,683 WARN com.bytedance.bitsail.base.packages.LocalFSPluginFinder [] - The class com.bytedance.bitsail.connector.legacy.mongodb.source.MongoDBInputFormat not exists in the class loader, try load from plugin store. 2023-06-12 17:56:05,684 INFO com.bytedance.bitsail.base.packages.LocalFSPluginFinder [] - Plugin class loader add plugin url: file:/data/src_git/bitsail-dist/target/bitsail-dist-0.2.0-SNAPSHOT-bin/bitsail-archive-0.2.0-SNAPSHOT/./libs/connectors/bitsail-connector-mongodb-0.2.0-SNAPSHOT.jar. 2023-06-12 17:56:05,691 INFO com.bytedance.bitsail.core.flink.bridge.program.FlinkDAGBuilderFactory [] - Writer class name is com.bytedance.bitsail.connector.legacy.hive.sink.HiveOutputFormat 2023-06-12 17:56:05,691 WARN com.bytedance.bitsail.base.packages.LocalFSPluginFinder [] - The class com.bytedance.bitsail.connector.legacy.hive.sink.HiveOutputFormat not exists in the class loader, try load from plugin store. 2023-06-12 17:56:05,692 INFO com.bytedance.bitsail.base.packages.LocalFSPluginFinder [] - Plugin class loader add plugin url: file:/data/src_git/bitsail-dist/target/bitsail-dist-0.2.0-SNAPSHOT-bin/bitsail-archive-0.2.0-SNAPSHOT/./libs/connectors/bitsail-connector-hive-0.2.0-SNAPSHOT.jar. 2023-06-12 17:56:05,708 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, bigdata-test 2023-06-12 17:56:05,708 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123 2023-06-12 17:56:05,708 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m 2023-06-12 17:56:05,708 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m 2023-06-12 17:56:05,708 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 8 2023-06-12 17:56:05,708 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 6 2023-06-12 17:56:05,708 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region 2023-06-12 17:56:05,720 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component log loaded in clazz interface com.bytedance.bitsail.base.metrics.reporter.MetricReporterBuilder. 2023-06-12 17:56:05,720 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component nop loaded in clazz interface com.bytedance.bitsail.base.metrics.reporter.MetricReporterBuilder. 2023-06-12 17:56:05,721 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component prometheus loaded in clazz interface com.bytedance.bitsail.base.metrics.reporter.MetricReporterBuilder. 2023-06-12 17:56:05,721 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component prometheus_pushgateway loaded in clazz interface com.bytedance.bitsail.base.metrics.reporter.MetricReporterBuilder. 2023-06-12 17:56:05,724 INFO com.bytedance.bitsail.base.metrics.ScheduledMetricReporterWrap [] - Periodically reporting metrics in intervals of 1 MINUTES for reporter com.bytedance.bitsail.base.metrics.reporter.impl.LogMetricReporter@6cff61fc of type com.bytedance.bitsail.base.metrics.reporter.impl.LogMetricReporter. 2023-06-12 17:56:05,728 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component hive loaded in clazz interface com.bytedance.bitsail.base.catalog.TableCatalogFactory. 2023-06-12 17:56:05,729 INFO com.bytedance.bitsail.base.component.DefaultComponentBuilderLoader [] - Component hive loaded in clazz interface com.bytedance.bitsail.base.catalog.TableCatalogFactory. 2023-06-12 17:56:05,729 WARN com.bytedance.bitsail.flink.core.execution.configurer.FlinkDAGBuilderInterceptor [] - Ignore align engine catalog, reader or writer not support table catalog factory. 2023-06-12 17:56:05,745 INFO com.bytedance.bitsail.connector.legacy.mongodb.source.MongoDBInputFormat [] - No reader_parallelism_num, use default value: [5] 2023-06-12 17:56:05,829 INFO org.mongodb.driver.cluster [] - Cluster created with settings {hosts=[10.206.117.32:18009, 10.206.117.33:18009, 10.206.117.34:18009], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500} 2023-06-12 17:56:05,829 INFO org.mongodb.driver.cluster [] - Adding discovered server 10.206.117.32:18009 to client view of cluster 2023-06-12 17:56:05,861 INFO org.mongodb.driver.cluster [] - Adding discovered server 10.206.117.33:18009 to client view of cluster 2023-06-12 17:56:05,862 INFO org.mongodb.driver.cluster [] - Adding discovered server 10.206.117.34:18009 to client view of cluster 2023-06-12 17:56:05,904 INFO org.mongodb.driver.cluster [] - Cluster description not yet available. Waiting for 30000 ms before timing out 2023-06-12 17:56:05,955 INFO org.mongodb.driver.connection [] - Opened connection [connectionId{localValue:2, serverValue:86774}] to 10.206.117.34:18009 2023-06-12 17:56:05,958 INFO org.mongodb.driver.connection [] - Opened connection [connectionId{localValue:1, serverValue:86405}] to 10.206.117.33:18009 2023-06-12 17:56:05,959 INFO org.mongodb.driver.connection [] - Opened connection [connectionId{localValue:3, serverValue:92210}] to 10.206.117.32:18009 2023-06-12 17:56:05,968 INFO org.mongodb.driver.cluster [] - Monitor thread successfully connected to server with description ServerDescription{address=10.206.117.33:18009, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 17]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2865775, setName='rs18007', canonicalAddress=10.206.117.33:18009, hosts=[10.206.117.34:18009, 10.206.117.32:18009, 10.206.117.33:18009], passives=[], arbiters=[], primary='10.206.117.32:18009', tagSet=TagSet{[]}, electionId=null, setVersion=19, lastWriteDate=Mon Jun 12 17:56:05 CST 2023, lastUpdateTimeNanos=1529366443204121} 2023-06-12 17:56:05,968 INFO org.mongodb.driver.cluster [] - Monitor thread successfully connected to server with description ServerDescription{address=10.206.117.34:18009, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 17]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=10408233, setName='rs18007', canonicalAddress=10.206.117.34:18009, hosts=[10.206.117.34:18009, 10.206.117.32:18009, 10.206.117.33:18009], passives=[], arbiters=[], primary='10.206.117.32:18009', tagSet=TagSet{[]}, electionId=null, setVersion=19, lastWriteDate=Mon Jun 12 17:56:05 CST 2023, lastUpdateTimeNanos=1529366447624037} 2023-06-12 17:56:05,969 INFO org.mongodb.driver.cluster [] - Monitor thread successfully connected to server with description ServerDescription{address=10.206.117.32:18009, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 2, 17]}, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=6948713, setName='rs18007', canonicalAddress=10.206.117.32:18009, hosts=[10.206.117.34:18009, 10.206.117.32:18009, 10.206.117.33:18009], passives=[], arbiters=[], primary='10.206.117.32:18009', tagSet=TagSet{[]}, electionId=7fffffff0000000000000004, setVersion=19, lastWriteDate=Mon Jun 12 17:56:05 CST 2023, lastUpdateTimeNanos=1529366447629341} 2023-06-12 17:56:05,969 INFO org.mongodb.driver.cluster [] - Discovered cluster type of REPLICA_SET 2023-06-12 17:56:05,970 INFO org.mongodb.driver.cluster [] - Setting max election id to 7fffffff0000000000000004 from replica set primary 10.206.117.32:18009 2023-06-12 17:56:05,970 INFO org.mongodb.driver.cluster [] - Setting max set version to 19 from replica set primary 10.206.117.32:18009 2023-06-12 17:56:05,971 INFO org.mongodb.driver.cluster [] - Discovered replica set primary 10.206.117.32:18009 2023-06-12 17:56:06,033 INFO org.mongodb.driver.connection [] - Closed connection [connectionId{localValue:4}] to 10.206.117.32:18009 because there was a socket exception raised by this connection. 2023-06-12 17:56:06,035 ERROR com.bytedance.bitsail.core.Engine [] -
The cause of the job failure maybe due to:
com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='data_collect_user', source='data_collect_prod', password=
2023-06-12 17:56:06,037 INFO com.bytedance.bitsail.flink.core.execution.FlinkExecutionEnviron [] - Flink job start terminal. 2023-06-12 17:56:06,038 INFO com.bytedance.bitsail.base.metrics.ScheduledMetricReporterWrap [] - Close metric reporter. 2023-06-12 17:56:09,039 WARN com.bytedance.bitsail.base.metrics.ScheduledMetricReporterWrap [] - Metric executor service did not terminate in time, shutting it down now. 2023-06-12 17:56:09,040 INFO com.bytedance.bitsail.flink.core.execution.FlinkExecutionEnviron [] - Flink job terminal finished.
What do you expect to happen
连接Mongo会报错
How to reproduce
使用账户密码连接Mongo读连接器，可能因为密码中有#号的原因
Build Environment
centos
Execution Environment
No response
BitSail version
master
BitSail Component or Code Module
BitSail Core
Are you willing to submit PR?
- [X] Yes, I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
