canal
canal copied to clipboard
alibaba
canal同步mysql 8.4到elasticsearch 8.15使用transport https时会报错，使用rest正常
环境介绍： canal 1.1.8 elasticsearch 8.15.0 mysql 8.4.2
elasticsearch开启了https和安全认证：
# egrep -v '(#|^$)' /etc/elasticsearch/elasticsearch.yml
cluster.name: my-application
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.11.240
http.port: 9200
transport.port: 9300
discovery.seed_hosts: []
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: []
http.host: 0.0.0.0
错误日志： 配置完成canal后，启动都是正常的。 我使用curl命令同步数据时报错：
curl http://127.0.0.1:8081/etl/es8/dpm_account_tdoad.yml -X POST -d "params=2021-07-22 00:00:00;2024-12-31 23:59:59"
{"succeeded":false,"resultMessage":"导入ES 数据：0 条","errorMessage":"dpm_account_tdoad etl failed! ==>java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\ndpm_account_tdoad etl failed! ==>java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
我的canal adapter配置的es如下：
- name: es8
hosts: https://192.168.11.240:9300 # 127.0.0.1:9200 for rest mode
properties:
mode: transport # or rest
security.ca.path: /etc/elasticsearch/certs/http_ca.crt
security.auth: elastic:xxxxxxxxxxxx # only used for rest mode
cluster.name: my-application
我在网上搜索说将浏览器访问的https证书下载下来导入jdk信任cacerts：
keytool -import -file /root/es_ca.pem -alias es8_http_ca -keystore /usr/local/jdk-11.0.24+8/lib/security/cacerts
导入后删除命令：
keytool -delete -alias es8_http_ca -keystore /usr/local/jdk-11.0.24+8/lib/security/cacerts
canal adapter的配置改为主机名形式：
hosts: https://hostname:9300 # 127.0.0.1:9200 for rest mode
properties:
mode: transport # or rest
security.ca.path: /etc/elasticsearch/certs/http_ca.crt
security.auth: elastic:xxxxxxxxxxxx # only used for rest mode
cluster.name: my-application
然后再次请求报错：
curl http://127.0.0.1:8081/etl/es8/dpm_account_tdoad.yml -X POST -d "params=2021-07-22 00:00:00;2024-12-31 23:59:59"
{"succeeded":false,"resultMessage":"导入ES 数据：0 条","errorMessage":"dpm_account_tdoad etl failed! ==>java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate\ndpm_account_tdoad etl failed! ==>java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate"}
2024-09-04 12:53:25.525 [pool-4-thread-2] ERROR c.a.otter.canal.client.adapter.es8x.etl.ESEtlService - java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118)
at org.elasticsearch.client.IndicesClient.getMapping(IndicesClient.java:538)
at com.alibaba.otter.canal.client.adapter.es8x.support.ESConnection.getMapping(ESConnection.java:136)
at com.alibaba.otter.canal.client.adapter.es8x.support.ES8xTemplate.getEsType(ES8xTemplate.java:393)
at com.alibaba.otter.canal.client.adapter.es8x.support.ES8xTemplate.getValFromRS(ES8xTemplate.java:183)
at com.alibaba.otter.canal.client.adapter.es8x.etl.ESEtlService.lambda$executeSqlImport$1(ESEtlService.java:82)
at com.alibaba.otter.canal.client.adapter.support.Util.sqlRS(Util.java:62)
at com.alibaba.otter.canal.client.adapter.es8x.etl.ESEtlService.executeSqlImport(ESEtlService.java:64)
at com.alibaba.otter.canal.client.adapter.support.AbstractEtlService.lambda$importData$1(AbstractEtlService.java:91)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
... 15 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:347)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:186)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:278)
at org.apache.http.nio.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:497)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:540)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
... 1 common frames omitted
最后改为了rest模式就可以同步了，这是什么原因导致的，有什么解决方案吗？
- name: es8
hosts: https://192.168.11.240:9200 # 127.0.0.1:9200 for rest mode
properties:
mode: rest #transport # or rest
security.ca.path: /etc/elasticsearch/certs/http_ca.crt
security.auth: elastic:xxxxxxxxx # only used for rest mode
cluster.name: my-application
导入数据成功：
curl http://127.0.0.1:8081/etl/es8/dpm_txncore_order.yml -X POST -d "params=2021-07-22 00:00:00;2024-12-31 23:59:59"
{"succeeded":true,"resultMessage":"导入ES 数据：36362 条"}
