ecommerce_realtime_data_pipeline
ecommerce_realtime_data_pipeline copied to clipboard
Published
20 hours ago •
behnamyazdan
behnamyazdan
dependency failed to start: container debezium exited (127)
trafficstars
Hi @behnamyazdan,
After running the "docker-compose up" command, I encountered the following error. Do you have any ideas on how to fix it? I am using a Mac with an M1 chip and allocating 12GB of RAM for the Docker environment.System info
Error log :
Attaching to airflow-init, airflow-scheduler, airflow-webserver, clickhouse, debezium, debezium-connector-init, debezium-ui, grafana, kafka-broker, kafka-ui, postgres-main, postgres_airflow, zookeeper
debezium | Using BOOTSTRAP_SERVERS=kafka-broker:19092
debezium | Plugins are loaded from /kafka/connect
debezium | Debezium Scripting enabled!
debezium | Using the following environment variables:
debezium | GROUP_ID=1
debezium | CONFIG_STORAGE_TOPIC=connect_configs
debezium | OFFSET_STORAGE_TOPIC=connect_offsets
debezium | STATUS_STORAGE_TOPIC=connect_statuses
debezium | BOOTSTRAP_SERVERS=kafka-broker:19092
debezium | REST_HOST_NAME=172.20.0.7
debezium | REST_PORT=8083
debezium | ADVERTISED_HOST_NAME=172.20.0.7
debezium | ADVERTISED_PORT=8083
debezium | KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
debezium | VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
debezium | OFFSET_FLUSH_INTERVAL_MS=60000
debezium | OFFSET_FLUSH_TIMEOUT_MS=5000
debezium | SHUTDOWN_TIMEOUT=10000
debezium | --- Setting property from CONNECT_REST_ADVERTISED_PORT: rest.advertised.port=8083
debezium | --- Setting property from CONNECT_OFFSET_STORAGE_TOPIC: offset.storage.topic=connect_offsets
debezium | --- Setting property from CONNECT_KEY_CONVERTER: key.converter=org.apache.kafka.connect.json.JsonConverter
debezium | --- Setting property from CONNECT_CONFIG_STORAGE_TOPIC: config.storage.topic=connect_configs
debezium | --- Setting property from CONNECT_GROUP_ID: group.id=1
debezium | --- Setting property from CONNECT_REST_ADVERTISED_HOST_NAME: rest.advertised.host.name=172.20.0.7
debezium | --- Setting property from CONNECT_REST_HOST_NAME: rest.host.name=172.20.0.7
debezium | --- Setting property from CONNECT_VALUE_CONVERTER: value.converter=org.apache.kafka.connect.json.JsonConverter
debezium | --- Setting property from CONNECT_REST_PORT: rest.port=8083
debezium | --- Setting property from CONNECT_STATUS_STORAGE_TOPIC: status.storage.topic=connect_statuses
debezium | --- Setting property from CONNECT_OFFSET_FLUSH_TIMEOUT_MS: offset.flush.timeout.ms=5000
debezium | --- Setting property from CONNECT_PLUGIN_PATH: plugin.path=/kafka/connect
debezium | --- Setting property from CONNECT_OFFSET_FLUSH_INTERVAL_MS: offset.flush.interval.ms=60000
debezium | --- Setting property from CONNECT_BOOTSTRAP_SERVERS: bootstrap.servers=kafka-broker:19092
debezium | --- Setting property from CONNECT_TASK_SHUTDOWN_GRACEFUL_TIMEOUT_MS: task.shutdown.graceful.timeout.ms=10000
debezium | 2024-06-26 10:00:14,356 INFO || Kafka Connect worker initializing ... [org.apache.kafka.connect.cli.AbstractConnectCli]
debezium | 2024-06-26 10:00:14,364 INFO || WorkerInfo values:
debezium | jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/kafka/logs, -Dlog4j.configuration=file:/kafka/config/log4j.properties
debezium | jvm.spec = Red Hat, Inc., OpenJDK 64-Bit Server VM, 11.0.20, 11.0.20+8
debezium | jvm.classpath = /kafka/libs/activation-1.1.1.jar:/kafka/libs/aopalliance-repackaged-2.6.1.jar:/kafka/libs/argparse4j-0.7.0.jar:/kafka/libs/audience-annotations-0.12.0.jar:/kafka/libs/caffeine-2.9.3.jar:/kafka/libs/checker-qual-3.19.0.jar:/kafka/libs/commons-beanutils-1.9.4.jar:/kafka/libs/commons-cli-1.4.jar:/kafka/libs/commons-collections-3.2.2.jar:/kafka/libs/commons-digester-2.1.jar:/kafka/libs/commons-io-2.11.0.jar:/kafka/libs/commons-lang3-3.8.1.jar:/kafka/libs/commons-logging-1.2.jar:/kafka/libs/commons-validator-1.7.jar:/kafka/libs/connect-api-3.6.1.jar:/kafka/libs/connect-basic-auth-extension-3.6.1.jar:/kafka/libs/connect-json-3.6.1.jar:/kafka/libs/connect-mirror-3.6.1.jar:/kafka/libs/connect-mirror-client-3.6.1.jar:/kafka/libs/connect-runtime-3.6.1.jar:/kafka/libs/connect-transforms-3.6.1.jar:/kafka/libs/error_prone_annotations-2.10.0.jar:/kafka/libs/hk2-api-2.6.1.jar:/kafka/libs/hk2-locator-2.6.1.jar:/kafka/libs/hk2-utils-2.6.1.jar:/kafka/libs/jackson-annotations-2.13.5.jar:/kafka/libs/jackson-core-2.13.5.jar:/kafka/libs/jackson-databind-2.13.5.jar:/kafka/libs/jackson-dataformat-csv-2.13.5.jar:/kafka/libs/jackson-datatype-jdk8-2.13.5.jar:/kafka/libs/jackson-jaxrs-base-2.13.5.jar:/kafka/libs/jackson-jaxrs-json-provider-2.13.5.jar:/kafka/libs/jackson-module-jaxb-annotations-2.13.5.jar:/kafka/libs/jackson-module-scala_2.13-2.13.5.jar:/kafka/libs/jakarta.activation-api-1.2.2.jar:/kafka/libs/jakarta.annotation-api-1.3.5.jar:/kafka/libs/jakarta.inject-2.6.1.jar:/kafka/libs/jakarta.validation-api-2.0.2.jar:/kafka/libs/jakarta.ws.rs-api-2.1.6.jar:/kafka/libs/jakarta.xml.bind-api-2.3.3.jar:/kafka/libs/javassist-3.29.2-GA.jar:/kafka/libs/javax.activation-api-1.2.0.jar:/kafka/libs/javax.annotation-api-1.3.2.jar:/kafka/libs/javax.servlet-api-3.1.0.jar:/kafka/libs/javax.ws.rs-api-2.1.1.jar:/kafka/libs/jaxb-api-2.3.1.jar:/kafka/libs/jersey-client-2.39.1.jar:/kafka/libs/jersey-common-2.39.1.jar:/kafka/libs/jersey-container-servlet-2.39.1.jar:/kafka/libs/jersey-container-servlet-core-2.39.1.jar:/kafka/libs/jersey-hk2-2.39.1.jar:/kafka/libs/jersey-server-2.39.1.jar:/kafka/libs/jetty-client-9.4.52.v20230823.jar:/kafka/libs/jetty-continuation-9.4.52.v20230823.jar:/kafka/libs/jetty-http-9.4.52.v20230823.jar:/kafka/libs/jetty-io-9.4.52.v20230823.jar:/kafka/libs/jetty-security-9.4.52.v20230823.jar:/kafka/libs/jetty-server-9.4.52.v20230823.jar:/kafka/libs/jetty-servlet-9.4.52.v20230823.jar:/kafka/libs/jetty-servlets-9.4.52.v20230823.jar:/kafka/libs/jetty-util-9.4.52.v20230823.jar:/kafka/libs/jetty-util-ajax-9.4.52.v20230823.jar:/kafka/libs/jline-3.22.0.jar:/kafka/libs/jolokia-jvm-1.7.2.jar:/kafka/libs/jopt-simple-5.0.4.jar:/kafka/libs/jose4j-0.9.3.jar:/kafka/libs/jsr305-3.0.2.jar:/kafka/libs/kafka-clients-3.6.1.jar:/kafka/libs/kafka-group-coordinator-3.6.1.jar:/kafka/libs/kafka-log4j-appender-3.6.1.jar:/kafka/libs/kafka-metadata-3.6.1.jar:/kafka/libs/kafka-raft-3.6.1.jar:/kafka/libs/kafka-server-common-3.6.1.jar:/kafka/libs/kafka-shell-3.6.1.jar:/kafka/libs/kafka-storage-3.6.1.jar:/kafka/libs/kafka-storage-api-3.6.1.jar:/kafka/libs/kafka-streams-3.6.1.jar:/kafka/libs/kafka-streams-examples-3.6.1.jar:/kafka/libs/kafka-streams-scala_2.13-3.6.1.jar:/kafka/libs/kafka-streams-test-utils-3.6.1.jar:/kafka/libs/kafka-tools-3.6.1.jar:/kafka/libs/kafka-tools-api-3.6.1.jar:/kafka/libs/kafka_2.13-3.6.1.jar:/kafka/libs/lz4-java-1.8.0.jar:/kafka/libs/maven-artifact-3.8.8.jar:/kafka/libs/metrics-core-2.2.0.jar:/kafka/libs/metrics-core-4.1.12.1.jar:/kafka/libs/netty-buffer-4.1.100.Final.jar:/kafka/libs/netty-codec-4.1.100.Final.jar:/kafka/libs/netty-common-4.1.100.Final.jar:/kafka/libs/netty-handler-4.1.100.Final.jar:/kafka/libs/netty-resolver-4.1.100.Final.jar:/kafka/libs/netty-transport-4.1.100.Final.jar:/kafka/libs/netty-transport-classes-epoll-4.1.100.Final.jar:/kafka/libs/netty-transport-native-epoll-4.1.100.Final.jar:/kafka/libs/netty-transport-native-unix-common-4.1.100.Final.jar:/kafka/libs/osgi-resource-locator-1.0.3.jar:/kafka/libs/paranamer-2.8.jar:/kafka/libs/pcollections-4.0.1.jar:/kafka/libs/plexus-utils-3.3.1.jar:/kafka/libs/reflections-0.10.2.jar:/kafka/libs/reload4j-1.2.25.jar:/kafka/libs/rocksdbjni-7.9.2.jar:/kafka/libs/scala-collection-compat_2.13-2.10.0.jar:/kafka/libs/scala-java8-compat_2.13-1.0.2.jar:/kafka/libs/scala-library-2.13.11.jar:/kafka/libs/scala-logging_2.13-3.9.4.jar:/kafka/libs/scala-reflect-2.13.11.jar:/kafka/libs/slf4j-api-1.7.36.jar:/kafka/libs/slf4j-reload4j-1.7.36.jar:/kafka/libs/snappy-java-1.1.10.5.jar:/kafka/libs/swagger-annotations-2.2.8.jar:/kafka/libs/trogdor-3.6.1.jar:/kafka/libs/zookeeper-3.8.3.jar:/kafka/libs/zookeeper-jute-3.8.3.jar:/kafka/libs/zstd-jni-1.5.5-1.jar
debezium | os.spec = Linux, amd64, 6.6.31-linuxkit
debezium | os.vcpus = 8
debezium | [org.apache.kafka.connect.runtime.WorkerInfo]
debezium | 2024-06-26 10:00:14,364 INFO || Scanning for plugin classes. This might take a moment ... [org.apache.kafka.connect.cli.AbstractConnectCli]
debezium | 2024-06-26 10:00:14,428 INFO || Loading plugin from: /kafka/connect/debezium-connector-spanner [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:14,573 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
zookeeper | [2024-06-26 10:00:14,693] INFO Processing srvr command from /172.20.0.2:46630 (org.apache.zookeeper.server.NIOServerCnxn)
debezium | 2024-06-26 10:00:14,815 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-spanner/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:14,818 INFO || Loading plugin from: /kafka/connect/debezium-connector-oracle [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:14,880 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:14,933 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-oracle/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:14,933 INFO || Loading plugin from: /kafka/connect/debezium-connector-vitess [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:14,972 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:15,010 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-vitess/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,010 INFO || Loading plugin from: /kafka/connect/debezium-connector-sqlserver [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,026 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:15,063 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-sqlserver/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,242 INFO || Loading plugin from: /kafka/connect/debezium-connector-informix [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,266 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:15,304 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-informix/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,305 INFO || Loading plugin from: /kafka/connect/debezium-connector-mongodb [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,318 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:15,392 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-mongodb/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,392 INFO || Loading plugin from: /kafka/connect/debezium-connector-mysql [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,410 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:15,443 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-mysql/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,447 INFO || Loading plugin from: /kafka/connect/debezium-connector-db2 [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,480 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:15,521 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-db2/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,522 INFO || Loading plugin from: /kafka/connect/debezium-connector-jdbc [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,552 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:15,589 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-jdbc/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,654 INFO || Loading plugin from: /kafka/connect/debezium-connector-postgres [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,674 INFO || Using up-to-date JsonConverter implementation [io.debezium.converters.CloudEventsConverter]
debezium | 2024-06-26 10:00:15,704 INFO || Registered loader: PluginClassLoader{pluginLocation=file:/kafka/connect/debezium-connector-postgres/} [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,706 INFO || Loading plugin from: classpath [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,716 INFO || Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@3d4eac69 [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,717 INFO || Scanning plugins with ServiceLoaderScanner took 1291 ms [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | 2024-06-26 10:00:15,721 INFO || Loading plugin from: /kafka/connect/debezium-connector-spanner [org.apache.kafka.connect.runtime.isolation.PluginScanner]
debezium | #
debezium | # A fatal error has been detected by the Java Runtime Environment:
debezium | #
debezium | # SIGSEGV (0xb) at pc=0x00007ffffea68481, pid=1, tid=736
debezium | #
debezium | # JRE version: OpenJDK Runtime Environment (Red_Hat-11.0.20.0.8-1.fc37) (11.0.20+8) (build 11.0.20+8)
debezium | # Java VM: OpenJDK 64-Bit Server VM (Red_Hat-11.0.20.0.8-1.fc37) (11.0.20+8, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
debezium | # Problematic frame:
debezium | # V [libjvm.so+0x7f5481] G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x51
debezium | #
debezium | # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
debezium | #
debezium | # An error report file with more information is saved as:
debezium | # /kafka/hs_err_pid1.log
debezium | #
debezium | # If you would like to submit a bug report, please visit:
debezium | # https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora&component=java-11-openjdk-portable&version=37
debezium | #
debezium |
debezium | [error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007fffff5ec899]
debezium |
[Kdebezium exited with code 127
Hi @behnamyazdan ,
Here is the docker version
Docker version 26.1.4, build 5650f9b