Got "Servname not supported for ai_socktype" when deploy Karapace Schema Registry

[hongbo-miao]

Originally asked in Stack Overflow. Below is a copy. Any guide would be appreciate, thanks! 😃

---

I am trying to deploy Karapace Schema Registry in a local Kubernetes based on Karapace's docker-compose.yml. Here is a copy:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9101:9101"  # JMX
      - "9092:9092"  # Kafka
    environment:
      # Listeners:
      # PLAINTEXT_HOST -> Expose kafka to the host network
      # PLAINTEXT -> Used by kafka for inter broker communication / containers
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://karapace-registry:8081
      # Metrics:
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      # Keep in sync with tests/integration/conftest.py::configure_and_start_kafka
      KAFKA_BROKER_ID: 1
      KAFKA_BROKER_RACK: "local"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_INTER_BROKER_PROTOCOL_VERSION: 2.4
      KAFKA_LOG_CLEANER_ENABLE: "true"
      KAFKA_LOG_MESSAGE_FORMAT_VERSION: 2.4
      KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 300000
      KAFKA_LOG_SEGMENT_BYTES: 209715200
      KAFKA_NUM_IO_THREADS: 8
      KAFKA_NUM_NETWORK_THREADS: 112
      KAFKA_NUM_PARTITIONS: 1
      KAFKA_NUM_REPLICA_FETCHERS: 4
      KAFKA_NUM_RECOVERY_THREADS_PER_DATA_DIR: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_SOCKET_RECEIVE_BUFFER_BYTES: 102400
      KAFKA_SOCKET_REQUEST_MAX_BYTES: 104857600
      KAFKA_SOCKET_SEND_BUFFER_BYTES: 102400
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_NUM_PARTITIONS: 16
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"

  karapace-registry:
    image: ghcr.io/aiven/karapace:develop
    entrypoint:
      - /bin/bash
      - /opt/karapace/start.sh
      - registry
    depends_on:
      - kafka
    ports:
      - "8081:8081"
    environment:
      KARAPACE_ADVERTISED_HOSTNAME: karapace-registry
      KARAPACE_BOOTSTRAP_URI: kafka:29092
      KARAPACE_PORT: 8081
      # For `KARAPACE_HOST`, I also tried `localhost`, `karapace-registry.hm-karapace.svc`, but none of them works.
      KARAPACE_HOST: 0.0.0.0
      KARAPACE_CLIENT_ID: karapace
      KARAPACE_GROUP_ID: karapace-registry
      KARAPACE_MASTER_ELIGIBILITY: "true"
      KARAPACE_TOPIC_NAME: _schemas
      KARAPACE_LOG_LEVEL: WARNING
      KARAPACE_COMPATIBILITY: FULL

  karapace-rest:
    image: ghcr.io/aiven/karapace:develop
    entrypoint:
      - /bin/bash
      - /opt/karapace/start.sh
      - rest
    depends_on:
      - kafka
      - karapace-registry
    ports:
      - "8082:8082"
    environment:
      KARAPACE_PORT: 8082
      KARAPACE_HOST: 0.0.0.0
      KARAPACE_ADVERTISED_HOSTNAME: karapace-rest
      KARAPACE_BOOTSTRAP_URI: kafka:29092
      KARAPACE_REGISTRY_HOST: karapace-registry
      KARAPACE_REGISTRY_PORT: 8081
      KARAPACE_ADMIN_METADATA_MAX_AGE: 0
      KARAPACE_LOG_LEVEL: WARNING

As I already have Kafka in my local Kuberentes, and karapace-rest depends on karapace-registry, so I hope to make karapace-registry work first.

Here is my attempt converting karapace-registry to use Kubernetes YAML files:

hm-karapace-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: hm-karapace

karapace-registry-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: karapace-registry-deployment
  namespace: hm-karapace
  labels:
    app.kubernetes.io/name: karapace-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: karapace-registry
  template:
    metadata:
      labels:
        app: karapace-registry
    spec:
      containers:
        - name: karapace-registry
          # I changed from tag `develop` to latest version `3.4.6`
          image: ghcr.io/aiven/karapace:3.4.6
          command:
            - /bin/bash
            - /opt/karapace/start.sh
            - registry
          env:
            - name: KARAPACE_ADVERTISED_HOSTNAME
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_advertised_hostname
            - name: KARAPACE_BOOTSTRAP_URI
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_bootstrap_uri
            - name: KARAPACE_CLIENT_ID
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_client_id
            - name: KARAPACE_COMPATIBILITY
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_compatibility
            - name: KARAPACE_GROUP_ID
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_group_id
            - name: KARAPACE_LOG_LEVEL
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_log_level
            - name: KARAPACE_MASTER_ELIGIBILITY
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_master_eligibility
            - name: KARAPACE_HOST
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_host
            - name: KARAPACE_PORT
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_port
            - name: KARAPACE_TOPIC_NAME
              valueFrom:
                configMapKeyRef:
                  name: karapace-registry-configmap
                  key: karapace_topic_name
          ports:
            - name: registry
              protocol: TCP
              containerPort: 8081

karapace-registry-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: karapace-registry-configmap
  namespace: hm-karapace
  labels:
    app.kubernetes.io/name: karapace-registry
data:
  karapace_advertised_hostname: karapace-registry.hm-karapace.svc
  # My Kafka bootstrap is at hm-kafka-kafka-bootstrap.hm-kafka.svc:9092
  karapace_bootstrap_uri: hm-kafka-kafka-bootstrap.hm-kafka.svc:9092
  karapace_client_id: karapace
  karapace_compatibility: FULL
  karapace_group_id: karapace-registry
  karapace_log_level: WARNING
  karapace_master_eligibility: "true"
  karapace_host: 0.0.0.0
  karapace_port: "8081"
  karapace_topic_name: _schemas

karapace-registry-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: karapace-registry
  namespace: hm-karapace
  labels:
    app.kubernetes.io/name: karapace-registry
spec:
  type: ClusterIP
  selector:
    app: karapace-registry
  ports:
    - name: registry
      protocol: TCP
      targetPort: 8081
      port: 8081

However, when I deploy, I got error log for this Karapace Registry pod:

Starting Karapace Schema Registry
Cannot enable Sentry.io sending: importing 'sentry_sdk' failed
asyncio             	MainThread	ERROR   	unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-1' coro=<_run_app() done, defined at /usr/local/lib/python3.9/dist-packages/aiohttp/web.py:289> exception=gaierror(-8, 'Servname not supported for ai_socktype')>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web.py", line 516, in run_app
    loop.run_until_complete(main_task)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web.py", line 415, in _run_app
    await site.start()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web_runner.py", line 121, in start
    self._server = await loop.create_server(
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1460, in create_server
    infos = await tasks.gather(*fs, loop=self)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1400, in _create_server_getaddrinfo
    infos = await self._ensure_resolved((host, port), family=family,
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1396, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
  File "/usr/lib/python3.9/asyncio/base_events.py", line 856, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -8] Servname not supported for ai_socktype
karapace.statsd     	MainThread	ERROR   	Unexpected exception in statsd send
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/karapace/karapace_all.py", line 66, in main
    app.run()
  File "/usr/local/lib/python3.9/dist-packages/karapace/rapu.py", line 471, in run
    aiohttp.web.run_app(
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web.py", line 516, in run_app
    loop.run_until_complete(main_task)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web.py", line 415, in _run_app
    await site.start()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web_runner.py", line 121, in start
    self._server = await loop.create_server(
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1460, in create_server
    infos = await tasks.gather(*fs, loop=self)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1400, in _create_server_getaddrinfo
    infos = await self._ensure_resolved((host, port), family=family,
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1396, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
  File "/usr/lib/python3.9/asyncio/base_events.py", line 856, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -8] Servname not supported for ai_socktype

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/karapace/statsd.py", line 82, in _send
    self._socket.sendto(b"".join(parts), self._dest_addr)
OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/dist-packages/karapace/karapace_all.py", line 74, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/karapace/karapace_all.py", line 66, in main
    app.run()
  File "/usr/local/lib/python3.9/dist-packages/karapace/rapu.py", line 471, in run
    aiohttp.web.run_app(
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web.py", line 516, in run_app
    loop.run_until_complete(main_task)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web.py", line 415, in _run_app
    await site.start()
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/web_runner.py", line 121, in start
    self._server = await loop.create_server(
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1460, in create_server
    infos = await tasks.gather(*fs, loop=self)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1400, in _create_server_getaddrinfo
    infos = await self._ensure_resolved((host, port), family=family,
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1396, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
  File "/usr/lib/python3.9/asyncio/base_events.py", line 856, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -8] Servname not supported for ai_socktype

For KARAPACE_HOST inside, I tried 0.0.0.0, localhost, karapace-registry.hm-karapace.svc, but none of them works.

I found a similar issue with a potential solution at https://github.com/aiven/karapace/issues/397#issuecomment-1108559512

But I still didn't get what it means.

Plus I am not using Terraform, so I won't be able to use enable_service_links in the reply.

Any guide would be appreciate. Thanks!

[tvainika]

Pointed out in Stack Overflow seems to be on the spot: A configuration toggle to disable StatsD client fully is missing. Need to add that so that karapace does not assume telegraf client would be running on same host 8125