Unable to setup authentication for schema-registry

[radhikari-3]

What specific settings must I make in order to enable basic authentication so that I may access the Karapace Schema Registry? I can currently create schemas using the curl code below with or without a password. It appears to be a configuration issue.

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\": \"record\", \"name\": \"Obj\", \"fields\":[{\"name\": \"age\", \"type\": \"int\"}]}"}' \
  http://localhost:8081/subjects/test-key/versions

Please outline the specific processes for me. My objective is to prohibit unauthorised access to this endpoint at https://localhost:8081.

And after the authentication is enabled, I assume the curl to create schema should look something like below :

curl -u username:password -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\": \"record\", \"name\": \"Obj\", \"fields\":[{\"name\": \"age\", \"type\": \"int\"}]}"}' \
  http://localhost:8081/subjects/test-key/versions

or may be I should pass some auth headers to create schema

curl -X POST 'http://localhost:8081/subjects/test-key/versions' \
-H'Content-Type: application/vnd.schemaregistry.v1+json' \
-H 'Authorization: <password>' \
--data '{"schema": "{\"type\": \"record\", \"name\": \"Obj\", \"fields\":[{\"name\": \"age\", \"type\": \"int\"}]}"}'

[tvainika]

You need to add registry_authfile setting to Karapace config json. The details of the configuration are in the chapter Authentication and authorization of Karapace Schema Registry REST API https://github.com/aiven/karapace#authentication-and-authorization-of-karapace-schema-registry-rest-api

Hope this help. Please let me know if something is unclear on the documentation.

[radhikari-3]

Just to get this correctly, I am writing down the steps to be performed:

Step 1: I need to create a hashed secret using the command karapace_mkpasswd -u <user-name> -a <algorithm> <password> [hashkey (optional)] Note: Giving the hash key is optional; if it's not provided the command generates one on its own.

Step 2: Create an authfile.json having the user credentials and authorization in it.

{
    "users": [
        {
            "username": "admin",
            "algorithm": "sha512",
            "salt": "<put salt for randomized hashing here>",
            "password_hash": "<put hashed password here>"
        }
    ],
    "permissions": [
        {
            "username": "admin",
            "operation": "Write",
            "resource": ".*"
        }
    ]
}

Step 3: Pass the file path of authfile.json in environment variable KARAPACE_REGISTRY_AUTHFILE and restart the server.

Do we need to enable any additional flags in addition to the above configuration mentioned above? If not, how can I test the aforementioned scenario for the admin user in this situation? Please outline the steps.

[tvainika]

You're correct here. If KARAPACE_REGISTRY_AUTHFILE or registry_authfile in karapace config json is set, then all schema registry requests are authorized using those credentials and permissions.

However please note that REST API is separate in case you are running both Schema Registry and Kafka REST API in same Karapace instance.

[radhikari-3]

Thank you for you help earlier, I am able to setup the BASIC authentication on subjects/* endpoint. However, I am still not been able to test the authorization.

I was attempting to understand your last statement and, in light of that, I was attempting to determine what it meant to have both the karapace_rest and karapace_registry options set to true. Then I understood that it meant I was using the same Karapace instance to run both the Kafka REST API and the Schema Registry.

What do you mean by REST API being separate, based on your previous comment?

Also, I tried to create a schema using this curl curl --location 'https://karapace.app.mydomain.com/subjects/example/versions' \ --header 'Content-Type: application/vnd.schemaregistry.v1+json' \ --header 'Authorization: Basic ZGVhci11bmljb3JuOmJhbmtpbmdzdWl0ZQ==' \ --data '{ "schema": "{\"namespace\": \"example.avro\", \"type\": \"record\", \"name\": \"simple\", \"fields\": [{\"name\": \"name\", \"type\": \"string\"}]}" }'

To which I am getting a successful response. { "id": 2 }

But, when I am trying to execute the below curl given in karapace documentation

Produce a message backed up by schema registry:

curl --location 'https://karapace.app.mydomain.com/topics/example' \
--header 'Content-Type: application/vnd.kafka.avro.v2+json' \
--header 'Authorization: Basic ZGVhci11bmljb3JuOmJhbmtpbmdzdWl0ZQ==' \
--data '{
    "value_schema": "{\"namespace\": \"example.avro\", \"type\": \"record\", \"name\": \"simple\", \"fields\": [{\"name\": \"name\", \"type\": \"string\"}]}",
    "records": [
        {
            "value": {
                "name": "name0"
            }
        }
    ]
}'

Response: { "error_code": 500, "message": "Internal server error" }

Also, I am getting below error in stacktrace:

karapace                MainThread    ERROR       Internal server error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/karapace/rapu.py", line 326, in _handle_request
    data = await callback(**callback_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/karapace/kafka_rest_apis/__init__.py", line 403, in topic_publish
    await proxy.topic_publish(topic, content_type, request=request)
  File "/usr/local/lib/python3.9/dist-packages/karapace/kafka_rest_apis/__init__.py", line 715, in topic_publish
    await self.publish(topic, None, content_type, request)
  File "/usr/local/lib/python3.9/dist-packages/karapace/kafka_rest_apis/__init__.py", line 665, in publish
    await self.validate_publish_request_format(data, formats, content_type, topic)
  File "/usr/local/lib/python3.9/dist-packages/karapace/kafka_rest_apis/__init__.py", line 936, in validate_publish_request_format
    await self.validate_schema_info(data, prefix, content_type, topic, formats["embedded_format"])
  File "/usr/local/lib/python3.9/dist-packages/karapace/kafka_rest_apis/__init__.py", line 768, in validate_schema_info
    data[f"{prefix}_schema_id"] = await self.get_schema_id(data, topic, prefix, schema_type)
  File "/usr/local/lib/python3.9/dist-packages/karapace/kafka_rest_apis/__init__.py", line 749, in get_schema_id
    schema_id = await self.serializer.get_id_for_schema(data[f"{prefix}_schema"], subject_name, schema_type)
  File "/usr/local/lib/python3.9/dist-packages/karapace/serialization.py", line 186, in get_id_for_schema
    schema_id = await self.registry_client.post_new_schema(subject, schema_typed)
  File "/usr/local/lib/python3.9/dist-packages/karapace/serialization.py", line 90, in post_new_schema
    result = await self.client.post(f"subjects/{quote(subject)}/versions", json=payload)
  File "/usr/local/lib/python3.9/dist-packages/karapace/client.py", line 141, in post
    async with client.post(
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/client.py", line 560, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/client_reqrep.py", line 899, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/usr/local/lib/python3.9/dist-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
aiohttp.access          MainThread    INFO        0.877757s - "POST /topics/example HTTP/1.1" 500 "PostmanRuntime/7.29.3" response=357b request_body=274b

Could you please explain why this problem is occurring?

[radhikari-3]

Hi @tvainika , do you have any update on the above issue ?

[tvainika]

Karapace is able to run both Kafka REST API and schema registry in same process. Internally, for authentication and other reasons, it will make REST calls from REST component to schema registry. Based on your stack trace it seems there something fails, but I was not able to reproduce this way. It should not get internal server errors. Can you share more details about this setup and Karapace version?

In my testing I received 400 based error codes if my configuration did not allow REST component to access the schema registry.