only basicauth available for SchemaRegistry? require ssl auth (with certs)

[msrijita18]

For example to consume avro msgs via schema registry using following cmd:

./kafka-avro-console-consumer \
         --topic topicname \
         --bootstrap-server server.net:443 \
         --consumer.config client.properties \
         --property schema.registry.url=https://server-schema.net/ \
         --property schema.registry.ssl.keystore.location=/Users/ljo100/PP-SR-CERT/sr.keystore.jks \
         --property schema.registry.ssl.truststore.location=/Users/ljo100/PP-SR-CERT/sr.truststore.jks \
         --property schema.registry.ssl.keystore.password=XXXXXXXXWMy \
         --property schema.registry.ssl.truststore.password=XXXXXXXXWMy \
         --from-beginning --property parse.key=true --property key.separator=, --property print.key=true

[mostafa]

Hey @msrijita18,

The SchemaRegistry object accepts a SchemaRegistryConfig object, which also contains TLSConfig and there you can add your TLS information and certs.

[msrijita18]

Hi @mostafa , Thanks, i'll try it. I did try something like this, but it was not working. I'll research a bit about the Schema registry SASL SSL auth with TLS certs.

const schemaRegistry = SchemaRegistry({ 
    url: "https://server.net",
    saslConfig: {
        username:"YjM4MGFlZjcwN",
        password:"YjM4MGFlZjcwN",
        algorithm: SASL_SSL,
    },
    tls: {
        clientCertPem: /tmp/sr.truststore.jks,
        clientKeyPem: /tmp/sr.keystore.jks,
        enableTls: true,
        insecureSkipTlsVerify:true,
        minVersion: TLS_1_2,
    },
});

[mostafa]

@msrijita18 Also, JKS format is not supported for key and certificate, as mentioned in this ticket. As indicated in the variable names, clientCertPem and clientKeyPem, they require PEM-formatted files. Also, for proper TLS authentication, you need to set insecureSkipTlsVerify to false.

Also, saslConfig is not a supported argument for schema registry object.

[msrijita18]

@mostafa Oh okay, i'll look into it. Thanks :)

[msrijita18]

Hi @mostafa ,

truststore.jks and keystore.jks are there for Schema registry TLS Auth. I converted the truststore.jks to .pem files so that tlsConfig works.

Did the following to convert it: keytool -importkeystore -srckeystore /SR/keystore.jks -destkeystore keystore.p12 -deststoretype PKCS12 openssl pkcs12 -in /SR/keystore.p12 -nokeys -out sr.cer.pem openssl pkcs12 -in /SR/keystore.p12 -nodes -nocerts -out sr.key.pem

But this error pops up:

ERRO[0000] Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "cluster-ca v0") {0xc00324c580 0xc00007a3d0 0xc00324cb00}}) error="Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster-ca v0\") {0xc00324c580 0xc00007a3d0 0xc00324cb00}})" ERRO[0000] Failed to create dialer., OriginalError: %!w(*fmt.wrapError=&{failed to dial: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "cluster-ca v0") {0xc00324c580 0xc00007a3d0 0xc00324cb00}}) at file:///tmp/test_sr_auth.js:49:19(97) at native hint="script exception"

Would appreciate some help, thanks!

[mostafa]

@msrijita18 In your case, since you are using a self-signed certificate, you need to set insecureSkipTlsVerify to true.

[msrijita18]

@mostafa

Its working now. I followed this to convert jks files to pem files

But now I'm facing this error:

ERRO[0000] File not found: , OriginalError: %!w(*fs.PathError=&{stat 2}) at github.com/mostafa/xk6-kafka.(*Kafka).schemaRegistryClientClass-fm (native) at file:///tmp/test_sr_auth.js:64:8(119) at native hint="script exception"

I'm using this image.

I'm looking at this issue. Is it an outdated image problem?

[mostafa]

@msrijita18 Good to hear that you were able to make it work! 💪

Also, the issue you face is that you're using an image with no mounted volume, while using your local addresses in your script. Mount a directory that contains your script and certificates and then use valid paths inside the container in your script.

[msrijita18]

@mostafa Thanks! Oh okay I was using the image and copying the files from local to tmp folder. I'll try to use mounted volume.

[mostafa]

@msrijita18 Using the image and copying should also work, as long as you use the path inside the container, not the host, in your script.

[mostafa]

I suppose this is resolved, so I'll close the issue. If you still have any questions related to this issue, feel free to reopen the issue.

[msrijita18]

Hi @mostafa ,

I tried to do the SR auth a couple of ways but it still shows me the same error:

ERRO[0000] File not found: , OriginalError: %!w(*fs.PathError=&{stat 2}) at github.com/mostafa/xk6-kafka.(*Kafka).schemaRegistryClientClass-fm (native) at file:///tmp/test_sr_auth.js:61:8(119) at native hint="script exception"

[mostafa]

@msrijita18 It happens if you don't pass the server certificate at the same time for mutual TLS, and it seems that it isn't needed by the schema registry either. It is pretty similar to this issue, but not the same exact thing.

[mostafa]

@msrijita18 Created #169 to fix this issue.

[msrijita18]

Thanks! I look forward to it :)

[mostafa]

@msrijita18 Fixed in #170.

[msrijita18]

@mostafa Thanks a lot :-)

[msrijita18]

Hi @mostafa

ERRO[0003] Failed to get schema from schema registry, OriginalError: %!w(*url.Error=&{Get https://.../versions/latest 0xc00117fb60})
	at github.com/mostafa/xk6-kafka.(*Kafka).schemaRegistryClientClass.func1 (native)
	at file://scripts-spt/test_avro_with_schema_registry.js:286:4(144)
	at native  hint="script exception"

This error gets thrown when I try to get schemas from remote SR with TLS:

Config provided:

const valueSubjectName = schemaRegistry.getSubjectName({
    topic:topic,
    element:VALUE,
    subjectNameStrategy:RECORD_NAME_STRATEGY,
    schema:valueSchema,
});
const valueSchemaObject = schemaRegistry.getSchema({
    subject:valueSubjectName,
    schema:valueSchema,
    schemaType:SCHEMA_TYPE_AVRO,
});

I tried on local schema registry, it works then and if schema is not found then it shows this error

ERRO[0000] Failed to get schema from schema registry, OriginalError: %!w(srclient.Error={40401 Subject 
'com.example.perso-key' not found. 0xc0006fc1b0})
	at github.com/mostafa/xk6-kafka.(*Kafka).schemaRegistryClientClass.func1 (native)
	at file:///Users/srijita.mallick/maersk_workspace/scripts/test_avro_with_schema_registry.js:83:4(109)
	at native  hint="script exception

I feel the problem lies with handling tls auth schema registry for getting schemas, can you pl look into it, thanks

[mostafa]

@msrijita18 This is not related to TLS handling at all. You are trying to get a schema from Schema Registry that doesn't exist. You should first create the schema if it doesn't exist.

[msrijita18]

@mostafa No actually it exists. Also I can't create schemas. Basically the problem occurs whenever I try to request GET/POST to TLS auth SR for creating/fetching schemas. Is it that these functions need TLS config like writer and reader functions? Because this function works fine while authenticating to SR:

const schemaRegistry = SchemaRegistry({ 
    url:"https://pdigital.net",
    tls:{
        enableTls:true,
        insecureSkipTlsVerify:false,
        minVersion:TLS_1_2,
        clientCertPem:"/SR/client-cert.pem",
        clientKeyPem:"/SR/client-key.pem",
        serverCertPem:"/SR/server-cert.pem",
    },
});

I wanted to show what error it throws if a schema doesn't exist which is this: Error says schema not found

ERRO[0000] Failed to get schema from schema registry, OriginalError: %!w(srclient.Error={40401 Subject 

'com.example.perso-key' not found. 0xc0006fc1b0})
	at github.com/mostafa/xk6-kafka.(*Kafka).schemaRegistryClientClass.func1 (native)
	at file:///Users/srijita.mallick/maersk_workspace/scripts/test_avro_with_schema_registry.js:83:4(109)
	at native  hint="script exception

And what error I'm getting in TLS auth SR get schema: Errors says cant get https://.../versions/latest

ERRO[0003] Failed to get schema from schema registry, OriginalError: %!w(*url.Error=&{Get https://.../versions/latest 0xc00117fb60})
	at github.com/mostafa/xk6-kafka.(*Kafka).schemaRegistryClientClass.func1 (native)
	at file://scripts-spt/test_avro_with_schema_registry.js:286:4(144)
	at native  hint="script exception"

When I curl the url to fetch schemas, I get the results easily in my terminal

[msrijita18]

Hi @mostafa ,

I wanted to share the whole script that I'm using: script.js

I'm able to log the constant valueSchemaObject , that means function getSubjectName works fine (Earlier i was using the wrong subjectNameStrategy)

But I'm facing error while using the function getSchema. In the documentation it says we can pass in the following parameters:

const valueSchemaObject = schemaRegistry.getSchema({
    data: <??>,
    schema: valueSchema, 
    schemaType: SCHEMA_TYPE_AVRO,
});

But what value should i put in data parameter (it says we can put any which is an interface in go)?

[mostafa]

@msrijita18 Good to hear! I am reusing the Schema struct here, but you don't need to use the data parameter. Only enableCaching, subject and version is used. So the signature you're using is wrong.

[msrijita18]

@mostafa I'm using the below config:

const valueSubjectName = schemaRegistry.getSubjectName({
    topic: topic,
    element: VALUE,
    subjectNameStrategy: RECORD_NAME_STRATEGY,
    schema: valueSchema,
});

const valueSchemaObject = schemaRegistry.getSchema({
    enableCaching: false,
    subject:valueSubjectName,
    version:1
});

But still getting the following error:

ERRO[0006] Failed to get schema from schema registry, OriginalError: %!w(*url.Error=&{Get https://pp-
digital.net/subjects/com.alerting.topic/versions/1 0xc0022a4cc0})
	at github.com/mostafa/xk6-kafka.(*Kafka).schemaRegistryClientClass.func1 (native)
	at file:///Users/schema_registry_test.js:107:12(116)
	at native  hint="script exception"

[mostafa]

Hey @msrijita18,

Is this resolved or do you still have the issue?

[mostafa]

@msrijita18 I'll close this ticket due to inactivity. Feel free to re-open it if the issue persists.

[msrijita18]

@mostafa

I'm still facing the same issue, so i used kafka clients for my testing

[mostafa]

@msrijita18

I suppose this is resolved. Feel free to reopen it if the issue persists.