nifikop icon indicating copy to clipboard operation
nifikop copied to clipboard

Published 20 hours ago • verified publisher icon konpyutaika

Nifi Cluster SitetoSite Protocol communication

Open Premfer opened this issue 2 years ago • 5 comments

Question

What did you do?

we have installed the nifi cluster with 2 nodes using NiFiKop operator. we are trying to send the file through s2s protocol from standalone nifi application. we are trying this for the hybrid model where we send files from on-prem to AWS. By s2s protocol it tries to send data to the nodes using their name but it is a DNS name after installation and it is not resolved outside the cluster so standalone nifi out of the cluster could not connect to it. Kindly advise how to proceed or change the node names to IPs.

Environment

  • nifikop version: 0.7.6

  • Kubernetes version information:

1.20.11

  • Kubernetes cluster kind:

  • NiFi version:

1.12.1

Premfer avatar Jul 04 '22 07:07 Premfer

Perhaps if each node's identity/hostname was exposed on an Ingress in k8s, this would be possible. I can't think of another way since the Nifi node hostnames wouldn't be known external to the Kubernetes cluster.

An alternative is to just go through InvokeHTTP/ListenHTTP processors or processors enabling some other protocol (e.g. FTP/SFTP, grpc, etc). Yet another alternative would be to go through a buffer/queue like kafka, RabbitMQ, etc.

mh013370 avatar Jul 05 '22 09:07 mh013370

Thanks @michael81877 . We are using istio for service mesh. I am new and quite unclear in the ingress part you were suggesting to do in the first half. will it be route based? Kindly help..

Premfer avatar Jul 05 '22 10:07 Premfer

Thanks @michael81877 . We are using istio for service mesh. I am new and quite unclear in the ingress part you were suggesting to do in the first half. will it be route based? Kindly help..

I'm just referring to a Kubernetes Ingress. If you're using istio as your ingress controller, then i believe you'd create a Service Entry for each host in your NiFI cluster.

I haven't done this, though. So i'm not certain whether this approach would work for S2S between on-prem to k8s NiFi clusters.

mh013370 avatar Jul 05 '22 11:07 mh013370

Hi @michael81877 Thanks for the reply. but service entry is to resolve external service in the cluster according to my understanding and usage. Issue is that DNS or host names of the nodes wont resolve outside the cluster and the standalone nifi outside cluster tries to send data to the nodes in case of large files. so I am not sure how to use this. Kindly let me know if my understanding is wrong

Premfer avatar Jul 05 '22 13:07 Premfer

I'm sorry - i'm not an istio expert. I believe I meant an Istio Gateway, which is the equivalent of an Ingress

mh013370 avatar Jul 05 '22 13:07 mh013370