nifikop
nifikop copied to clipboard
Orange-OpenSource
Simplenifi cluster is running but unaccessible
Bug Report
What did you do? I've installed simple nifi cluster following https://orange-opensource.github.io/nifikop/docs/2_setup/1_getting_started
What did you expect to see? Running nifi cluster with 2 nodes accessible through web UI
NAME READY STATUS RESTARTS AGE pod/nifikop-586867994d-lkmgc 1/1 Running 0 6h56m pod/nifikop-586867994d-pvnmn 0/1 Terminating 0 25h pod/simplenifi-1-nodew5925 1/1 Running 0 6h52m pod/simplenifi-2-nodegt8rh 1/1 Running 0 22h pod/zookeeper-0 1/1 Running 1 6h52m pod/zookeeper-1 1/1 Running 1 6h52m pod/zookeeper-2 1/1 Running 1 6h52m
What did you see instead? Under which circumstances? UI is not accessible through svc service/simplenifi-all-node. Moreover I failed to curl http:localhost:8080 from inside a container
$ curl http://localhost:8080/nifi curl: (7) Failed to connect to localhost port 8080: Connection refused
Environment
-
nifikop version: 0.5.1
-
Kubernetes version information:
1.18
- Kubernetes cluster kind: Yandex cloud
Hi, there was an issue for clusters created with services in non-headless mode (when the spec.service.headlessEnabledfield is set to false). With the lastest release v0.5.2-release this should be fixed.
So you either you set service in headless mode or you migrate to the lastest version. Once done, just keep in mind that since version v0.5.0-release there is no more default external service (nodePort or loadbalancer service) created by the operator, you must declare them explicitly as we did in the example : https://github.com/Orange-OpenSource/nifikop/blob/master/config/samples/simplenificluster.yaml#L52. So if you want to access to your cluster, all you have to do is declare your service using the spec.externalServices field.
@erdrix I think the issue might be something different; I've encountered this as well & it seems Nifi is only listening on the external pod IP. When using .e.g kubectl port-forward it does not forward the port properly as it defaults to the containers localhost.
Can confirm the described bug with nifikop:v0.6.0-release.
Additionally, this is the log output for nifi:
Could not resolve host: simplenifi-headless.nifi-lab.svc.cluster.local
Expire in 4 ms for 1 (transfer 0x557c03967f50)
Closing connection 0
curl: (6) Could not resolve host: simplenifi-headless.nifi-lab.svc.cluster.local
Port-forward to 8080 results in connection refused as well. simplenifi-headless service exists.
Same issue here, unable to access to nifi 8080 port using port forward in release-0.6.3
nifi namespace pods:
❯ kubectl get po -n nifi
NAME READY STATUS RESTARTS AGE
datos-0-nodeqbnwp 1/1 Running 0 22m
datos-2-nodess74q 1/1 Running 0 22m
green-nifikop-7877b7dc94-5btjr 1/1 Running 0 14h
green-zookeeper-0 1/1 Running 0 14h
green-zookeeper-1 1/1 Running 0 14h
green-zookeeper-2 1/1 Running 0 14h
Services:
❯ kubectl get svc -n nifi
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
datos-clusterip ClusterIP 10.100.213.62 <none> 8080/TCP 22m
datos-headless ClusterIP None <none> 8080/TCP,6007/TCP,10000/TCP,9020/TCP,10001/TCP,9090/TCP 22m
green-zookeeper ClusterIP 10.100.17.181 <none> 2181/TCP,2888/TCP,3888/TCP 4d20h
green-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 4d20h
green-zookeeper-metrics ClusterIP 10.100.12.218 <none> 9141/TCP 4d20h
nifikop-metrics ClusterIP 10.100.87.7 <none> 9710/TCP 4d21h
Service port-forward
❯ kubectl port-forward --namespace nifi service/datos-headless 8082:8080
Forwarding from 127.0.0.1:8082 -> 8080
Forwarding from [::1]:8082 -> 8080
❯ curl -IL http://localhost:8080 --insecure
curl: (7) Failed to connect to localhost port 8080: Connection refused
Pod port-forward
❯ kubectl get po -n nifi
NAME READY STATUS RESTARTS AGE
datos-0-nodeqbnwp 1/1 Running 0 25m
datos-2-nodess74q 1/1 Running 0 25m
green-nifikop-7877b7dc94-5btjr 1/1 Running 0 14h
green-zookeeper-0 1/1 Running 0 14h
green-zookeeper-1 1/1 Running 0 14h
green-zookeeper-2 1/1 Running 0 14h
❯ kubectl port-forward --namespace nifi pod/datos-0-nodeqbnwp 8082:8080
Forwarding from 127.0.0.1:8082 -> 8080
Forwarding from [::1]:8082 -> 8080
Handling connection for 8082
E0823 15:27:21.527808 89707 portforward.go:400] an error occurred forwarding 8082 -> 8080: error forwarding port 8080 to pod 72a8e9c4fa537c1fff4a15d042af53c24473a7705fdfa0cdb79ac9cdcfb4be6d, uid : exit status 1: 2021/08/23 15:27:21 socat[30859] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused
For future googler's:
You must configure the following:
- WebProxyHosts:
- Configure nifi properties: nifi.web.http.network.interface.default=eth0 nifi.web.http.network.interface.lo=lo
I attach sample cluster and registry
apiVersion: nifi.orange.com/v1alpha1
kind: NifiRegistryClient
metadata:
name: test-registry
namespace: nifi
spec:
# Contains the reference to the NifiCluster with the one the registry client is linked.
clusterRef:
name: test-cluster
namespace: nifi
# The Description of the Registry client.
description: "Simple nifi registry demo"
# The URI of the NiFi registry that should be used for pulling the flow.
uri: "http://test-registry:18080"
---
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: test-cluster
namespace: nifi
spec:
service:
headlessEnabled: true
zkAddress: "green-zookeeper:2181"
zkPath: "/test_cluster"
clusterImage: "apache/nifi:1.12.1"
oneNifiNodePerNode: false
managedAdminUsers:
- identity: "[email protected]"
name: "eavendanoa"
nodeConfigGroups:
default_group:
isNode: true
provenanceStorage: "10 GB"
runAsUser: 1000
imagePullPolicy: IfNotPresent
storageConfigs:
# Name of the storage config, used to name PV to reuse into sidecars for example.
- name: provenance-repository
# Path where the volume will be mount into the main nifi container inside the pod.
mountPath: "/opt/nifi/provenance_repository"
# Kubernetes PVC spec
# https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-persistentvolumeclaim
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "gp2"
resources:
requests:
storage: 10Gi
serviceAccountName: "nifi"
resourcesRequirements:
limits:
cpu: "1"
memory: 4Gi
requests:
cpu: "1"
memory: 3Gi
nodeSelector:
node.kubernetes.io/role: batch
nodes:
- id: 0
nodeConfigGroup: "default_group"
- id: 1
nodeConfigGroup: "default_group"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
readOnlyConfig:
maximumTimerDrivenThreadCount: 30
logbackConfig:
replaceConfigMap:
data: logback.xml
name: test-cluster-logback-config
namespace: nifi
bootstrapProperties:
nifiJvmMemory: "2g"
nifiProperties:
overrideConfigs: |
nifi.web.http.network.interface.default=eth0
nifi.web.http.network.interface.lo=lo
webProxyHosts:
- test-clusterip:8080
- localhost:8080
listenersConfig:
internalListeners:
- type: "http"
name: "test-http"
containerPort: 8080
- type: "cluster"
name: "test-cluster"
containerPort: 6007
- type: "s2s"
name: "test-s2s"
containerPort: 10000
- type: "prometheus"
name: "test-prometh"
containerPort: 9090
externalServices:
- name: "test-clusterip"
spec:
type: ClusterIP
portConfigs:
- port: 8080
internalListenerName: "test-http"
serviceAnnotations:
toto: tata
sidecarConfigs:
- name: app-log
image: "busybox:1.32.0"
args: [ tail, -n+1, -F, /var/log/nifi-app.log ]
resources: &log_resources
requests:
cpu: 50m
memory: 50Mi
limits:
cpu: 50m
memory: 50Mi
volumeMounts:
- name: logs
mountPath: /var/log
- name: bootstrap-log
image: "busybox:1.32.0"
args: [ tail, -n+1, -F, /var/log/nifi-bootstrap.log ]
resources: *log_resources
volumeMounts:
- name: logs
mountPath: /var/log
- name: user-log
image: "busybox:1.32.0"
args: [ tail, -n+1, -F, /var/log/nifi-user.log ]
resources: *log_resources
volumeMounts:
- name: logs
mountPath: /var/log
Next steps add security, ldap and ingress. I hope it helps.
thank you @esteban1983cl for your answer, I am a case quite similar to yours and yet I still have the same concerns,
here is my conf:
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: simplenifi
spec:
service:
headlessEnabled: true
zkAddress: zookeeper:2181
zkPath: "/simplenifi"
clusterImage: "apache/nifi:1.12.1"
oneNifiNodePerNode: false
nodeConfigGroups:
default_group:
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "hostpath"
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
resourcesRequirements:
limits:
cpu: "0.5"
memory: 2Gi
requests:
cpu: "0.5"
memory: 2Gi
nodes:
- id: 1
nodeConfigGroup: "default_group"
- id: 2
nodeConfigGroup: "default_group"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
readOnlyConfig:
maximumTimerDrivenThreadCount: 30
logbackConfig:
replaceConfigMap:
data: logback.xml
name: test-cluster-logback-config
namespace: nifi
bootstrapProperties:
nifiJvmMemory: "2g"
nifiProperties:
overrideConfigs: |
nifi.web.http.network.interface.default=eth0
nifi.web.http.network.interface.lo=lo
webProxyHosts:
- clusterip:8080
- localhost:8080
listenersConfig:
internalListeners:
- type: "http"
name: "http"
containerPort: 8080
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
- type: "prometheus"
name: "prometheus"
containerPort: 9090
externalServices:
- name: "clusterip"
spec:
type: ClusterIP
portConfigs:
- port: 8080
internalListenerName: "http"
serviceAnnotations:
toto: tata
- name: "loadbalancer"
spec:
type: LoadBalancer
portConfigs:
- port: 8080
internalListenerName: "http"
serviceAnnotations:
toto: tata
- name: "nodepart"
spec:
type: NodePort
portConfigs:
- port: 8080
internalListenerName: "http"
serviceAnnotations:
toto: tata
`
and when I start the cluster:
kubectl -n nifi get pods
NAME READY STATUS RESTARTS AGE
nifikop-79bb46b754-9trpg 1/1 Running 4 3h12m
simplenifi-1-nodetxzjg 1/1 Running 0 25m
simplenifi-2-nodetmvj5 1/1 Running 0 23m
zookeeper-0 1/1 Running 31 3h28m
zookeeper-1 1/1 Running 35 3h28m
zookeeper-2 1/1 Running 33 3h28m`
kubectl describe pods -n nifi simplenifi-1-nodetxzjg
* Trying 10.1.0.57...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x5623b6c9bf50)
* connect to 10.1.0.57 port 8080 failed: Connection refused
* Failed to connect to simplenifi-1-node.simplenifi-headless.nifi.svc.cluster.local port 8080: Connection refused
* Closing connection 0
curl: (7) Failed to connect to simplenifi-1-node.simplenifi-headless.nifi.svc.cluster.local port 8080: Connection refused
I can't find the source of the problem, thank you for your help :)
I have the same problem `
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: sslnifi
namespace: apache-nifi
spec:
service:
headlessEnabled: true
zkAddress: "nifi-zookeeper:2181"
zkPath: "/ssllnifi"
clusterImage: "apache/nifi:1.12.1"
initContainerImage: "busybox"
oneNifiNodePerNode: false
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
readOnlyConfig:
nifiProperties:
webProxyHosts:
- nifi-infra.test:8443
overrideConfigs: |
nifi.security.user.oidc.discovery.url=*****
nifi.security.user.oidc.client.id=****
nifi.security.user.oidc.client.secret=*****
nifi.security.identity.mapping.pattern.dn=CN=([^,]*)(?:, (?:O|OU)=.*)?
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
nodeConfigGroups:
default_group:
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "nifi-sc"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/data"
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "nifi-sc"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/flowfile_repository"
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "nifi-sc"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/nifi-current/conf"
name: conf
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "nifi-sc"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/content_repository"
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "nifi-sc"
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/provenance_repository"
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "nifi-sc"
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
nodeSelector:
node-purpose: infra
resourcesRequirements:
limits:
cpu: "2"
memory: 3Gi
requests:
cpu: "1"
memory: 1Gi
nodes:
- id: 0
nodeConfigGroup: "default_group"
- id: 1
nodeConfigGroup: "default_group"
- id: 2
nodeConfigGroup: "default_group"
listenersConfig:
internalListeners:
- type: "https"
name: "https"
containerPort: 8443
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
sslSecrets:
tlsSecretName: "tls-nifi"
create: false
`
- Could not resolve host: sslnifi-headless.apache-nifi.svc.cluster.local
- Expire in 1 ms for 1 (transfer 0x55b7d6a74f50)
- Closing connection 0 curl: (6) Could not resolve host: sslnifi-headless.apache-nifi.svc.cluster.local `
apache-nifi sslnifi-headless ClusterIP None <none> 8443/TCP,6007/TCP,10000/TCP 49m
I'm experiencing this as well, any ideas or suggestions would be greatly appreciated @erdrix
-
nifikop version: 0.7.0
-
Kubernetes version information: 1.18.0
-
Kubernetes cluster kind: kind
-
NiFi version: 1.12.1
Config:
apiVersion: nifi.orange.com/v1alpha1
kind: NifiCluster
metadata:
name: simplenifi
spec:
service:
headlessEnabled: true
zkAddress: "zookeeper.default.svc.cluster.local:2181"
zkPath: "/simplenifi"
clusterImage: "apache/nifi:1.12.1"
oneNifiNodePerNode: false
nodeConfigGroups:
default_group:
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
storageClassName: "examplestorageclass"
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
resourcesRequirements:
limits:
cpu: "0.5"
memory: 2Gi
requests:
cpu: "0.5"
memory: 2Gi
nodes:
- id: 1
nodeConfigGroup: "default_group"
- id: 2
nodeConfigGroup: "default_group"
propagateLabels: true
readOnlyConfig:
nifiProperties:
overrideConfigs: |
nifi.web.http.network.interface.default=eth0
nifi.web.http.network.interface.lo=lo
webProxyHosts:
- clusterip:8080
- localhost:8080
nifiClusterTaskSpec:
retryDurationMinutes: 10
listenersConfig:
internalListeners:
- type: "http"
name: "http"
containerPort: 8080
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
- type: "prometheus"
name: "prometheus"
containerPort: 9090
externalServices:
- name: "clusterip"
spec:
type: ClusterIP
portConfigs:
- port: 8080
internalListenerName: "http"
serviceAnnotations:
toto: tata
- name: "nodepart"
spec:
type: NodePort
portConfigs:
- port: 8080
internalListenerName: "http"
serviceAnnotations:
toto: tata
I'm able to port-forward to access the UI on localhost but the error remains:
* Expire in 200 ms for 4 (transfer 0x55fda1c09f50)
* connect to 10.96.58.102 port 8080 failed: Connection refused
* Failed to connect to simplenifi-all-node.nifi.svc.cluster.local port 8080: Connection refused
* Closing connection 0
curl: (7) Failed to connect to simplenifi-all-node.nifi.svc.cluster.local port 8080: Connection refused
Thanks for your suggestion @esteban1983cl I tried your config and similarly could port forward successfully but still got this message in the pod:
* Could not resolve host: simplenifi-headless.nifi.svc.cluster.local
* Expire in 1 ms for 1 (transfer 0x5595b58a6f50)
* Closing connection 0
curl: (6) Could not resolve host: simplenifi-headless.nifi.svc.cluster.local
