helm3-charts
helm3-charts copied to clipboard
Published
20 hours ago •
sonatype
sonatype
CrashLoopBackOff for nexus-iq pod after installing nexus-iq chart
I've installed repository manager and it works fine, but nexus-iq-server giving me troubles. I am not sure how to debug it since i can not check logs of nexus-iq-server pod because of ./start.sh: line 2: /var/log/nexus-iq-server/stderr.log: Permission denied error.
Both repository manager and nexus-iq are in one nexus namespace.
RM storing data in one pv, and nexus-iq storing logs and error logs in different pvs.
I am not sure if this is an issue, or i am simply missing something in my configs.
Nexus IQ Config:
---
# Default values for iqserver.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
statefulset:
# This is not supported
enabled: false
replicaCount: 1
deploymentStrategy: Recreate
image:
# Sonatype Official Public Image
repository: sonatype/nexus-iq-server
tag: 1.142.0
pullPolicy: IfNotPresent
imagePullSecrets:
# for image registries that require login, specify the name of the existing
# kubernetes secret
# - name: <pull-secret-name>
iq:
name: nxiq
hostname: iq-server.mydomain.com
applicationPort: 8070
adminPort: 8071
# base 64 encoded license file with no line breaks
licenseSecret: ''
# add this line with this file path and the `licenseSecret` above to autoconfigure licensing
# licenseFile: /etc/nexus-iq-license/license_lic
extraLabels:
# add the following two lines to mount a secrets volume within the container at the specified location
# secretName: secret-jks
# secretMountName: /etc/secret-volume
env:
- name: JAVA_OPTS
value: '-Djava.util.prefs.userRoot=$(SONATYPE_WORK)/javaprefs'
# In conjunction with 'secretName' and 'secretMountName' above, this is an example of how to inject required password
# secrets into the runtime environment, and how to modify the startup of the server to utilize custom Java SSL stores.
# - name: TRUSTSTORE_PASSWORD
# valueFrom:
# secretKeyRef:
# name: secret-jks
# key: truststorePassword
# - name: KEYSTORE_PASSWORD
# valueFrom:
# secretKeyRef:
# name: secret-jks
# key: keystorePassword
# - name: JAVA_OPTS
# value: "-Djavax.net.ssl.keyStoreType=jks -Djavax.net.ssl.keyStore=/etc/secret-volume/keystore.jks -Djavax.net.ssl.keyStorePassword=$(KEYSTORE_PASSWORD) -Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStore=/etc/secret-volume/truststore.jks -Djavax.net.ssl.trustStorePassword=$(TRUSTSTORE_PASSWORD) -Djava.util.prefs.userRoot=${SONATYPE_WORK}/javaprefs"
# Configures the liveness probe for IQ pod
livenessProbe:
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
timeoutSeconds: 2
successThreshold: 1
# Configures the readiness probe for IQ pod
readinessProbe:
initialDelaySeconds: 30
periodSeconds: 30
failureThreshold: 10
timeoutSeconds: 2
successThreshold: 1
nameOverride: ''
fullnameOverride: ''
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name:
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
service:
type: 'ClusterIP'
port: 80
ingress:
enabled: true
ingressClassName: nginx
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: '0'
kubernetes.io/ingress.allow-http: 'false'
cert-manager.io/cluster-issuer: letsencrypt-issuer
hostUI: iq-server.mydomain.com
hostUIPath: /
hostAdmin: admin-iq-server.mydomain.com
hostAdminPath: /
tls:
- secretName: nxiq-tls
hosts:
- iq-server.mydomain.com
- admin-iq-server.mydomain.com
resources:
# We usually recommend not to specify default resources and to leave this as a
# conscious choice for the user. This also increases chances charts run on
# environments with little resources, such as Minikube. If you do want to
# specify resources, uncomment the lines below. These values ensure minimum
# resources for production.
# requests:
# cpu: 8
# memory: 8Gi
# Optionally limit the resources to the recommended minimum requirements.
# limits:
# cpu: 8
# memory: 8Gi
nodeSelector: {}
tolerations: []
affinity: {}
persistence:
enabled: true
accessMode: ReadWriteOnce
storageSize: 1Gi
# # If defined, storageClass: <storageClass>
# # If set to "-", storageClass: "", which disables dynamic provisioning
# # If undefined (the default) or set to null, no storageClass spec is
# # set, choosing the default provisioner. (gp2 on AWS, standard on
# # GKE, AWS & OpenStack)
# storageClass: "-"
# # annotations are only applied to persistentVolumeClaim
# annotations:
# "helm.sh/resource-policy": keep
# # For existing PVCs
# existingClaim: nexus-data-disk
# existingLogClaim: nexus-log-disk
# # persistentVolume settings below here
# # Uncomment these to create a persistentVolume for data and/or logs volumes
# pvName: nexus-data-disk
# logpvName: nexus-log-disk
# # Uncomment this for gcePersistentDisk
# gcePersistentDisk:
# # Uncomment the next two lines if you want to use an existing GCE PersistentDisk for IQ Data.
# pdName: nexus-data-disk
# fsType: ext4
# # Uncomment the next two lines if you want to use an existing GCE PersistentDisk for IQ Logs.
# logpdName: nexus-log-disk
# logfsType: ext4
# #Uncomment the following section for using an existing AWS EBS Volume for data and/or logs
# awsElasticBlockStore:
# # Uncomment the next two lines if you want to use an existing AWS EBS Volume for IQ Data.
# volumeID: vol-xxxxxxxxxxxxxxxxx
# fsType: ext4
# # Uncomment the next two lines if you want to use an existing AWS EBS Volume for IQ Data.
# logvolumeID: vol-xxxxxxxxxxxxxxxxx
# logfsType: ext4
# # Uncomment the following block if you want to use ANY CSI Storage Driver
# # The YAML under 'csi:' or 'logCSI:' are taken as you choose to write them in order to support any CSI
# # The following is the csi block for the Data PV
# csi:
# driver: ebs.csi.aws.com
# volumeHandle: vol-xxxxxxxxxxxxxxxxx
# # The following is the csi block for the Log PV
# logCSI:
# driver: ebs.csi.aws.com
# volumeHandle: vol-xxxxxxxxxxxxxxxxx
# # Uncomment the following block if you need PV affinity
# # The nodeSelectorTerms block is taken as you choose to write it in whole
# affinity:
# nodeSelectorTerms:
# - matchExpressions:
# - key: topology.ebs.csi.aws.com/zone
# operator: In
# values:
# - us-east-2a
# configYaml is the full text of the config.yml file that will be passed to IQ Server
configYaml:
baseUrl: http://iq-server.mydomain.com
sonatypeWork: /sonatype-work
licenseFile: /etc/nexus-iq-license/license_lic
initialAdminPassword: admin123
features:
enableUnauthenticatedPages: true
server:
applicationConnectors:
- type: http
port: 8070
adminConnectors:
- type: http
port: 8071
# HTTP request log settings.
requestLog:
appenders:
# All appenders set to console
- type: file
currentLogFilename: /var/log/nexus-iq-server/request.log
# Do not display log statements below this threshold to stdout.
# threshold: INFO
logFormat: '%clientHost %l %user [%date] "%requestURL" %statusCode %bytesSent %elapsedTime "%header{User-Agent}"'
archivedLogFilenamePattern: /var/log/nexus-iq-server/request-%d.log.gz
archivedFileCount: 50
createSampleData: true
logging:
# The default level of all loggers. Can be OFF, ERROR, WARN, INFO, DEBUG, TRACE, or ALL.
level: DEBUG
# Logger-specific settings.
loggers:
'com.sonatype.insight.scan': INFO
'eu.medsea.mimeutil.MimeUtil2': INFO
'org.apache.http': INFO
'org.apache.http.wire': ERROR
'org.eclipse.birt.report.engine.layout.pdf.font.FontConfigReader': WARN
'org.eclipse.jetty': INFO
'org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter': INFO # WARNING: This reveals credentials at DEBUG level
'com.sonatype.insight.audit':
appenders:
# All appenders set to console
- type: file
currentLogFilename: /var/log/nexus-iq-server/audit.log
# Do not display log statements below this threshold to stdout.
# threshold: INFO
archivedLogFilenamePattern: /var/log/nexus-iq-server/audit-%d.log.gz
archivedFileCount: 50
'com.sonatype.insight.policy.violation':
appenders:
- type: console
# Do not display log statements below this threshold to stdout.
threshold: INFO
appenders:
# Settings for logging to stdout.
- type: console
# Do not display log statements below this threshold to stdout.
threshold: INFO
logFormat: "%d{'yyyy-MM-dd HH:mm:ss,SSSZ'} %level [%thread] %X{username} %logger - %msg%n"
- type: file
# Do not display log statements below this threshold to stdout.
threshold: ALL
logFormat: "%d{'yyyy-MM-dd HH:mm:ss,SSSZ'} %level [%thread] %X{username} %logger - %msg%n"
# The file to which current statements will be logged.
currentLogFilename: /var/log/nexus-iq-server/clm-server.log
archivedLogFilenamePattern: /var/log/nexus-iq-server/clm-server-%d.log.gz
archivedFileCount: 50
Nexus Repository Manager Config:
---
statefulset:
# This is not supported
enabled: false
deploymentStrategy: Recreate
image:
# Sonatype Official Public Image
repository: sonatype/nexus3
tag: 3.41.0
pullPolicy: IfNotPresent
imagePullSecrets:
# for image registries that require login, specify the name of the existing
# kubernetes secret
# - name: <pull-secret-name>
nexus:
docker:
enabled: true
registries:
- port: 5000
host: docker-registry.mydomain.com
secretName: docker-registry-tls
env:
# minimum recommended memory settings for a small, person instance from
# https://help.sonatype.com/repomanager3/product-information/system-requirements
- name: INSTALL4J_ADD_VM_PARAMS
value: |-
-Xms2048m -Xmx2048m
-XX:ActiveProcessorCount=4
-XX:MaxDirectMemorySize=2703M
-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap
-Djava.util.prefs.userRoot=/nexus-data/javaprefs
- name: NEXUS_SECURITY_RANDOMPASSWORD
value: 'true'
properties:
override: false
data:
nexus.scripts.allowCreation: true
# See this article for ldap configuratioon options https://support.sonatype.com/hc/en-us/articles/216597138-Setting-Advanced-LDAP-Connection-Properties-in-Nexus-Repository-Manager
# nexus.ldap.env.java.naming.security.authentication: simple
# nodeSelector:
# cloud.google.com/gke-nodepool: default-pool
resources:
# minimum recommended memory settings for a small, person instance from
# https://help.sonatype.com/repomanager3/product-information/system-requirements
# requests:
# cpu: 4
# memory: 8Gi
# limits:
# cpu: 4
# memory: 8Gi
# The ports should only be changed if the nexus image uses a different port
nexusPort: 8081
# Default the pods UID and GID to match the nexus3 container.
# Customize or remove these values from the securityContext as appropriate for
# your deployment environment.
securityContext:
runAsUser: 200
runAsGroup: 200
fsGroup: 200
podAnnotations: {}
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 30
failureThreshold: 6
timeoutSeconds: 10
path: /
readinessProbe:
initialDelaySeconds: 30
periodSeconds: 30
failureThreshold: 6
timeoutSeconds: 10
path: /
# hostAliases allows the modification of the hosts file inside a container
hostAliases: []
# - ip: "192.168.1.10"
# hostnames:
# - "example.com"
# - "www.example.com"
nameOverride: ''
fullnameOverride: ''
deployment:
# # Add annotations in deployment to enhance deployment configurations
annotations: {}
# # Add init containers. e.g. to be used to give specific permissions for nexus-data.
# # Add your own init container or uncomment and modify the given example.
initContainers:
# - name: fmp-volume-permission
# image: busybox
# imagePullPolicy: IfNotPresent
# command: ['chown','-R', '200', '/nexus-data']
# volumeMounts:
# - name: nexus-data
# mountPath: /nexus-data
# Uncomment and modify this to run a command after starting the nexus container.
postStart:
command: # '["/bin/sh", "-c", "ls"]'
preStart:
command: # '["/bin/rm", "-f", "/path/to/lockfile"]'
terminationGracePeriodSeconds: 120
additionalContainers:
additionalVolumes:
additionalVolumeMounts:
ingress:
enabled: true
ingressClassName: nginx
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: '0'
kubernetes.io/ingress.allow-http: 'false'
cert-manager.io/cluster-issuer: letsencrypt-issuer
hostPath: /
hostRepo: storage.mydomain.com
tls:
- secretName: nexus-tls
hosts:
- storage.mydomain.com
service:
name: nexus3
enabled: true
labels: {}
annotations: {}
type: ClusterIP
route:
enabled: false
name: docker
portName: docker
labels:
annotations:
# path: /docker
nexusProxyRoute:
enabled: false
labels:
annotations:
# path: /nexus
persistence:
enabled: true
accessMode: ReadWriteOnce
## If defined, storageClass: <storageClass>
## If set to "-", storageClass: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClass spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# existingClaim:
annotations:
'helm.sh/resource-policy': keep
# storageClass: "-"
storageSize: 8Gi
# If PersistentDisk already exists you can create a PV for it by including the 2 following keypairs.
# pdName: nexus-data-disk
# fsType: ext4
tolerations: []
# Enable configmap and add data in configmap
config:
enabled: false
mountPath: /sonatype-nexus-conf
data: []
# # To use an additional secret, set enable to true and add data
secret:
enabled: false
mountPath: /etc/secret-volume
readOnly: true
data: []
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ''
Having the exact same issue here--not able to see anything missing in our configuration either. PVC/PV mounted properly, and we are running Nexus IQ in its own dedicated namespace
To review the Nexus IQ Server logs, connect to the pod via bash shell. The logs are in /var/log/nexus-iq-server.
Here are the commands I used to shell into a pod:
$ kubectl get pods --all-namespaces $ kubectl --namespace <namespace> exec -it <pod> -- bash
We can't view the logs as the container comes up and immediately crashes. I think the filesystem permission error is a hard exit
Right so I edited the startup command for the container so I could go in and manually try starting it to see what happens. The file permissions are set as follows:
$ ls -la /var/log
total 292
drwxr-xr-x 1 root root 44 Jul 27 16:36 .
drwxr-xr-x 1 root root 30 May 3 08:55 ..
-rw-rw-r-- 1 root utmp 292292 Jul 27 16:36 lastlog
drwxr-xr-x 3 root root 4096 Aug 30 09:52 nexus-iq-server
$ whoami
nexus
The nexus user doesn't have permissions to this directory, and therefore can't write logs as the error message states
Found the fix for this @grigoryevandrey, you need to set the fsGroup under podSecurityContext so the volumes are writeable. I've raised a PR to change the default values file to reflect this, as the chart will not work out-of-the-box without this uncommented.
https://github.com/sonatype/helm3-charts/pull/226/files
This PR has merged and should apply a similar change as #226. Please let us know if it works better in your security environment. https://github.com/sonatype/helm3-charts/pull/227
