Harbor K8s running behind a OPNSense

Open gsaudade99 opened this issue 1 year ago • 0 comments

Hello, Im currently testing k8s harbor setup in our new datacenter. The datacenter is behind a opnsense. I have managed to migrate services and resolve them using the HAProxy in the opnsense.

I have managed to install harbor, login and resolve the dns correctly. Harbor is exposed using metallb in l2 mode and the storage is longhorn. Im currently facing the issue where It keeps retrying to push the image to the repository and eventually gives a 500 erorr.

➜  harbor docker login harbordc.ubiwhere.com -u admin
Password: 
Login Succeeded

➜  harbor time docker push harbordc.ubiwhere.com/test-speed/1gb-random-file:latest

The push refers to repository [harbordc.ubiwhere.com/test-speed/1gb-random-file]
613d42ba11ea: Pushing [==================================================>]  1.074GB
e154057080f4: Pushing [==================================================>]   4.23MB
received unexpected HTTP status: 500 writing request for harbordc.ubiwhere.com:80: write tcp 10.255.2.3:43868->91.209.16.33:80: write: broken pipe
docker push harbordc.ubiwhere.com/test-speed/1gb-random-file:latest  0.15s user 0.09s system 0% cpu 3:19.58 total

I think the issue might be my configurations HaProxy and Harbor's internal proxy conflict. All our dns and CA is resolved in the haproxy.

Bellow Is my values.yml configuration:

expose:
  type: loadBalancer
  tls:
    enabled: false
    certSource: auto
    auto:
      commonName: "harbordc.ubiwhere.com"
    secret:
      secretName: ""
  ingress:
    hosts:
      core: harbordc.ubiwhere.com
    controller: default
    kubeVersionOverride: ""
    className: ""
    annotations:
      ingress.kubernetes.io/ssl-redirect: "true"
      ingress.kubernetes.io/proxy-body-size: "0"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
    harbor:
      annotations: {}
      labels: {}
  clusterIP:
    name: harbordc
    staticClusterIP: ""
    annotations: {}
    ports:
      httpPort: 80
      httpsPort: 443
  nodePort:
    name: harbordc
    ports:
      http:
        port: 80
        nodePort: 30002
      https:
        port: 443
        nodePort: 30003
  loadBalancer:
    name: harbordc
    IP: "<some_ip>"
    ports:
      httpPort: 80
      httpsPort: 443
    annotations: {}
    sourceRanges: []

externalURL: https://harbordc.ubiwhere.com

internalTLS:
  enabled: false
  strong_ssl_ciphers: false
  certSource: "auto"
  trustCa: ""
  core:
    secretName: ""
    crt: ""
    key: ""
  jobservice:
    secretName: ""
    crt: ""
    key: ""
  registry:
    secretName: ""
    crt: ""
    key: ""
  portal:
    secretName: ""
    crt: ""
    key: ""
  trivy:
    secretName: ""
    crt: ""
    key: ""

ipFamily:
  ipv6:
    enabled: true
  ipv4:
    enabled: true

persistence:
  enabled: true
  resourcePolicy: "keep"
  persistentVolumeClaim:
    registry:
      existingClaim: ""
      storageClass: "longhorn"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
    jobservice:
      jobLog:
        existingClaim: ""
        storageClass: "longhorn"
        subPath: ""
        accessMode: ReadWriteOnce
        size: 1Gi
        annotations: {}
    database:
      existingClaim: ""
      storageClass: "longhorn"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
      annotations: {}
    redis:
      existingClaim: ""
      storageClass: "longhorn"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 1Gi
      annotations: {}
    trivy:
      existingClaim: ""
      storageClass: "longhorn"
      subPath: ""
      accessMode: ReadWriteOnce
      size: 5Gi
      annotations: {}
  imageChartStorage:
    disableredirect: false

    type: filesystem
    filesystem:
      rootdirectory: /storage
    azure:
      accountname: accountname
      accountkey: base64encodedaccountkey
      container: containername
      existingSecret: ""
    gcs:
      bucket: bucketname
      encodedkey: base64-encoded-json-key-file
      existingSecret: ""
      useWorkloadIdentity: false
    s3:
      region: us-west-1
      bucket: bucketname
    swift:
      authurl: https://storage.myprovider.com/v3/auth
      username: username
      password: password
      container: containername
      existingSecret: ""
    oss:
      accesskeyid: accesskeyid
      accesskeysecret: accesskeysecret
      region: regionname
      bucket: bucketname
      existingSecret: ""

imagePullPolicy: IfNotPresent

imagePullSecrets:

updateStrategy:
  type: RollingUpdate

logLevel: info

existingSecretAdminPasswordKey: HARBOR_ADMIN_PASSWORD
harborAdminPassword: "<Password>"

caSecretName: ""

secretKey: "not-a-secure-key"
existingSecretSecretKey: ""

proxy:
  httpProxy:
  httpsProxy:
  noProxy: 127.0.0.1,localhost,.local,.internal
  components:
    - core
    - jobservice
    - trivy

enableMigrateHelmHook: false



nginx:
  image:
    repository: goharbor/nginx-photon
    tag: v2.10.1
  serviceAccountName: ""
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  priorityClassName:

portal:
  image:
    repository: goharbor/harbor-portal
    tag: v2.10.1
  serviceAccountName: ""
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  serviceAnnotations: {}
  priorityClassName:

core:
  image:
    repository: goharbor/harbor-core
    tag: v2.10.1
  serviceAccountName: ""
  automountServiceAccountToken: false
  replicas: 1
  revisionHistoryLimit: 10
  startupProbe:
    enabled: true
    initialDelaySeconds: 10
  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  serviceAnnotations: {}
  configureUserSettings:
  quotaUpdateProvider: db # Or redis
  secret: ""
  existingSecret: ""
  secretName: ""
  tokenKey: |
  tokenCert: |
  xsrfKey: ""
  existingXsrfSecret: ""
  existingXsrfSecretKey: CSRF_KEY
  priorityClassName:
  artifactPullAsyncFlushDuration:
  gdpr:
    deleteUser: false
    auditLogsCompliant: false

jobservice:
  image:
    repository: goharbor/harbor-jobservice
    tag: v2.10.1
  replicas: 1
  revisionHistoryLimit: 10
  serviceAccountName: ""
  automountServiceAccountToken: false
  maxJobWorkers: 10
  jobLoggers:
    - file
  loggerSweeperDuration: 14 #days
  notification:
    webhook_job_max_retry: 3
    webhook_job_http_client_timeout: 3 # in seconds
  reaper:
    max_update_hours: 24
    max_dangling_hours: 168

  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints:
  podAnnotations: {}
  podLabels: {}
  secret: ""
  existingSecret: ""
  existingSecretKey: JOBSERVICE_SECRET
  priorityClassName:

registry:
  serviceAccountName: ""
  automountServiceAccountToken: false
  registry:
    image:
      repository: goharbor/registry-photon
      tag: v2.10.1
    extraEnvVars: []
  controller:
    image:
      repository: goharbor/harbor-registryctl
      tag: v2.10.1

    extraEnvVars: []
  replicas: 1
  revisionHistoryLimit: 10
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  priorityClassName:
  secret: ""
  existingSecret: ""
  existingSecretKey: REGISTRY_HTTP_SECRET
  relativeurls: false
  credentials:
    username: "harbor_registry_user"
    password: "harbor_registry_password"
    existingSecret: ""
    htpasswdString: ""
  middleware:
    enabled: false
    type: cloudFront
    cloudFront:
      baseurl: example.cloudfront.net
      keypairid: KEYPAIRID
      duration: 3000s
      ipfilteredby: none
      privateKeySecret: "my-secret"
  upload_purging:
    enabled: true
    age: 168h
    interval: 24h
    dryrun: false

trivy:
  enabled: true
  image:
    repository: goharbor/trivy-adapter-photon
    tag: v2.10.1
  serviceAccountName: ""
  automountServiceAccountToken: false
  replicas: 1
  debugMode: false
  vulnType: "os,library"
  severity: "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL"
  ignoreUnfixed: false
  insecure: false
  gitHubToken: ""
  skipUpdate: false
  skipJavaDBUpdate: false  
  offlineScan: false
  securityCheck: "vuln"
  timeout: 5m0s
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 1
      memory: 1Gi
  extraEnvVars: []
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  podAnnotations: {}
  podLabels: {}
  priorityClassName:

database:
  type: internal
  internal:
    serviceAccountName: ""
    automountServiceAccountToken: false
    image:
      repository: goharbor/harbor-db
      tag: v2.10.1
    password: "changeit"
    shmSizeLimit: 512Mi
    livenessProbe:
      timeoutSeconds: 1
    readinessProbe:
      timeoutSeconds: 1
    extraEnvVars: []
    nodeSelector: {}
    tolerations: []
    affinity: {}
    priorityClassName:
    initContainer:
      migrator: {}
      permissions: {}
  external:
    host: "192.168.0.1"
    port: "5432"
    username: "user"
    password: "password"
    coreDatabase: "registry"
    existingSecret: ""
    sslmode: "disable"
  maxIdleConns: 100
  maxOpenConns: 900
  podAnnotations: {}
  podLabels: {}

redis:
  type: internal
  internal:
    serviceAccountName: ""
    automountServiceAccountToken: false
    image:
      repository: goharbor/redis-photon
      tag: v2.10.1
    extraEnvVars: []
    nodeSelector: {}
    tolerations: []
    affinity: {}
    priorityClassName:
    jobserviceDatabaseIndex: "1"
    registryDatabaseIndex: "2"
    trivyAdapterIndex: "5"
  external:
    addr: "192.168.0.2:6379"
    sentinelMasterSet: ""
    coreDatabaseIndex: "0"
    jobserviceDatabaseIndex: "1"
    registryDatabaseIndex: "2"
    trivyAdapterIndex: "5"
    username: ""
    password: ""
    existingSecret: ""
  podAnnotations: {}
  podLabels: {}

exporter:
  replicas: 1
  revisionHistoryLimit: 10
  extraEnvVars: []
  podAnnotations: {}
  podLabels: {}
  serviceAccountName: ""
  automountServiceAccountToken: false
  image:
    repository: goharbor/harbor-exporter
    tag: v2.10.1
  nodeSelector: {}
  tolerations: []
  affinity: {}
  topologySpreadConstraints: []
  cacheDuration: 23
  cacheCleanInterval: 14400
  priorityClassName:

metrics:
  enabled: false
  core:
    path: /metrics
    port: 8001
  registry:
    path: /metrics
    port: 8001
  jobservice:
    path: /metrics
    port: 8001
  exporter:
    path: /metrics
    port: 8001
  serviceMonitor:
    enabled: false
    additionalLabels: {}
    interval: ""
    metricRelabelings:
      []
    relabelings:
      []

trace:
  enabled: false
  provider: jaeger
  sample_rate: 1
  jaeger:
    endpoint: http://hostname:14268/api/traces
  otel:
    endpoint: hostname:4318
    url_path: /v1/traces
    compression: false
    insecure: true
    timeout: 10

cache:
  enabled: false
  expireHours: 24

Some of the logs while trying to push to the registry:

[15/Apr/2024:10:58:59 +0000]:10.42.3.0 - "GET /v2/ HTTP/1.1" 401 76 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.002 0.002 .
[15/Apr/2024:10:58:59 +0000]:10.42.3.0 - "GET /service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry HTTP/1.1" 200 633 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.009 0.009 .
[15/Apr/2024:10:59:00 +0000]:10.42.3.0 - "GET /v2/ HTTP/1.1" 200 2 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.068 0.069 .
[15/Apr/2024:10:59:01 +0000]:10.42.3.0 - "GET /v2/ HTTP/1.1" 401 76 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.002 0.002 .
[15/Apr/2024:10:59:01 +0000]:10.42.3.0 - "GET /service/token?account=admin&scope=repository%3Atest-speed%2F1gb-random-file%3Apush%2Cpull&service=harbor-registry HTTP/1.1" 200 722 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.011 0.011 .
[15/Apr/2024:10:59:01 +0000]:10.42.3.0 - "HEAD /v2/test-speed/1gb-random-file/blobs/sha256:abf85a4cf2dd657ef8721648a7f6122c6758b9390220bc95e8369a9d1df123b4 HTTP/1.1" 404 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.003 0.003 .
[15/Apr/2024:10:59:01 +0000]:10.42.3.0 - "HEAD /v2/test-speed/1gb-random-file/blobs/sha256:b2388ca7fa65a68824f137dc4184ea3ea789570753d795042d9af40fc9383448 HTTP/1.1" 404 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.004 0.005 .
[15/Apr/2024:10:59:02 +0000]:10.42.3.0 - "HEAD /v2/test-speed/1gb-random-file/blobs/sha256:026cb769511a65d93a9a24aadff9124af782f85b558d1e981931d1f947ae2ee0 HTTP/1.1" 404 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.003 0.003 .
[15/Apr/2024:10:59:02 +0000]:10.42.3.0 - "POST /v2/test-speed/1gb-random-file/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.087 0.087 .
[15/Apr/2024:10:59:02 +0000]:10.42.3.0 - "POST /v2/test-speed/1gb-random-file/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.086 0.086 .
[15/Apr/2024:10:59:07 +0000]:10.42.3.0 - "HEAD /v2/test-speed/1gb-random-file/blobs/sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299 HTTP/1.1" 404 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.004 0.004 .
[15/Apr/2024:10:59:07 +0000]:10.42.3.0 - "POST /v2/test-speed/1gb-random-file/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.15.49-linuxkit-pr os/linux arch/amd64 UpstreamClient(Docker-Client/26.0.0 \x5C(linux\x5C))" 0.086 0.087

I think the problem here is that the harbor as TLS Disabled and tryes to connect to port 80 instead of port 443 and docker doenst allow that while trying to push (?). If I enable TLS and change haproxy to forward to port 443 I get this info:

Any help provided is welcome. Thanks.

Apr 15 '24 11:04 gsaudade99