goharbor
Replacing nginx with Istio VirtualService
I'm trying to replace the nginx-based Ingress in my Harbor deployment with an Istio VirtualService, as follows:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: harbor
namespace: istio-system
spec:
hosts:
- "registry.elpenguino.net"
gateways:
- istio-ingressgateway.istio-system.svc.cluster.local
http:
- name: "core"
match:
- uri:
prefix: "/api"
- uri:
prefix: "/service"
- uri:
prefix: "/v2"
- uri:
prefix: "/chartrepo"
- uri:
prefix: "/c"
route:
- destination:
host: harbor-core.harbor.svc.cluster.local
port:
number: 80
timeout: 30s
- name: "portal"
route:
- destination:
host: harbor-portal.harbor.svc.cluster.local
port:
number: 80
timeout: 30s
As far as I can tell, this is all working, with the exception of uploading charts into ChartMuseum. It seems as if my POST request goes like this:
Me -> (A) Istio Ingress -> (B) Harbor Core -> (C) Harbor Core, and I never see the request actually hit the ChartMuseum pod.
Curiously, in connection (C) above, Istio tries to send the post to the EXTERNAL_URL of my Harbor instance, not to the in-cluster URL used for A and B.
Am I missing anything re how nginx / core deals with the prefixes detailed above?
Thanks! David
Which version of harbor did you try with?
I managed to get harbor 1.10.2 working with Istio using config I posted in https://github.com/goharbor/harbor-helm/issues/77. Not using the chartmusem so don't know if that actually works.
I spent a great deal of time debugging why the requests towards the registry (/v2/) didn't work, until I discovered that the core handling them is just acting as a proxy so I rerouted them directly to the registry service and got it working.
I've not managed to get harbor 2.x working with Istio.
@jabbors did you not encounter the problem of having empty database registration of the registry actions? If we do a push directly bypassing core, we see nothing in the database.
@Morriz, no I did not. I analysed the core source code but couldn't find anything indicating that a request towards /v2/ would trigger any read/writes in the database. It's just a proxy in front of the registry.
Later I found an Istio misconfiguration in our harbor deployment which resulted in dropping the bypass. See https://github.com/goharbor/harbor/issues/11906 for more information.
We have it all working also (login, pull, push), directly proxying to the registry backend, but we don't see any database entries reflecting the docker commands.
When we routed requests via core we got all kinds of errors on docker commands, while curl GET commands would work nicely. Might it be the difference between curls https and dockers protocol?
I am not sure if anybody out there on the web has a proven setup with istio that results in DB updates showing in the UI. Hope to find the missing link. I would assume core is responsible for proxying to the registry, AND instrumenting some callbacks triggering db updates. But since you seem to have inspected the code and found nothing noteworthy, I am left in the dark on the internals responsible for the docker+postgres integration.
I was talking about the same issue here btw: https://github.com/goharbor/harbor/issues/12804#issuecomment-694732997
I found this comment in the core app (https://github.com/goharbor/harbor/blob/73981062a9e9b76426b28029e14a5755b5cb6777/src/core/middlewares/middlewares.go#L44):
// For the PUT Blob Upload API, we will make a transaction manually to write blob info to the database when put blob upload successfully.
So I think bypassing core is not the way...
We migrated our harbor instance to k8s around the 1.10 release and have been running the 1.10.x releases with Istio for some while. No nginx deployed and the Istio virtual service proxying /v2/ requests through core.
Ensure that the service names for you harbor components are prefixed with http- or tcp-, eg. http-core and http-registry, as this is a requirement by Istio. I had forgotten about this and when we fixed our deploy we could revert our hack bypassing /v2/ requires directly to registry.
I got it working just now. We have all service paths behind an auth proxy and had to also make /service/ public (next to /v1/ and /v2/). I have to investigate the impact of that and might close down the rest that can go back under auth.
Ensure that the service names for you harbor components are prefixed with
http-ortcp-, eg.http-coreandhttp-registry, as this is a requirement by Istio. I had forgotten about this and when we fixed our deploy we could revert our hack bypassing/v2/requires directly to registry.
That is only a "requirement" for mTLS I believe, and might not even be a real requirement (it is old info that goes around imo).
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
I had similar problems with my istio installation. Basically, I took the same approach as @funkypenguin and hit the errors with the registry connection mentioned in this issue. Thanks to @dcherman's comment https://github.com/goharbor/harbor/issues/14605#issuecomment-877695309 I was able to figure out that the core proxy will preserve the Host header, which will mess up internal mesh routing.
The solution was relatively easy
- Add a
ServiceEntrywith the expected contents of the host header (i.e. your external domain name)
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: harbor-registry
spec:
hosts:
- myregistry.example.com
- myalias.example.com
ports:
- number: 5000
name: http-registry
location: MESH_INTERNAL
resolution: NONE
- Add an additional route to the
VirtualServiceand include the specialmeshgateway
gateways:
- istio-system/default-gateway
+ - mesh
...
http:
+ - match:
+ - port: 5000
+ route:
+ - destination:
+ host: harbor-registry.harbor.svc.cluster.local
+ port:
+ number: 5000
...
This will correctly route the internal request from the core to the registry component.
We don't use chartmuseum, but I guess a similar solution might fix that problem, too.
Thanks @heilerich for you patches.
I deployed harbor with helm (chart version 1.7.5) with Istio mTLS enabled and it worked right away.
Just to be extra clear, the helm chart deploys a nginx pod and a service but with mTLS in the mix the nginx service will not become available. But the nginx pod is not needed if one creates an VirtualService as described by the author os this issue.
The next step is to figure out how to not deploy nginx at all or contribute with an upstream change to make it possible to enable/disable the nginx module.
Thanks @heilerich for you patches.
I deployed harbor with helm (chart version 1.7.5) with Istio mTLS enabled and it worked right away.
Just to be extra clear, the helm chart deploys a nginx pod and a service but with mTLS in the mix the nginx service will not become available. But the nginx pod is not needed if one creates an VirtualService as described by the author os this issue.
The next step is to figure out how to not deploy nginx at all or contribute with an upstream change to make it possible to enable/disable the nginx module.
We have done just that in redkubes/otomi-core: disable nginx and use istio to wire it up. Works great!
@Morriz
Nice, did you have any troubles with trivy image scnanning? At least for us trivy installs just fine, but but during a scan job it fails to pull the image to scan.
This is what we see in the logs
scan error: unable to initialize a scanner: unable to initialize a docker scanner: 3 errors occurred:
* unable to inspect the image (harbor-core:80/test/hello@sha256:5b47eac1ccd2e864ed4d98342fee7204eabbb6e8693ffc87f402cd0de615c5ee): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
* GET http://harbor-core:80/v2/test/hello/manifests/sha256:5b47eac1ccd2e864ed4d98342fee7204eabbb6e8693ffc87f402cd0de615c5ee: unsupported status code 503; body: upstream connect error or disconnect/reset before headers. reset reason: connection termination
I assume it's just a hostname that is missing from the Istio mesh but I haven't had time to investigate it further.
We have the exact same issue with trivy scanning. Since we don't use it at the moment, I didn't investigate possible solutions so far.
@Morriz
Nice, did you have any troubles with trivy image scnanning? At least for us trivy installs just fine, but but during a scan job it fails to pull the image to scan.
This is what we see in the logs
scan error: unable to initialize a scanner: unable to initialize a docker scanner: 3 errors occurred: * unable to inspect the image (harbor-core:80/test/hello@sha256:5b47eac1ccd2e864ed4d98342fee7204eabbb6e8693ffc87f402cd0de615c5ee): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? * unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory * GET http://harbor-core:80/v2/test/hello/manifests/sha256:5b47eac1ccd2e864ed4d98342fee7204eabbb6e8693ffc87f402cd0de615c5ee: unsupported status code 503; body: upstream connect error or disconnect/reset before headers. reset reason: connection terminationI assume it's just a hostname that is missing from the Istio mesh but I haven't had time to investigate it further.
No, we connected everything correctly using virtualservice resources. Trivvy scanning (or any scanning) works fine.
@Morriz
Are you able to share the VirtualService resources that you create? Or guide me where I can see them?
I took a look at the otomo-core source code but couldn't spot them there. I also tried to install otomi on my laptop to inspect them, but installing otomi on my local cluster ( kind 1.21 1+3 workers) failed.
Sure, you can just install the cli (see otomi.io) and follow the docs to bootstrap a values repo locally. Then you can template all resources with otomi template, or only render the ingress related resources with otomi template -l ingress=true.
No cluster needed ;)
Thanks, was able to render the resources and compare them with our VirtualService. They looked mostly the same as the one posted by the author of this issue, which is the one we use as well.
The difference between our setups is that we don't use a nginx ingress at all. Instead we use Istio's own ingressgateway, which is exposed outside the cluster as a LoadBalancer service with the help of MetalLB.
FYI, I tried to apply your harbor and istio resources in one of our testing clusters but got to same results as before. Even had to apply the ServiceEntry patch to be able to perform a succesful docker login
in our setup nginx is just the front gateway, but passes the traffic to istio. It is important though to see what ingress is mapped to which harbor service. You can find that config in otomi-core/core.yaml
@jabbors Did you have the Helm installation for Harbor at hand. As far as I'm aware I've got the correct Istio gateway yamls and Virtual Services but am still unsure of all the configuration options in the Harbor helm chart.
I've also found that I can communicate with harbor UI using http but not https
@ChrisJBurns with Helm chart are you referring to? The one I extracted from otomi-core or the official chart which is the one I'm trying to get working.
I deployed harbor with helm (chart version 1.7.5) with Istio mTLS enabled and it worked right away.
Mainly the config for that you used to get Harbor working with mTLS. I'm finding that when the istio-proxy sidecars are included in the Harbor pods, notary-server and notary-signer fail to start due to connection issues.
I'm also unable to get to a front end when going to a https link.
I must say we skipped install of notary because of similar issues. We don't see the added value when automatic scanning is enabled and harbor is used as the source registry for k8s
I guess the added value is that it ensures images are signed, and you can somewhat verify that only signed images are deployed to Kubernetes - I guess it's that extra layer of security
For reference. Here are my Gateway and VirtualService yaml
Gateway
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-gateway
namespace: istio-system
spec:
selector:
istio: gateway # use Istio default gateway implementation
servers:
- port:
number: 443
name: harbor
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: harbor-ingress-cert # This should match the Certificate secretName
hosts:
- harbor-istio2.domain.com # This should match a DNS name in the Certificate
- notary-istio2.domain.com
VirtualService
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: harbor-vs
namespace: harbor
spec:
hosts:
- harbor-istio2.domain.com
gateways:
- istio-system/istio-gateway
http:
- name: "harbor"
match:
- uri:
prefix: '/api/'
- uri:
prefix: '/c/'
- uri:
prefix: '/chartrepo/'
- uri:
prefix: '/service/'
- uri:
prefix: '/v1/'
- uri:
prefix: '/v2/'
route:
- destination:
host: harbor-core
port:
number: 80
- match:
- uri:
prefix: /
route:
- destination:
host: harbor-portal
port:
number: 80
Here is how we are installing Harbor:
Harbor Helm Values
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: harbor
namespace: harbor
spec:
chart:
spec:
chart: harbor
version: 1.8.1
sourceRef:
kind: HelmRepository
name: harbor
interval: 1m
values:
externalURL: http://harbor-istio2.domain.com
updateStrategy:
type: Recreate
metrics:
enabled: true
persistence:
enabled: true
resoucePolicy: keep
persistentVolumeClaim:
registry:
size: 20Gi
chartmuseum:
size: 5Gi
expose:
tls:
enabled: true
type: clusterIP
So far, I've only got a 404 not found when going to https://harbor-istio2.domain.com. Not sure why it's not routing. Ideally, we'd like the TLS termination to be done at the IstioGateway which then initiates a new TLS connecting to Harbor services where Istio mTLS works from there.
404 error:
Not sure what I'm doing wrong to be honest, I'm suspecting it is the Helm installation of Harbor with regards to the values themselves as I've tried all sorts of VirtualService and Gateway combinations that I've seen around this issue - but I could be wrong - so open to comments/guidance.
I guess the added value is that it ensures images are signed, and you can somewhat verify that only signed images are deployed to Kubernetes - I guess it's that extra layer of security
IMO that adds no real "extra layer of security" security when the right measures are taken, such as full network directionality control. (By means of network policies icw egress rules allowing only traffic towards the harbor host.) I am then not worried about an attacker relaying traffic to another "fake" harbor/registry host (as this is simply not possible), and pulling signed images from such a trusted endpoint does not give any added security in that case.
Neither signing nor directionality control mitigates the scenario of a compromised harbor host. Once a harbor host is compromised an attacker can pull in any or overwrite any image and it will be signed automatically by notary.
About guidance: many days of work went into getting harbor to work well with istio, and that is reflected in redkubes/otomi-core. It has a cli and you can generate all the resources for your target cluster with otomi template -l pkg=harbor, but you have to follow the docs to initialize a values repo first: https://otomi.io/docs/cli/working-with