microsoft
A non public deployment of FinOps Hub (Use Private Endpoints)
⚠️ Problem
A non public deployment of FinOps Hub, no public endpoints and use private endpoints.
🛠️ Solution
Add conditional deployment of private endpoints for Data Factory, Key Vault and storage account.
Add a paramater called subnetResourceId, if the parameter is present, private endpoints will be created to the Subnet.
Add a parameter called publicNetworkAccess , if the parameter is set to Disabled public access is disabled for Data Factory, Key Vault and storage account.
Add a parameter called integrationRuntimeSubnetResourceId , if the parameter is present, an allow VNet rule for the subnet will be added to the storage account firewall.
Note 1: If the environment has policies preventing Public IP addresses for PaaS services, the deployment must be moved to a staging area where the policy does not apply during deployment, as otherwise the script steps will not function. Once the deployment is complete, you can move it back to the subscription.
Note 2: A private environment need an Self-Hosted Integration Runtime to access the content in the storage account.
ℹ️ Additional Information
A draft is made and tests have been done to confirm functionality over Private Endpoints and no public acccess. I have updated the Bicep files with the parameters and private endpoints and made the necessary changes to allow a private deployment. There are som manual steps involving changing the Integration Runtime within the Data Factory, this is included in the deployment description added to the README.md.
Let me know if there is interest in these changes.
A few questions, since I don't have enough context on private endpoints:
- The
subnetResourceIdandintegrationRuntimeSubnetResourceIdparameters both point to existing resources, correct? - Are these generally different subnets? (I'm assuming so. Just confirming.)
- How does the staging area process work? I'm not familiar with that.
- What are the manual steps? (Although, I'm sure I'll see that in the PR.)
- How will Power BI connect to storage?
At a high level, this all sounds good. I created a new features/private branch as I'm sure we'll need a few PRs to finalize things before we merge into dev. I'll add this to 0.3 tentatively for tracking purposes. Thanks for proposing a contribution!
@MSBrett / @Springstone / @arthurclares any thoughts you want to add?
A few questions, since I don't have enough context on private endpoints:
- The
subnetResourceIdandintegrationRuntimeSubnetResourceIdparameters both point to existing resources, correct?
Correct, these must exists.
- Are these generally different subnets? (I'm assuming so. Just confirming.)
Normally yes, so the FinOps resource Private Endpoints will be deployed to the subnetResourceId subnet and you probably already have an IR deployed somewhere in the environment. However, if someone would put the IR on the same subnet, thats not a problem, then you would not need to define integrationRuntimeSubnetResourceId, since this creates an Allow rule in the storage acconts FW for that specific subnet. And if they are on the same subnet, that won't be an issue.
- How does the staging area process work? I'm not familiar with that.
We have a strict private Azure environment, we dont allow Public service and we control this with policy.
We have leaned on the Cloud Adoption Framework Enterprise scale architecture for structure, so the policy applies to all Management Groups, except one called Staging where no policies are applied. If you use Private Endpoint but don't prevent the use of Public Services with policy, this will not be an issue, but for us it is.
So the process is quite simple, move the target subscription to a place where the policy Public network access should be disabled for PaaS services don't apply, install the FinOps Hub and move it back in the structure.
- What are the manual steps? (Although, I'm sure I'll see that in the PR.)
First you must share the IR with the FinOps Data Factory so it will be accessibe, then you have to go to the Linked Services, click on storage and change Connect via integration runtime to the Self-Hosted IR.
Then you are good to go. I have added some links to the README How to Share an ADF and add a Self-Hosted IR in the ADF as well.
- How will Power BI Connect to storage?
Using the DNS name as you would accessing it from Internet. If you have configured your environment right and followed the documentation for Private Endpoints and DNS at scale, I have added links to the README, this would work perfectly. When you are on your private network, in Azure or VPN, the DNS name will resolve a private IP and you can access the storage account privatley. If you are on internet, the DNS will resolve a public IP and won't be accessible since the storage account prevents public access.
I guess that if you use Private Endpoints, you are familiar with this, however, there are a few checks that can be added to the Troubleshooting guide.
At a high level, this all sounds good. I created a new
features/privatebranch as I'm sure we'll need a few PRs to finalize things before we merge into dev. I'll add this to 0.3 tentatively for tracking purposes. Thanks for proposing a contribution!@MSBrett / @Springstone / @arthurclares any thoughts you want to add?
@flanakin I made the change in the branch features/private but I wasn't allowed to push it. Should I create a new branch or how should I do it?