Databricks+ADF E2E Sample (Milestone 2)

[ydaponte]

Milestone 2: end-TBD for the catcher team.

E2E Sample "Technical Problem"

• Technical Problem: improve sample usability, ai and security features.
• Persona: Data Engineers and Developers.

Modularization

Incorporate checkpoint mechanism during deployment, so that re-deployments are faster and resilient improving the user experience.

• [ ] Split sample in 3 main parts for re-deployment: Bicep deployment, Services configuration and AzDo deployment.
• [ ] Incorporate checkpoints so re-deployment starts where it left of on the last run.

Security

Incorporate security features so that the repository showcases a high-security standards beyond the DataOps principal function.

• [ ] Remove root user from Docker deployment
• [ ] Perform a inventory of used SP - plan and implement to replace with MSIs
• [ ] Replace SQL Dedicated pool by a Azure SQL instance.
  • [ ] Allow Entra ID authentication only in the SQL resource.
  • [ ] Replace SQL authentication by a MSI.
• [ ] Disable on the storage accounts the "Allow Key access" configuration.
• [ ] Disable all public networks setting from all applicable azure resources present in the current deployment (ex: storage accounts, key vault resource, azure sql, etc).
  • [ ] Make necessary changes for the services to be able to connect with each other

AI

Incorporate AI features so that the repository is easier to be understood and used by new users and contributors.

Backlog existing items

There are existing items on the backlog that are worth to follow-up on and prioritize accordingly. There are also eligible items to drive some design sessions moving forward.