localgov
localgov copied to clipboard
localgovdrupal
Hosting LocalGov Drupal on Microsoft Azure (and other options)
@willguv has been asked a few times about hosting LocalGov Drupal on Microsoft Azure.
I know that @Adnan-cds at Croydon has some experience of this.
From what I know, Azure is a cloud platform that can offer virtual machines, including Ubuntu images.
It can also do Docker / Kubernetes / Openshift...
From a LocalGov Drupal perspective, it would be good to be able to advise councils on hosting considerations within Azure.
As a starting point, perhaps we could gather information on the setup at Croydon and any other councils that are using Azure to run Drupal sites.
@Adnan-cds are you able to summarise how you have it set up and perhaps list any issues that you had to work around?
@willguv do we know of any other councils hosting Drupal on Azure?
are you able to summarise how you have it set up and perhaps list any issues that you had to work around?
There are two things to consider here:
- Drupal hosting.
- CI/CD
Hosting
We use Azure Webapps (AKA the "App service") for hosting our LocalGov Drupal sites. There are three ways you can provide your PHP code to a Webapp:
- Upload code from a Git repo or FTP. Azure then uses its own PHP Docker container to serve the site. We avoided it as we found it slightly wanting for hosting Drupal.
- Bring your own Docker container. Azure's own PHP container has good integration with the rest of Azure. So we extend the Azure PHP container to roll our own. This way we retain all the Azure goodness while fine tuning for Drupal.
- Docker compose. This allows multiple containers. But is perennially in "Preview", so we skipped it.
The underlying VM, which we don't manage, is of P2V2 category. It has 2 virtual cores (whatever that means) and 7GB of memory. It can host 4-6 Webapps.
Pros
- Easy to maintain consistency between local development sites and Azure hosted sites.
- Extending Azure's PHP container saves us from OS upgrades.
- Azure App service manages auto scaling, TLS certificates, load balancers, etc. which leaves us with a lot less hassle.
- Built-in support for Blue-green deployment.
- Can't remember of any downtime in our nearly 3 years of Drupal hosting in Azure.
Cons
- Having to use a single container means Apache and PHP (and some other services) had to be bundled into one container which is less than ideal.
Other services
- Database: Azure's hosted MySQL with 2 virtual cores and 333GB disk space dispensing 999 IOPS. MySQL's rolling backup takes up a lot of space (1-3TB) due to constant update of various cache_* tables.
- Drupal's "files/" directory: Azure storage File share mounted as a shared directory within the PHP container.
- Microsoft CDN. Cheap but effective.
- Redis on Azure isn't cheap, so couldn't justify so far due to cost-benefit considerations.
Cost
Monthly total hosting expenses for a production environment and a dev environment are in the lower four figures. The MySQL instances (including a replica) are costing the lion's share.
CI/CD
Azure DevOps pipelines was a natural choice as it has good integration with rest of the Azure services. The definitions of various pipelines are provided as Yamls and live as part of the Drupal codebase.
Pros
- It works.
Cons
- Running drush commands inside a running container from a pipeline involves some gymnastics.
- There have been a few incidences of bugs creeping into the pipeline product causing us some head scratching.
Wow, thanks @Adnan-cds!
@willguv a couple of questions for you:
- who else uses Azure?
- who has been asking about Azure that we can bring into the conversation?
Cheers!
Thanks @Adnan-cds for your amazing reply - really helpful
Hi @finnlewis, thanks very much for kicking off this discussion. The original Azure request came from Westminster. Our original contact has moved on, but we have a new contact there to try
The latest request came from Southwark a couple of weeks ago, and I believe they're meeting with Adnan separately
We've not recorded Azure requests between these two. Let's post on #general and consider for the next newsletter
We (well, Nomensa on our behalf) also use Azure.
Our setup has:
- Azure Front Door instead of Azure CDN
- Container Apps instead of App Service
- Azure managed MariaDB instead of Azure managed MySQL
- Gitlab CI instead of Azure DevOps Pipelines
Other than that, I think the architecture is pretty similar to what Adnan describes. A lot of the detail is in our repo https://github.com/essexcountycouncil/essex-gov-uk-drupal.
@finnlewis I've just been asked the following
Do you have any case studies or examples of councils using LocalGov Drupal in the cloud – specifically AWS or Azure?
@finnlewis I've just been asked by another council about technical discovery work done by LGD councils, principally around hosting infrastructure. This is a very popular topic!
I wondered if we could run a show and tell involving a few councils to show the variations (copying @TJHDCR and @aaronhirtenstein) and then write up as documentation?
We're discussing this again today in our Technical Drop-in.
Tamworth are also trying to get a dev site setup in Azure. Anushyia at Colchester are also trying to get setup in Azure.
We'd like to get some guidance written to help people to get started on Azure, how-tos, maybe some videos.
For rebuild of the website of https://www.criminaljusticehub.org.uk (CJH), I have investigated various approaches to hosting LocalGov on Microsoft Azure. As my last experience with Azure Cloud (AZ) was in 2018, I anticipated that my knowledge might be outdated. Consequently, I delved deeply to justify my choices.
My primary objectives were to:
- Deploy a technically sound solution. Since CJH is a non-profit organization, the DevOps team is likely to experience frequent changes. Each shortcut taken could become a potential hurdle for people I have yet to meet.
- Ensure easy scalability, utilizing the cloud.
- Maintain cost control.
I hope the following reasoning and choices are helpful and informative.
Technical breakdown
CDN
CJH has previously used CloudFlare (CF) for their website hosted on Cloudways. I discovered no limitations that would prevent me from using CF with Azure.
CI/CD
Initially, I intended to use Azure DevOps pipelines. However, the requirement to apply for a parallelism grant for even a single worker (possibly just a glitch?) discouraged me, and I opted for GitHub Actions instead.
Hosting stack
While Azure Web Apps' PHP container can be updated to host Drupal, it incurs considerable vendor lock-in, as it requires Azure-specific settings and offers no additional benefits. Moreover, even the fastest Storage account drives' File shares (possibly due to SMB mounting?) occasionally exhibited slow speeds, causing intermittent timeouts, particularly during Composer installs. Although CI/CD automation could mitigate such issues, the unreliable file system stability remained a deal-breaker.
The second limitation, which applies to all Azure containerized services, is the sandboxed Linux environment that disallows direct SSH. Though CI/CD pipelines can mitigate this, the pipelines become Azure-specific, which could limit hiring DevOps personnel and increase costs.
These issues also affect Azure Container Instances (ACI). The instances, especially when formed as a multi-container group via Azure Resource Manager (ARM), work well and serve as drop-in replacements for Docker Compose. However, in addition to the aforementioned issues, they have limited volumeMounts, as they always bind with the root folder of the File share. Overcoming this limitation would require amending my PHP and Nginx Docker images, which necessitates an Azure-specific approach.
Unsurprisingly, Azure K8s File mounts have similar limitations.
After a day of testing, I determined that Azure Linux VM (VM) was the most rational choice. It is a standard Linux machine with Bring Your Own Image capability and linked SSD for source code, providing the expected speed.
It scales both vertically and horizontally.
Database
Azure Database for MariaDB server (1 core, 2GB RAM) was an obvious choice due to its cloud-native scalability. However, to manage costs, I decided to use it solely for the production environment, integrating MariaDB into the VM for staging and development servers.
File storage
I chose File shares at Azure Storage account with Locally-redundant storage for binary files. This allows me to keep the drive size within the VM low and share binary data from a single resource as needed. This resource scales organically.
Price breakdown (Azure)
- VM (Standard B2s): ~£28/month
- Code (HTML, CSS, JS) traffic: ~£3/month based on 1000 page visits/day, predominantly from the UK
- Database (Azure Database for MariaDB server, 1 core, 2GB RAM): ~£25/month
- Binary file storage data (locally-redundant storage): ~£0.5 based on approximately 5GB of data stored
- Binary file storage data (transactions and data transfer costs, with CDN): within the free tier
Storage without CDN
- Binary file storage data (transaction costs, based on 1.5 million requests/month): ~£2
- Binary file storage data (data transfer cost, based on ~700GB outbound traffic): ~£65
Should you identify any inconsistencies or have recommendations for improvement, please don't hesitate to share your insights!
What are the tasks to be done on this @willguv ?
Publish a summary / how to on the docs site?
Hi @finnlewis yes, write up the common/ best way (or ways) or hosting LGD on Azure. We can create other issues for common hosting methods
Do you have enough info above to complete this?
Is it possible to get this information on our docs site?
It comes up a lot, most recently with SRS Wales who are evaluating Azure vs AWS and I have shared this issue with them, but having it more easily accessible in Docs would make it easier to signpost to
