Hundreds of websites in a Drupal multisite factory

The Belgian Police was looking for a solution to migrate their outdated website infrastructure to a more modern alternative. In consultation with Calibrate, they chose to build the websites in the most recent version of Drupal.

At the outset of the project, we faced several major technical challenges. First, the existing look-and-feel must not be changed. This is because it has been defined by the internal committee according to an approved specification.

In addition, we will be transferring over 180 websites from both the local and federal police. Our developers brought all these websites together under one common portal site, while the Belgian Police web team can still manage each website separately.

So the challenge is to make the migration as smooth as possible, taking into account that each website can also have one or more languages and needs to be accessible under 3 domain names. As an extension to the existing functionality, we also provide a mechanism to centrally manage content and synchronize it between all websites.

In the first major phase of the project we migrate the old content from the Joomla and Nooku websites to Drupal 8. For this, we make a comprehensive functional analysis where we inventory all data types. In this phase we also map out the functional operation of the website. The release of Drupal 8 was still recent when we started this project, so we started working with the beta versions of Drupal 8 as early adopters.

Many functionalities that we now consider obvious still had to be developed or adapted. The police themselves are also working hard on innovation and digitalization of their existing infrastructure and are committed to open source. Partly because of this we were able to share our 'pioneering work' in the Drupal community. In a later phase we exchanged the legacy code for modern code, in order to achieve a stable and secure web platform.

Not only building the platform itself was a challenge. The migration of the websites turned out to be a real undertaking. We completely transferred 180 websites without interrupting the operation of the existing websites. The police websites not only have an informative role but also an active one.

To give an example, the website of the Federal Police is consulted 160,000 times a month. Traces of missing persons, information on wanted criminals, reports of crimes, etc. All crucial data that we handle with the necessary care.

With the help of Calibrate, the complexity of 4 languages, 3 domain names, 1 install profile, 180 websites, 2 themes was minimized. The Drupal install profile contains a configuration according to Drupal standards, several modules (contrib and custom) and a number of visual themes.

To meet the requirement of multilingualism, we modify Drupal's standard installation process so that multiple languages are installed immediately. Using the two basic themes, each site can choose whether it is a Local Police or Federal Police website. By working with a custom installation profile, we can guarantee uniformity and reduce development complexity to a minimum. 

Initially, we used Aegir in conjunction with a Jenkins deploy server for the entire setup. Through Aegir, separate instances were created based on the installation profile and each site was updated separately. Then Jenkins took over and the sites were transferred from the acceptance environment to production.

This ensures that when we roll out a new release, we have to update about 180 website instances. The full release cycle (local - dev - acc - production) would come out to about 3 days of lead time. However, this was not acceptable in the case of security releases so we came up with an alternative approach.

We swap the entire deploy and hosting logic. We leave out Aegir and now let Jenkins take care of everything. A Drupal multisite setup replaces the 180 individual website instances. The result was a significant reduction in deploy time to eight hours, but we were not yet satisfied and aim for an even shorter period. By modifying the scripts to parallel executable blocks, we can now provide 180 sites with a full database update within a two to three hour timeframe. Code changes or a security release happen practically instantaneously thanks to the Drupal multisite setup, where only 1 codebase needs to be updated.

Data security is a hot-topic today. Of course, we are therefore fully up-to-date with the GDPR legislation. Here too, we closely follow the latest guidelines and develop tools ourselves to comply with these guidelines. Privacy and disclaimer statements are managed centrally and rolled out across all 180 websites. In this way, the various police districts do not have to worry about the regulations. If there are specific needs, each zone can add additional texts.