Improving the robustness of our databases

We really don’t like storing data, but there are some we can't avoid storing. At least we can keep that safe.

We recently migrated our backend databases to new hardware, with the intention of improving robustness and reliability. What does this mean exactly? Did we upgrade something to improve our scaling due to a mass influx of users? What changed? Why new hardware?

First, a little background

We really don’t like storing any data about our customers. It is in our privacy model and our reason for being. What we do want to do is provide a reliable and robust service for you.

Our backend servers provide, among other things, the list of servers and their working status, the account tokens, and payment status of your account – inner workings so that we can allow you to use our service. As you can imagine it is a very critical part of our internal infrastructure.

So what exactly did we do?

Our backend servers are made up of a number of worker servers. These take in the queries that our website sends it, and returns information to you or the Mullvad app. These worker servers are connected to a number of our databases. That is where the account information (16-digit tokens, payment status) as well as server details are stored.

These databases were moved to new hardware for a number of reasons:

• We need to scale up. We have a lot of new customers coming in, and we need to ensure that we provide a reliable service for new and current customers.
• We cannot afford to lose your trust. We want to have redundant data so that in case of a panic, we do not lose your trust, or your account status.
• We want to simplify and internally share the knowledge of how our services are set up.

Some slightly more technical technicalities

We run a replicating PostgreSQL set up on our database servers. They are located in a number of data centers in Sweden, all with powerful hardware, multiple disks, dedicated power supplies, and their own UPS each (failover batteries).

Our databases replicate from a leader database. The leader is what our backend servers connect to, where the queries that the website, app or other client get sent. The followers are continuously archiving and keeping a local copy of any updates that the leader gets, but they’re in read-only mode: only the leader can answer backend servers' queries.

Having many databases in multiple locations means we have the reliability of different power supplies and redundancy against an entire data center losing power.

To connect these databases together we use our trusted WireGuard® protocol. We are big fans of WireGuard, and we found it fitting to implement it to secure the data sent between each database, especially if they are to be located in many locations. We also make use of TLS certificates and strong encryption on the replication and backend servers' queries. We only want the relevant services to be able to get the relevant information.

Toward the future

Simplification of services is our future goal. We will be improving the deployment of all our services moving forward, aiming for trust, reliability and robustness in all that we provide.

For your right to privacy,
Mullvad VPN

 

"WireGuard" is a registered trademark of Jason A. Donenfeld.