A distributed public key server for 90 million users

The problem

A software development company was building a large-scale email encryption system. Every encrypted message needs the recipient’s public key, so the key server sits on the path of every email sent: tens of millions of users, a lookup for every message, and reads that outnumber writes many times over, with sharp peaks during the working day.

They needed a public key infrastructure that could start small, to keep costs down at launch, and scale out seamlessly as the user base grew.

What we did

• Multiple primary databases, each replicated to a standby for failover and to read-only databases.
• Custom sharding splits the data across the primaries. This is what takes performance up to 90 million users.
• Primaries added with zero downtime. The system redistributes data onto the new server automatically. This took the system to 40,000 writes per minute.
• Read-only databases added the same way, with no downtime and automatic redistribution. They are memory-only, which gives extreme performance: over a million reads per minute.
• Load-balanced front-end servers for redundancy and scalability.
• An architecture that starts small. A small number of servers keeps costs down at the start, and the same design scales to hundreds as load increases.

What changed

The key server went live on a small footprint and grew with the product to 90 million users, 40,000 writes and over a million reads per minute, without a maintenance window for adding capacity.