How to Manage Data Protection and Scale Your Business Fast
Pipedrive started working on GDPR requirements over four years ago.
But we can’t claim to have psychic powers.
We didn’t forecast the GDPR.
We just knew exactly how important data privacy and management is to our customers.
We wanted to do everything we possibly could to safeguard the security of any data we control and manage.
The GDPR didn’t force us to start working on this project four years ago.
But our journey reached a significant milestone on May 25 when the GDPR comes into full effect. I want to share some of the many learnings we have drawn from our project that started all those years ago – so you too can proactively improve your data management practices and secure the growth of your business.
One Simple Exercise That Changed the Engineering Team’s Entire Approach
At the beginning of 2014 I asked the founders of Pipedrive to complete a simple exercise.
I wanted them to explain to us what Pipedrive would look like in three years.
One of the questions asked about the number of customers Pipedrive would be serving by 2018. The founders reached a consensus on a figure of around 70,000 customers worldwide. This turned out to be quite a precise estimate. Maybe the gang did have some psychic powers after all 😉
We’re edging past that 70,000 figure right now.
It is hard to tell if we were good fortune tellers or spelling out the number made us to work harder to achieve that goal. I can tell you the latter is definitely true for the Engineering team.
When you are operating a system with 10,000 customers and you suddenly learn that relatively soon you will have to serve 8-10 times more customers – you get just a little nervous! After the beads of sweat were wiped away, we composed ourselves and prepared to start evolving our system in a totally different way.
Enter Project Megaparsec!
If we wanted to manage a system with 10 times more customers – we couldn’t afford to rely on solving local bottlenecks here and there. An ad-hoc approach could not survive.
We decided we needed the possibility to serve customers from multiple areas within the whole system.
Our project was internally called “Megaparsec”. A sexy name for those in the know. A megaparsec is a million parsecs. Simple, right?
Not so much. You just need to know there are 3.3 light-years to a parsec. That means a megaparsec is quite a long way.
Externally we were referring to the project as “multi-dc architecture”. Slightly easier to understand, I guess! Regardless, we were settling in for a long and productive journey to improve the experience for Pipedrive customers during this period of rapid growth.
But you may be wondering why Project Megaparsec would be so mega.
What is so hard about allowing multiple instances of our system serving different sets of customers?
There are already many companies who use a similar approach.
The complexity came from our second goal:
We needed to scale our engineering organization while keeping really fast release cycles of our code changes to make sure our customers could still rely on fast and regular product improvements.
If we wanted to be able to scale our engineering organization fast and allow new people to be effective and productive in a manner of weeks not years – we needed three critical things to actively support these goals:
- Organization structure
- Development processes
- System architecture
We had to organize our engineering department into small full-stack teams – each working on a dedicated area of our overall responsibilities.
Connection with GDPR
A huge portion of our customers are based in the European Union and many more are also dealing with European customer data. We always wanted to have our second data centre in Europe.
Once the GDPR regulation was adopted by European Parliament we knew how strategically important it would be for us to house the data of our customers in the EU. A multi-dc (multi-data-center) architecture would be the most customer friendly approach.
After 3 years of true dedication from every single engineer on the Pipedrive team – I was so proud to be able to tell our customers that we finally had our second data center up and running in Frankfurt, Germany by the end of the 2017. This was an exciting Christmas present for us data management junkies!
This project had a lot of risks and unknowns, but fortunately we were able to grind through all of them and successfully finish our “Megaparsec” mission.
But this is far from the point where all our team’s effort ends.
The Challenge of Migrating Data to a New European Data Center
Our new data center was able to start serving new customers, but we still had a big number of existing customers served from our US data center in Chicago. While we were focusing on the development of the multiple data centers architecture, we had no time to focus on anything else – but we absolutely had to complete the customer data migration to the new center at some point.
Once our German data center was functioning, our core team took a new challenge of implementing the data migration tool. Coordinating this process across the rest of the engineering organization was quite the challenge.
The complexity came from the our approach to software development and microservices architecture. Customer data scattered across many different data storage types and locations was a real problem.
Our data migration process
We collect all of the data on one side of the ocean. Then the fun starts…
- We start by compressing this data
- Encrypt for the transportation
- Move the data over the ocean
- Decrypt the data
- Uncompress the data
- Then distribute across all of our different microservices in correct order and format
And this process had to be repeatable and executable for thousands of customers in parallel.
We didn’t have the luxury of dedicating years to the process of moving terabytes of customer data. We have to grow and scale fast!
Somehow, this intimidating task of migration tool implementation was finished by our team in a matter of months.
Pipedrive has moved all of our European customers to our German data center in Frankfurt. It’s a huge achievement, but with careful planning and a passionate team – you can make these changes happen without compromising the growth of your business.
European Data Migration is Just One Part of Preparing for GDPR
Even though we achieved the monumental task of implementing the European datacenter and moving our customers data – there was so much more work going on at the same time to make sure Pipedrive was prepared for the specific GDPR requirements.
I asked Martin Ojala, our Data Protection Officer and resident GDPR expert, to explain the most important proactive measures we have taken to make sure our customers can rest assured before and after May 25.
“I feel privileged to work for a company like Pipedrive who takes data management so seriously. After being appointed as our GDPR specialist Data Protection Officer (DPO), we have taken a collection of very important measures to safeguard our customers.”
- We prioritize staff training and governance to safeguard of all assets
- We provide assets to customers to help them roll-out GDPR compliant sales processes
- We have three European offices and a data center in Frankfurt, Germany
- We rigorously monitor our product, it’s features and data centers for compliance
- We only work with privacy shield certified third parties like Google, Rackspace, and Amazon
- We give you control of your data to access, suppress and permanently delete as you wish
Use GDPR as a Prompt to Keep Optimizing Your Customer Experience
Pipedrive has always been passionately committed to developing the best possible experience for our customers.
Proactively managing the safety and security of our customer data is the core foundation of this experience. Data privacy and protection is critically important. Without safely managed data – our user interface, our price, our customer support, our product quality – the rest of our entire customer experience falls apart.
We will continue to optimize our data management practices after May 25. GDPR is intended to encourage customer friendly practices.
We’re proud of our journey to make sure we’re more than prepared for May 25 and we’re already looking to the future to find more ways to safeguard the data we manage for our customers. You too should take this chance to focus on continually optimizing your customer experience as you speed up the growth of your business.