Update on “Poodle” SSL 3.0 vulnerability (and Shellshock)

Earlier today, Google reported a vulnerability in SSL 3.0 technology that allows the plaintext of secure connections to be calculated by a network attacker.

I just wanted to inform that we here at Pipedrive had dropped SSL 3.0 support quite some time ago earlier this year— specifically on security grounds. Thus, Google’s recently reported vulnerabilities do not concern Pipedrive. Furthermore, we recently introduced extended validation on all our primary application and website domain premises.

If you are concerned with our security in terms of SSL encryption, you can verify our SSL security reports about our primary application domains app.pipedrive.com and api.pipedrive.com by Qualys SSL Labs at any time.

Update: In addition to SSL 3.0 vulnerability, I wanted to share an update regarding another major security vulnerability reported a few weeks ago, called Shellshock, which allowed malicious code execution through Unix Bash shell, present on majority of servers. Pipedrive’s servers got patched against Shellshock a few weeks ago already — immediately after the patches for the Linux distribution that we use on our production servers got published.

How to Keep Customers Happy By Using the Pipedrive+Yesware Combo

Yesware-PipedrivePipedrive can be used for tons of different reasons. Not least, we’ve seen that Pipedrive can successfully be used for managing customer success and its communications like they do at Falcon Social. Now for us, customer success gets fired up and supercharged when we combine Pipedrive with Yesware – and boy does the combo make customer success communication easy.

My job as the Customer Success Manager here at Pipedrive is to make sure that our clients are happy. I proactively approach customers, introduce the different available Pipedrive features and solutions that could prove to be useful for them, and help them make the most of Pipedrive.

Much of this communication with customers takes place over email – in fact it’s one of my main tools, and I send out hundreds every week. To keep track of what works and what doesn’t (like which emails people open, and which ones they don’t), I use Yesware. And to have an overview of all open deals and details, I use Pipedrive.

This combo gets me super organized, and makes sure I get to keep our customers happy.

How Pipedrive and Yesware work together:

Since I work a lot from Gmail, Pipedrive’s drop-box feature sync information across Yesware and Pipedrive. Essentially, the forwarding feature connects my emails with specific contacts in Pipedrive. I simply forward my emails to Pipedrive by using a drop-box link in the Bcc line. If I have an open deal with the contact, the whole email thread will be visible under deal details.

Yesware Gmail compose window

Having customer details and the related deal history all in one place keeps everyone in my team on the same page, since we can always check the contact’s history before contacting them.

A brief overview of how the Yesware-Gmail-Pipedrive combo works together was given by our CEO in a recent blog post.

My favorite Yesware features are:

  • The opportunity to save template answers and emails, and categorize them appropriately for easy-access sometime later.
  • Tracking the customers’ email openings
  • The visual graph for the last 30-day email opening tracking report.
  • Reminder notifications, which help me to remember when I last sent an email to a customer, and when I should ping them next for a response.

To keep track of where I am with every customer, and help me manage customer success proactively, the second part to my combo is Pipedrive. Even though I don’t use Pipedrive for sales, the pipeline is great for managing customer success, since I can get a really good overview of who requires my urgent attention.

My favorite Pipedrive features are:

  • The pipeline view, which is my main working view gives me a bird’s-eye view of all processes.
  • Deal “rotting” feature, which highlights the deals that need my urgent attention, coloring them red. This indicates the deals I should work on first.

Customer Success Pipeline

One final tip for managing success

One final and really cool way I organize my work is using Zapier trigger actions within different pipelines in Pipedrive. For example, whenever someone from the sales team closes a deal (= wins a deal with specific criteria), a new deal gets automatically added to my Customer Success pipeline into the “Ready for approach” stage.

This nifty automation option saves me a lot of time, and helps me stay on top of everything.

That’s it – that’s how I manage customer success communications. If you like it, why not try it out with a 60-day trial using the promo code CS60. 

Andy McLoughlin joins our board

As you may have noticed, we’re growing fast both in terms of number of customers and employees. This means we have to get better at everything all the time, and this includes setting the longer term course for our company and business.

Andy McLoughlin - Hy! Summit - March 20, 2014 - Image by Dan Taylor-2I am therefore extremely pleased that one of our earliest investors, Andy McLoughlin, has joined the board of Pipedrive, Inc. Andy is co-founder of Huddle, one of Europe’s most prominent and awarded SaaS startups with offices in London, San Francisco, New York and Washington DC. Andy is also a highly active angel investor, working with great SaaS companies like Pipedrive, Rolepoint, Apiary, Buffer, Intercom, Bugsnag, and Import.io, as well as exciting B2C startups like Postmates, Thread, Hullabalu and Secret Escapes. Andy lives in San Francisco, is an enthusiastic (rather than skillful) skier, and an avid consumer of fine food and adult beverages.

In his own words:

“I first met the Pipedrive founders in 2011 and was immediately impressed by their hustle, vision and early metrics. Having participated in their first two seed rounds, I’ve been lucky enough to stay close to the team and watch the company mature into the great business that it is today. Joining their board and being a part of their continuing journey is a huge honor and I’m excited for their next phase of growth.”

Andy has been very helpful to us as a seed investor, and I’m really excited to be working with him on this slightly more formalized level as well.

We owe you an apology

I wanted to take a moment to address the unfortunate downtime last Friday that impacted the majority of our Americas-based customers who couldn’t use our application in the middle of a workday.

First off, we’re very sorry for disrupting your workday. I want to assure you that we take this very seriously and find it absolutely unacceptable. As a team we feel bad for letting you down.

Secondly, we’ve learned some important lessons and have already made some changes to improve issue detection, technical processes and communication. While this doesn’t reduce the frustration you must have felt last Friday, we’ll be both quicker and more efficient in responding to any issues in the future.

What happened on Friday

For background: for some months now we’ve been working on significantly improving the infrastructure underlying our application. This project is ongoing and is meant to dramatically improve stability and performance of our app – along with giving us enough headroom to support our rapid growth.

Some configuration elements (namely, switch ports) with our new infrastructure setup failed on Friday and a rare technical event (a spanning tree failure event) took down our network.

Our operations team was immediately aware as we monitor the performance of Pipedrive 24×7 through many automated tests and alarms. We urgently coordinated a response with our network hosting providers. While the effort was immediate parts of our application take time to recover from a complete shutdown so some customers faced unreliable functionality for up to two hours.

Steps taken to avoid such situations in the future

Together with our hosting providers we have taken extra precautions in migrating to our new infrastructure setup. In the last couple of days we’ve also discussed and agreed upon several new internal workflows that will help to identify and fix any issues faster.

As I mentioned, there already were quite a few things in our roadmap that will reduce our exposure to issues like this. For example, reducing the size of databases and using multiple hosting locations. We’ll continue executing along these plans.

I hope this explains the reasons behind the downtime on Friday. I hope it also sheds light on our commitment to avoid outages and keep improving the speed and reliability of our app. Last but not least, I hope you’ll accept our apologies.

I expect to be writing more upbeat blog posts in the future.

PS. If you have questions or if you’d like to know more about our infrastructure improvements please contact us via our support email.

Interruption on Sept 26, 2014

On Friday, Sept 26, 2014 at 6:32 PM GMT (11:32 AM PST), a network failure caused Pipedrive app to be unavailable. Our infrastructure team was immediately notified. Because the network failure affected our entire physical infrastructure, we had to get on the phone with Rackspace, our hosting provider, who started looking at the issues immediately to restore access to the services for us.

By 6:57 PM GMT (11:57 AM PST), our central physical networking layer had been restored and we started to eradicate the effects of the network failure. However the external storage volume mounts had gone into read-only mode, and thus our databases were not operating correctly.

Update #1: As of 7:21 PM GMT (12:21 PM PST) we are actively working on regaining read/write access to the storage volumes which would allow us to bring the app fully up again. We are still on the direct line with Rackspace engineers and work actively to get the services restored fully.

Update #2: As of 7:36 PM GMT (12:36 PM PST) we have regained network access to our environment and are currently restarting core services as fast as possible.

Update #3: As of 7:50 PM GMT (12:50 PM PST) we have restored functionality across roughly 65% of our database clusters. The remaining DBs are in the process of restarting now. Side services are being started now as well but some of these such as Google sync may continue to be unreliable until they are all up and running again.

Update #4: As of 8:01 PM GMT (1:01 PM PST) we have restored functionality across roughly 90% of our database clusters. The remaining DBs are in the process of restarting now. Side services continue to progress toward restoration and our operations team will begin validating performance across the application soon.

Update #5: As of 8:16 PM GMT (1:16 PM PST) we have restored functionality across all database clusters. Side services are generally available too but some users may continue to experience issues with search and mailbox-beta as the components facing these features are still being restored. Pipedrive operations is now commencing performance validation and monitoring the situation closely.

Update #6: As of 9:24 PM GMT (2:24 PM PST) search functionality has been restored for all users.  Our elasticsearch clusters are now syncing to their replica shards and this will improve search performance throughout the day. Mailbox beta users should see delayed mail delivered within the next few minutes, and mail will return to real-time delivery status shortly thereafter.

Fortunately, for those of you that need to make use of search today we do have a proposed workaround. You can actually use Pipedrive Filters as a form of advanced search, and this is a best practice we often recommend for people trying to search deeper than the search bar allows them to normally.

To take advantage of this all you need to do is create a Filter that you can then edit later each time you want to search for something different. An example screenshot is below:

Using filters for advanced search
Using filters for advanced search

Below is our support center documentation on Filters as well, in case you wanted to brush up on the power of searching cross-item records easily.

http://support.pipedrive.com/customer/portal/articles/1076914

Update #7: As of 11:24 PM GMT (4:24 PM PST) we’ve completed validation of the app and all services are fully operational. Search performance will continue to improve throughout the day as our elasticsearch clusters sync back to their replica shards. I want to assure you that we take matters like these very seriously – today’s events should simply never happen and we will be conducting a full root cause analysis investigation in concert with our datacenter provider to ensure precisely that.

We are profusely apologetic to all our customers for the interruption today and sincerely appreciate the patience you’ve shown us. For the technical crowd out there, initial reports indicate today’s outage was caused by a rare and unexpected spanning tree event during a planned network extension as part of our ongoing infrastructure migration project.

We have temporarily suspended any further network extension efforts today and will cautiously proceed with the scheduled maintenance tomorrow morning, however the recent events have prompted us to extend tomorrow’s maintenance window hours to 2:00 AM – 9:00 AM PDT.

 

Scheduled maintenance on September 27, 2014 (updated)

Scheduled maintenance window:

  • Saturday, 9/27/2014 @ 2:00 am – 11:30 am PDT  (Saturday, 9/27/2014 @ 9:00 am –  6:30 pm GMT)

As part of our ongoing infrastructure migration project we’ll be working with our datacenter partners to perform a scheduled maintenance of various network components comprising the Pipedrive infrastructure on Saturday, September 27th, 2014 between 2:00 AM and 9:00 AM PDT. We expect a short interruption of service for all Pipedrive users during this maintenance window. Please be aware that Pipedrive may be briefly unavailable at times during this window and plan your work accordingly.

We will update this blog post again once the maintenance has been completed successfully.

UPDATE 1: As of Friday 9/26 at 4:35 pm PDT we have extended the maintenance window hours from 6-9 am PDT (originally posted) to 2-9am PDT (new updated time). We are very sorry for the short notice about this update.

UPDATE 2: As of Saturday 9/27 at 9:05 am PDT maintenance is still underway and the window will need to be extended by a further 1-2.5 hours. During the planned maintenance we encountered abnormal performance from one of our loadbalancers and decided to replace it entirely – this added to the work we needed to perform during the window and is the basis for extending the hours again. The Pipedrive application remains generally available throughout and we do not expect significant impact to logged in users as the application is online, fully functional, and operating normally now.

UPDATE 3: As of Saturday 9/27 at 12:05 pm PDT we are conducting a validation of the environment. The application should remain stable throughout and we will close the maintenance window upon completion.

UPDATE 4: As of Saturday 9/27 at 12:40 pm PDT the environment has been validated and the maintenance window has been closed successfully.There will be no further updates to this blog post.

Shh… we have updated our iOS and Android apps

We’re always improving our iOS and Android mobile apps. Here are some of the things that got done over the last months:

iOS updates: Android and iOS phones feature

  • Global search
  • Deal and contact custom fields in read-only mode
  • Contact list filters – you can now filter your contacts based on the same filters as in the web app
  • Delete deals, contacts, activities

Android updates:

  • Google Sign-in
  • Notes for deals and contacts – you can now see the notes added in the web, as well as add new notes directly from the mobile app
  • Custom fields now available, currently read-only
  • Add and edit organizations

Coming soon on iOS:

  • Google Sign-in
  • Custom fields editing for deals and contacts
  • New activity scheduler

Coming soon on Android:

  • Global search

And as the ancient app store saying goes: “if you don’t have it, now’s the perfect time to get it”.

appstore googleplay

Sept 17th degraded search functionality (FIXED)

UPDATE #3: As of 4:05 PM PDT (11:05 PM GMT) all replica shards for the affected elasticsearch cluster have been synced and search performance is back to normal. There will be no further updates to this blog post as this issue has been resolved.

 

UPDATE #2: As of 1:05 PM PDT (8:05 PM GMT) the primary shard index for the affected elasticsearch cluster has been rebuilt successfully. Search is functioning normally again and search performance will continue to improve throughout the day as our replicas are synced back to the primary shard.

 

UPDATE #1: As of 12:39 PM PDT (7:39 PM GMT) we have manually restarted the service and the index is being rebuilt. Search functionality for affected customers will be restored once the index is rebuilt which we estimate within the next 15-20 minutes. Search performance will continue to improve throughout the day as our replicas are synced back to the primary shards.

As of 10:15 AM PDT (5:15 PM GMT) we are aware of degraded search functionality for some select customers. This morning a few of our elasticsearch shards failed leaving one elastic cluster with a suboptimal number of active shards to handle incoming requests. We are very sorry for the impact this may have on your use of Pipedrive today but our operations team is on it and actively investigating a recovery plan.

In the meantime users that were connected to this cluster may continue to experience some degraded search functionality. Closed beta testers in our Gmail Mailbox Beta are also affected as mail may be delayed today. We recommended defaulting back to your normal inbox for the remainder of the day if you are a beta tester of Pipedrive mailbox.

Fortunately, for those of you that need to make use of search today we do have a proposed workaround. You can actually use Pipedrive Filters as a form of advanced search, and this is a best practice we often recommend for people trying to search deeper than the search bar allows them to normally.

To take advantage of this all you need to do is create a Filter that you can then edit later each time you want to search for something different. An example screenshot is below:

Using filters for advanced search
Using filters for advanced search

Below is our support center documentation on Filters as well, in case you wanted to brush up on the power of searching cross-item records easily.

http://support.pipedrive.com/customer/portal/articles/1076914

This blog post will be updated again later today when the index is successfully rebuilt and all functionality is restored.

 

Bulk-editing of activities: one small step for mankind, a giant leap for some of our customers

Pipedrive have now added a feature for bulk-editing activities. This will relieve headaches for many of you, I’m sure.

When I went through our case-log history this morning, I saw that some of you had already asked bulk-editing a couple of years ago. We have a product development principle of only developing features that are needed by more than just a few. The requests about bulk-editing for activities were few and far apart up until recently. As soon as it became apparent how many of you need it, we quickly built the feature.

bulk-editing activities

An example of bulk-editing activities is having a person leave work. With bulk-editing you are now able to reassign all the cases that belong to that person to someone else – all in one go.

We know you guys want bulk-delete for activities as well, which is why we’re happy to say it’s coming soon.

Billing update

We have a slice of news we’d like to share. We are introducing seat quotas to make adding and removing users to your account easier.

In January this year, Pipedrive launched seat-based pricing, giving total control over the seat count to the customer. This was all great and merry, but the separate invoice that was created every time a new user was added to an account meant an awful lot of headache to companies’ accountants. So rather than continue with headache generation, we decided to implement a small change.

screen_shot_2014-09-04_at_11.04.04_am

We have re-introduced seat quotas, however with the difference that the pricing will remain seat-based. The change is simple – you can now buy seats before adding new users. To reduce the number of seats, remove users from the account first and then click on ‘Remove unused seats’. And when you just want to change users in your account, but keep the same amount of seats, all you need to do is remove one user and add the other – doing this won’t incur any changes to your billing.

The billing page will now show the number of active users / available seats. And the quota progress bar is back in the Settings -> Users page. Otherwise things are pretty similar – you can still buy one seat at a time, but you can also easily buy a large number of seats all in one go.

If you have any questions or feedback, let us know in comments or get in touch.