Interruption on Sept 26, 2014

On Friday, Sept 26, 2014 at 6:32 PM GMT (11:32 AM PST), a network failure caused Pipedrive app to be unavailable. Our infrastructure team was immediately notified. Because the network failure affected our entire physical infrastructure, we had to get on the phone with Rackspace, our hosting provider, who started looking at the issues immediately to restore access to the services for us.

By 6:57 PM GMT (11:57 AM PST), our central physical networking layer had been restored and we started to eradicate the effects of the network failure. However the external storage volume mounts had gone into read-only mode, and thus our databases were not operating correctly.

Update #1: As of 7:21 PM GMT (12:21 PM PST) we are actively working on regaining read/write access to the storage volumes which would allow us to bring the app fully up again. We are still on the direct line with Rackspace engineers and work actively to get the services restored fully.

Update #2: As of 7:36 PM GMT (12:36 PM PST) we have regained network access to our environment and are currently restarting core services as fast as possible.

Update #3: As of 7:50 PM GMT (12:50 PM PST) we have restored functionality across roughly 65% of our database clusters. The remaining DBs are in the process of restarting now. Side services are being started now as well but some of these such as Google sync may continue to be unreliable until they are all up and running again.

Update #4: As of 8:01 PM GMT (1:01 PM PST) we have restored functionality across roughly 90% of our database clusters. The remaining DBs are in the process of restarting now. Side services continue to progress toward restoration and our operations team will begin validating performance across the application soon.

Update #5: As of 8:16 PM GMT (1:16 PM PST) we have restored functionality across all database clusters. Side services are generally available too but some users may continue to experience issues with search and mailbox-beta as the components facing these features are still being restored. Pipedrive operations is now commencing performance validation and monitoring the situation closely.

Update #6: As of 9:24 PM GMT (2:24 PM PST) search functionality has been restored for all users.  Our elasticsearch clusters are now syncing to their replica shards and this will improve search performance throughout the day. Mailbox beta users should see delayed mail delivered within the next few minutes, and mail will return to real-time delivery status shortly thereafter.

Fortunately, for those of you that need to make use of search today we do have a proposed workaround. You can actually use Pipedrive Filters as a form of advanced search, and this is a best practice we often recommend for people trying to search deeper than the search bar allows them to normally.

To take advantage of this all you need to do is create a Filter that you can then edit later each time you want to search for something different. An example screenshot is below:

Using filters for advanced search
Using filters for advanced search

Below is our support center documentation on Filters as well, in case you wanted to brush up on the power of searching cross-item records easily.

http://support.pipedrive.com/customer/portal/articles/1076914

Update #7: As of 11:24 PM GMT (4:24 PM PST) we’ve completed validation of the app and all services are fully operational. Search performance will continue to improve throughout the day as our elasticsearch clusters sync back to their replica shards. I want to assure you that we take matters like these very seriously – today’s events should simply never happen and we will be conducting a full root cause analysis investigation in concert with our datacenter provider to ensure precisely that.

We are profusely apologetic to all our customers for the interruption today and sincerely appreciate the patience you’ve shown us. For the technical crowd out there, initial reports indicate today’s outage was caused by a rare and unexpected spanning tree event during a planned network extension as part of our ongoing infrastructure migration project.

We have temporarily suspended any further network extension efforts today and will cautiously proceed with the scheduled maintenance tomorrow morning, however the recent events have prompted us to extend tomorrow’s maintenance window hours to 2:00 AM – 9:00 AM PDT.

 

Scheduled maintenance on September 27, 2014 (updated)

Scheduled maintenance window:

  • Saturday, 9/27/2014 @ 2:00 am – 11:30 am PDT  (Saturday, 9/27/2014 @ 9:00 am –  6:30 pm GMT)

As part of our ongoing infrastructure migration project we’ll be working with our datacenter partners to perform a scheduled maintenance of various network components comprising the Pipedrive infrastructure on Saturday, September 27th, 2014 between 2:00 AM and 9:00 AM PDT. We expect a short interruption of service for all Pipedrive users during this maintenance window. Please be aware that Pipedrive may be briefly unavailable at times during this window and plan your work accordingly.

We will update this blog post again once the maintenance has been completed successfully.

UPDATE 1: As of Friday 9/26 at 4:35 pm PDT we have extended the maintenance window hours from 6-9 am PDT (originally posted) to 2-9am PDT (new updated time). We are very sorry for the short notice about this update.

UPDATE 2: As of Saturday 9/27 at 9:05 am PDT maintenance is still underway and the window will need to be extended by a further 1-2.5 hours. During the planned maintenance we encountered abnormal performance from one of our loadbalancers and decided to replace it entirely – this added to the work we needed to perform during the window and is the basis for extending the hours again. The Pipedrive application remains generally available throughout and we do not expect significant impact to logged in users as the application is online, fully functional, and operating normally now.

UPDATE 3: As of Saturday 9/27 at 12:05 pm PDT we are conducting a validation of the environment. The application should remain stable throughout and we will close the maintenance window upon completion.

UPDATE 4: As of Saturday 9/27 at 12:40 pm PDT the environment has been validated and the maintenance window has been closed successfully.There will be no further updates to this blog post.

Shh… we have updated our iOS and Android apps

We’re always improving our iOS and Android mobile apps. Here are some of the things that got done over the last months:

iOS updates: Android and iOS phones feature

  • Global search
  • Deal and contact custom fields in read-only mode
  • Contact list filters – you can now filter your contacts based on the same filters as in the web app
  • Delete deals, contacts, activities

Android updates:

  • Google Sign-in
  • Notes for deals and contacts – you can now see the notes added in the web, as well as add new notes directly from the mobile app
  • Custom fields now available, currently read-only
  • Add and edit organizations

Coming soon on iOS:

  • Google Sign-in
  • Custom fields editing for deals and contacts
  • New activity scheduler

Coming soon on Android:

  • Global search

And as the ancient app store saying goes: “if you don’t have it, now’s the perfect time to get it”.

appstore googleplay

Sept 17th degraded search functionality (FIXED)

UPDATE #3: As of 4:05 PM PDT (11:05 PM GMT) all replica shards for the affected elasticsearch cluster have been synced and search performance is back to normal. There will be no further updates to this blog post as this issue has been resolved.

 

UPDATE #2: As of 1:05 PM PDT (8:05 PM GMT) the primary shard index for the affected elasticsearch cluster has been rebuilt successfully. Search is functioning normally again and search performance will continue to improve throughout the day as our replicas are synced back to the primary shard.

 

UPDATE #1: As of 12:39 PM PDT (7:39 PM GMT) we have manually restarted the service and the index is being rebuilt. Search functionality for affected customers will be restored once the index is rebuilt which we estimate within the next 15-20 minutes. Search performance will continue to improve throughout the day as our replicas are synced back to the primary shards.

As of 10:15 AM PDT (5:15 PM GMT) we are aware of degraded search functionality for some select customers. This morning a few of our elasticsearch shards failed leaving one elastic cluster with a suboptimal number of active shards to handle incoming requests. We are very sorry for the impact this may have on your use of Pipedrive today but our operations team is on it and actively investigating a recovery plan.

In the meantime users that were connected to this cluster may continue to experience some degraded search functionality. Closed beta testers in our Gmail Mailbox Beta are also affected as mail may be delayed today. We recommended defaulting back to your normal inbox for the remainder of the day if you are a beta tester of Pipedrive mailbox.

Fortunately, for those of you that need to make use of search today we do have a proposed workaround. You can actually use Pipedrive Filters as a form of advanced search, and this is a best practice we often recommend for people trying to search deeper than the search bar allows them to normally.

To take advantage of this all you need to do is create a Filter that you can then edit later each time you want to search for something different. An example screenshot is below:

Using filters for advanced search
Using filters for advanced search

Below is our support center documentation on Filters as well, in case you wanted to brush up on the power of searching cross-item records easily.

http://support.pipedrive.com/customer/portal/articles/1076914

This blog post will be updated again later today when the index is successfully rebuilt and all functionality is restored.

 

Bulk-editing of activities: one small step for mankind, a giant leap for some of our customers

Pipedrive have now added a feature for bulk-editing activities. This will relieve headaches for many of you, I’m sure.

When I went through our case-log history this morning, I saw that some of you had already asked bulk-editing a couple of years ago. We have a product development principle of only developing features that are needed by more than just a few. The requests about bulk-editing for activities were few and far apart up until recently. As soon as it became apparent how many of you need it, we quickly built the feature.

bulk-editing activities

An example of bulk-editing activities is having a person leave work. With bulk-editing you are now able to reassign all the cases that belong to that person to someone else – all in one go.

We know you guys want bulk-delete for activities as well, which is why we’re happy to say it’s coming soon.

Billing update

We have a slice of news we’d like to share. We are introducing seat quotas to make adding and removing users to your account easier.

In January this year, Pipedrive launched seat-based pricing, giving total control over the seat count to the customer. This was all great and merry, but the separate invoice that was created every time a new user was added to an account meant an awful lot of headache to companies’ accountants. So rather than continue with headache generation, we decided to implement a small change.

screen_shot_2014-09-04_at_11.04.04_am

We have re-introduced seat quotas, however with the difference that the pricing will remain seat-based. The change is simple – you can now buy seats before adding new users. To reduce the number of seats, remove users from the account first and then click on ‘Remove unused seats’. And when you just want to change users in your account, but keep the same amount of seats, all you need to do is remove one user and add the other – doing this won’t incur any changes to your billing.

The billing page will now show the number of active users / available seats. And the quota progress bar is back in the Settings -> Users page. Otherwise things are pretty similar – you can still buy one seat at a time, but you can also easily buy a large number of seats all in one go.

If you have any questions or feedback, let us know in comments or get in touch.

 

 

Suporte em português a partir de setembro / Portuguese-speaking support now available

Good news for Pipedrive users in Brazil and other Portuguese-speaking markets – Pipedrive is offering Portuguese-speaking support from the 1st of September. 

Boas novas para quem fala português no Brasil e no mundo: suporte em português a partir de setembro

O Brasil foi um dos mercados que mais surpreendeu positivamente o Pipedrive – o crescimento foi constante e rápido. Como resposta, agora oferecemos suporte ao idioma português, o segundo idioma de suporte oficial além do inglês.

E para servir aos mercados de língua portuguesa, temos dois caras sensacionais para vocês – conheçam Joel e Jei!

Jay and Joel

Joel é de Portugal e é um excelente e recente reforço no nosso time, e o Jei passou anos morando no Rio e também é a voz de nossos vídeos de suporte em língua portuguesa. Ambos estão muito animados de passar para o português e ajudá-los em sua língua materna. Eles estarão à disposição entre as 3 e 18h, horário do Rio (estamos na Europa) nos dias úteis.

Você pode falar com eles escrevendo para suporte@pipedrive.com ou ligando para +55 41 3907-5403.

Além de oferecer suporte ativo, também traduzimos os artigos e vídeos mais importantes para português. Eles contêm as respostas à maioria das perguntas que possam existir, então dê uma olhada nelas aqui.

Esse é só o primeiro passo para o Pipedrive na busca de servir melhor nossos clientes no Brasil e em outros mercados que falam português. Vocês vão ver nossa base de conhecimento em português continuar crescendo e outras medidas do nosso lado para atender às necessidades dos mercados brasileiro e outros que também falam português!

Para celebrar a ocasião, Nós falamos com o CEO do WTM do Brasil Mr. Lisandro Vieira, longo cliente nosso, em relação á experiência da empresa com o Pipedrive.

Como chegamos a decisar de utilizar o Pipedrive e que diferença isto fez?

Decidimos utilizar o Pipedrive após testarmos vários outros softwares de CRM, decidimos pelo Pipedrive em função da facilidade de uso, possibilidade de personalizar os funis de venda com as fases, período de estagnação, etc. Também levamos em conta o fato de podermos personalizar as atividades e adequá-las a nossa necessidade.

Percebemos no Pipedrive uma ferramenta moderna, dentro dos conceitos atuais de gestão de vendas e relacionamento, com grande capacidade de personalização e integração com outras ferramentas que utilizamos nas demais áreas da empresa.

O que mais gostamos no Pipedrive?

Gostamos muito de sempre termos uma surpresa positiva, da quantidade e da qualidade de atualizações e de novos recursos que estão sendo sempre incorporados a ferramenta. Isto nos faz sentir que não vamos ficar parados no tempo, sentimos que estamos sempre acompanhando o que acontece lá fora. O mundo é dinâmico, nosso CRM também é!

P.S. Um cofundador do Pipedrive estará no Brasil em novembro e ficará feliz de se encontrar com clientes atuais e potenciais. Entre em contato via ragnar@pipedrive.com.

Boa caça aos negócios!

Scheduled maintenance on August 23, 2014

UPDATE 3: As of 12:00 am PDT (7:00 am GMT) the maintenance window has been closed and all work has completed successfully. Search performance may be degraded temporarily as our elastic shard indexes are automatically rebuilt, but the system is otherwise stable and all services are online.

UPDATE 2: As of 11:15 pm PDT (6:15 am GMT) our datacenter partners have corrected the issue that was preventing our validation and services are being restarted accordingly. We estimate a further testing period of approximately 30m as we measure performance across the network before we close the maintenance window.

UPDATE 1: As of 10:35 pm PDT (5:35 am GMT) the scheduled work has been completed by our datacenter partners however it has not yet passed validation with the Pipedrive operations team. We are continuing to test and validate until we are satisfied with the internode connectivity this maintenance was meant to address.

We will update this post with more information as we have it, but currently we must extend the maintenance window to ensure proper validation of the new components.

Scheduled maintenance window:

  • Saturday, 8/23/2014 @ 8:30 pm – 10:30 pm PDT  (Sunday, 8/24/2014 @ 3:30 am –  5:30 am GMT)

To enhance stability we’ll be working with our datacenter partners to perform a scheduled maintenance of various network components comprising the Pipedrive infrastructure on Saturday, Aug 23, 2014 between 8:30 PM and 10:30 PM PDT. The Pipedrive application should be generally available throughout the maintenance window but performance may suffer and some services may be delayed temporarily. Following the completion of work some services may need to be restarted which could disrupt active sessions temporarily for a brief period of a few minutes at a time. We recommend you plan accordingly and apologize for the short notice.

We will update this blog post again once the maintenance has been completed successfully.

Aug 20th degraded search functionality (FIXED)

UPDATE #2: As of 1:39 PM PDT (8:39 PM GMT) mailbox beta has completed syncing and mail should be back to real-time delivery again.

UPDATE: As of 1:12 PM PDT (8:12 PM GMT) the impacted shard’s elastic index has been successfully rebuilt and search should be functioning again. Search speed/performance may be slightly degraded for some users while our replicas are synced but will speed up throughout the day as these tasks are completed automatically. Mailbox beta users should begin seeing mail again though there may be a slight delay as a large batch of messages needs to be processed now.

As of 10:05 AM PDT (5:05 PM GMT) we are aware of degraded search functionality for some select customers. We are very sorry for the impact this may have on your use of Pipedrive today. This morning it appears one of our elasticsearch shards failed and the automatic recovery did not function properly. We are investigating why to mitigate this in the future and we have manually restarted the service – it is running again but rebuilding the index will take several hours today.

In the meantime users that were connected to this shard may continue to experience degraded search functionality. Closed beta testers in our Gmail Mailbox Beta are also affected as mail may be delayed today. We recommended defaulting back to your normal inbox for the remainder of the day if you are a beta tester of Pipedrive mailbox.

Fortunately, for those of you that need to make use of search today we do have a proposed workaround while the index is rebuilding. You can actually use Pipedrive Filters as a form of advanced search, and this is a best practice we often recommend for people trying to search deeper than the search bar allows them to normally.

To take advantage of this all you need to do is create a Filter that you can then edit later each time you want to search for something different. An example screenshot is provided below:

Using filters for advanced search

Using filters for advanced search

Below is our support center documentation on Filters as well, in case you wanted to brush up on the power of searching cross-item records easily.

http://support.pipedrive.com/customer/portal/articles/1076914

This blog post will be updated again later today when the index is successfully rebuilt and all functionality is restored.

 

Scheduled maintenance on Aug 9, 2014

Update 9:15 AM GMT: This maintenance is completed by now. During the maintenance we were able to have an important piece of configuration of a storage mount patched that had previously caused some trouble during moments of higher system load. Thank you for your patience.

Original info, posted yesterday: To increase stability, we’ll be doing a scheduled maintenance to one of the database hosts on Saturday, Aug 9, 2014 between 7 AM and 9 AM GMT time. A portion of our users will be seeing the maintenance notice for up to 30 minutes. We are very sorry for the inconvenience and the short notice. We’ll keep this post updated with details as the maintenance happens.