We owe you an apology

I wanted to take a moment to address the unfortunate downtime last Friday that impacted the majority of our Americas-based customers who couldn’t use our application in the middle of a workday.

First off, we’re very sorry for disrupting your workday. I want to assure you that we take this very seriously and find it absolutely unacceptable. As a team we feel bad for letting you down.

Secondly, we’ve learned some important lessons and have already made some changes to improve issue detection, technical processes and communication. While this doesn’t reduce the frustration you must have felt last Friday, we’ll be both quicker and more efficient in responding to any issues in the future.

What happened on Friday

For background: for some months now we’ve been working on significantly improving the infrastructure underlying our application. This project is ongoing and is meant to dramatically improve stability and performance of our app – along with giving us enough headroom to support our rapid growth.

Some configuration elements (namely, switch ports) with our new infrastructure setup failed on Friday and a rare technical event (a spanning tree failure event) took down our network.

Our operations team was immediately aware as we monitor the performance of Pipedrive 24×7 through many automated tests and alarms. We urgently coordinated a response with our network hosting providers. While the effort was immediate parts of our application take time to recover from a complete shutdown so some customers faced unreliable functionality for up to two hours.

Steps taken to avoid such situations in the future

Together with our hosting providers we have taken extra precautions in migrating to our new infrastructure setup. In the last couple of days we’ve also discussed and agreed upon several new internal workflows that will help to identify and fix any issues faster.

As I mentioned, there already were quite a few things in our roadmap that will reduce our exposure to issues like this. For example, reducing the size of databases and using multiple hosting locations. We’ll continue executing along these plans.

I hope this explains the reasons behind the downtime on Friday. I hope it also sheds light on our commitment to avoid outages and keep improving the speed and reliability of our app. Last but not least, I hope you’ll accept our apologies.

I expect to be writing more upbeat blog posts in the future.

PS. If you have questions or if you’d like to know more about our infrastructure improvements please contact us via our support email.

Looking for a Front-End Developer

_MG_6135

Q: Why did the developer go broke? A: Because he used up all his cache.

We at Pipedrive are building the greatest ever sales management software. We also like cheesy puns and dogs. If you like those things too, and if you want to join a company growing over 100% a year with paying clients in over 155 countries, then we’re probably a really good match. So join our ranks…

Front-End Developer 

We’re seeking a Front-End Developer to join our ongoing quest to create the most epic, innovative sales management solution around.

We’re looking for smart, passionate, and capable software engineers who want to make an impact in the real world using technology to do so. As a software engineer, you work on a small team and can switch teams and projects as our fast-paced business grows and evolves. We need our engineers to be versatile and passionate to tackle new problems as we continue to push technology forward.

As a key member of a small and versatile team, you design, test, deploy and maintain software solutions.

Responsibilities

  • Develop next-generation web applications with a focus on the client side.
  • Redesign existing UI’s and implement new UI’s.
  • Design and build scalable framework for web applications.

The following experience is relevant to us:

  • Proven software development track record
  • Personal projects that show an aptitude for engineering and product sense
  • Ability to work in areas outside of their usual comfort zone and get things done quickly
  • Understanding of security threats
  • Experience with following technologies: HTML5, JavaScript, CSS3, PHP, MySQL, Node.JS

What we have to offer

  • Competitive base pay
  • No corporate BS
  • One of the coolest offices in Estonia
  • A chance to be a part of the next success story out of Estonia

If this is something for you, please get in touch via jobs-admin@pipedrive.com. Send your resume (in English) or link to your LinkedIn profile and please add a few lines on why we should pay extra attention to your application.

 

Pun courtesy to Elijah Manor.

Looking for a Back-End Developer

Pipedrive OfficeWe at Pipedrive are building the best sales software. We’re growing at a crazy speed, which is why we need to add some nitro to our engine (or something like that – we know more about code than we do about engines and petrol). If you’re a back-end whiz with skills that leave people in awe, then you’re the person we’re looking for the role of…

Back-End Developer

We’re seeking a Back-End Developer to join our ongoing quest to create the most epic, innovative sales management mobile solution around.

We’re looking for smart, passionate, and capable software engineers who want to make an impact in the real world using bleeding edge technology. In this role you will be a central point of integration for Pipedrive different system components and will be working with experienced engineers, devops and product managers in a challenging, stimulating startup environment.

You are expected to work independently and deliver high quality code in a timely manner across a variety of product engagements. You are also responsible for code reviews and standards and best practices enforcement. You pride yourself in being an early adopter of powerful and fun tech; and don’t fall into the trap of supporting legacy software. You look for ways to improve your stack and constantly pay off technical debt.

The following experience is relevant to us:

  • Proven software development track record
  • Personal projects that show an aptitude for engineering and product sense
  • Ability to work in areas outside of their usual comfort zone and get things done quickly
  • Interest and contributions to OpenSource community
  • Understanding of security threats
  • Experience with following technologies: Node.js, PHP, JavaScript, MySQL, ElasticSearch, Redis, OpenStack

What we have to offer

  • Competitive base pay
  • No corporate BS
  • One of the coolest offices in Estonia
  • A chance to be a part of the next success story out of Estonia

If this is something for you, please get in touch via jobs-admin@pipedrive.com. Send your resume (in English) or link to your LinkedIn profile and please add a few lines on why we should pay extra attention to your application.

We’re hiring a (top) Quality Engineer

quality stamp

The people at Pipedrive are all about building the greatest sales software ever. We believe in this stamp on the right, which says 100% quality. Are you a person who loves spotting errors? Do you think that quality should be built into systems, rather than checked after completion? If so, then we want you to become our…

Quality Engineer with automation experience

Join our ongoing quest to create the most epic, innovative sales management solution around.

In this role, you’ll be the guardian of our outstanding customer experience. You will able to dive into many aspects of sales management, define a strategy for testing new features and functionality, and be responsible for quantifying and improving data quality across our entire product. Your job is to keep our ridiculously high-quality bar set ridiculously high, and to help us identify and execute ways to raise it even higher.

You are our ideal candidate if you want to work on challenging problems with smart people. You have a keen attention to detail and are exceptionally good at identifying inconsistencies and flaws. You can juggle several balls and not lose track of the little things. You’re looking for a role with substantial opportunities to learn, grow, and take on more (the good news is, we have that stuff in spades!).

You also meet most (if not all) of the following requirements:

  • A passion for technology and a knack for details.
  • Ability to create test plans, prepare testing strategies, and get a kick out of uncovering those hard-to-find bugs.
  • Experience with Java or another object-oriented programming language.
  • Experience with test automation tools: Selenium and others.
  • Strong communication and collaboration skills.
  • Excitement and passion for start-ups. We’re cool; we hope you are too.
  • Experience with SQL and database is a plus.
  • Programming experience in a Linux/Unix environment. We mostly use PHP, JavaScript and Node.JS.

What we have to offer

  • Competitive base pay
  • No corporate BS
  • One of the coolest offices in Estonia
  • A chance to be a part of the next success story out of Estonia

If this is something for you, please get in touch via jobs-admin@pipedrive.com. Send your resume (in English) or link to your LinkedIn profile and please add a few lines on why we should pay extra attention to your application.

Would the best Infrastructure Engineer please stand up?

Pipedrive's team (+ the President of Estonia)

Pipedrive’s team (+ the President of Estonia)

We’re a startup building the fastest, easiest and most useful sales management software ever. To do it quicker and better, we need the crème de la crème of talent, which is why we want you – the best Infrastructure Engineer – to join our team. If you’re the one we’re looking for, then there are already over 50 smiling faces here to greet you with the number growing fast.

Infrastructure Engineer

We’re seeking for an Infrastructure Engineer to join our ongoing quest to create the most epic, innovative sales management solution around.

In this role, you will be responsible for evaluation, implementation, integration, maintenance, upgrade, documentation, and troubleshooting of all remote systems and services. You will work closely with our high-paced agile development as well as customer support teams. You also will be expected to maintain your knowledge and experience level with the ever evolving IT landscape.

You are our ideal candidate if you want to work on challenging problems with smart people. You have a keen attention to detail and are exceptionally good at identifying inconsistencies and flaws. You can juggle several balls and not lose track of the little things. You’re looking for a role with substantial opportunities to learn, grow, and take on more (the good news is, we have that stuff in spades!).

You also meet most (if not all) of the following requirements:

  • Solid technical background in a hosted services environment including infrastructure networks, hardware, software and telecommunications.
  • Good knowledge of security as it relates to cloud based infrastructure.
  • Interest or passion for security and/or identity services is a plus.
  • Experience with DevOps and automation.
  • Experience with automated monitoring tools.
  • Experience with OpenStack is a plus.
  • Ability to maintain good working relationships with third party service providers.
  • Strong communication and collaboration skills.
  • Fluent in spoken and written English.
  • Excitement and passion for start-ups. We’re cool; we hope you are too.

What we have to offer

  • Competitive base pay
  • No corporate BS
  • One of the coolest offices in Estonia
  • A chance to be a part of the next success story out of Estonia

If this is something for you, please get in touch via jobs-admin@pipedrive.com. Send your resume (in English) or link to your LinkedIn profile and please add a few lines on why we should pay extra attention to your application.

Interruption on Sept 26, 2014

On Friday, Sept 26, 2014 at 6:32 PM GMT (11:32 AM PST), a network failure caused Pipedrive app to be unavailable. Our infrastructure team was immediately notified. Because the network failure affected our entire physical infrastructure, we had to get on the phone with Rackspace, our hosting provider, who started looking at the issues immediately to restore access to the services for us.

By 6:57 PM GMT (11:57 AM PST), our central physical networking layer had been restored and we started to eradicate the effects of the network failure. However the external storage volume mounts had gone into read-only mode, and thus our databases were not operating correctly.

Update #1: As of 7:21 PM GMT (12:21 PM PST) we are actively working on regaining read/write access to the storage volumes which would allow us to bring the app fully up again. We are still on the direct line with Rackspace engineers and work actively to get the services restored fully.

Update #2: As of 7:36 PM GMT (12:36 PM PST) we have regained network access to our environment and are currently restarting core services as fast as possible.

Update #3: As of 7:50 PM GMT (12:50 PM PST) we have restored functionality across roughly 65% of our database clusters. The remaining DBs are in the process of restarting now. Side services are being started now as well but some of these such as Google sync may continue to be unreliable until they are all up and running again.

Update #4: As of 8:01 PM GMT (1:01 PM PST) we have restored functionality across roughly 90% of our database clusters. The remaining DBs are in the process of restarting now. Side services continue to progress toward restoration and our operations team will begin validating performance across the application soon.

Update #5: As of 8:16 PM GMT (1:16 PM PST) we have restored functionality across all database clusters. Side services are generally available too but some users may continue to experience issues with search and mailbox-beta as the components facing these features are still being restored. Pipedrive operations is now commencing performance validation and monitoring the situation closely.

Update #6: As of 9:24 PM GMT (2:24 PM PST) search functionality has been restored for all users.  Our elasticsearch clusters are now syncing to their replica shards and this will improve search performance throughout the day. Mailbox beta users should see delayed mail delivered within the next few minutes, and mail will return to real-time delivery status shortly thereafter.

Fortunately, for those of you that need to make use of search today we do have a proposed workaround. You can actually use Pipedrive Filters as a form of advanced search, and this is a best practice we often recommend for people trying to search deeper than the search bar allows them to normally.

To take advantage of this all you need to do is create a Filter that you can then edit later each time you want to search for something different. An example screenshot is below:

Using filters for advanced search
Using filters for advanced search

Below is our support center documentation on Filters as well, in case you wanted to brush up on the power of searching cross-item records easily.

http://support.pipedrive.com/customer/portal/articles/1076914

Update #7: As of 11:24 PM GMT (4:24 PM PST) we’ve completed validation of the app and all services are fully operational. Search performance will continue to improve throughout the day as our elasticsearch clusters sync back to their replica shards. I want to assure you that we take matters like these very seriously – today’s events should simply never happen and we will be conducting a full root cause analysis investigation in concert with our datacenter provider to ensure precisely that.

We are profusely apologetic to all our customers for the interruption today and sincerely appreciate the patience you’ve shown us. For the technical crowd out there, initial reports indicate today’s outage was caused by a rare and unexpected spanning tree event during a planned network extension as part of our ongoing infrastructure migration project.

We have temporarily suspended any further network extension efforts today and will cautiously proceed with the scheduled maintenance tomorrow morning, however the recent events have prompted us to extend tomorrow’s maintenance window hours to 2:00 AM – 9:00 AM PDT.

 

Scheduled maintenance on September 27, 2014 (updated)

Scheduled maintenance window:

  • Saturday, 9/27/2014 @ 2:00 am – 11:30 am PDT  (Saturday, 9/27/2014 @ 9:00 am –  6:30 pm GMT)

As part of our ongoing infrastructure migration project we’ll be working with our datacenter partners to perform a scheduled maintenance of various network components comprising the Pipedrive infrastructure on Saturday, September 27th, 2014 between 2:00 AM and 9:00 AM PDT. We expect a short interruption of service for all Pipedrive users during this maintenance window. Please be aware that Pipedrive may be briefly unavailable at times during this window and plan your work accordingly.

We will update this blog post again once the maintenance has been completed successfully.

UPDATE 1: As of Friday 9/26 at 4:35 pm PDT we have extended the maintenance window hours from 6-9 am PDT (originally posted) to 2-9am PDT (new updated time). We are very sorry for the short notice about this update.

UPDATE 2: As of Saturday 9/27 at 9:05 am PDT maintenance is still underway and the window will need to be extended by a further 1-2.5 hours. During the planned maintenance we encountered abnormal performance from one of our loadbalancers and decided to replace it entirely – this added to the work we needed to perform during the window and is the basis for extending the hours again. The Pipedrive application remains generally available throughout and we do not expect significant impact to logged in users as the application is online, fully functional, and operating normally now.

UPDATE 3: As of Saturday 9/27 at 12:05 pm PDT we are conducting a validation of the environment. The application should remain stable throughout and we will close the maintenance window upon completion.

UPDATE 4: As of Saturday 9/27 at 12:40 pm PDT the environment has been validated and the maintenance window has been closed successfully.There will be no further updates to this blog post.

Shh… we have updated our iOS and Android apps

We’re always improving our iOS and Android mobile apps. Here are some of the things that got done over the last months:

iOS updates: Android and iOS phones feature

  • Global search
  • Deal and contact custom fields in read-only mode
  • Contact list filters – you can now filter your contacts based on the same filters as in the web app
  • Delete deals, contacts, activities

Android updates:

  • Google Sign-in
  • Notes for deals and contacts – you can now see the notes added in the web, as well as add new notes directly from the mobile app
  • Custom fields now available, currently read-only
  • Add and edit organizations

Coming soon on iOS:

  • Google Sign-in
  • Custom fields editing for deals and contacts
  • New activity scheduler

Coming soon on Android:

  • Global search

And as the ancient app store saying goes: “if you don’t have it, now’s the perfect time to get it”.

appstore googleplay

Sept 17th degraded search functionality (FIXED)

UPDATE #3: As of 4:05 PM PDT (11:05 PM GMT) all replica shards for the affected elasticsearch cluster have been synced and search performance is back to normal. There will be no further updates to this blog post as this issue has been resolved.

 

UPDATE #2: As of 1:05 PM PDT (8:05 PM GMT) the primary shard index for the affected elasticsearch cluster has been rebuilt successfully. Search is functioning normally again and search performance will continue to improve throughout the day as our replicas are synced back to the primary shard.

 

UPDATE #1: As of 12:39 PM PDT (7:39 PM GMT) we have manually restarted the service and the index is being rebuilt. Search functionality for affected customers will be restored once the index is rebuilt which we estimate within the next 15-20 minutes. Search performance will continue to improve throughout the day as our replicas are synced back to the primary shards.

As of 10:15 AM PDT (5:15 PM GMT) we are aware of degraded search functionality for some select customers. This morning a few of our elasticsearch shards failed leaving one elastic cluster with a suboptimal number of active shards to handle incoming requests. We are very sorry for the impact this may have on your use of Pipedrive today but our operations team is on it and actively investigating a recovery plan.

In the meantime users that were connected to this cluster may continue to experience some degraded search functionality. Closed beta testers in our Gmail Mailbox Beta are also affected as mail may be delayed today. We recommended defaulting back to your normal inbox for the remainder of the day if you are a beta tester of Pipedrive mailbox.

Fortunately, for those of you that need to make use of search today we do have a proposed workaround. You can actually use Pipedrive Filters as a form of advanced search, and this is a best practice we often recommend for people trying to search deeper than the search bar allows them to normally.

To take advantage of this all you need to do is create a Filter that you can then edit later each time you want to search for something different. An example screenshot is below:

Using filters for advanced search
Using filters for advanced search

Below is our support center documentation on Filters as well, in case you wanted to brush up on the power of searching cross-item records easily.

http://support.pipedrive.com/customer/portal/articles/1076914

This blog post will be updated again later today when the index is successfully rebuilt and all functionality is restored.

 

We’re hiring: Digital Marketing Manager, come join our team

Digital marketing toolkit2

We at Pipedrive build great sales software. So much so that our most important marketing channel by far is our customers telling their co-workers and friends about us. This has helped us grow to more than 50,000 paying users around the world and we’re growing fast.

Good old digital marketing has also worked well for us, and we’re looking for a superstar Digital Marketing Manager to take our performance marketing to the next level.

Here’s the to do list:

  • Testing and scaling paid media channels, including (but not limited to) PPC, retargeting, social ads, paid listings and referral programs. You’ll have a lot of freedom to execute your own ideas
  • Implementing campaign tracking prior to every campaign launch and tracking relevant KPI’s.
  • Constant data analysis to identify areas for improvement and improving our analytics stack, where necessary.
  • Working with our content marketers to optimize distribution of our content and helping to measure ROI.
  • Communicate you progress to the rest of the team via weekly and monthly digital marketing reports. Rest assured, not too much paperwork.

Experience and skills we’re looking for:

  • A proven track record in running successful online acquisition campaigns for a fast-growing business or within an agency. Previous B2B and SaaS marketing experience is a big plus.
  • Hands-on experience in Paid Search, Facebook Power Editor and other major digital marketing platforms. We expect you to hit the ground running.
  • Technically proficient with configuring analytics tools and building reports.
  • Analytical mindset. You like to test the options and let the numbers guide your decisions.
  • Results-driven. Enough said.
  • Good at building constructive relationships with colleagues, partners, media owners and agencies.

What we have to offer:

  • Competitive base pay + stock options package.
  • No corporate BS.
  • Loads of room for career development as you grow with the company.

The role is ideally based in our Menlo Park, CA office but you’re also welcome to work out of our Tallinn, Estonia office.

If this is something for you, please get in touch via jobs-admin@pipedrive.com. Send your resume (in English) or link to your LinkedIn profile and please add a few lines on why we should pay extra attention to your application.