As you are aware, we experienced three interruptions to Olark service over the last week. Here is a short overview of what happened and why:
On Wednesday, November 16, we received reports that users were being kicked out of their Olark chat platform and could not log back in. This was due to a cascading failure we experienced caused by one third of our chat infrastructure failing at once, which overloaded our remaining infrastructure.
On Thursday, November 17, we began receiving reports that users were being kicked out of the chat.olark.com and could not log back in. Other users were unable to log in to start a new session. In this case the issue was caused by a defective piece of our chat infrastructure. When this defect was triggered, it caused our other related servers to be taken offline. We were able to load balance and correct this issue. We began the process of replacing the older infrastructure with newer and better-supported versions.
On Monday, November 21, users were once again being removed from their chat dashboards and could not login. We identified that the previous defect, which hadn’t yet been fully mitigated, was once again causing our servers to come offline. We deployed our newer infrastructure to production immediately, which stabilized the service.
What's been done to prevent this from happening again?
For the incident caused on November 16, we've increased our capacity by more than 30% in response to the cascading failure.
Yesterday (Monday, November 21) we scheduled and completed maintenance that rolled out a new version of our chat infrastructure and directly addresses the identified defect.
A message from our COO, Matt Pizzimenti:
We realize these downtimes raise a lot of questions about our system stability as we move into a busy shopping season. Rest assured we have corrected the issues that caused these recent outages, and our engineering team's highest priority in the coming weeks is monitoring and maintaining system stability.
Our team appreciates your patience and understanding; we do our best to be transparent and active when it comes to issues that may affect service. As always, you can chat with us on Olark.com, and feel free to ask for me if you need to discuss further!