We know you rely on Olark to help support your customers and we take that responsibility very seriously and we apologize for the outage on November 14th. Whenever we don’t live up to our end of the agreement, it erodes trust and that takes time to rebuild. Over the past year, we have been working hard to improve stability and make our infrastructure more resilient to unforeseen issues.
On November 14th, 2019 we shipped a minor code change to production which causes our software to reconnect to our database provider. Unknown to us, there was a change made on our provider’s side which made our database connections fail, rendering our login and website as unusable. Our engineers immediately began investigating the issue. As soon as we identified the root cause, we immediately began making changes to work around the database connection issue. Login and our website came back up an hour after they went down.
Even though this issue was ultimately caused by our service provider, it’s our responsibility to provide reliable service for you and your customers. We are working with our service provider to understand what happened in more detail and come up with ways to improve communication around changes impacting our services.