Incident Alerted

Incident Report for Olark Live Chat

Postmortem

On June 10th we received reports from our customers that agent commands were not working, message delivery was inconsistent and users were having trouble launching chat.

Our team tracked the issue to a recent code change on one of our systems. We have since reverted this change.

This issue revealed a gap in our internal monitoring. We have added an additional metric to detect these sorts of issues quickly should they ever happen again.

This issue also revealed a scheduled deployment process that is no longer needed and that exasperated this downtime. We are evaluating removing this process.

Posted Jun 11, 2021 - 11:15 EDT

Resolved

This incident has been resolved.

Posted Jun 10, 2021 - 19:38 EDT

Monitoring

A fix has been implementing and we're monitoring the results. You should be free to resume chatting as normal.

Posted Jun 10, 2021 - 18:48 EDT

Identified

We believe we've identified the cause of the issues with message delivery and commands and are working to rectify things. We'll update as soon as we have additional information.

Posted Jun 10, 2021 - 17:31 EDT

Update

We're investigating reports of issues with chat message delivery and commands not working properly. We'll update as soon as we know more.

Posted Jun 10, 2021 - 17:15 EDT

Investigating

We've detected an issue and are working to resolve this quickly. We'll have an update within the hour.

Posted Jun 10, 2021 - 17:10 EDT

This incident affected: chat.olark.com, Operator messages, and Visitor messages.