On June 10th we received reports from our customers that agent commands were not working, message delivery was inconsistent and users were having trouble launching chat.
Our team tracked the issue to a recent code change on one of our systems. We have since reverted this change.
This issue revealed a gap in our internal monitoring. We have added an additional metric to detect these sorts of issues quickly should they ever happen again.
This issue also revealed a scheduled deployment process that is no longer needed and that exasperated this downtime. We are evaluating removing this process.