MURAL Collaboration Issue

Incident Report for Mural

Postmortem

What happened?

At 06:40 UTC we started to receive notifications from Mural users that were unable to access their content in our European datacenter. This was caused by a configuration change that resulted in connections to our EU database being rejected.

Once the cause of the issue was identified, an emergency fix was created, tested and deployed. Normal service was restored at 07:17 UTC. The impact of the emergency fix was monitored for an hour, after which the incident was declared as ‘resolved’.

Summary

Murals in our EU datacenter were unavailable from 06:40 to 07:17 UTC, for a total of 37 minutes of service interruption.

What we've done to avoid this happening again

We are improving our deployment processes and monitoring to include anomalies in the volume of realtime connections.

Posted Mar 08, 2023 - 16:50 GMT-03:00

Resolved

This incident has been resolved.

Posted Nov 30, 2022 - 08:12 GMT-03:00

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Nov 30, 2022 - 07:17 GMT-03:00

Update

We are continuing to work on a fix for this issue.

Posted Nov 30, 2022 - 07:13 GMT-03:00

Identified

The issue has been identified and a fix has been implemented.

Posted Nov 30, 2022 - 07:13 GMT-03:00

Investigating

MURAL is experiencing connection issues, leaving users unable to collaborate, seeing a 'Trying to Reconnect' message. We're investigating and will restore regular service ASAP.

Please check our status page for the most up-to-date info 👉 status.mural.co/

Posted Nov 30, 2022 - 07:05 GMT-03:00

This incident affected: Mural Application (Realtime collaboration).