Users are unable to sign in to MURAL
Incident Report for MURAL
Postmortem

What happened?

At 21:51 UTC on September 27 2021, we executed a routine operation in our database which inadvertently caused database writes to fail. This in turn overloaded the database and resulted in access requests also failing. Once the original operation finished at 22:42, the database returned to normal operations and access to the MURAL application was restored.

Details and corrective actions

The specifics of how the operation was performed resulted in a less-known bug on our database server to be triggered. We now understand the steps and will update our processes to avoid repeating this scenario.

Summary

The outage resulted in 50 minutes of downtime. No data from prior to the outage was lost during this time.

What we've done to avoid this happening again

We have updated our processes for running database operations to avoid triggering the now-known bug, and will be performing an upgrade to fully mitigate the issue. This upgrade will be announced on the MURAL status page as routine maintenance with the appropriate notice period.

Posted Oct 04, 2021 - 10:51 GMT-03:00

Resolved
The fix we implemented earlier has been successful in resolving the issue and full service has been resolved.

We apologize for the inconvenience this interruption caused, and will be conducting a full review to avoid a repeat in the future.
Posted Sep 27, 2021 - 21:03 GMT-03:00
Monitoring
We've implemented a fix that should resolve the issue. We're monitoring the results to make sure this has the intended effect and will update this message as soon as we've been able to confirm this.
Posted Sep 27, 2021 - 20:04 GMT-03:00
Identified
The issue has been identified and our team is working on fixing it. We'll update this message as soon as we have more details.
Posted Sep 27, 2021 - 19:08 GMT-03:00
Investigating
Some members are currently unable to sign in to MURAL. We’re investigating the issue and will restore regular service ASAP. Thank you for your patience.

Please check our status page for the most up-to-date info: status.mural.co/
Posted Sep 27, 2021 - 19:02 GMT-03:00
This incident affected: MURAL Application (Authentication).