Access blocked when logging in to MURAL

Incident Report for Mural

Postmortem

What happened?

At around 16:15 UTC we started to receive notifications from Mural users that were unable to log in to Mural. At 17:01 the issue was escalated to our major incident team. Initial investigations indicated the issue coincided with an update to to the Mural web application earlier in the day. Following the update, a higher volume of web traffic was being generated. This triggered security alerts in our firewall and blocked Mural users from logging in.

At 17:18 we adapted the web traffic thresholds in line with the new expected volume of traffic. This enabled Mural users to log in again and restored full service.

‌

Summary

This incident impacted logins from 16:15 UTC to 17:18 UTC, for a total of 1 hour 3 minutes of service interruption.

‌

What we've done to avoid this happening again

The previous firewall ruleset had a low tolerance for deviations from the expected volume of web traffic. We have updated the firewall ruleset to account for the increased web traffic and ensure there are no further service interruptions.

Posted Dec 07, 2022 - 17:34 GMT-03:00

Resolved

At around 15:00 UTC we started to receive notifications from MURAL users that were unable to access their content. The issue appears to have been caused by one of MURAL's security measures, which identified certain actions as potentially malicious. This resulted in a number of MURAL users having their IP addresses blocked.

We have modified this configuration to ensure users can log in. We are investigating the root cause and will publish a postmortem on our Status Page once we have full details and corrective actions.

Posted Nov 24, 2022 - 17:00 GMT-03:00