At around 16:15 UTC we started to receive notifications from Mural users that were unable to log in to Mural. At 17:01 the issue was escalated to our major incident team. Initial investigations indicated the issue coincided with an update to to the Mural web application earlier in the day. Following the update, a higher volume of web traffic was being generated. This triggered security alerts in our firewall and blocked Mural users from logging in.
At 17:18 we adapted the web traffic thresholds in line with the new expected volume of traffic. This enabled Mural users to log in again and restored full service.
This incident impacted logins from 16:15 UTC to 17:18 UTC, for a total of 1 hour 3 minutes of service interruption.
What we've done to avoid this happening again
The previous firewall ruleset had a low tolerance for deviations from the expected volume of web traffic. We have updated the firewall ruleset to account for the increased web traffic and ensure there are no further service interruptions.