MURAL Notifications Delay

Incident Report for Mural

Postmortem

What happened?

On Friday, June 17 2022 at 22:20 UTC, our email service provider incorrectly identified a security risk with MURAL’s account. As a precaution, our email service provider suspended MURAL’s account. This resulted in MURAL product emails (for example: invitations to join a mural) not sending.

MURAL worked with our email service provider to restore service as quickly as possible. Together with our service provider we implemented a fix at 02:50 UTC on June 18th, which restored full service and allowed the pending emails to be sent. We continued to monitor the results of the fix until 04:00 UTC, at which time the incident was closed.

‌

Summary

MURAL’s platform remained available throughout this incident. There was a delay of 2 hours and 50 minutes sending product emails, but this did not impact service availability.

Posted Aug 23, 2022 - 12:43 GMT-03:00

Resolved

The fix we implemented earlier has been successful in resolving the issue.

We apologize for the inconvenience this interruption caused and will be conducting a full review to avoid a repeat in the future.

Posted Jun 18, 2022 - 00:57 GMT-03:00

Update

We are continuing to monitor the results to make sure the fix has the intended effects.

Posted Jun 18, 2022 - 00:53 GMT-03:00

Monitoring

We've implemented a fix that should resolve the issue. We're monitoring the results to make sure this has the intended effect and will update this message as soon as we've been able to confirm this.

Posted Jun 17, 2022 - 23:51 GMT-03:00

Update

Our team is still working on resolving this issue. We'll update this message as soon as we have more details.

Posted Jun 17, 2022 - 23:01 GMT-03:00

Update

We're continuing to work on resolving this issue.

Posted Jun 17, 2022 - 22:04 GMT-03:00

Update

We have identified the path to return to full service and are working towards this as quickly as possible.

Posted Jun 17, 2022 - 20:48 GMT-03:00

Identified

The issue has been identified and our team is working on fixing it. We'll update this message as soon as we have more details.

Posted Jun 17, 2022 - 19:59 GMT-03:00

Investigating

We're experiencing a service disruption that is causing notifications, including password reset emails, to not be delivered. We're investigating the issue and will restore regular service ASAP.

Please check our status page for the most up-to-date info 👉 status.mural.co/

Posted Jun 17, 2022 - 19:37 GMT-03:00

This incident affected: Mural Application (Notifications).