Users experience icons disappearing on the mural and are no longer available in the icon library when searched
Incident Report for MURAL
Postmortem

What happened?

At 19:00 UTC on November 18 2021 we started to receive a small number of notifications of some Noun Project icons disappearing from murals and not being visible in the search bar. During our initial investigations we were unable to reproduce the issue. We continued to receive reports and at 09:40 UTC on November 19 2021 we classified this as a critical incident. The relevant code for displaying icons was retained and no data was lost, however the icons were not being displayed.

Details and corrective actions

At 16:26 UTC we updated the configuration for the Noun Project integration, which restored full service with no data loss.

Summary

This incident resulted in a partial outage for a core service, lasting 7 hours and 44 minutes. MURAL remained operational throughout this incident and members were able to log in and access their data; however, some Noun Project icons were not visible in the search menu or on murals.

What we've done to avoid this happening again

We have resolved the configuration issue and restored full service with the use of icons. We are working on several short- and medium-term initiatives to improve the reliability of this service going forward.

Posted Nov 24, 2021 - 17:21 GMT-03:00

Resolved
The fix we implemented has been successful in resolving the issue with icons and full service has been resolved.

We apologize for the inconvenience this interruption caused and will be conducting a full review to avoid a repeat in the future.
Posted Nov 19, 2021 - 14:25 GMT-03:00
Monitoring
We've implemented a fix that should resolve the issue with icons. We're monitoring the results to make sure this has the intended effect and will update this page within the next hour to confirm.
Posted Nov 19, 2021 - 13:26 GMT-03:00
Update
Our team continues to work on this issue. We will post an update as soon as we have more information or in an hour’s time. We appreciate your patience.
Posted Nov 19, 2021 - 13:11 GMT-03:00
Update
We are continuing to work on a fix for this issue. We will post an update as soon as we have more information or in an hour’s time.
Posted Nov 19, 2021 - 12:10 GMT-03:00
Update
Our team is still working on this issue. We will post an update as soon as we have more information or in an hour’s time.
Posted Nov 19, 2021 - 11:07 GMT-03:00
Update
Our team continues to work on this issue. We will post an update as soon as we have more information or in an hour’s time.
Posted Nov 19, 2021 - 10:06 GMT-03:00
Update
Our team continues to work on fixing the issue. We will post an update as soon as we have more information or in an hour’s time. We appreciate your patience and apologize for this inconvenience.
Posted Nov 19, 2021 - 09:06 GMT-03:00
Identified
The cause of the issue has been identified and our team is working on fixing it.
Posted Nov 19, 2021 - 07:16 GMT-03:00
Investigating
We're experiencing a service disruption causing icons to disappear on the mural and are no longer available on the icon library when searched. We're investigating the issue and will restore regular service ASAP.
Posted Nov 19, 2021 - 06:40 GMT-03:00
This incident affected: MURAL Application (Realtime collaboration).