Customers on private tenants with BYOK are not able to log in
Incident Report for MURAL
Postmortem

What happened?

A small number of customers using BYOK experienced an issue with logging in to the MURAL application. Authentication requests were being redirected from their customer tenants to the main app.mural.co tenant, resulting in their users being unable to log in. Initial reports of this were received from MURAL customers on Sunday May 1st. Further reports were received on Monday May 2nd, at which point the issue was classified as a major incident and our task force was engaged.

Summary

MURAL’s task force started investigating at 13:58 UTC on Monday May 2nd. The cause of the issue was identified as being an incorrect configuration in how users on custom tenants were being routed through the login process. Once the issue was identified, a fix was rolled out at 14:07 UTC. This immediately corrected the login flow for clients using BYOK.

What we've done to avoid this happening again

Parts of the login flow for BYOK clients had been hard-coded. We have changed this to be dynamically generated, reducing the risk of code-based errors. We are also researching avenues to improve the automated monitoring of login issues.

Posted Jun 24, 2022 - 10:59 GMT-03:00

Resolved
This incident has been resolved
Posted May 02, 2022 - 13:24 GMT-03:00
Update
We are continuing to monitor for any further issues.
Posted May 02, 2022 - 12:05 GMT-03:00
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 02, 2022 - 11:17 GMT-03:00
Investigating
Some users are currently unable to sign in to MURAL. We know this is a major service disruption. We're investigating the issue and working to restore regular service as soon as possible. Please check our status page for the most up-to-date info status.mural.co
Posted May 02, 2022 - 10:58 GMT-03:00
This incident affected: MURAL Application (Authentication, Billing, Canvas, Exports, Integrations, MURAL Database, Notifications, Realtime collaboration, Search) and Learning, Website.