What happened?

A small number of customers using BYOK experienced an issue with logging in to the MURAL application. Authentication requests were being redirected from their customer tenants to the main app.mural.co tenant, resulting in their users being unable to log in. Initial reports of this were received from MURAL customers on Sunday May 1st. Further reports were received on Monday May 2nd, at which point the issue was classified as a major incident and our task force was engaged.

‌

Summary

MURAL’s task force started investigating at 13:58 UTC on Monday May 2nd. The cause of the issue was identified as being an incorrect configuration in how users on custom tenants were being routed through the login process. Once the issue was identified, a fix was rolled out at 14:07 UTC. This immediately corrected the login flow for clients using BYOK.

‌

‌What we've done to avoid this happening again

Parts of the login flow for BYOK clients had been hard-coded. We have changed this to be dynamically generated, reducing the risk of code-based errors. We are also researching avenues to improve the automated monitoring of login issues.