On 13th July 2023, we implemented a change to our SAML SSO login flow. This change was released to production at 7pm UTC and we started receiving sporadic reports of login issues that were treated as isolated incidents, as the vast majority of Mural users were able to continue accessing without issues.
By 11am UTC on the 14th July, the volume of login issues had increased and we activated our taskforce to investigate the cause. By 11:25 UTC we identified a potential cause of the issue and started preparing a fix. At 11:45 UTC we made the decision to roll back the change from the previous day, which was completed at 12:15 UTC.
As of 12:15 UTC login behavior returned to normal parameters.
What we've done to avoid this happening again
The change we implemented on Thursday afternoon was rolled back and we are preparing a new change that will avoid impacting the SAML SSO login flow.