Invalid signature error when signing in with SSO

Incident Report for Mural

Postmortem

Summary
On 13th July 2023, we implemented a change to our SAML SSO login flow. This change was released to production at 7pm UTC and we started receiving sporadic reports of login issues that were treated as isolated incidents, as the vast majority of Mural users were able to continue accessing without issues.

By 11am UTC on the 14th July, the volume of login issues had increased and we activated our taskforce to investigate the cause. By 11:25 UTC we identified a potential cause of the issue and started preparing a fix. At 11:45 UTC we made the decision to roll back the change from the previous day, which was completed at 12:15 UTC.

As of 12:15 UTC login behavior returned to normal parameters.‌

‌What we've done to avoid this happening again
The change we implemented on Thursday afternoon was rolled back and we are preparing a new change that will avoid impacting the SAML SSO login flow.

Posted Jul 18, 2023 - 12:56 GMT-03:00

Resolved

The fix we implemented earlier has been successful in resolving the issue and full service has been resolved.

We apologize for the inconvenience this interruption caused, and will be conducting a full review to avoid a repeat in the future.

Posted Jul 14, 2023 - 10:56 GMT-03:00

Monitoring

We've implemented a fix that should resolve the issue. We're monitoring the results to make sure this has the intended effect and will update this message as soon as we've been able to confirm this.

Posted Jul 14, 2023 - 09:59 GMT-03:00

Update

We are continuing to investigate this issue.

Posted Jul 14, 2023 - 09:24 GMT-03:00

Investigating

Some users are unable to log in to Mural via SSO, with an error message of “invalid signature”. We are investigating and will restore regular service as soon as possible.

Posted Jul 14, 2023 - 08:26 GMT-03:00

This incident affected: Mural Application (Authentication) and Learning, Website.