What happened:
At 8:09am Pacific Time on Feb 22, 2021 we updated the IP Allowlisting settings for one of our Enterprise customers. The list of IP addresses for that customer contained an incorrectly formatted IP range, causing an exception in the code that updates our firewall settings. Our firewall configuration system responded to this exception in the most conservative manner possible, temporarily shutting off all customer access.
At 8:27am Pacific Time on Feb 22, 2021 we corrected the malformed entry and re-applied the IP Allowlisting settings for the customer in question, restoring access to the system.
What we’ve done to avoid this happening again:
At 5:14am Pacific Time on Feb 23, 2021 we published an update to our Firewall configuration system. This update provides more granular exception handling and ensures that Firewall updates will apply only to the customer being modified.