On Friday, February 18, 2022, the search engine in MURAL was upgraded from ElasticSearch 6 to ES7. The change proceeded as planned and no immediate issues were detected during our post-release checks.
Later that day our engineers detected unexpected behavior with MURAL search results. An immediate decision was taken to roll back to the previous version. The process to roll back the upgrade was unsuccessful. Our team worked through the weekend to restore search functionality within MURAL workspaces. Public API endpoints for searching were however inoperative.
Our team continued working through the next week to identify the root of the issue with ElasticSearch. On Friday, February 25, the indexes were fully rebuilt on a new ES6 cluster, tested successfully in beta, and then enabled in production. This restored full functionality using ES6, including the public API endpoints for search.
This incident resulted in a partial outage for search functionality within MURAL workspaces, lasting 54 hours and 56 minutes. MURAL remained operational throughout this incident and members were able to log in and access their data.
What we've done to avoid this happening again
We are re-evaluating our upgrade process for ES6 to ES7 to ensure this does not happen again during the next upgrade.