RIZZASTER

Incident Review: Backup Failure and Resolution

/archive/2018/misc/old/ · last edited 21 May 2022 · Partnerships

Last Friday, at around 3 PM, the Rizzaster game server experienced an unexpected issue with the world file backups. The server logs indicated that the backup process had failed, although no error messages were immediately available. A quick check revealed that the backup had not been completed successfully. This failure was significant as it jeopardized the integrity and accessibility of the server’s world files, which are crucial for maintaining the game's continuity and the players' experience.

Upon reviewing the server settings, it became clear that the backup job had been inadvertently disabled due to a recent update to the server’s maintenance scripts. The update aimed to streamline the backup process but inadvertently introduced a bug that prevented the backup from running. As a result, the last successful backup was from a week ago, which was not sufficient to cover recent events and player progress. The team quickly re-enabled the backup job and manually initiated a backup process to ensure the current state of the world files was saved.

The incident served as a reminder to the moderators to thoroughly test new changes before deploying them, and to include more detailed error logging to prevent such issues in the future. We also decided to set up a more robust monitoring system to alert us to any issues related to backups, ensuring that such incidents can be addressed swiftly.

Related pages

Up to /archive/2018/misc/old/ · Archive home