RIZZASTERIncident Review: Backup Failure of World Files
/archive/old/2021/ · last edited 25 Dec 2019 · Billing
On the 15th of October, 2023, a critical issue arose with the backup process for the world files of Rizzaster. During the nightly backup routine, the system failed to complete the process, leaving the most recent world files unprotected.
The primary cause of the failure was identified as a misconfiguration in the backup script. Specifically, the script had an unexpected condition where it attempted to compress the files before they were fully written to the server, leading to incomplete backup files. This issue was exacerbated by the fact that the server was experiencing high traffic during the backup window, causing delays and further complications.
To address this issue, we have updated the backup script to ensure that files are fully written before compression begins. Additionally, the backup window has been shifted to a less busy time, reducing the likelihood of similar issues arising in the future. Regular audits and testing of the backup process will be implemented to prevent similar incidents.