Basically one of the backup processes borked and stopped deleting old backup files, and also filled a log with errors. The combination of both of those caused the server to run out of diskspace, and then the database crashed.
Ordinarily, deleting the old files, clearing out the log, and restarting the database would have worked.
However once I did the first two, the server failed to start. Due to the crash, some data got corrupted. I got Claude Code to write some recovery scripts, which worked to first extract all the good data to a backup, then restore that data to a new database server. Thankfully the corrupted data wasn't actually anything critical, so I don't believe we lost anything of value.
Even if we had, we still had the last backup from the day before the database crashed (FYI, the forum gets backed up locally every morning). I designed some redundancies into the system, haha. But I'll work with the admins to figure out a plan going forward since we do need to upgrade some parts of the server in the future.
Ordinarily, deleting the old files, clearing out the log, and restarting the database would have worked.
However once I did the first two, the server failed to start. Due to the crash, some data got corrupted. I got Claude Code to write some recovery scripts, which worked to first extract all the good data to a backup, then restore that data to a new database server. Thankfully the corrupted data wasn't actually anything critical, so I don't believe we lost anything of value.
Even if we had, we still had the last backup from the day before the database crashed (FYI, the forum gets backed up locally every morning). I designed some redundancies into the system, haha. But I'll work with the admins to figure out a plan going forward since we do need to upgrade some parts of the server in the future.
