This happened yesterday — I don’t think I’ve ever felt such a sharp “oh no” moment.
I was running disk check commands, and then suddenly I couldn’t type anything.
Rebooted — and “kernel panic!” appeared. The system wouldn’t start.
I knew this was bad, but I thought: this is exactly what backups are for, so we should be able to recover.
And then I was genuinely reminded of Linux’s strengths.
I’d been making backups thinking “there’s probably some redundancy, but better to have it” — covering /boot /lib /etc /usr /home /var and most of the important directories. That decision paid off completely.
I kept the original partition layout intact, did a minimal fresh install, then used rsync to copy everything back from the backup drive.
The result was a perfect restoration of the previous state.
Email, web, SSH, user configuration — everything was exactly as before.
The config files didn’t need any changes. The whole operation was done in about 3 hours, most of which was the copy time itself. (Total data was about 90 GB.)
This might have been a good opportunity to actually test a backup recovery in practice.
I can now run the server going forward with real confidence.
So glad I had those backups.