Having worked in the IT field for over two decades I've seen backup tools evolve from tape magazines to cloud services, and I've seen simple backup methods at SMB's to more complex ones at larger organizations.
The variables at play in this story:
Bitrot, likely due to a environment unfriendly to a single server running an older version of Citrix Xen
Inadequate storage on the NAS
Image-based backups without verification or "application-aware backup" enabled
No regular reboots or backup testing
Back in 2015-16, when I was working at an MSP, there was a hotel which started having account login failures. We were unable to access the Active Director domain controller remotely for some reason, so we went onsite (as one does). What we found was the domain controller with failed services left and right. Initially expecting the problem to be that it hadn't been rebooted in ages (which is a common thing in SMB's), we confirmed the backups had been completed recently and rebooted it.
Upon reboot, the hard drive was shown as failed. Uh oh.
We attempted to perform a restore from the backups, which were stored on a small Synology NAS on some excellent 2TB WD Gold drives. We restored the most recent backup. Same issue. A previous backup, same issue.
We went back to the last few weeks, but all had the same error. We would have gone back farther, but the backups only went back a month or so due to space limitations.
So...it was all gone. No offsite backups, no cloud backups, and all the onsite backups were corrupted and not bootable.
Honestly, we were kind of dumbfounded. This was very unexpected. We were relieved that this was not our setup, as it was inherited only recently, but we also knew that we had to come up with a solution ASAP and get this property up and fully running again.
The resulting solution:
The cost to the client was pretty steep, but we had them up and running again in less than 36 hours by working all night. We pulled a spare HPE ProLiant DL380G7 out of our stock, installed ESXi 6.0 with a free license, set up a new domain controller with the client's existing Windows Server 2008R2 licensing, and then went through the time consuming process of moving each computer to the new domain with all new user accounts. Backups were also set up again on the Synology (which we added more storage to) and tested periodically.