The tldr; is at the bottom, after the story mode writeup. Feel free to CTRL+END or hyper-scroll your way there, or read the machinations of my troubleshooting on your way there.
There was a time at Manufacturing Company 1 (AKA MC1) when they had an IBM SAN. It worked well and the file shares were backed up regularly to tape from a tape drive with a magazine. Then came the end-times. The ransomware times. A time when trust was broken, data was lost and chaotic data anarchy reigned in the void of SOP & functional design.
This is a story about rising up and taking control of the data again.
Like the remnants of civilization ravaged by a zombie apocalypse, MC1's IT team had attempted to salvage what they could and establish what looked like a bajillion file shares across several NAS devices and an old Exchange server repurposed as a file server. The SAN was shut down due to distrust by the entire organization. The fear of being reinfected was real. The backups? Abandoned, but also impossible to do now with the spread of data everywhere. Also the server that had performed the backups was encrypted, as were the backups, so everything was seen as bleak, futile or impossible.
A newer Synology had been purchased, though vastly under provisioned on storage, it could not hope to replace more than 1 of the salvaged storage devices on its own.
The older Buffalo NAS' couldn't handle the change in dates to daylight savings that had occurred years before, and each cycle I would have to manually change the time on them so that the LDAP auth would allow users to access the files again.
The repurposed Exchange server was...well, old. It was running 2003 R2 (2019 was the current release) and did not have much storage available either.
MC1 had resorted to storing a lot of data in their Dropbox account, and new users were given accounts more often than not. Much of the data on Dropbox was also stored on 1 or more network drives, much was new data that was not. This setup was not sustainable, advisable, even remotely close to "good practice".
The need for standardizing our data was here, and the solution was to get a new storage array. My best guess put our capacity needs at ~20TB, so I ordered a larger Synology RS unit with 144TB raw. That would be more than enough to handle the data plus backups.
This was the easy part. It's easy to get new hardware and set it up. It's also not hard to migrate all the data from the small NAS' and old Exchange server and remap the network drives for folks.
There were a few hiccups, the biggest one being that for a few years a number of product managers had been pasting links to files located on an older Buffalo NAS. Those links broke when we removed the NAS, but it was easy enough to repoint the DNS record to the new Synology so that the links worked again. We had to add in another network connection with the IP address of a different NAS for the same reason, but those links had been using the IP address instead of the FQDN.
Over a few months I'd had everything moved over and removed all of the old storage devices. I was especially pleased when I could power off the old Exchange server. I did not like having 2003R2 on the network.
The next phase was more difficult. I needed to sync the files with Dropbox and vice versa, because we needed uniform data organization and for years new hires had gotten into the habit of storing data in different places with no regard to organization or duplication.
Synology makes the connectivity pretty easy, just use their Cloud Sync tool to sync folders up and down with Dropbox. The real issue was the organization of the files. There were not only duplicates between Dropbox and the file servers, there were duplicates between the different file servers as well!
There is no easy way to fix this. No "One more thing" tool or trick to this. We had to collapse the number of file shares and reorganize the data. It's a tedious type of work that requires a lot of participation from users. There were some users who were excited about the idea, because it would make some aspects of their jobs so much easier!
But...most people just didn't care. Or, to be honest, it was more that they didn't like change, even a change like this. They'd been putting this daily report in the N:\Sales\Department\Year\Month\Week\Day folder for 15 years! They really did not want to save it to W:\Sales\Department\Reports\Quarter\Month instead. Why should they have to change instead of Team2 in the same department? Why couldn't Team2 jus save their files in the N:\Sales\Department\Year\Month\Week\Day folders instead?
Most teams had this same attitude about it, so it appeared we'd never get this resolved. Even the stakeholders just didn't care enough about it. Why should they get in the middle of their teams and dictate something like this? This is an IT problem, not a VP of Sales problem! Just find a way to make it work.
So I did. I made it work. Not in a way that everyone was please with, but it worked. There comes a time when collaboration fails. Not because you aren't collaborating, but because the collaborators are like that one person in your project group in University who didn't participate, but expected to get credit for something.
Instead of being discouraged I took the comments from the stakeholders as directive instead of dismissal, and I reorganized the data myself. I organized it the way it made sense to me.
And I'm very particular about the way I organize my data.
So I set up new shares on the server and synced them to new team folders in Dropbox, then I started syncing the data from select areas of file shares into them so that they started to mostly organize automatically. I just didn't add any permissions, so know one knew. Once everything was ready I sent out an email about the changes that would take effect in 2 weeks, followed by a couple of reminders. On the weekend before cutover for Sales I removed access from the old shares and added the users into the new Dropbox groups and AD groups that I'd already set up, then activated the GPOs that would remove the old mapped drives and add the new ones.
Come Monday people had...comments. But everything was still there, it was just somewhere different. And I'd provided a complete mapping of the before and after on each email notice prior to cutover, so no one could actually complain in any meaningful way. But it was good! The new organization worked well for them, even if it took a few weeks for the grumbling about how "it was fine the way it was" to die down.
But I did the same for Accounting, Manufacturing, Product Development, HR, Shipping/Warehouse/Logistics, and even *gasp* IT.
We did have some issues though. There was so much disorganization that after the data was reorganized some people lost access to some reports. And in the process of getting approval to add permission for these Accounting would be surprised! No, that person should never have had access to that! Please do not give them access! Only their director should have that access!
Lot's of those occurred. It seems a lot of improper delegation had been happening over the years and security was basically tossed out of the window. Lot's of auditable events that we fixed over a few month's time.
Then the cleanup began. Removing all of the old file shares and Dropbox shares no one had access to for 6 months. Firing up all of the old NAS' to wipe them. Pulling the old Exchange server drives and shredding them.
Years of chaos undone in a few months due to a plan and the ability to take decisive action.
But that's not the end of our story. Because over the years a new issue was lurking. An issue about spend. An issue about redundancy.
tldr; data was stored all over the place, so I purchased a new Synology RS unit with tons of space, migrated all the data to it, reorganized it and synced it to Dropbox.
Total Cost: $7,500 + the sacrifice of some brownie points when I made the decisions others wouldn't.