Far from the bustle of Manhattan and its skyscrapers, I walked into the 25-person marketing firm in New Jersey not knowing what to expect. An agency persuaded me to go see a potential client which wanted to Migrate a Mac Server and replicate it at a DR host. They presented it as an “interview” (i.e. unpaid time) for my services but I took it because I knew there was more to the story. The IT Director “Jake” was a great guy but as I sized up the situation, it became apparent that Jake and his man were edging into dangerous waters. I started thinking of Winston Wolf from Pulp Fiction…
The Set Up
Had I understood correctly that they had only one file server? No DNS, DHCP, FTP? Any of the other dozen cool services available in OSX server? Nope, “SERVERD1” only served files. Are there network logins or…? The lack of Open Directory or AD meant that they would have to hand-create each client that joined or left the network. When I gently probed why they felt they needed a consultant, it emerged that there had been several recent outages of Email and File Services. Jake laughed nervously about how one Sunday the Email server was “unavailable for a few hours” while the building shut down the power. “For some reason the creative team decided they really needed to Email something out on a Sunday.” You can just picture the icy, gaze of management turning slowly towards the IT cubes… In no time flat, Email was outsourced, gobbled up by Rackspace’s Managed Email. Now they wanted to replicate the bulk of their files to an off-site file server.
Their entire storage infrastructure fits on a Drobo 5N, only 1TB of which is being used. If I was reading it correctly, the server being down one Sunday caused a chain reaction that resulted in someone questioning (perhaps unfairly) what was “going on with IT”. Without more budget or hands, Jake would have a hard time keeping his file server available 24/7/365.
I did not hold back in telling them exactly how to create a warm site for the file server: “You take SERVERD1 and you clone it to an image file. You back up your directories. You then back up your backups. And their backups. You put the server in Thunderbolt or Firewire target disk mode attached to an OS 10.9 Mini Server. You patch the Mini up to the latest version, then use Migration Assistant to import your original server configuration and your users. You make another clone of this machine for your Warm Mini over in– Who did you say your host was again?”
“Mac Mini Vault”.
“I’ve never heard of them but for your backup location you use the software you mentioned, Goodsync— which I’ve read good things about but never used– and you start your Warm Mini syncing with the Mini at HQ. By the way, that’s just the server side. Your clone needs to be fed and bathed continually now as well! To make it a warm site you have to replicate the contents of that server constantly…” I became I aware that I was talking for a while and looked up to see them both staring at me with their mouths slightly open and blinking.
“Have you guys thought about your firewall?”
“Firewall?” Jake asked, looking a little alarmed.
“Yeah, you’re going to have to create a tunnel between your locations so you can replicate your Drobo to the Drobo at MMV.”
“I need another Drobo at MMV?”
“Yes, you need equal or greater storage at your new location”.
They started hemming and hawing perhaps for good reason. Even though Jake had dismissed the cost of a 2nd Mini Server with dual SSDs as being “only $1700″, you could see the dollar signs starting to rack up . There was the monthly cost for the hosting, software license for Goodsync, a 2nd Drobo… If it is an unmanaged host, they would have to also throw in a second Sonicwall TZ.
Jake said, “Uh, I don’t want to touch the firewall”. He leaned in closer and said quietly “I haven”t touched it in 8 years.” I can almost sympathize with this sentiment. But what will happen when (not if) his firewall”s power supply fails? In his case, he will lose DHCP and VPN access. Jake will have to bone up quickly on the ins and outs of his firewall if he wants to keep HQ going, let alone replicate across sites. Change (and mirrored configurations) are not for the faint of heart.
Tips for Creating a Remote Hot Spare File Server:
1) Right size network bandwidth for your number of simultaneous users and workflow
2) Have a plan for failing back, only then will you be the hero
3) Grow up and face the DNS: name servers by their fully qualified domain name (FQDN)
4) Budget time for project planning and scheduling; and
5) Expect questions: Have ready your budget and schedule for your boss and messaging for your users
6) Purchase all hardware components in 2s — one day you will be overjoyed you had a spare.
Overview of Client Process:
1) Clone live production server
2) Set up a new 10.9 Server and migrate users and groups
3) Test like crazy
4) Verify users can login and check permissions carried over
5) Install Goodsync on new server
6) Clone the new server and apply to a 3rd Mini Server for the Warm Site
7) Test logging in
8) Teach the firewalls at each location to trust each other
9) Configure Goodsync to replicate