Since his tenure at City of Florence began in 1989, Systems Administrator John Moody has helped the city integrate and streamline the networking and telecom systems connecting more than 700 employees at nearly 30 remote locations. While their conversion to a Cisco voice platform in 2006 was one of their largest undertakings to date, the next project on the horizon for City of Florence would prove even more crucial in light of the Southeast’s deadly tornado outbreak of 2011.
Designing a Failover Plan
“We had begun planning for a secondary disaster recovery location for our primary operations site at City Hall well before April 2011, but seeing firsthand the destruction left in a tornado’s wake really upped our sense of urgency,” said Moody. “We knew it was imperative for us to develop a business continuity strategy that would allow the people of Florence and their businesses to operate in spite of a natural disaster, to be able to do something as simple as paying a utility bill. So the question we needed to answer was one of redundancy and business continuity: ‘How do we make sure we can continue functioning from another site?’ It was critical that we have the ability to fail over at a moment’s notice. Call it ‘just being on the safe side’ if you like, but if you think you can do just enough to get by, it doesn’t work like that in this world.”
Once the secondary location for the City’s core operating, telecommunication, and emergency (911) systems had been selected, Moody’s team began mapping out the infrastructure design the City would need to meet their recovery objectives. Moody’s team held multiple whiteboard strategy sessions to produce the criteria and scope of the project. When it was time to engage a technology solutions provider for the project, Moody was looking for a business partner who could demonstrate the most cost-effective delivery of his design.
The proposal submitted by TekLinks Corporate Enterprise Account Manager Jim Griffith called for a continuous data protection system by EMC and VMware that would work seamlessly with the City’s existing Cisco server platform. With EMC VNX Series Storage arrays installed at both locations along with the requisite licensing for vSphere Enterprise Plus and vCenter, TekLinks could implement VMware Site Recover Manager (SRM) and EMC RecoverPoint to enable point-in-time replication and recovery across a 10GB connection between their main site at City Hall to the secondary failover site several blocks away.
“City of Florence had previously partnered with TekLinks to upgrade their network, phone platform, and video surveillance systems, so our engineers were already very familiar with their environment,” explained Griffith.
In addition to TekLinks’ in-depth knowledge of the City’s current systems, Moody complimented the team’s focus on relationship over financial reward. “Customer service is really important to me. For instance, I appreciated the time the TekLinks team took to walk me through RecoverPoint’s journaling feature, even down to how it operates on the level of a RAID group. To have a vendor sit down with me to explain a solution - knowing that he may not get the sale - spoke volumes about the relationship I could count on, as well as the level of service TekLinks was ready to provide.”
Ultimately, the City’s decision hinged on which vendor could execute most efficiently on their proposed implementation plan within the given budget. “The solution TekLinks proposed worked well with our Cisco servers, and the design of the EMC arrays for both locations ensured the kind of failover protection City of Florence needed at a reasonable total cost of ownership,” said Moody.
RecoverPoint in Action
With TekLinks Solutions Architect Chad Sanders at the helm, the City’s business continuity project proceeded as planned and without any major delays. Much of this was due to the efforts of Florence Network Administrators Kim Denton and Keith Hovater, who worked closely with Sanders and were instrumental in the project’s success. Moody also attributes the smooth implementation to clear project management and frequent communication. Because of this transparency, Moody’s team was able to use that time as an educational opportunity. “We could look over TekLinks’ shoulder, so to speak, and learn from what they were doing at each step. We knew we could trust their engineers to adhere to best practices in any scenario,” explained Moody.
Roughly four months after the project’s completion, a situation arose that forced Moody to fully test the failover capabilities of the new RecoverPoint solution.
Around 1:15 am, Moody began receiving alerts that the equipment at the City’s production site was failing. He immediately called on Sanders to check the health of the VNX array. Sanders quickly ascertained that an A/C unit had malfunctioned, allowing the temperature of the server room to reach nearly 100 degrees Fahrenheit. Realizing it could be hours before the A/C could be restored, Moody and Sanders decided to cut-over all servers to the secondary site. Within just one hour, all of the City’s data, email, and applications were fully synced at the DR site with zero data loss. City of Florence’s entire environment could operate out of the DR site as long as necessary. The faulty A/C unit was soon resolved, and within an hour Sanders moved the environment back over to their main site to resume normal operations.
“As the City’s first unplanned ‘do-it-now’ DR scenario, there were definitely some lessons learned that we will account for in the future,” said Sanders. “But RecoverPoint is, without a doubt, one of the strongest enterprise-level replication solutions available right now.”
For Moody, the success of the recovery also stems from the reliability of his TekLinks team. “We had only tested one server to failover prior to this event,” said Moody. “So when I realized that our entire environment was at risk, I needed a guy to answer that middle-of-the-night call for help. Chad Sanders was that guy. A technology partnership is really about relationship and trust, and I believe TekLinks understands that.”
Business Results
City of Florence can now continue operating from a secondary facility in the event of a outage or natural disaster. All city employees can now rely on continuous replication to protect their data, phone system, and applications each day to deliver services to the citizens of Florence without interruption.
“RecoverPoint did exactly what it was supposed to do,” said Moody. “When I watched the failover go through its processes that night, I knew that we had chosen the right solution for the City’s needs. Ultimately, that second facility is not just a DR site, it’s a business continuity site. Everything you have lives and works on your network, and it has to be up 100% of the time. Business continuity and DR are simply the next step to where all businesses will need to be. Being down is not an option anymore.”
Download this Customer Success Story (PDF).
Interested in Business Continuity and Disaster Recovery solutions for your business?
Contact us today at 877.301.3125 or info@teklinks.com, or enter your information below!
[gravityform id="1" name="General Inquiries"]
