A Beneficial Bet
A natural disaster can disrupt even the most robust technology system. Two institutions are at the starting gate to serve as data recovery backup for each other.
By Sherry B. Mondou
Let's face it: You can never completely eliminate risk from campus operations. What you can do is seize opportunities and take safety precautions to help strengthen and protect your institution. That was the goal at the University of Puget Sound, Tacoma, Washington, a couple of years ago when we undertook enterprise risk management.
For our residential liberal arts institution-with a budget of $113 million and enrollment of 2,650 FTEs, our executive team and board identified what we considered our top 10 risk events. We examined strategic, financial, operating, and life safety risks. We concluded that one of our major risks was a breakdown in our information technology (IT) system, which we rely on for academic and administrative operations. We asked ourselves, "How can we minimize the risks associated with business interruption during a disaster?"
We had already built an alternate server location on campus that duplicates our core services. In the event of a localized problem, such as a fire, our seismic-reinforced emergency center could take over pretty quickly. However, if we experienced a regional event, such as an earthquake, it's possible that both buildings—although they are situated across campus from each other—could be damaged. We wanted another layer of protection, but we were reluctant to replicate our entire data system using a third-party provider, which could mean five- or perhaps even six-digit costs per year.
As we studied our options, we got lucky. Kenneth Pflueger, chief information officer of Pomona College, Claremont, California, serendipitously reached out to us with a query: Would we be interested in a possible disaster-recovery partnership?
While we were immediately enthusiastic, we knew we were venturing into uncharted territory. Puget Sound's chief technology officer, William Morse Jr., set up a meeting with his counterpart at Pomona to explore the concept and determine whether both parties shared similar goals and ideas. Over lunch, Morse and Pflueger created a three-page memorandum of understanding.
"Our idea was that this would be a cooperative agreement in which each partner would contribute space in its respective server room for one rack that would house the necessary redundant servers," Morse explains. "We didn't have expectations of either institution providing or doing work on the other's servers. In addition, each partner's server room would provide power, a secure location, and cooling and fire protection."
A Parameter-Defining Agreement
With technology services and risk management both in my portfolio of responsibilities, I was eager to sit down with Morse to review the potential benefits and risks of the proposed partnership.
The benefit was immediately obvious to both of us. The partnership was a relatively easy, low-cost strategy for getting us up and running—probably within 24 to 48 hours—in the event of a natural disaster.
Although the risks were a little harder to pin down, the bottom line was clear: Neither institution wanted to be sued if the arrangement didn't work out perfectly. "The challenge—definitely a legitimate one—was to define how the relationship would work to make sure the university's legal interests were protected," Morse says. "We didn't want to end up with a situation where a good deed got punished."
Our respective lawyers created a contract that took effect July 1. Because we didn't want to leave anything to chance, the background language in the agreement articulates the intent of both parties:
- Pomona and Puget Sound each wish to prepare for the smooth continuation or recovery of operation of the critical computer services and related technology infrastructure needed to conduct the essential business of their organizations after a natural or human-induced disaster or other extended business interruption, and each is in need of a location remote from its current facilities to locate backup computer equipment for disaster recovery.
- Pomona and Puget Sound are located sufficiently remote from each other, and each institution has unused capacity within its data center.
- Pomona and Puget Sound would like to provide for a mutually beneficial arrangement where each provides the other with the use of certain data center space, without charge, for disaster-recovery purposes such that both can avoid the cost of obtaining such space from commercial providers.
- Pomona and Puget Sound envision that each party's disaster-recovery equipment would be installed in the other party's data center—preloaded with the guest institution's software, and left in a state of readiness—so that the guest institution's critical applications and data can be more quickly restored in the event a disaster disrupts the technology infrastructure of the guest institution but not of the host institution.
Technical Takeaways
The process of establishing a disaster-recovery agreement will have taken about a year from the time Pomona first contacted us to the expected final installation date of our servers at Pomona this fall. During this time, we've learned a lot that could help other institutions with an interest in creating a similar arrangement.
Sign a contract. Our cost for legal counsel and contract preparation was $10,000, which has proven to be worth every penny. It answers all kinds of questions, including: Who's going to do what? How much space will be given? Who's responsible for maintaining the equipment? How do you get access to the equipment? What kind of effort will be involved in helping the other institution in the case of disaster? What turnaround time is acceptable? How can the partnership be dissolved?
"Our initial 3-page memorandum of understanding got translated into a 17-page document by the lawyers," Pflueger explains. "The lawyers made sure we covered things like indemnification,-essentially covering liability in the event that there is harm done to a staff member visiting the host institution, or if there is harm or damage to the hardware/system by being housed at the host institution. They helped us very clearly articulate what the agreement included and the responsibilities and liabilities of each institution."
Take it to the top. The magnitude of this partnership required presidential approval at both institutions. "Don't keep this on the IT level," recommends Pflueger. "It's really important to have institutional commitment at the highest levels. After William and I talked about this, I immediately went to our executive staff, which includes the president and vice presidents, and got their buy-in. That's key."
Include technical staff in the loop. "Our technical folks helped assess whether it was feasible to do this," Pflueger says. "Our network services staff at both schools worked out the protocols for transfer of data, which will be done in a batch mode every evening. They identified the hours of the transfer in order to minimize the impact on the network itself. They were helpful in assessing the plan's potential of success."
Keep hands off hardware. "The hardware that we deploy is left up to each institution," Pflueger says. "We've agreed we're providing rack space. What the institution puts in that rack is the institution's business. We don't have access to anything else."
Morse adds that the two institutions' systems do not have to be compatible for the recovery plan to work. "Their information system does not have anything to do with ours," he says. "We'll never touch their data system. The only thing we're depending on is their network. There will be no integration with their infrastructure."
If Disaster Strikes
If we never experience a disaster on campus, our students and faculty will never realize that we operate backup servers at Pomona. If disaster does strike, Morse will notify Pflueger of the emergency, and we will send some of our technical staff to Pomona. Upon arrival, our staff will have access to our equipment and be able to rebuild our core systems and put them back online using Pomona's network.
Morse emphasizes, and I agree, that today's institutions can't survive for an extended period of time without their computer networks. "Frankly, the virtual infrastructure of a college is as important as the physical structure these days," he says. "If you can't pay people, if you can't register or recruit students, if you can't deal with the issues of retention, your college could suffer long-term damage in reputation and infrastructure. There's not enough value placed on these data systems, which are as important as any academic building. After all, you might be able to relocate faculty offices and classes if you lose a building, but if you lose your data system, you won't be able to buy things, pay people, or do much
of anything."
Even in these difficult economic times—or perhaps even because of them—we think partnering with Pomona was a smart investment. Including legal bills, other fees, and the value of staff time, we estimate we spent less than $30,000 on this plan. We believe some strategic investments are worth taking, because they improve the institution's health over the long haul.
Our data systems are integral to campus operations. Why leave them at risk?
SHERRY B. MONDOU is vice president, finance and administration, University of Puget Sound, Tacoma, Washington. The article was coordinated and edited by Margo Vanover Porter, Locust Grove, Virginia, who covers higher education business issues for Business Officer.