NACUBO

My NacuboWhy Join: Benefits of Membership

E-mail:   Password:   

 Remember Me? | Forgot password? | Need an online account?

Business Officer Magazine
Loading

A Beneficial Bet

A natural disaster can disrupt even the most robust technology system. Two institutions are at the starting gate to serve as data recovery backup for each other.

By Sherry B. Mondou

*Let's face it: You can never completely eliminate risk from campus operations. What you can do is seize opportunities and take safety precautions to help strengthen and protect your institution. That was the goal at the University of Puget Sound, Tacoma, Washington, a couple of years ago when we undertook enterprise risk management.

For our residential liberal arts institution-with a budget of $113 million and enrollment of 2,650 FTEs, our executive team and board identified what we considered our top 10 risk events. We examined strategic, financial, operating, and life safety risks. We concluded that one of our major risks was a breakdown in our information technology (IT) system, which we rely on for academic and administrative operations. We asked ourselves, "How can we minimize the risks associated with business interruption during a disaster?"

We had already built an alternate server location on campus that duplicates our core services. In the event of a localized problem, such as a fire, our seismic-reinforced emergency center could take over pretty quickly. However, if we experienced a regional event, such as an earthquake, it's possible that both buildings—although they are situated across campus from each other—could be damaged. We wanted another layer of protection, but we were reluctant to replicate our entire data system using a third-party provider, which could mean five- or perhaps even six-digit costs per year.

As we studied our options, we got lucky. Kenneth Pflueger, chief information officer of Pomona College, Claremont, California, serendipitously reached out to us with a query: Would we be interested in a possible disaster-recovery partnership?

While we were immediately enthusiastic, we knew we were venturing into uncharted territory. Puget Sound's chief technology officer, William Morse Jr., set up a meeting with his counterpart at Pomona to explore the concept and determine whether both parties shared similar goals and ideas. Over lunch, Morse and Pflueger created a three-page memorandum of understanding.

"Our idea was that this would be a cooperative agreement in which each partner would contribute space in its respective server room for one rack that would house the necessary redundant servers," Morse explains. "We didn't have expectations of either institution providing or doing work on the other's servers. In addition, each partner's server room would provide power, a secure location, and cooling and fire protection."

A Parameter-Defining Agreement

With technology services and risk management both in my portfolio of responsibilities, I was eager to sit down with Morse to review the potential benefits and risks of the proposed partnership.

The benefit was immediately obvious to both of us. The partnership was a relatively easy, low-cost strategy for getting us up and running—probably within 24 to 48 hours—in the event of a natural disaster.

Although the risks were a little harder to pin down, the bottom line was clear: Neither institution wanted to be sued if the arrangement didn't work out perfectly. "The challenge—definitely a legitimate one—was to define how the relationship would work to make sure the university's legal interests were protected," Morse says. "We didn't want to end up with a situation where a good deed got punished."

Our respective lawyers created a contract that took effect July 1. Because we didn't want to leave anything to chance, the background language in the agreement articulates the intent of both parties:

  • Pomona and Puget Sound each wish to prepare for the smooth continuation or recovery of operation of the critical computer services and related technology infrastructure needed to conduct the essential business of their organizations after a natural or human-induced disaster or other extended business interruption, and each is in need of a location remote from its current facilities to locate backup computer equipment for disaster recovery.
  • Pomona and Puget Sound are located sufficiently remote from each other, and each institution has unused capacity within its data center.
  • Pomona and Puget Sound would like to provide for a mutually beneficial arrangement where each provides the other with the use of certain data center space, without charge, for disaster-recovery purposes such that both can avoid the cost of obtaining such space from commercial providers.
  • Pomona and Puget Sound envision that each party's disaster-recovery equipment would be installed in the other party's data center—preloaded with the guest institution's software, and left in a state of readiness—so that the guest institution's critical applications and data can be more quickly restored in the event a disaster disrupts the technology infrastructure of the guest institution but not of the host institution.

Designated racks will house servers similar to this one for backup recovery equipment at the partner installations.

Technical Takeaways

The process of establishing a disaster-recovery agreement will have taken about a year from the time Pomona first contacted us to the expected final installation date of our servers at Pomona this fall. During this time, we've learned a lot that could help other institutions with an interest in creating a similar arrangement.

Sign a contract. Our cost for legal counsel and contract preparation was $10,000, which has proven to be worth every penny. It answers all kinds of questions, including: Who's going to do what? How much space will be given? Who's responsible for maintaining the equipment? How do you get access to the equipment? What kind of effort will be involved in helping the other institution in the case of disaster? What turnaround time is acceptable? How can the partnership be dissolved?

"Our initial 3-page memorandum of understanding got translated into a 17-page document by the lawyers," Pflueger explains. "The lawyers made sure we covered things like indemnification,-essentially covering liability in the event that there is harm done to a staff member visiting the host institution, or if there is harm or damage to the hardware/system by being housed at the host institution. They helped us very clearly articulate what the agreement included and the responsibilities and liabilities of each institution."

Take it to the top. The magnitude of this partnership required presidential approval at both institutions. "Don't keep this on the IT level," recommends Pflueger. "It's really important to have institutional commitment at the highest levels. After William and I talked about this, I immediately went to our executive staff, which includes the president and vice presidents, and got their buy-in. That's key."

Include technical staff in the loop. "Our technical folks helped assess whether it was feasible to do this," Pflueger says. "Our network services staff at both schools worked out the protocols for transfer of data, which will be done in a batch mode every evening. They identified the hours of the transfer in order to minimize the impact on the network itself. They were helpful in assessing the plan's potential of success."

Keep hands off hardware. "The hardware that we deploy is left up to each institution," Pflueger says. "We've agreed we're providing rack space. What the institution puts in that rack is the institution's business. We don't have access to anything else."

Morse adds that the two institutions' systems do not have to be compatible for the recovery plan to work. "Their information system does not have anything to do with ours," he says. "We'll never touch their data system. The only thing we're depending on is their network. There will be no integration with their infrastructure."

If Disaster Strikes

If we never experience a disaster on campus, our students and faculty will never realize that we operate backup servers at Pomona. If disaster does strike, Morse will notify Pflueger of the emergency, and we will send some of our technical staff to Pomona. Upon arrival, our staff will have access to our equipment and be able to rebuild our core systems and put them back online using Pomona's network.

Morse emphasizes, and I agree, that today's institutions can't survive for an extended period of time without their computer networks. "Frankly, the virtual infrastructure of a college is as important as the physical structure these days," he says. "If you can't pay people, if you can't register or recruit students, if you can't deal with the issues of retention, your college could suffer long-term damage in reputation and infrastructure. There's not enough value placed on these data systems, which are as important as any academic building. After all, you might be able to relocate faculty offices and classes if you lose a building, but if you lose your data system, you won't be able to buy things, pay people, or do much
of anything."

Even in these difficult economic times—or perhaps even because of them—we think partnering with Pomona was a smart investment. Including legal bills, other fees, and the value of staff time, we estimate we spent less than $30,000 on this plan. We believe some strategic investments are worth taking, because they improve the institution's health over the long haul.

Our data systems are integral to campus operations. Why leave them at risk?

SHERRY B. MONDOU is vice president, finance and administration, University of Puget Sound, Tacoma, Washington. The article was coordinated and edited by Margo Vanover Porter, Locust Grove, Virginia, who covers higher education business issues for Business Officer.

Tips for Finding the Right Partner

For private liberal arts colleges, it's not been a cultural phenomenon to work together on the type of exchange we've put in place with Pomona College," says William Morse Jr., chief technology officer and associate vice president for technology services, University of Puget Sound, Tacoma, Washington. "Given the economic realities of what's going on in the world, these types of partnerships will help us make education more affordable for our students and yet meet the needs of our institution by protecting us from any sort of unexpected natural disaster."

If you decide to look for a partner, here's our best advice:

  • Tap into your networks. At Puget Sound, we do a lot of collaborating and information sharing. We have made meaningful and productive relationships through Pacific Conference (PACCON), a 32-year-old consortium of 28 small, private liberal arts colleges and universities on the West Coast serving CFOs and their chief accounting officers. Because Pomona is part of the group, we were already comfortable with our respective organizations.
  • Check out reputations. You need to know that your partner institution maintains a really strong team of IT professionals. You're trusting that the staff demonstrate a certain level of competence and care on a daily basis.
  • Seek flexibility. Remember, you're entering into a cooperative agreement, not hiring a company, and your expectations can't be the same for both. If a potential partner insists: "I expect you to do X, Y, and Z in an emergency," you might want to keep looking. We prefer a partner who says, "Yeah, I know you're doing this for our institution as a favor. What we need to do is ensure that both of our institutions are protected." That attitude will make for a good partner.
  • Consider size and type. Morse is fond of saying: "Your most expensive student is your first one. Everything else is free because you have to provide all the resources and work to make sure your first student can complete an education." There's some truth in that, but it's unlikely we could have reached an agreement with a large institution, such as Duke, or Emory, or Harvard, because they would need more than one rack and probably require instant backup and other services.
  • Talk to institutions that share your concerns. Not all institutions recognize the potential for disaster in their IT business continuity plans, explains Kenneth Pflueger, chief information officer, Pomona College, Claremont, California. "You have to have some mutual appreciation for the potential of a disaster," he says. "Without that, there's not the same kind of commitment." He explains that when he initially talked to a few colleagues, their initial response was, "Oh, that sounds like a great idea," but when he tried to press it, the conversations never went very far. "Puget Sound and Pomona were on the same page with the same concerns," he says.
  • Go the distance. "Being in southern California, the most likely disaster we face is an earthquake," Pflueger says, "so we wanted to find an institution that was out of our immediate area. We figured Puget Sound was far enough away from us that it would be highly unlikely that an earthquake would affect both of us at the same time." Of course, you might want to consider commuting time for setup and maintenance. Both of our institutions appreciate that direct flights operate between our two cities, notes Pflueger.
  • Investigate turnover. "A major risk is that of too much turnover in an institution, and suddenly the commitment to the agreement doesn't exist at one place or the other," Pflueger points out. "For example, while William and I agreed that many clauses in the legal agreement were just common sense, the lawyers countered, 'That's fine if you and William are there, but we need to think about what happens if you are not.'"

Sherry Mondou