Hurricane Sandy caused devastating effects to both personal and business operations. Coping with its aftermath has been particularly tough for those business professionals whose responsibility is dealing with IT disaster recovery. At Bellevue, New York City’s flagship public hospital, the challenges were nearly insurmountable.
Bellevue planned to stay open during Hurricane Sandy. However, as conditions in the hospital deteriorated in the days following landfall, hospital staff and members of the National Guard evacuated Bellevue’s approximately 725 patients.
Prior to the evacuation, the storm surge flooded elevator shafts, knocked out landline and wireless communications and disrupted the hospital’s supply of running water. The storm surge also flooded the basement and shut down the pumps supplying Bellevue’s backup generators. It was in darkness that hundreds of critically ill patients were successfully evacuated and transferred to other hospitals.
6 IT Disaster Recovery Lessons from Bellevue’s Hurricane Sandy Experience
IT administrators can learn a great deal from considering what kinds of issues arose at Bellevue and how well the hospital’s IT disaster recovery plans worked.
The basement is not the ideal place for mission critical systems. Bellevue had hardened their systems after Hurricane Irene and NYU engineers believed that backup power systems would work so long as the storm surge did not exceed 12 feet. Unfortunately, Sandy produced a 14-foot surge, Bellevue’s cavernous basement flooded, and the pumps delivering fuel to generators shorted out. Relocating pumps and controls is likely to be expensive and require solving code-related issues, but moving mission critical controls higher in the building may be the only way for Bellevue to ensure it has electric power and water during a similar storm.
The storm knocked out landline and cellular service in the hospital. Communication inside the hospital was poor once the phone system went offline. New York Times’ reporters Amemona Hartocollis and Nina Bernstein reported Bellevue doctors and nurses wished that “some other form of communication, like walkie-talkies, had been available.” Does your IT disaster recovery plan include a way for hospital staff to stay connected when neither landline nor wireless telephones work?
The last two patients at Bellevue stayed because they needed an elevator to evacuate. Unfortunately, all 32 elevators at Bellevue stopped working when the shafts began to fill with water and the backup generators failed. Writing in the Journal of American Medical Association, Tia Powell, MD and Dan Hanfling, MD and Lawrence Gostin, JD, note, “A timely preemptive move [of these two patients] to a room on a lower floor might have minimized the risk.” That may be true.
Is the last similar disaster thinking apply to this disaster too? The NYT reported that Dr. Farley, New York’s health commissioner, based his decision not to order the closing of Bellevue and other hospitals during Sandy partly upon his experience with Hurricane Irene. Dr. Farley and others were monitoring predictions about the magnitude of the expected storm surge based on Irene. Those predictions underestimated the size of the surge.
Consider how accurately simulations capture disaster conditions. Pam Grier, CEO of Maimodes Medical Center in Brooklyn, shared one take away from her hospital’s experience coping with Sandy, “As prepared as we think we are we’ve never had a mock disaster drill where we carried patients downstairs. I’m shocked that we didn’t do that. Now we’re going to.” Does your disaster recovery training include simulating evacuation under difficult conditions?
Another crucial element of the evacuation is sending data with the transferred business. Without electricity and systems or even light, getting patients out with important data such as copies of charts and records was much more difficult for the hospital. The New York Times reported that Bellevue was successful in sending rudimentary records with each patient, highlighting the need to replicate offsite not only data but complete systems to access the data in a meaningful way. Image based backups and the ability to easily convert those to virtual systems at a remote data center could have kept those records accessible. Is your business able to quickly get systems operational under such conditions?
Healthcare workers at Bellevue and throughout the Northeast have done exceptional work in helping communities recover from Hurricane Sandy. They are still working at it. The experience of Bellevue and other affected businesses may help improve hospital disaster recovery plans. For some specific ways to improve your hospital’s IT disaster recovery plan, check out this webinar on 5 Reasons Why Disaster Recovery Plans Fail and How to Fix Yours.
“At Bellevue, a Desperate Fight To Ensure the Patients’ Safety.” The New York Times. The New York Times, 02 Nov. 2012. http://www.nytimes.com/2012/11/02/nyregion/at-bellevue-a-desperate-fight-to-ensure-the-patients-safety.html?pagewanted=all
“Emergency Preparedness and Public Health: The Lessons of Hurricane Sandy.” JAMA Network. American Medical Association, 16 Nov. 2012. http://jama.jamanetwork.com/article.aspx?articleid=1392488
image source: reuters