|
As a network of regional destination hospitals specializing in complex and advance-stage cancer care, Cancer Treatment Centers of America (CTCA) has taken steps to change not only the face of cancer care, but also the entire hospital environment.
CTCA hospitals offer state-of-the-art clinical care and technology as well as deluxe features such as hotel-style lobbies with concierge desks, airport transportation service, caregiver accommodations and complementary therapies (e.g., nutrition therapy, oncology rehabilitation and pain management) all under one roof and all designed to deliver what CTCA calls the 'Mother Standard' of care.
 That quest to go the extra mile extends to the group's data strategy. In authorizing a new colocated data center to accommodate growth, for example, one mandate was that no technology stone be left unturned to keep operations up and running in the event of a failure. That directive was inspired by the need to prevent service disruptions affecting access to electronic medical records and other critical information to support the CTCA promise of high-quality medical treatment. Accordingly, the new data center was designed, engineered and implemented in conjunction with Chicago-based IT consultancy Ahead with even more redundancy than most facilities of its kind.
Among the extras was the use of new EMC VPLEX Metro technology to eliminate the need to fail over to a disaster recovery site 120 miles away in the event of an outage affecting one of the two virtual storage arrays in the new production center. Instead, using VPLEX, data is synchronously mirrored between the two on-site storage arrays to avoid the delays associated with remote data recovery.
The new data center in Wisconsin-live since April 5, 2011-also utilizes advanced data center infrastructure (Cisco Nexus), mid-tier and enterprise storage arrays (EMC Symmetrix VMAX) and server virtualization (VMware vSphere 4.1). It's a gold standard of data delivery that corresponds to the CTCA Mother Standard of patient care.
Spurred by EHR Upgrade
CTCA's former production data center was located in company headquarters in Schaumburg, IL. It dated back to the days when the group had only two hospitals-its original facility in Chicago's far north suburbs in Zion and a second in Tulsa, OK.
By 2010 CTCA had added regional hospitals in Philadelphia and Phoenix, opened an outpatient oncology clinic in Seattle, and announced plans to open a fifth hospital in Atlanta. Its legacy data center was at capacity with no room in the building to expand. That challenge, combined with an upcoming upgrade to the Eclipsys (now Allscripts) EHR platform used in all CTCA facilities, sparked a decision to move and update the IT infrastructure.
"Because of our space constraints, we were unable to build a new environment for our EHR upgrade in Schaumburg and run both the new and old versions in parallel for a safe migration. We knew we had to relocate our data center to do it," says Chad A. Eckes, Chief Information Officer for CTCA. "That gave us the opportunity to build an enterprise-class infrastructure with newer technologies that would help us become more efficient as well as deliver better service. Once we began outlining our requirements and synchronizing them with our commitment to the patients and families we serve, it became clear that our No. 1 goal was 100% uptime with automatic failover in the event of a system failure."
Achieving that goal would not only ensure continuous access to electronic patient records but also eliminate the effort of manual restart and recovery. In addition, it would avoid many hours of backfilling electronic charts with paper SOAP notes that must be maintained if an EHR system is offline.
Next-Generation Architecture
Working with Ahead for engineering support and technology recommendations, CTCA decided to lease space from a colocation provider in Madison. The original Schaumburg data center would be turned into a disaster recovery site with some infrastructure upgrades to take advantage of current technologies.
Then, engineers from both organizations began designing the new data center architecture, keeping in mind the need to provide redundancy for critical components and services such as switches, servers, storage, power and cooling so that a faulty part or connection will not cause the data operations to grind to a halt.
For the core building blocks, the team selected next-generation Cisco Nexus 7000 switches that reduce cabling requirements and improve performance; EMC Symmetrix VMAX and EMC Celerra NS-960 storage arrays for production and backup storage, respectively; and VMware vSphere 4.1 to virtualize 100% of the server infrastructure for server consolidation, space savings and a full range of other operational and administrative efficiencies.
The high availability features built into some of these solutions are assisting CTCA in meeting their zero-downtime objectives. The Cisco Nexus switches, for example, are built to withstand multiple failures through duplication of critical components and switchover capabilities.
Also, CTCA designed their network with OTV (Overlay Transport Virtualization) technology that extends Layer 2 connectivity across sites, making it possible to automatically fail over and move virtual machines from Madison to Schaumburg when needed for business continuity.
Similarly, VMware's vMotion feature allows virtual machines within the same data center to be moved from one physical server to another, even while doctors or nurses are reading records or inputting data, with no interruption to users.
High Availability Extras
Additional layers of protection are provided with add-on solutions like EMC RecoverPoint and EMC Avamar.
EMC RecoverPoint automatically replicates key data from the primary storage array in Madison to the disaster recovery site in Schaumburg and saves it for 24 hours, providing redundancy in case of a power outage, hardware failure or other service interruption. If there is a problem like database corruption, RecoverPoint has the ability to restore data almost to the point of failure (if a failure happens at 3 pm, for example, the data can be rolled back to 2:59), virtually eliminating data loss.
Once the 24-hour interval has elapsed, EMC Avamar automatically performs backups with advanced deduplication that substantially reduces disk, physical storage and bandwidth needs. Each CTCA hospital also uses Avamar to deduplicate files and transmit backup copies to the Schaumburg site, where they are, in turn, replicated and sent to the primary data center for redundancy.
Finally, in an innovative strategy for building redundancy into the virtual storage system, CTCA became one of the first organizations to use EMC's new VPLEX Metro technology to replicate data between two virtualized EMC storage arrays at the same site-not at two different sites as originally intended by EMC developers. In the CTCA deployment:
-
If one of the two virtual storage arrays in Madison becomes unavailable for any reason, VPLEX will continue to serve new input and output to the server from the second array, as well as log data until the first array is restored and data can be resynchronized.
-
As a result, workload can be actively migrated between arrays in the same data center with zero downtime for end users in the event of a connectivity loss or other failure.
-
This eliminates service interruptions and recovery time by avoiding the need to fail over to CTCA's secondary site in Schaumburg.
"Every day, CTCA patients and their families see the front end of our operations through contact with our clinical staff, hospital buildings and extensive ancillary services, but what goes on behind the scenes in the data center also affects the quality of care. Access to patient information is vital to make a diagnosis, deliver a treatment plan, prescribe medication, check physician orders, perform surgery and more, and we must ensure that access 24x7x365," Eckes notes. "For that reason, every aspect of our new data center was designed to minimize our outage exposure, whether from a failed processor, a faulty fan or power supply, loss of connectivity or any number of other scenarios. We wanted the best technology available to ensure continuous data availability."
Sean Burns is VP of Business Development at Ahead, a Chicago-based IT consultancy specializing in designing and deploying next-generation data centers.
|