On July 1, 2021, CU*Answers completed a bare-metal restoration of their production core-processing system in a test environment as part of their Disaster Recovery Program. With a robust and mature high availability environment hosting CU*BASE for more than 190 credit unions spanning data centers located 750 miles apart, preparing for catastrophic scenarios is not overlooked.

Every six months, teams at CU*Answers conduct a high availability rollover to direct live production of CU*BASE to the secondary data center. After a period of up to three weeks, production is redirected back to the primary site. “Regular HA rollovers give us the confidence we need, knowing what to expect when disruptive incidents occur,” states Jim Lawrence, VP of Operations and Business Continuity. “Worst-case scenario planning requires you to design and test tertiary recovery plans, should the primary and secondary systems fail.”

While performed less frequently than HA rollovers, restoring systems from backup tape media requires a series of well-documented and orchestrated processes. The timing of this restoration test followed a change in tape encryption vendors to keep up with changes in security and performance enhancements. “In a constantly changing IT environment, regular testing is critical for validating procedures and confirming recovery time objectives. Almost as quickly as the data on the tape becomes stale, so do the procedures to restore it,” says Jim Lawrence.

After debriefing and collection of recovery test notes, teams at CU*Answers will publish a gap analysis report for credit unions to review detailing the event, including challenges observed and lessons learned. Reports from all recent high availability rollover and disaster recovery tests are available on the corporate website.