Evaluation of the May 2023 Power Outage at the Hines Information Technology Center in Illinois

The OIG evaluated a power outage that occurred at the Hines Information Technology Center on May 4, 2023. The outage lasted approximately 22 hours and adversely affected more than 10,000 VA employees nationwide, preventing them from accessing critical VA data and systems such as compensation, pension, and education benefits applications. The OIG found the Hines center’s physical access controls were generally adequate but did not have an effective physical control to prevent the activation of a circuit breaker that caused an inadvertent outage at the data center. Also, contrary to design standards that require redundant power distribution paths, the Hines center circuit breaker functions as a master power switch between the uninterruptible power supplies and the information technology equipment. Consequently, when an authorized employee activated the circuit breaker on May 4, 2023, electricity stopped flowing to the data center equipment and critical applications the center hosts. The OIG did not find evidence the employee intended to disrupt power, nor did it make a determination as to whether the incident was accidental. The Hines center also did not have a detailed contingency plan to guide staff in the recovery of facility information systems following a power outage. When the power outage occurred, engineering staff did not coordinate with the Office of Information and Technology and did not correctly prioritize restoration of network devices, prolonging the system downtime. Such power outages could delay veterans and their families in receiving benefits processed by staff using the affected VA data and applications. Therefore, the OIG recommended the Office of Information and Technology provide redundant electrical distribution paths, cover and add warnings to circuit breakers at Hines and other core data centers, and develop and test a detailed contingency plan to reduce system downtime in the event of a power outage.