The OIG evaluated a power outage that occurred at the Hines Information Technology Center on May 4, 2023. The outage lasted approximately 22 hours and adversely affected more than 10,000 VA employees nationwide, preventing them from accessing critical VA data and systems such as compensation, pension, and education benefits applications.The OIG found the Hines center’s physical access controls were generally adequate but did not have an effective physical control to prevent the activation of a circuit breaker that caused an inadvertent outage at the data center. Also, contrary to design standards that require redundant power distribution paths, the Hines center circuit breaker functions as a master power switch between the uninterruptible power supplies and the information technology equipment. Consequently, when an authorized employee activated the circuit breaker on May 4, 2023, electricity stopped flowing to the data center equipment and critical applications the center hosts. The OIG did not find evidence the employee intended to disrupt power, nor did it make a determination as to whether the incident was accidental.The Hines center also did not have a detailed contingency plan to guide staff in the recovery of facility information systems following a power outage. When the power outage occurred, engineering staff did not coordinate with the Office of Information and Technology and did not correctly prioritize restoration of network devices, prolonging the system downtime.Such power outages could delay veterans and their families in receiving benefits processed by staff using the affected VA data and applications. Therefore, the OIG recommended the Office of Information and Technology provide redundant electrical distribution paths, cover and add warnings to circuit breakers at Hines and other core data centers, and develop and test a detailed contingency plan to reduce system downtime in the event of a power outage.
Report Date | Agency Reviewed / Investigated | Report Title | Type | Location | |
---|---|---|---|---|---|
Department of Veterans Affairs | Evaluation of the May 2023 Power Outage at the Hines Information Technology Center in Illinois | Review | Agency-Wide | View Report | |
Social Security Administration | Office of the Inspector General Audit Recommendations that Had Not Been Implemented as of March 22, 2024 | Other | Agency-Wide | View Report | |
National Aeronautics and Space Administration | NASA’s Compliance with the Payment Integrity Information Act for Fiscal Year 2023 | Audit | Agency-Wide | View Report | |
National Archives and Records Administration | Audit of NARA’s Compliance with the Payment Integrity Information Act of 2019 for Fiscal Year 2023 | Audit | Agency-Wide | View Report | |
Department of the Treasury | BILL AND COIN MANUFACTURING: The Mint Needs to Enhance Controls over Gold Acquisitions | Audit | Agency-Wide | View Report | |
Department of Defense | Audit of the Defense Digital Service Support of DoD Programs and Operations | Audit | Agency-Wide | View Report | |
General Services Administration | Restricted Memorandum on GSA System Access Control | Other | Agency-Wide | View Report | |
Department of Homeland Security | DHS' Fiscal Year 2023 Compliance with the Payment Integrity Information Act of 2019 | Audit | Agency-Wide | View Report | |
Office of Personnel Management | Audit of the U.S. Office of Personnel Management's Compliance with the Payment Integrity Information Act of 2019 | Audit | Agency-Wide | View Report | |
U.S. Postal Service | U.S. Postal Service Emergency Preparedness: Hurricane Ian | Audit | Agency-Wide | View Report | |