The Department of Homeland Security (DHS) invests billions of dollars each year in new technologies such as facial recognition software, X-ray equipment, surveillance aircraft, and U.S. Coast Guard ships.
A Government Accountability Office (GAO) review has found that DHS has strengthened its policies, guidance, and training to encourage testing and evaluation earlier in the acquisition process. However, GAO questioned whether DHS has enough workers with the right skills to adequately oversee test and evaluation activities.
Since August 2010, the DHS Office of Test and Evaluation (OTE) has assessed major acquisition programs’ test results and DHS leadership has used OTE’s assessments to make informed acquisition decisions. Programs generally received approval to progress through the acquisition life cycle, but DHS placed conditions on its approvals in more than half the cases GAO reviewed.
Since May 2017, OTE has updated its policies and released new guidance that met nearly all of the key test and evaluation (T&E) practices GAO identified as contributing to successful acquisition outcomes. For example, OTE’s policy directs program managers to designate a T&E manager who is required to be certified to the highest level in the T&E career field—level III. This met the key practice that programs establish an appropriately trained test team. However, OTE’s guidance partially met the key practice to demonstrate that subsystems work together prior to finalizing a system’s design. Specifically, the guidance instructs programs to conduct integration testing, but not until after the design is finalized. Changes after finalizing design can increase costs or delay schedules.
The review found that DHS faces challenges with its T&E workforce to effectively provide oversight, and most programs do not have a level III certified T&E manager.
OTE also compiles data to monitor programs and the status of T&E managers, but GAO found that this data was unreliable and said OTE had inaccurate data for about half of the programs reviewed.
DHS has expanded OTE’s responsibilities for T&E oversight in recent years. However, OTE officials told GAO that executing these responsibilities has been difficult because DHS has not authorized changes to its federal workforce since 2014. These officials added that they have had to prioritize their oversight efforts to programs actively engaged in testing and, as such, are unable to assist programs that are early in the acquisition life cycle.
GAO is therefore calling for the Under Secretary for Science and Technology to assess OTE’s workforce to ascertain the extent to which it has the appropriate number of staff with the necessary skills to fulfill its responsibilities.
GAO further recommends that T&E policy or guidance is revised as necessary to fully meet the key practice for programs to test that components and subsystems work together as a system in a controlled setting before finalizing a system’s design. The policy or guidance should also specify when in the acquisition life cycle a major acquisition program manager should designate a level III certified T&E manager.
In addition, the watchdog wants to see an assessment process developed and performance measures established to help ensure T&E training achieves desired results as well as an internal control process to ensure that data collected and maintained on major acquisition programs’ T&E managers is reliable.
DHS concurred with all recommendations and intends to implement at least some of this work by the end of September 2020.