The Department of Homeland Security (DHS) is required by law to report annually on 43 specific measures of border security effectiveness. DHS reported on 35 of those metrics in its 2017 report and generally used data and methods that should produce reliable results.
However a Government Accountability Office (GAO) review found the report disclosed some, but not all, of the data’s limitations. For example, the report didn’t indicate that different locations may have recorded some border-related data differently.
The United States has approximately 6,000 miles of land borders, 95,000 miles of coastline, and more than 300 ports of entry where travelers and cargo are inspected and processed for entry. Securing U.S. border areas is a key part of DHS’s mission, and the department’s ability to measure its border security efforts is essential for it to manage its responsibilities effectively and efficiently.
The National Defense Authorization Act (NDAA) for Fiscal Year 2017 requires DHS to report annually on 43 border security metrics. DHS issued its first report in May 2018. The Act also includes a provision for GAO, within 270 days of receipt and biennially for the following 10 years, to review and report on the data and methodology contained in DHS’s report. GAO’s March 21 report assesses the extent to which DHS reported metrics as outlined in the NDAA using quality information; and validated assumptions and conveyed statistical uncertainty for unlawful entry metrics, among other objectives.
GAO found that DHS reported on 35 of 43 metrics called for by the NDAA for Fiscal Year 2017 and generally used quality information, but did not identify some data limitations. About half of the 35 metrics generally included elements as called for by the NDAA, while 17 metrics differed, such as in scope or calculation. For example, DHS only provided information on the southwest border for some metrics, such as the estimate of undetected unlawful border crossers for which a methodology for estimating unlawful crossings for the northern border had not yet been completed.
DHS also identified and disclosed limitations for some, but not all, of the data elements and metrics used. For example, GAO found that DHS did not disclose limitations on data related to apprehensions of individuals that were assisted by unmanned aerial systems.
The GAO investigation revealed that DHS used a statistical model to estimate three metrics on unlawful border entries but did not validate some assumptions the model employs through sensitivity analyses and provide measures of statistical uncertainty in accordance with standards for federal agencies. For example, DHS’s model assumes that 100 percent of families unlawfully crossing the border will be apprehended, but DHS did not provide information on the extent to which the assumption affected its metrics. DHS also did not provide information on the level of statistical uncertainty for the metrics, such as margins of error. Providing such information would allow Congress and the public to better understand the potential limitations and accuracy of these metrics of unlawful entry. Additionally, DHS’s statistical model, which is based on Mexican adults not seeking asylum, represents a small and declining share of those apprehended at the border. DHS is developing a new model to account for current border conditions.
GAO concludes that DHS should implement a process to systematically review the reliability of the data used in its Border Security Metrics Report and comprehensively identify any limitations with the data and methodologies that underlie its metrics. It should also ensure the communication of the limitations of the metrics identified through the systematic review in the department’s annual Border Security Metrics Report.
In addition, GAO recommends that the Under Secretary for the Office of Strategy, Policy, and Plans should include the results of sensitivity analyses to key assumptions in its statistical models of unlawful entry estimates in its annual Border Security Metrics Report, and include measures of statistical uncertainty for all metrics based on estimates derived from statistical mode.
DHS concurred with the recommendations and stated that it planned to implement three of these by October 2020. With respect to the second recommendation regarding the communication of the limitations of the metrics, DHS requested that GAO consider it closed as implemented because the department already detailed some of the limitations in its fiscal year 2017 report, and plans to continue to identify known limitations and the progress made to mitigate previously identified limitations in future reports. GAO agrees that DHS identified and disclosed limitations for some metrics in its fiscal year 2017 Border Security Metrics Report; however, it identified at least one additional limitation for 21 of the 35 metrics on which DHS reported that DHS did not disclose or about which it could have been more transparent.