The Log4j vulnerability “somewhat surprisingly” has had impacts that were less than feared but exposed organizational challenges in cyber threat response including resources, confusion and even “patching fatigue,” according to the first report from the Cyber Safety Review Board.
The Cyber Safety Review Board was established in February per President Biden’s May 2021 executive order Improving the Nation’s Cybersecurity. The panel is led by DHS Under Secretary for Policy Rob Silvers and co-chaired by Google Vice President of Security Engineering Heather Adkins. The 15-member board is a mix of public- and private-sector representatives.
“To advance the overall security and resiliency of our digital ecosystem, we applaud the application of this highly effective lessons-learned model from other industries,” Silvers and Adkins wrote in a message at the beginning of the report. “With the discovery of major software vulnerabilities and gaps in our capabilities to effectively mitigate them, we believe this effort will help drive improvements in our overall cyber resiliency.”
The board undertook its first review with the Log4j open-source software vulnerability that was disclosed in December 2021. “Log4j is very broadly used in a variety of consumer and enterprise services, websites, and applications—as well as in operational technology products—to log security and performance information,” CISA said in its guidance. “An unauthenticated remote actor could exploit this vulnerability to take control of an affected system.” In January, CISA officials said there had been no major intrusions yet in the United States tied to the flaw, with CISA Director Jen Easterly warning that “we do expect Log4j to be used in intrusions well into the future.”
Silvers and Adkins said the review of the Log4j event “presented the Board with several challenges,” including “no crash site or damaged vehicle to inspect, no stress tests to perform on failed equipment, and no wiring diagrams to review.”
“Instead, we reviewed practices used to create and adopt technology, ecosystems, and processes. We relied on subject matter experts in open source software, its development, deployment, and maintenance,” they said. “In our discussions, we observed enthusiasm for tackling these issues, and also the community’s desire to have a more fulsome picture of where vulnerabilities lie, which of them are exploitable, and the effectiveness of remediations.”
The co-chairs stressed that the Log4j event is not over. “Log4j remains deeply embedded in systems, and even within the short period available for our review, community stakeholders have identified new compromises, new threat actors, and new learnings,” they wrote. “We must remain vigilant against the risks associated with this vulnerability, and apply the best practices described in this review.”
Silvers and Adkins also emphasized “a real need to drive widespread development and adoption of capabilities, tooling, and automated frameworks that support developers with the daunting task of building secure software.”
“Just as the software industry has enabled the democratization of software programming—the ability for anyone to generate software with little or no formal training—we must also democratize security by baking security by default into the platforms used to generate, build, deploy, and manage software at scale,” they said.
The pair noted that industry was especially helpful in supporting the review with insights and data, adding that they “believe industry has come to understand that the Board is not an enforcement or regulatory body and is not focused on assigning blame.”
After the Log4j vulnerability was first reported, network defenders “faced a particularly challenging situation; the vulnerability impacted virtually every networked organization and the severity of the threat required fast action,” the report said.
“The fact that there is no comprehensive ‘customer list’ for Log4j, or even a list of where it is integrated as a sub-system, hindered defender progress. Enterprises and vendors scrambled to discover where they used Log4j,” it continued. “The pace, pressure, and publicity compounded the defensive challenges: security researchers quickly found additional vulnerabilities in Log4j, contributing to confusion and ‘patching fatigue’; defenders struggled to distinguish vulnerability scanning by bona fide researchers from threat actors; and responders found it difficult to find authoritative sources of information on how to address the issues. This culminated in one of the most intensive cybersecurity community responses in history.”
The review board noted that the response included “high levels of cooperation, extensive use of social media for rapid sharing of mitigation advice, innovative response actions from the Department of Homeland Security’s (DHS) Cybersecurity and Infrastructure Security Agency (CISA), and the creation of new shared community resources.” Organizations that responded most effectively “understood their use of Log4j and had technical resources and mature processes to manage assets, assess risk, and mobilize their organization and key partners to action.” However, executing a response with the necessary speed was found to be a challenge for most organizations, as when the Apache Software Foundation made upgrades for Log4j available “deploying them was itself a risk decision, forcing a tradeoff between possible operational disruption and timeliness, completeness, and compensating controls.”
One federal cabinet department reported that it spent 33,000 hours on Log4j vulnerability defense. The board noted that “these costs, often sustained over many weeks and months, delayed other mission-critical work, including the response to other vulnerabilities.”
“For the long term, we will continue to see the tension between our collective need for crisis-driven risk management and the foundational investments that would support more rapid response for future incidents,” the report said. “Perhaps most significantly, the force exerted on the urgent response and the challenges in managing risk also contributed to professional ‘burnout’ among defenders that may, compounded with the generally intense pace of many cybersecurity jobs, have a long-term impact on the availability of cybersecurity talent.”
The review found that “somewhat surprisingly” to date “exploitation of Log4j occurred at lower levels than many experts predicted, given the severity of the vulnerability.”
“It has been difficult to arrive at this conclusion,” the report added. “While cybersecurity vendors were able to provide some anecdotal evidence of exploitation, no authoritative source exists to understand exploitation trends across geographies, industries, or ecosystems. Many organizations do not even collect information on specific Log4j exploitation, and reporting is still largely voluntary.”
“Most importantly, however, the Log4j event is not over. The Board assesses that Log4j is an ‘endemic vulnerability’ and that vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer. Significant risk remains.”
Log4j “also called attention to security risks unique to the thinly-resourced, volunteer-based open source community,” the report said, and “it is essential that public and private sector stakeholders create centralized resourcing and security assistance structures that can support the open source community going forward.”
The board’s first group of recommendations involves addressing continued risks of Log4j: Organizations “should be prepared to address Log4j vulnerabilities for years to come” and “should continue to report (and escalate) observations of Log4j exploitation.”
“CISA should expand its capability to develop, coordinate, and publish authoritative cyber risk information,” the report continued. “Federal and state regulators should drive implementation of CISA guidance through their own regulatory authorities.”
To drive existing best practices for security hygiene, the report recommends that organizations “invest in capabilities to identify vulnerable systems,” “develop the capacity to maintain an accurate information technology (IT) asset and application inventory,” have “a documented vulnerability response program” and “a documented vulnerability disclosure and handling process,” and “software developers and maintainers should implement secure software practices.”
The recommendations focused on building a better software ecosystem state that “open source software developers should participate in community-based security initiatives,” while organizations should “invest in training software developers in secure software development,” “improve Software Bill of Materials (SBOM) tooling and adoptability,” “increase investments in open source software security,” and “pilot open source software maintenance support for critical services.”
The final set of recommendations from the Cyber Safety Review Board focuses on investments in the future, recommending that organizations “explore a baseline requirement for software transparency for federal government vendors,” “examine the efficacy of a Cyber Safety Reporting System (CSRS),” “explore the feasibility of establishing a Software Security Risk Assessment Center of Excellence (SSRACE),” “study the incentive structures required to build secure software,” and “establish a government-coordinated working group to improve identification of software with known vulnerabilities.”
“The impact to organizations over the long term will be difficult to assess without better tools for discerning real exploitation and centralized reporting of successful compromises,” the report states.
