Open-source intelligence (OSINT), as defined by the SANS Institute, refers to the process of producing intelligence through collecting, evaluating and analyzing publicly available information. This is done with the purpose of answering specific intelligence questions. Its origins trace back to just before World War II, when monitoring radio broadcasts and news outlets was crucial for military and political decision-making. Over time, OSINT has evolved in scope and complexity and is now an integral part of intelligence operations across government, military, and private sectors.
In November 1989, Mark Schwalm, Senior Vice President of Business Development at Babel Street, was an Air Force Special Operations intelligence officer stationed at Clark Air Base in the Philippines. His story in this Q&A highlights two pivotal moments in the use of OSINT and its evolution.
In the late 1980s, OSINT was still manual and rudimentary, relying on physical documents, such as maps, hotel brochures, and reports from the Foreign Broadcast Information Service (FBIS). During the November 1989 coup attempt in Manila, these basic tools helped then Captain Schwalm and other military personnel assess the situation and plan for the possible evacuation of Americans.
By March 2012, the technology had transformed significantly. At that time, now Schwalm was an executive for a start-up technology company when he witnessed the use of social media platforms such as Twitter for near-instant collection of real-time intelligence during a coup in Bamako, Mali.
Schwalm’s journey from the early days of manual intelligence gathering to the present, with the arrival of artificial intelligence (AI) and big data, captures the profound changes in the evolution of OSINT collection, processing, and analysis. Now, with the high volumes of data being collected, AI and other technologies are force multipliers, compiling and processing data at faster-than-human speeds while improving human decision-making accuracy — enabling the ability to extract actionable insights at scale.
In this Q&A, Mark Schwalm walks readers through the evolution of OSINT and how new technologies are shaping the future of modern intelligence operations.
- What was the role of the Crisis Action Team (CAT) during the 1989 coup attempt in Manila?
When a coup happens in a foreign country in which Americans live or frequent, one of the first orders of business is locating Americans in the event they need to be evacuated. This was one role we undertook as the chaos erupted throughout the country.
Prior to the November 1989 coup attempt in Manila, we assembled a Crises Action Team (CAT) with representatives from operations, plans, intelligence, and communications from the 353rd Special Operations Wing located at Clark Air Base in the Philippines. Our CAT would tie into 13th Air Force’s team and begin feeding vital updates to Special Operations Command, Pacific (SOCPAC).
- How was open-source intelligence (OSINT) gathered during the coup attempts in the late 1980s?
From the outset, the 1989 coup in Manila seemed better organized and coordinated than others we had lived through.
Because this wasn’t our first such operation, we had already done some initial planning for getting Americans out of Manila. This included collecting hotel brochures, local maps, and ground and aerial photos of hotels where American citizens frequented. These kinds of items, combined with Foreign Broadcast Information Service (FBIS) reports, were the foundation for our OSINT.
The coup progressed and a Joint Special Operations Task Force was established in our facility with SOCPAC deployed forward, and the Defense Intelligence Agency even deployed one of their National Intelligence Support Teams (NIST). While we had unprecedented access to intelligence resources, ironically it was a Green Beret in a Manila hotel room relaying spot intelligence reports via a landline phone that provided the most timely updates to aid our situational awareness.
- What significant change in intelligence gathering did you witness between the 1980s and 2012?
By March 2012, I had traded my camouflage for a business suit and was one of the first employees at a technology start-up that created an open-source, automated network analytics tool that measures influence within the analysts’ search criteria. One day, we were collecting and analyzing information coming out of Bamako, Mali, as a military coup d’etat unfolded. Instead of talking with the Special Forces Sergeant reporting Philippine rebel movements from his hotel, I now had access to a virtual sensor network with a variety of voices and perspectives. These sources were posting on Twitter and other Internet sources, in real-time and describing what was happening on the ground in Bamako—priceless information!
Essentially, technology had enabled us to collect and report near-real time intelligence on what was happening during a military coup d’etat. Twitter had effectively become the world’s greatest mass communications tool as it was the driver behind the Arab Spring a year before and was gaining traction as a medium for sharing ideas (both good and bad) worldwide.
- What technological changes have you seen in the years since 2012?
My career has seen many dramatic changes in the world, but with AI embedded in intelligence tools, retrieving and interpreting vital details will happen faster and cleaner, enabling problem solving to happen at an unprecedented pace and level.
Since 2012, I’ve been in the open-source, big data technology world of the Internet and social media. I’ve witnessed how OSINT has become even more integral to how the world works.
Like other forms of intelligence – signal intelligence (SIGINT), imagery intelligence (IMINT), human intelligence (HUMINT), etc. – the key is still finding and isolating the valuable, actionable information amongst all the noise. And, while sources of data have exploded, the number of hours in a day haven’t, and analysts and investigators need capabilities that make them more efficient and effective. This is why I was ecstatic when, in 2020, Babel Street acquired Dunami, where I was working as an executive vice president, to combine the industry leading data and language company with leading-edge analytics.
Later, Babel Street enhanced its already strong offerings by incorporating Rosette’s advanced natural language processing (NLP) and Vertical Knowledge’s custom data collection capabilities. This powerhouse of data, analytics, and insights makes us a valuable resource for today’s Air Force and many other organizations.
- What are some of the challenges modern intelligence analysts face despite having access to a vast amount of data?
Experts have predicted that by 2025 over 463 exabytes of data will be generated each day globally. Not only are we seeing exponential growth in the volume of data, we are also experiencing growth in the veracity and the variety of data. This is being compounded by the fact that, with every new app added around the globe there is an endless evolution in emerging data formats.
Both government and commercial entities need better and faster ways to extract intel from all this data. Capabilities like ChatGPT have begun to revolutionize how we work and find needles in the haystacks of data. However, like any code or program, these extraordinary resources can tunnel our attention and reduce our workloads. However, we still need humans to review and verify the results. We are currently experimenting with our own version of ChatGPT via our Insights tool and have seen some positive results when trying to condense or summarize large numbers of documents.
- Your story provides a real-life view of OSINT’s evolution. Based on your experience, where do you see OSINT going?
It’s been roughly 35 years since that Manila coup attempt and OSINT’s evolution has been remarkable. We have unprecedented access to information now, but OSINT is still considered to be somewhat of a stepchild in the Intelligence Community (IC).
In The IC OSINT Strategy 2024-2026, the subtitle is The INT of First Resort: Unlocking the Value of OSINT is an encouraging sign. However, while the document’s Strategic Focus Areas layout a solid plan, I learned from my time at the Pentagon that “vision without funding is hallucination.” The IC must commit resources to attain this vision and not be so locked into their INT that they do not see how OSINT can benefit their area of focus.
If that happens, I can see OSINT becoming perhaps as good as, if not greater than, the other mainstream INTs within the next decade.
The evolution of OSINT from the 1989 Manila coup to the present illustrates a remarkable journey, as told by someone who lived it, from manual data collection and analysis to advanced, technology-driven processes.
Initially, OSINT focused on gathering basic open-source materials. Since then, real-time data collection and analysis have made OSINT an even more effective resource. Going forward, OSINT continues to integrate more sophisticated technologies and methods, with the aim of providing a clearer and more actionable intelligence picture for decision making.