Vulnerability Intelligence. The untold story of the Solarwinds and Centreon compromises by nation-state hackers.
Which other IT Monitoring solution has been hacked by nation-state actors like UNC2452, Dark Halo or Sandworm?
Disclaimer: The views, methods, and opinions expressed at Anchored Narratives are those of the author and do not necessarily reflect the official policy or position of my employer.
Introduction
As shared in earlier reporting, listening on social media platforms for relevant threats indicates what is popular amongst security researchers and nation-state cyber activity. An interesting study from 2019 revealed that social media activity like Twitter and Reddit could be used to accurately predict software exploit development on GitHub, a platform to develop software.1
One of those relevant “trigger keywords” is “CVE-” to monitor these relevant software exploits. Those letters stand for Common Vulnerabilities and Exposures. This system is maintained by the National Institute Standards of Technology (NIST) that maintains a database of software vulnerabilities.2 So when a vulnerability has been found, a unique number per year will be assigned with a severity level, a so-called CVSS score. The vulnerabilities are scored on several factors, for example, if hackers can locally or remotely exploit the vulnerability, if the user must be authenticated, etcetera. CVE-2020-14005, a Solarwinds software vulnerability, scored a 9, which means hackers could remotely exploit it. Since December 2020, a lot of chatter on critical vulnerabilities from Solarwinds on social media took place. This, of course, had everything to do with an advanced breach that was announced by FireEye, which also triggered security researchers.
This article will look back on the Solarwinds and Centreon breaches' initial reporting by setting the scene. Both vendors deliver IT Monitoring solutions, and we will assess if these breaches are connected. By leveraging Vulnerability Intelligence data in a simulated attack scenario, it is possible to identify additional IT Monitoring vendors as likely targets in a larger nation-state intrusion campaign. What kind of strategy did the actors take to compromise these companies, and were only Centreon and Solarwinds picked? And finally, it will be closed off with the conclusion and some learnings.
Setting the scene
On December 8, 2020, Kevin Mandia, CEO of the well-known cybersecurity vendor FireEye, reported that nation-state hackers broke into their network and obtained their Red Team testing tools. They immediately shared Indicators of Compromise (IOC) as they were concerned that their test hacking tools might be used against other companies. His initial statement, however, did not disclose how these nation-state actors broke into their network. In a statement made on 13 December 2020, it was revealed that a malicious software update from a known vendor, called Solarwinds, provided access to their environment but also the networks of many other companies who utilized the Solarwinds Orion software package. Solarwinds is a US company based in Austin, Texas, and develops and sells IT Monitoring software solutions for many large companies and governments. One of their top software products is the Orion Platform and used by many global companies and governments.
Although FireEye provided some details, they did not release a full forensic root cause report on what actually happened on their network and how they detected this stealthy actor, dubbed UNC2452. Their statement reports about working together with the FBI and Microsoft. It remains unclear if FireEye was notified of the breach by one of them. To the author, this is likely, because if they detected it themselves, they would have shared how they did it and with what kind of product, I assume. FireEye, the company that released the famous APT1 report detailing that Chinese nation-state actors were massively breaching companies worldwide, is now less transparent.
More than 18.000 customers used Solarwinds Orion. Another cybersecurity company, called Volexity, shared more details. Their report on 14 December referred to a similar actor, but they called them Dark Halo. Their report disclosed that the actor was active in three attacks against a think tank institute starting in 2019 to July 2020.
After the initial disclosures, many cybersecurity companies shared their insights on the very sophisticated threat actor. According to a joint statement from the Cybersecurity and Infrastructure Security Agency (CISA), the FBI, NSA alleged a nation-state actor, “likely Russian in origin,” that hacked the Solarwinds company already in 2019.
Then on February 15, 2020, The French National Agency for the Security of Information Systems (ANSSI) released a statement that “they were informed” of an intrusion campaign targeting the monitoring software called Centreon, distributed by a French company that holds the same name. According to their website, the French company develops and delivers IT Monitoring solutions to many enterprises and multinational customers worldwide. The campaign resulted in the breach of several French entities. The first victim seems to have been compromised from late 2017. The campaign lasted until 2020. Their report dated 27 January 2021 referred to a known Russian nation-state actor, dubbed Sandworm, which might be tight to Russia's GRU military intelligence agency. That outfit is held accountable for the power grid blackouts in Ukraine and the famous NotPetya case. In the NotPetya case, the Sandworm group hacked a tax accounting software company in Ukraine called M.E.Doc. In that case, the actor rolled out a malicious software update, of tax software that Ukraine organizations and foreign organizations doing business with Ukraine are obliged to use, resulting in a mass ransomware attack which led to a massive outage of Maersk, the container shipping company.
Geostrategy, cyberwar objectives, and a potential attack scenario
As pointed out in, earlier reporting nation-states have certain objectives to hack other nations. This can be an economic driver, technological but it can also have a military interest. Geopolitics, the global competition among the so-called Super Powers, nations, and its global companies, is fierce, relentless, and drives the geostrategy of a nation, the subsequent development and execution of cyberattacks by nations.
Let’s assume you would be the commanding officer of a nation-state cyber actor group from a certain country that gets directed to obtain a constant and reliable flow of strategic information of competing nation-states from an economic, political, or military perspective. So what would you be your strategy? Hack away? No, you would develop a plan with certain objectives and likely scenarios to ascertain persistent access to networks to meet that goal.
First, of course, the team you work in already has a professional hacking background and experience from earlier Computer Network Exploitation (CNE) assignments. For any new plan or direction, one of the first steps is to obtain relevant intelligence of certain market solutions that can provide persistent access to meet the directive of the mission. Keep in mind that these nation-state crews are usually consisting of multiple teams that are specialists in different fields and well-funded. One team is responsible for obtaining initial access to a network by exploiting known vulnerabilities or not known vulnerabilities, so-called zero-days. Others are responsible for expanding access or malware development. Some are responsible for obtaining and maintaining the hacking infrastructure. This also means different skill-sets that are needed by such teams. Crucial skill-sets are vulnerability research, exploit development, and malware development. Due to the complex nature, talent is scarce, and team members with these skills are in high demand.
So why would these nation-state actor groups pick the Centreon and Solarwinds IT monitoring solutions? That is what I find the most interesting piece of these planned cyber campaigns. IT monitoring solutions like Solarwinds have great visibility and access to large corporate or government networks. Many IT monitoring solutions also store credentials to monitor relevant network devices or services. So from an attacker’s perspective, they are ideal as a pivot point into networks and obtain a strong and persistent foothold in the compromised network. Furthermore, security monitoring versus availability monitoring functions are usually separated functions in larger organizations and seem to have less interest in each other.
To pick Solarwinds and Centreon, could it have been the result of a brainstorming session or an earlier successful experience with such a solution to do an inventory of the most popular IT monitoring solutions in the market? Also, keep in mind that many of these hackers are testing security applications for fun and profit. So let’s do a bit of digging on these IT Monitoring solutions. An internet search revealed the following Top IT Monitoring solutions.
List Of The Top System Monitoring SoftwareSolarWinds Server and Application Monitor
eG Innovations
NinjaRMM
Datadog
PRTG
Zabbix
Spiceworks Network Monitor
Nagios
OpManager by ManageEngine
WhatsUp Gold
Cacti
Icinga
OpenNMS
Note that Centreon is not listed in this top 13 from January 2021.
A likely next step the actor group would take is to obtain intelligence about those solutions from vulnerability research and exploit development perspective and determine who the clients are of these IT Monitoring solutions. This is crucial information from a hacking perspective to further mature the approach to get persistent access to correct networks. To do this from a vulnerability perspective, the nation-state actors could leverage the National Vulnerability Database (NVD) maintained by NIST to understand potential attack surfaces.
Let’s assess what kind of information is publicly available off some of the solutions listed above. In this article, I will use cve-search, a tool that can be leveraged to request and analyze the NVD locally. Of course, make sure that the local database is synced daily with the central NVD to obtain the most accurate view. Let’s start with Solarwinds.
Solarwinds Vulnerability Intelligence
By searching on the keyword ‘solarwinds,” you will receive an overview of vulnerabilities that match Solarwinds. A nation-state actor, but also regular pentesters or red teamers, would like to understand how many vulnerabilities have been reported by the security community and with what kind of severity. Actors are particularly interested in high CVSS scores as these usually provide them direct or remote access. The following command was executed to obtain an overview:
:~/cve-search$ ./bin/search.py -p solarwinds -o csv >solarwinds.csvIn total, 124 CVE-numbers on Solarwinds products were reported. The first recorded vulnerability was from 2001, CVE-2001-0054, affecting the FTP Serv-U, a product to transfer files. The last one from 2021 is CVE-2021-25276 and also affects the same product.
It is interesting to see an increase in reported vulnerabilities in 2015, 2020, and especially in 2021. As the chatter increased on Twitter media after disclosing the Solarwinds breach in December, this also resulted in more reported vulnerabilities. Several by the breach itself and several reported vulnerabilities we found after the breach by security researchers that gained an interest in their products.
By filtering on the “Serv-U” product, the following overview of 42 reported vulnerabilities can be found from 2005 until 2021 with many critical CVSS scores (>7 CVSS).
When assessing the Solarwinds Orion Platform, 21 vulnerabilities were reported from 2010 until 2021, also many highly critical vulnerabilities. An overview is displayed below.
Analysis of the reported vulnerabilities demonstrates that many of the Solarwinds products have had critical software vulnerabilities, also repetitive, in the same product.
A similar insight could be retrieved from Centreon.
:~/cve-search$ ./bin/search.py -p centreon -o csv >centreon.csvThey had 59 vulnerabilities reported over the period of 2007 until 2020.
Vulnerability Intelligence on the Top IT Monitoring vendors
By applying the same logic, the following overview of the Top IT Monitoring was generated. Keep in mind that Solarwinds scores relatively high because the search revealed multiple of their software products.
By quickly reviewing this information, many of the Top IT Monitoring vendors have had multiple critical vulnerabilities over a longer period of time in the same products. This is, of course, nothing new as software evolves as new functionality is being added, but it might be a sign of poor security developing standards. That said, a lot of vulnerabilities scored really high and were easily exploitable. I noticed many so-called SQL injections, authentication bypass flaws, command injections, and remote code executions. The exploit code or the proof of concepts are widely available on the internet from many critical vulnerabilities. Many of the top IT Monitoring solutions are based on open-source, like Centreon, Zabbix, Nagios, and Cacti, indicating potential smaller security monitoring budgets. Open-source could also support exploit developers in developing new vulnerabilities.
By obtaining such an overview, the nation-state actor group understands which products contain critical vulnerabilities and have exploit codes. By examining these critical vulnerabilities and the accompanying proof of concept codes, these well-funded and advanced actors can find additional security flaws in these products if the known released exploits do not work first. Keep in mind that software updating cycles in larger networks are usually managed and slow. Sometimes for good business continuity reasons. The Centreon company released a statement on February 16 that the potential Sandword nation-state actors breached old versions of their software.
Developing the potential attack scenario
We now have obtained essential starting information about the IT Monitoring vendors. The vulnerability information of the Top IT Monitoring solutions reveals multiple other interesting candidates for an attack scenario.
Solarwinds
Centreon
Nagios
Cacti
Zabbix
OpManager by ManageEngine
Icinga
By assessing the potential customers via their commercial website, the actors get an insight into the networks they can access if they establish a strong foothold. By bringing it all together, the initial access can be further matured via multiple attack paths. The options listed below are feasible to meet the initial economic, political or military objective to obtain strategic and persistent access. I have only listed several attack paths via the Internet are listed below and excluded physical attack paths.
A direct attack on the infrastructure, applications, and employees, via phishing, of the IT Monitoring solution. Separate teams with isolated attack infrastructure would execute such an attack on the selected IT Monitoring solutions. Attackers want to avoid the impact of the discovery of their operations. Each attack is likely to start by doing a reconnaissance of the IT-infrastructure and leveraging open-source information to find all potential vulnerabilities, misconfiguration, or even exposed passwords. This attack method has been observed in the Solarwinds breach, where it became known that access could easily be obtained to an update server with the password ‘solarwinds123’. If access cannot be obtained via regular hacking, the nation-state actors will develop or use a zero-day if access cannot be obtained or via any other means. Once the initial access has been obtained, the actors will establish their persistence mechanisms and expanding their access, and finally, move to the software development environments. In the Solarwinds and M.E.Doc breaches, the actors focused on the software updating mechanism to finally obtain stealthy access to the customers' networks.
Another path could be by compromising the product directly at the strategic target via the known reported critical vulnerabilities, as reported by Centreon, or a newly developed zero-day vulnerability and expand access from there.
Indirect attack on the open-source repositories of several IT Monitoring solutions by inserting backdoored functions in the code.
Conclusion
In December 2020 and January 2021, much chatter occurred on Twitter around the Solarwinds breach and vulnerabilities. It resulted in many additional reported vulnerabilities in the National Vulnerability Database after the breach was disclosed. Last week the Centreon breach was disclosed. Will we also observe an increase in the reported critical vulnerabilities on their platform in the coming period, as predicted in the reference research paper?
From an industry perspective, I personally hope that FireEye, and others as well, will fully disclose what actually happened in the attack and what defenders can learn from it. Looking at a potential timeline of the UNC2452/Dark Halo actor campaign and the immediately announced involvement of Microsoft and the FBI by FireEye, I suspect that FireEye and Microsoft were both informed by the FBI. Then ANSSI announced in February that “they were informed” of a breach on Centreon. It remains unclear who informed them. Is this announcement an outcome of the Solarwinds investigation? The fact remains that the similarity of the nation-state actors' modus operandi at play, Sandworm, and UNC2452 or Dark Halo, are both known to utilize the software updating mechanism to expand access to achieve strategic access.
By doing the actor simulation scenario exercise, multiple other IT Monitoring solutions were identified that experienced critical vulnerabilities in their products for many years. They could become an easy target for nation-state actors or were also part of such a nation-state campaign. Besides the already compromised Solarwinds and Centreon, Nagios, Cacti, and Zabbix raise severe concerns from an actor perspective. Future will tell if any of the aforementioned nation-state actors also targeted these companies. Make these IT monitoring deployments part of regular pentesting or red teaming exercises. Based upon historically reported vulnerabilities, it seems likely that additional vulnerabilities will be found.
Vulnerability information should also be used as input in threat hunting exercises. How does your own IT (Monitoring) landscape look like from a vulnerability perspective? Any other software solutions (with agents) running in your network environment and have had many critical vulnerabilities in the past years could be misused, like in the Solarwinds breach?
To the author, it is unclear if secure coding practices are already applied at these companies. A brief assessment of the recently identified vulnerabilities does not look like that, but I have not researched it.
I feel an opportunity to proactively leverage this information and score software vendors on their vulnerability program maturity. How fast are they patching the identified vulnerabilities? Are they implementing secure development practices? This vulnerability information should be part of any risk assessment or software procurement department from a client perspective. And, of course, design and implement your networks on the zero-trust principle.
Feel free to reach out if you have any comments or feedback.
S. Horawalavithana, 2019 Mentions of Security Vulnerabilities on Reddit, Twitter and GitHub, website:https://www.cse.usf.edu/dsg/data/publications/papers/wi19_sameera.pdf
CVE defines a vulnerability as:
"A weakness in the computational logic (e.g., code) found in software and hardware components that, when exploited, results in a negative impact to confidentiality, integrity, or availability. Mitigation of the vulnerabilities in this context typically involves coding changes. Still, it could also include specification changes or even specification deprecations (e.g., removal of affected protocols or functionality in their entirety)."