The definition of Advanced Persistent Threat varies dramatically depending on who you ask. The military (most likely the Air Force Computer Emergency Response Team) who coined the term in the 90’s used it to mean only state-sponsored, sophisticated threats without monetary motivation. Today, the term is used more broadly and those sophisticated threats with monetary motivation are included in most definitions. Regardless of where you draw the line, one thing is for sure…When you have one in your network, you have a very serious problem that you probably don’t know about.
Recently, I worked with a client whose business is identified as part of the nation’s critical infrastructure. They had consulted with many professional security organizations about strange network behavior they had discovered, quite by accident. All agreed that the activity looked like an Advanced Persistent Threat. The issue started when it seemed that the network analysis tools were misidentifying the browser running on the latest operating system (a combination that seemed impossible). This turned out to be a false-positive, but chasing this false-positive lead to the discovery of quite a large number of systems that demonstrated other unusual behavior.
Analyzing the hard drive of one particularly noisy system resulted in the identification of a number of pieces of malware that completely evaded the latest version of antivirus software. In addition, it appeared that a very old version of some network management software had been installed so as to be completely hidden from the administrator. Even more surprising was the discovery of a number of executable files that appeared to establish a High-Speed-Token-Ring (HSTR) network.
This was incredible! An attacker was quite possibly setting up an emulated HSTR to run over TCP/IP and completely evade the IPS. The malware looked to be the typical dropper, botnet, and evasion software used to distribute code, communicate back to the C&C and evade detection by antivirus and IPS tools. The biggest difference we found was that the communications to the C&C used a different port (and switched between TCP and UDP) than any of those documented on the usual list of malware detection sites. This allowed it to evade the rule in the IPS that otherwise would have identified the communications.
Before I tell you exactly how we went about catching this APT, let me tell you a little more about some tools that provided varying degrees of help in the process.
- Symantec tools. Symantec antivirus was completely ineffective in identifying any of the malware. Not only did it miss all the droppers and botnet software, but it also missed all the HSTR emulators. Worse yet, we found indications that the malware might be turning off or interrupting all new virus signature downloads. Eventually, we activated the Symantec host-based firewall, but we saw no difference in the activity.
- Network Intelligence – Using Sourcefire’s Real-time Network Awareness (RNA) and Real-time User Awareness (RUA) helped us to identify some suspicious devices, but in the end, it also generated a lot of information that was not accurate enough to be helpful. In fact, it led us to believe the problems we were seeing could be a lot wider spread than they actually were. With this in mind, we would have had nowhere to start had it not been for the intelligence information provided by RNA. Being able to track communication by port number became increasingly more important as the incident wore on.
- DNS Analysis – After pointing all the internal DNS servers to a third party DNS analysis company, they were able to confirm that we had a pretty good infestation of a botnet. Clearly, a number of devices were communicating with suspicious foreign networks. They could also tell us, generally speaking, where the C&C communications were coming from. Unfortunately, they were not able to provide any information about which devices were compromised.
- SEIM – Using the SEIM, an RSA product, we were able to analyze the firewall logs, but this was incredibly slow, time consuming and inefficient. Automatically correlating the IPS events we discovered with firewall logs wasn’t even possible. We had to use a manual process. Essentially, when we saw an event in the IPS, we would search the same time frame on the SEIM, find the firewall logs for that time and try to determine what was able to pass through. Most of the time, this revealed nothing. Every once in a while though, we would find other events that made us look harder. For example, when we saw the same external IP address used in the botnet use a different port, we took note. Unfortunately, sometimes running a single query on the SEIM would take minutes, even hours.
- Intrusion Prevention System – Using Sourcefire’s IPS – based on Snort(TM) – we had some initial indications of which botnet was being used. A time consuming analysis of the firewall logs led us to believe the botnet could be changing ports regularly. Since the IPS was Snort based, we were able to easily modify the existing rules and watch for the botnet activity on other ports. Doing so led us to discover whenever the botnet switched ports. (Another point to make here is that the Sourcefire rule was overly complex. Because it was identifying a specific botnet, it used a rather long string of the C&C banner communications. We discovered this had been modified by the APT and was subsequently evading the Sourcefire rule. Another small change to the “content” string in the Sourcefire rule and we were able to identify more botnet activity with zero false positives.)
We also used a number of network analysis tools like Wireshark and various forensic tools on the hard drives. All of these provided some support to the process and often either confirmed or questioned our hypothesis.
If we had better coverage with our IPS, we might have been able to shut this down a lot faster, but the truth is our IPS was primarily monitoring the perimeter and seeing traffic mostly after passing through a NAT device making identification of the affected systems much more difficult.
After two months plus, it seemed like we were chasing our tail. Other issues seemed to get mixed with the primary incident. The team was losing focus and getting frustrated. Management wanted answers and other department leaders were tired of losing some of their most talented people to work on a security incident that would never end. The security team considered planning a “go-dark” strategy where all outside network connectivity would be cut and literally hundreds of systems would be rebuilt. Then we caught a break…
In what seemed to be an unrelated incident, an external IP address was discovered by the IPS as a source on an internal network. How could this happen? We ventured a number of guesses, but the answer was found in a simple DNS query of the IP address. It was 4G/wifi device connected to a local 4G service provider. It appears the 4G side lost signal and attempted to pass traffic through the internal network. The ramifications of this are bad… really bad, and we didn’t expect it to be related to our incident. We tracked the IP address through the IPS, switch logs and found it in the area where contractors were working on a project funded by a government grant. It was possible that, in a separate incident, confidential documents were being downloaded from the internal network and sent out via the 4G device, by a contractor. After discussing this with our legal team, we determined to thoroughly investigate that host, a laptop, immediately. We thought we put aside our current incident as we likely had a second, possibly even more serious incident to tend with.
Again, the system was a laptop, owned by an employee of the contractor, not company owned. He was not too happy with us taking his laptop (for details on how we got his laptop, picture Tom Cruise hanging from the ceiling in Mission Impossible – not really – we waited until he went to the bathroom) but soon after, we disclosed the reason to the contractor and they were able to appease the employee. But what we found was shocking…
On the laptop were not only many of the same malware products that we were investigating in the original incident, but also all the necessary development tools to modify that malware. A complete version control system, compilers, assemblers, disassemblers, etc. were all present. Clearly, this was likely the source of our initial compromise and all the suspicious activity we had been seeing for the past two months. The answer came rather quickly. As soon as the system was removed from the network, all of the suspicious activity we had been tracking ceased.
Now for the icing on the cake. The technology being developed through federal funding would clearly be of use to other countries. The individual who owned the laptop was of Malaysian descent and according to a copy of his passport we found on the laptop, he had spent a significant amount of time in China.
APT? – I think so.
This story ends on a positive note, at least we hope. It seems that our APT was on our internal network and possibly had eyes on the development of technology that would be a part of the nation’s critical infrastructure. We were able to remove the threat, stop the activity, but we were never able to positively identify exactly what confidential information was lost. The 4G unit was obviously removed as well, but had it not malfunctioned, finding the source of our incident would have taken much longer.
The implementation of additional IPS sensors and network data access switches now provides a better foundation from which to monitor, analyze and react to serious security incidents. Now, by simply reconfiguring an out-of-band data access switch (supplied by Network Critical) we can monitor or record all traffic at virtually any distribution layer without touching the operational network. Re-engineering the network also eliminated blind spots and moved the contractors to an isolated internal network versus having an unmonitored external network that had access to internal assets. Finally, we now have the flexibility to add monitoring tools such as a network recorder, forensics tools, or basic network analysis tools without having to make changes to the operational network.
Want to know more about the politics, the organization and the operational challenges of catching the APT? Want to know more about the products that proved effective in fighting a difficult security incident? Need help with your own security incident? Contact David Thomason at Thomason Technologies, LLC, www.thomasontech.com, david[at]thomasontech.com.
About Thomason Technologies, LLC
Thomason Technologies, LLC is based in San Antonio, TX and is a proud reseller of best-in-breed security technologies. In addition, Thomason Technologies, LLC provides security consulting services including risk assessments, vulnerability assessments, incident response, product implementation and management services. For more information on Thomason Technologies, see http://www.thomasontech.com.