Thomason Tech Blog

In February of 2012, Thomason Tech helped identify and apprehend a subcontractor working to steal Smart Grid technology from our utility customer in the northwest. He was working on a contract for the utility company who received a government grant. This particular contractor was skilled in his work and in his hacking. He set up a small 4G network device and connected to it and the internal network at the same time. This way, he could gather data from the internal network and send it out via the 4G network. He also did a number of things that would lead an investigation away from him directly. For example, he hacked into a number of his fellow contractors’ laptops and planted pornography and malware that would make it look like they were the culprits. He also used an antiquated network protocol to tunnel information, making it extremely difficult for any system, including a DLP, to identify the malicious activity.

Fortunately, we caught him early enough in the contract. We don’t believe any significant technology developments were stolen.  Interesting is a little more information about the thief. He is of Malaysian decent, but his passport clearly indicated he had spent a lot time training in China. We were not able to determine if he was associated with the  APT1 organization, made famous by Mandiant.

Since that time, we have seen a number of significant attacks, other utility companies have been targeted and a number of major corporations have been compromised. But in recent weeks, there seems to be lull in malicious activity – at least as reported by mainstream media. One might believe  the Chinese have quieted down due to the Mandiant APT1 report. It is true, a lot of malicious activity comes from China, but there are certainly a number of other sources. I worry there is something more devastating coming.

We could now be experiencing the calm before the storm. I would like to take a moment and put myself in the place of a very organized hacking organization – maybe even state sponsored. In this case, I would exclude China as a suspect for political and economic reasons. They have too much to lose. However, a well funded organization or nation state wanting to maximize the economic or environmental impact may be quietly lurking in obscurity right now. Countries like Syria, Pakistan and Iran certainly have motive to hurt the United States. A well placed attack against our financial systems, Wall Street, our utilities or even the Internet infrastructure could have a dramatic effect on the United States economy. Confidence in network communications could be damaged, and online businesses could be heavily impacted.

I’m typically not a preacher of gloom and doom. I believe we can be prepared for such an attack, but the time to do it is now, not after the attack begins, not when our own networks are finally running perfectly. Whether or not your company is experiencing a calm pattern, you need to be preparing for an inevitable storm. Government and industry standards are a good place to start, but you have to keep in mind that all these standards lag behind real, risk-based, security needs. Government or industry standards didn’t require testing for SQL injection attacks until they were rampant. Going back even further, no standards required testing for buffer overflows until those attacks were out of control. Recently, the trends have been toward targeted attacks, spear phishing, advanced persistent threats and Java exploits. How can your company prepare or even be immune from the next wave of attacks?

What will be next? Do you have the ability to monitor all your network traffic? Does your system have the flexibility to be able to address unusual network activity, malware and file analysis? Can you respond to incidents quickly and decisively? Do you have an incident response plan in place? Can you react to data in transit or at rest? How do you know when your sensitive data is leaving your network? Be prepared for the worst and hope for the best.

Recently, one of my customers was able to identify the Java 0-day exploit two full days before it was identified as malware.

Now this customer has been a frequent target of spear phishing attacks. Because of the nature of their business, they are under constant attack and have a sophisticated, capable security team. In fact, they have identified and eliminated an advanced persistent threat. (This was validated by a number of highly reputable, independent organizations including one government office.)

The story begins shortly after I negotiated a 250 seat license for them to use FireAMP for one year. This accompanied a rather large order, so we were able to go beyond the typical 100 seat license.

FireAMP is a host and network based file analysis system. It is NOT AntiVirus, nor is it application whitelisting. Rather, FireAMP takes an inventory of every executable file, stores information about the file in the cloud and then tracks what the file does and where the file goes. FireAMP uses a very lightweight client and provides information back to the user via the cloud. (A system that stores all the information locally is being developed.) The cloud then reports back on whether the file is known malware, known good or unknown. FireAMP remembers where and when every file was seen, so even if the file later becomes known to be malicious, FireAMP can tell you who picked it up first, every system where the file is found and exactly what actions the file took including what it contacted on the Internet.

The Senior Security Consultant on site deployed a handful of the licenses and found the ZeroAccess trojan. Unfortunately, their AV vendor was not alerting on this malware and so it could have gone unnoticed for quite some time.

ZeroAccess has demonstrated the following functionality:

Because of the many ways ZeroAccess can be used, it was determined that additional security measures were needed to protect the company’s critical information. FireAMP was deployed on addition critical assets.

After just deploying on a few systems, it became clear that FireAMP provides a level of visibility into the hosts and their network activity that is not provided by other security tools. For example, any executable that makes network calls is identified including the destination of those calls. FireAMP can correlate a device’s network activity with known malware. Then by using the results, the analyst can search for additional results throughout the install base. FireAMP’s Device Tractory ability can illustrate the history of all files and the network activity they create.

The definition of Advanced Persistent Threat varies dramatically depending on who you ask. The military (most likely the Air Force Computer Emergency Response Team) who coined the term in the 90’s used it to mean only state-sponsored, sophisticated threats without monetary motivation. Today, the term is used more broadly and those sophisticated threats with monetary motivation are included in most definitions. Regardless of where you draw the line, one thing is for sure…When you have one in your network, you have a very serious problem that you probably don’t know about.

Recently, I worked with a client whose business is identified as part of the nation’s critical infrastructure. They had consulted with many professional security organizations about strange network behavior they had discovered, quite by accident. All agreed that the activity looked like an Advanced Persistent Threat. The issue started when it seemed that the network analysis tools were misidentifying the browser running on the latest operating system (a combination that seemed impossible). This turned out to be a false-positive, but chasing this false-positive lead to the discovery of quite a large number of systems that demonstrated other unusual behavior.

Analyzing the hard drive of one particularly noisy system resulted in the identification of a number of pieces of malware that completely evaded the latest version of antivirus software. In addition, it appeared that a very old version of some network management software had been installed so as to be completely hidden from the administrator. Even more surprising was the discovery of a number of executable files that appeared to establish a High-Speed-Token-Ring (HSTR) network.

This was incredible! An attacker was quite possibly setting up an emulated HSTR to run over TCP/IP and completely evade the IPS. The malware looked to be the typical dropper, botnet, and evasion software used to distribute code, communicate back to the C&C and evade detection by antivirus and IPS tools. The biggest difference we found was that the communications to the C&C used a different port (and switched between TCP and UDP) than any of those documented on the usual list of malware detection sites. This allowed it to evade the rule in the IPS that otherwise would have identified the communications.

Before I tell you exactly how we went about catching this APT, let me tell you a little more about some tools that provided varying degrees of help in the process.

• Symantec tools. Symantec antivirus was completely ineffective in identifying any of the malware. Not only did it miss all the droppers and botnet software, but it also missed all the HSTR emulators. Worse yet, we found indications that the malware might be turning off or interrupting all new virus signature downloads. Eventually, we activated the Symantec host-based firewall, but we saw no difference in the activity.

• Network Intelligence – Using Sourcefire’s Real-time Network Awareness (RNA) and Real-time User Awareness (RUA) helped us to identify some suspicious devices, but in the end, it also generated a lot of information that was not accurate enough to be helpful. In fact, it led us to believe the problems we were seeing could be a lot wider spread than they actually were. With this in mind, we would have had nowhere to start had it not been for the intelligence information provided by RNA. Being able to track communication by port number became increasingly more important as the incident wore on.

• DNS Analysis – After pointing all the internal DNS servers to a third party DNS analysis company, they were able to confirm that we had a pretty good infestation of a botnet. Clearly, a number of devices were communicating with suspicious foreign networks. They could also tell us, generally speaking, where the C&C communications were coming from. Unfortunately, they were not able to provide any information about which devices were compromised.

• SEIM – Using the SEIM, an RSA product, we were able to analyze the firewall logs, but this was incredibly slow, time consuming and inefficient. Automatically correlating the IPS events we discovered with firewall logs wasn’t even possible. We had to use a manual process. Essentially, when we saw an event in the IPS, we would search the same time frame on the SEIM, find the firewall logs for that time and try to determine what was able to pass through. Most of the time, this revealed nothing. Every once in a while though, we would find other events that made us look harder. For example, when we saw the same external IP address used in the botnet use a different port, we took note. Unfortunately, sometimes running a single query on the SEIM would take minutes, even hours.

• Intrusion Prevention System – Using Sourcefire’s IPS – based on Snort(TM) – we had some initial indications of which botnet was being used. A time consuming analysis of the firewall logs led us to believe the botnet could be changing ports regularly. Since the IPS was Snort based, we were able to easily modify the existing rules and watch for the botnet activity on other ports. Doing so led us to discover whenever the botnet switched ports. (Another point to make here is that the Sourcefire rule was overly complex. Because it was identifying a specific botnet, it used a rather long string of the C&C banner communications. We discovered this had been modified by the APT and was subsequently evading the Sourcefire rule. Another small change to the “content” string in the Sourcefire rule and we were able to identify more botnet activity with zero false positives.)

We also used a number of network analysis tools like Wireshark and various forensic tools on the hard drives. All of these provided some support to the process and often either confirmed or questioned our hypothesis.

If we had better coverage with our IPS, we might have been able to shut this down a lot faster, but the truth is our IPS was primarily monitoring the perimeter and seeing traffic mostly after passing through a NAT device making identification of the affected systems much more difficult.

After two months plus, it seemed like we were chasing our tail. Other issues seemed to get mixed with the primary incident. The team was losing focus and getting frustrated. Management wanted answers and other department leaders were tired of losing some of their most talented people to work on a security incident that would never end. The security team considered planning a “go-dark” strategy where all outside network connectivity would be cut and literally hundreds of systems would be rebuilt. Then we caught a break…

In what seemed to be an unrelated incident, an external IP address was discovered by the IPS as a source on an internal network. How could this happen? We ventured a number of guesses, but the answer was found in a simple DNS query of the IP address. It was 4G/wifi device connected to a local 4G service provider. It appears the 4G side lost signal and attempted to pass traffic through the internal network. The ramifications of this are bad… really bad, and we didn’t expect it to be related to our incident. We tracked the IP address through the IPS, switch logs and found it in the area where contractors were working on a project funded by a government grant. It was possible that, in a separate incident, confidential documents were being downloaded from the internal network and sent out via the 4G device, by a contractor. After discussing this with our legal team, we determined to thoroughly investigate that host, a laptop, immediately. We thought we put aside our current incident as we likely had a second, possibly even more serious incident to tend with.

Again, the system was a laptop, owned by an employee of the contractor, not company owned. He was not too happy with us taking his laptop (for details on how we got his laptop, picture Tom Cruise hanging from the ceiling in Mission Impossible – not really – we waited until he went to the bathroom) but soon after, we disclosed the reason to the contractor and they were able to appease the employee. But what we found was shocking…

On the laptop were not only many of the same malware products that we were investigating in the original incident, but also all the necessary development tools to modify that malware. A complete version control system, compilers, assemblers, disassemblers, etc. were all present.  Clearly, this was likely the source of our initial compromise and all the suspicious activity we had been seeing for the past two months. The answer came rather quickly. As soon as the system was removed from the network, all of the suspicious activity we had been tracking ceased.

Now for the icing on the cake. The technology being developed through federal funding would clearly be of use to other countries. The individual who owned the laptop was of Malaysian descent and according to a copy of his passport we found on the laptop, he had spent a significant amount of time in China.

APT? – I think so.

This story ends on a positive note, at least we hope. It seems that our APT was on our internal network and possibly had eyes on the development of technology that would be a part of the nation’s critical infrastructure. We were able to remove the threat, stop the activity, but we were never able to positively identify exactly what confidential information was lost. The 4G unit was obviously removed as well, but had it not malfunctioned, finding the source of our incident would have taken much longer.

The implementation of additional IPS sensors and network data access switches now provides a better foundation from which to monitor, analyze and react to serious security incidents. Now, by simply reconfiguring an out-of-band data access switch (supplied by Network Critical) we can monitor or record all traffic at virtually any distribution layer without touching the operational network. Re-engineering the network also eliminated blind spots and moved the contractors to an isolated internal network versus having an unmonitored external network that had access to internal assets. Finally, we now have the flexibility to add monitoring tools such as a network recorder, forensics tools, or basic network analysis tools without having to make changes to the operational network.

Want to know more about the politics, the organization and the operational challenges of catching the APT? Want to know more about the products that proved effective in fighting a difficult security incident? Need help with your own security incident? Contact David Thomason at Thomason Technologies, LLC, www.thomasontech.com, david[at]thomasontech.com.

About Thomason Technologies, LLC

Thomason Technologies, LLC is based in San Antonio, TX and is a proud reseller of best-in-breed security technologies. In addition, Thomason Technologies, LLC provides security consulting services including risk assessments, vulnerability assessments, incident response, product implementation and management services. For more information on Thomason Technologies, see http://www.thomasontech.com.

The past few months have seen a significant increase in “hacktivism”, a word derived from hacking and activism. Loosely organized groups like Anonymous and LulzSec seem to be playing a game of “one-ups”. These groups are so bold as to taunt law enforcement from Twitter and to go after the highest profile targets such as the US Senate and the CIA.

Who loses? We all do. Businesses lose because a loss of personal information, say from a credit card company, erodes confidence and they lose customers. Individuals lose when their personal information is made public or sold to those who would use it illegally. Additionally, money will be spent: to investigate the crimes, to investigate the security posture of the targeted organizations, and finally to add legislation. History tells us this legislation will do little or nothing to stem the tide of hacktivism, but will add costs to businesses, which in turn pass those costs to the consumer. You and me.

At one time, perimeter security seemed to be the dominant solution to prevent attacks. After all, hackers must come from the outside. Today, firewalls aren’t even a speed bump to hackers. Many external firewalls look more like a pegboard. In fact, more than ever the attacks are “client side”– your users visit websites that serve up malware like extra-cheese on your favorite pizza. Spyware, adware and other types of malware are consumed by our users when they visit their favorite gaming or movie download site. Those are just two examples, but malware has been hidden in everything from openly malicious sites to advertisements on the most popular news sites.

Some would claim that our defenses are useless, but the truth is, there are a number of things we can do to either prevent our companies from becoming the next victim or to give us a chance at detecting and responding to the security incident. Let me offer the following four steps as a “quick-start”.

1. Get the basics done. Too many companies today still don’t have a complete grasp on the basic security processes, such as patch management, firewall audits, password changes and protection, security logging, anti-malware updates, etc.
2. Lock down your systems and keep backup images so that systems can be restored quickly in the case of failure or a complete compromise. One of the most secure organizations for whom I have ever done consulting kept a complete set of clean virtual images and compared them nightly against the desktops and servers to make sure nothing changed or was added. If there was any variance, the variance was recorded and copied for forensic purposes and a fresh instance was immediately loaded.
3. Deploy monitoring tools at the perimeter and internally. This would include IPS, web application monitoring, network intelligence tools, next generation firewalls, etc. Many companies will stop after deploying firewalls and IPS at the perimeter or maybe even at a secondary perimeter that protects against partner networks, VPN connections, etc. I say that’s not enough. If you aren’t monitoring internally, you are missing all the traffic passing between your desktops and between desktops and internal servers. Overlooking these links could be a fatal mistake. I’ll explain more later.
4. Train your users. Professionals have known forever that security is only as strong as the weakest link. If security is not a part of your corporate culture, you need to change your culture. Contrary to some opinions, it won’t stifle creativity and won’t hurt productivity for people to understand the importance of security as well as the practical steps to protect corporate information.

The truth is, these may all sound like basics and number 4 should probably be done at the same time as number 1, but too many companies don’t make the effort to get all of these functions done effectively.

For example, let’s take firewall audits as just one part of #1 above. It is difficult for organizations with over 10 firewalls to regularly audit the rules on the firewalls. It is time consuming, complicated and often not very productive. However, with good firewall configurations, many vulnerabilities are completely mitigated. For example, in the typical e-commerce environment, a web front end processes transactions and then sends those transactions to a back end database via an encrypted SQL connection. By using a firewall to prevent any communications except SQL over SSL between the web front end and the database you eliminate virtually all the vulnerabilities except those that can be exploited via that SQL over SSL channel, or locally. (Stay tuned to later blog entries for more on auditing firewalls.)

Locking down system configurations is an important part of security. In the past, we have relied on patch management systems and antivirus products to keep us safe. These products have not evolved fast enough and are horribly inadequate to protect us from the most basic viruses, spyware, adware and other malware. They are even less effective against advanced persistent threats (APT) or even from the dreaded hacktivist. New technologies that provide cloud based software authentication, reputation services and application whitelisting are proving to be significantly more effective and are likely to replace traditional antivirus products. (Again, stay tuned to later blog entries for more information on anti-malware.)

Network monitoring is not an easy task. For mid-sized companies, the process can seem overwhelming. Auditors might require it and yet maintaining the expertise is difficult. However, this is often the most revealing part of the security process. By monitoring the network at the perimeter and internally, you not only identify what attacks are coming at you from the outside, but you also see what your users are doing to compromise security internally. Using IPS and internal network intelligence tools, I have discovered dozens, wait, hundreds if not thousands of systems that were infected with insidious malware. I’ve found multiple laptops that were completely compromised while on the home network and then carried into the corporate environment where they started replicating their maliciousness. I’ve discovered users unknowingly downloading malicious code and I’ve discovered devices inappropriately placed on the internal network. One major key to internal monitoring is proper deployment. A good deployment requires the use of TAPs, also known as Traffic Access Points. TAPs are used for a number of purposes. They copy traffic from a network link to a network monitoring tool. They replicate data from a SPAN port to multiple network tools. They aggregate the information from multiple links to a single tool. They eliminate a single point of failure or reduce downtime by providing an alternate traffic path for inline devices like IPS systems. They convert from 10G to 1G or from 1G to multiple 100Mb links. They can be used to filter traffic, and the list goes on and on. The point here is that with a little engineering workand by using the right taps, you can maximize monitoring capability for your network tools and save a ton of money.

Finally, nothing will provide a better return on investment when it comes to security than training your users. Teaching your users to look for security issues, to recognize poor security practices and to practice good security is essential for any organization to maintain high levels of security.

A final recommendation…don’t wait for government to mandate the measures you take, because by the time they send down a new requirement, the hacktivists have already figured out how to get around it.

NOTE TO THE READER:

Thomason Technologies, LLC is an authorized reseller of Network Critical products.

We represent Network Critical and their line of TAPs for the following reasons:

1. They have an excellent price point
2. Their feature set is second to none
3. Their form factor is the best in the industry, packing the most ports into the smallest rack space
4. Their performance and reliability is best-of breed. In four years of selling their product I have yet to see a TAP fail or be returned for a defect.

Today is a great day as we first start blogging about some of the exciting things that are happening at Thomason Tech.

This past month we completely redesigned our website. We updated a lot of the graphics changed a lot of the text to reflect the new messaging provided by our vendors and we added this blog.

This month we also signed a partnership agreement with Drobo.  Drobo is a fantastic technology for easily doing backups and restores. The product is very affordable for the small to medium sized business, easily upgradable with any size SATA drive (yes, even SATA III is supported.) Everyone from the home based photographer to the middle sized company with huge data requirements (up to 36TBs) should have a Drobo.

We have also announced Network Critical’s latest 10G products. This is the most flexible solution for accessing data on 10G networks. Just off the top of my head it provides the following advantages:

1. Aggregate multiple 1G links to a single 10G monitoring tool
2. Extend the life of existing 1G monitoring tools to monitor underutilized 10G links
3. Configure any port as 1G or 10G and as a monitoring port or tool port.
4. Reduce the number of monitoring tools you need – save money

See more about Network Critical.

Finally, we are very excited here at Thomason Tech with some of the advancements we see at Sourcefire.  (http://www.thomasontech.com/staging/Sourcefire-Products/Sourcefire_3D_System.html) Sourcefire has gone beyond its 3D line of products and is soon to add an enterprise version of AntiVirus with their recently acquired Immunet product. This product is an amazing breakthrough in antivirus technology as it works in the cloud with incredible accuracy. While the product has been available in a consumer product for some time, now it will be available in a centrally managed package that scales to the largest enterprises. The visibility and control provided by Enterprise Immunet is going to be in high demand by those who have felt the pain of failed antivirus solutions.

Sourcefire has also announced they will soon be releasing a Next Generation Firewall (aka NGFW). While we don’t have a lot of information on the NGFW, it is going to work closely with other Sourcefire technologies like Real-time Network Awareness (RNA) and Real-time User Awareness (RUA). That alone makes it pretty exciting. All of this and they are adding the new FirePOWER technology have really made Sourcefire an exciting company to work with.