By: Wes Riley and Erik Heuser
In twenty plus years navigating the complexities of the information security (InfoSec) industry a common theme emerges: the fascination with creating the digital panacea, or Easy Button. Marketing departments highlight their product in the best light possible and tell you it will solve all your InfoSec headaches. Years of incident response work tells us otherwise. Working with companies – lightly defended to virtual digital fortresses – on their complex intrusions reveals all were penetrated by an external attacker that stole data over the course of weeks, months, or even years. One incident saw a 5+ year continuous attacker dwell time.
Each of these organizations had a myriad of control and monitoring sets with no two alike. We’ve observed various two factor authentication (2FA), host and network based IDS/IPS, and application whitelisting and NextGen AV all overcome. There were multiple engagements where there was no malware used in the intrusion. One compromised partner had their remote access credentials compromised and used to connect to and move about their environment, finding and stealing data. No system devised yet can effectively determine intent. There are attempts and approaches using machine learning to detect malicious behavior and that may yet bear fruit in the future, but there are some very steep hills on that path. The data size and diversity of network traffic and endpoint behavior are the issues at hand. The data size issue is that no single organization can account for all the different activities on networks they don’t personally own and operate. As a security company developing machine learning products, getting enough diverse data to accurately model “the field” and properly train an algorithm is our number one problem. No two companies use the exact same toolsets, network layouts, or monitoring infrastructure, which makes that development very difficult. For example, deploying RSA NetWitness® Packets presents several options that impact the visibility. Are their proxies in the environment? Are they explicit or transparent? Are you using a network TAP or a network SPAN? If so, on which segments? Are you sure your security team has effectively communicated with the network team and found all ingress/egress points?
An effective InfoSec program utilizes a risk-based lifecycle. Risk is used to weight priorities and assign scarce resources to the issue or gap identified. Let’s walk through an approach using the RSA NetWitness Packets, Logs, and Host stack. It may seem a bit different, but some will recognize the approach immediately. Used by the US Army to evaluate the surrounding terrain in a programmatic method, OCOKA (Observation, Cover and Concealment, Obstacles, Key Terrain, and Avenues of Approach) serves as a sort of triage of the environment. This allows you to dedicate more time and resources to the critical portions of the large datasets at your disposal. Additionally, unlike many other security-focused environmental analysis methods, OCOKA focuses on the expectation of attack and the planning of an appropriate response.
- Where is my RSA NetWitness Packets device in relation to the proxy?
- Can I see all the ingress traffic into DMZ’s?
- Are DMZ’s properly segmented or can they create connections into my network?
- Can I observe unencrypted VPN traffic connecting into my network?
- What percentage of hosts are covered by RSA NetWitness® Endpoint?
- Are Internet facing DMZ hosts remotely logging into RSA NetWitness® Logs?
- Am I decrypting SSL/TLS and observing that data?
- Is my retention time shorter than the frequency in which I review network traffic?
- Is there contextual overlap between my observation points?
- Can I correlate observed RDP in network traffic to user logins in Microsoft Windows Event Logs?
- Can I determine what process or imported code or library is causing the network traffic that I am observing?
Cover and Concealment
- Are my Active Directory (AD) connected servers named for their functionality (e.g. nyc.pos1 for a Point of Sale server)? AD oftentimes serves as a roadmap for attackers after they gain an initial foothold and start their internal reconnaissance.
- Are my RSA NetWitness Endpoint services renamed so as not to alert a potential attacker of the presence of an advanced endpoint detection tool?
- Are the security infrastructure management and access interfaces sufficiently segmented from the rest of the network such that only those requiring access can connect to them? Advanced Attackers frequently attempt to subvert security controls because of the amount of visibility and access they provide.
- Have all the default passwords been changed? Has this been verified with scanners and pen-testing?
- What are the limits to my visibility? This could mean seeing aggregated HTTP traffic with an explicit proxy. RSA NetWitness Packets is session-based and an explicit proxy can aggregate multiple connections to different domains in a single session. This is sub-optimal for gaining good network visibility.
- To what extent can I observe DMZ traffic? Can I see inbound from the Internet to the DMZ, traffic originating from the DMZ to the Internet, traffic originating from the DMZ to the internal network?
- What are the specific DMZ firewall policies and is it a flat or multi-tiered DMZ?
- Have I made any assumptions that would cause me to limit my view? Ignoring anything outbound over TCP/443 is a perfect example.
- Can I observe the Active Directory (AD) or LDAP infrastructure via Packets, Logs and Host data?
- Can I observe the 2FA infrastructure via Packets, Logs and Host data?
- Has the organization identified server infrastructure and translated that to feeds in RSA NetWitness Logs & Packets, or machine groups in RSA NetWitness Endpoint?
- Where does the data I’m trying to protect live? Email, POS and Credit Card data, Intellectual Property? What other controls are active to protect them?
- What would the attacker consider ‘Key Terrain’? What systems have high-volume logins, utilize cross-domain or cross-realm trusts, have high-availability and Internet access, or bridge non-equivocal trust segments?
Avenues of Approach
- DMZ’s are popular vectors.
- VPN or remote access applications that listen for connections from the Internet.
- Outbound access policies for client networks and server networks could be a channel for command and control.
- File sharing applications, non-corporate email providers and other similar popular applications including social media can and have been used for command and control.
While not a comprehensive list, these are questions to resolve when analyzing an unfamiliar network. By answering most, if not all, it’s possible to gain a clearer picture of the network and effectively dedicate the proper attention to the important aspects. This serves as the pre-requisite background data needed to use the RSA NetWitness Hunting Guide & Investigation model.
Considering these steps, any ‘Easy Button’ solution appears silly. There are simply too many variables for security to be boiled down to a Boolean True/False decision. Effort must be made to identify the root cause for every piece of malware/malcode discovered. For example, the Carbanak group utilizes the Carberp Trojan for command and control to employ APT style TTP’s to steal credit card and other data related to finance. If an AV alert for a banking Trojan popped up in your AV console, it might trigger a normal reaction to have the machine reimaged or rescanned. This could be a normal criminal group harvesting local banking information or it could be an advanced attacker who has loftier goals than a single individual’s banking information.
And herein lies the second problem: The process does not stop at detection. Organizations have traditionally put a significant amount of resources into prevention. When that consistently failed, significant resources were put into detection. However, the response part of the triangle must be accounted for and invested in. Proper investigation and response must occur to determine the context in which the detected security event took place. Without this, detection is of little use, as either unproportioned, incomplete, or incorrect actions may be taken that put an organization at greater risk. Actions, such as treating advanced malware with the same response as Ad-ware due to the same number of Virus Total (VT) detections, ignoring detections due to the connection coming from a signed Windows binary (false positives), or not realizing that others in the organization were also successfully phished by a certain campaign, can create cascading, systemic failures that can plague an organization for years.
That’s why, when malware is discovered by any tool, investigating with RSA NetWitness Suite will aid a SOC in determining Risk and Intent. Situational Awareness gained with this Suite is invaluable in rapidly assessing Risk and Intent of cyber criminals and nation-state actors. The recent leaks regarding the digital arsenal available to nation-states should be a revelation. The rapid weaponization of these leaks for criminal activities is evidence that a proactive approach to security is orders of magnitude more effective than a passive approach at detecting and mitigating threats.
RSA NetWitness Suite is an abstraction layer presenting verbose data on protocol-specific information, as well as behavioral information, across the Network, Host, and Log domains of an environment. By understanding the environment and constantly reevaluating your security posture anomalous events should bubble to the surface very quickly, allowing an organization to shorten the attacker dwell time. At the end of the day, these attackers are intelligent participants in a never-ending cat and mouse game with security vendors and network defenders, alike. New methods, tools and vectors are constantly being created and reused to accomplish the attacker’s goal, whatever that may be.
Several emerging technologies and tactic improvements make verbose monitoring essential. The number one emerging technological threat is a new instruction set called the Software Guard Extension. This set of CPU instructions allow an application to create a protected enclave in memory that cannot be accessed by debuggers, AV, etc. This will limit the ability of an analyst to analyze a sample statically and dynamically which makes tracking an application’s behaviors and network communications essential.
Attackers are also using anti-forensics tools on machines with an SSD and TRIM enabled. Most file systems don’t remove a file when it’s deleted, it’s simply unlinked. NTFS operates this way and if the file system isn’t in much use, recently deleted files can be recovered with forensics software. An SSD operates differently than a traditional spinning platter disk. The bits in an SSD cell have 3 states instead of the traditional 2. In addition to 1 and 0 they have a NULL state. To change a bit to a 1 or 0 the bits in the cell must be in the NULL state. TRIM goes over the cells in the SSD. When the disk is at a low use state the controller will NULL out cells of bits that have been marked as deleted, or unmapped. This means that with host-based forensics tools it is impossible to retrieve deleted files. Again, the verbose tracking data available in RSA NetWitness Endpoint will show you the attacker’s activities, automatically downloading the tools used and supplying the arguments used, which significantly aids in the investigation at hand.
Attackers properly using encryption can also foil packet analysis of communications with the command and control server. Oftentimes malware employs basic encoding schemas which can be easily reverse engineered with a script that can decode to clear-text the command and control communications. Some malware utilizes actual encryption algorithms, some modified slightly, some directly via API calls. Many times, a static key is somehow obfuscated, but available in the malware making decryption of command and control communications very easy. If the authors take care to not use custom encryption/encoding algorithms, or leave static symmetric algorithm keys in the malware, say by using Diffie-Hellman to dynamically generate symmetric keys, it makes the forensic examiners job much harder. With a verbose tracking system, RSA NetWitness Endpoint does not need the encryption keys to determine what actions an attacker took on a host. RSA NetWitness Packets can be used to examine the volume of encrypted traffic, coupling that with RSA NetWitness Endpoint’s tracking data a forensic examiner can determine what was stolen and verify it with several data points.
From a less sophisticated perspective, the consistent utilization of legitimate, whitelisted operating system binaries and administrative tools to conduct attacker actions cause resident threats to go undetected, ignored, or even incorrectly determined to be legitimate use. Common and popular methods, such as injection into critical operating system binaries, DLL loading via Path Search Ordering, and resident scripting tool utilization (VB/Powershell/Java/Perl/Python/etc.) consistently appear as the legitimate binary conducting the illegitimate action, significantly reducing host-based indicators that incident responders often rely on.
By verbosely monitoring an environment an analyst can piece together actions taken when a security incident occurs. The mindset of 100% prevention is not in line with observations from incidents across many industry verticals in the past few years. The mindset of 100% detection is helpful, but still incomplete in its purpose. Even if a threat is detected, something has to be done about it, and there is a significant amount of context needed to take the correct action. InfoSec is a life cycle and will never be 100% perfect, but with the right tools and processes in place threats can be detected quickly, the appropriate scope of the threat and the context surrounding it can be determined, attacker dwell time can be reduced, remediation can be contextualized, rapid, and thorough, and the overall security posture of an organization can be seen to increase with each attacker engagement.