PowerShell Category Streams
In this issue:
- Defining Categories
- Parsing Commands
- Parsing Scripts with the AST
- Writing to Alternate Data Streams
- Retrieving from Alternate Data Streams
- Creating Tools
- Next Steps
- Summary
Last year I wrote a series of articles on using the Abstract Syntax Tree (AST) to analyze PowerShell script files and create metadata about them. One topic that I intended to cover was a process for automatically categorizing scripts based on their content. I thought I would cross that off my list by writing about it now.
Even if you are not interested in categorizing scripts, you might find some of the techniques I use interesting and potentially useful in other PowerShell projects.
Defining Categories
My initial goal is to organize PowerShell commands into groups or categories. Then, I can identify commands in a PowerShell script and assign one or more categories to the script. Once I have that information, I can use it however I want. I want to write to the file's alternate data stream. This will require NTFS which means Windows. The categorizing process should work cross-platform, but writing to the alternate data stream will be Windows-only.
The category information is data. Which means it should be separated from the code that processes it. You could use a JSON or XML file to store the category definitions, but I decided to use a PowerShell data file. These are .psd1 files that contain a hashtable. In my design, thee keys are the category names, and the values are arrays of command names.
# script-category.psd1
@{
HyperV = 'Get-VM', 'Get-VMNetworkAdapter', 'Get-VMProcessor', 'Get-VMMemory', 'Get-VMHardDiskDrive', 'Get-VMNetworkAdapterVlan', 'Get-VMIntegrationService', 'Get-VMFirmware', 'Get-VMReplication', 'Get-VMReplicationAuthorizationEntry', 'Get-VMReplicationConnection', 'Get-VMSnapshot', 'Export-VM', 'Import-VM', 'New-VHD', 'Mount-VHD', 'Dismount-VHD', 'Convert-VHD'
Data = 'ConvertTo-Json', 'ConvertFrom-Json', 'ConvertTo-Csv', 'ConvertFrom-Csv', 'ConvertTo-Xml', 'ConvertFrom-Xml' , 'Export-Csv', 'Import-Csv', 'Import-PowerShellDataFile', 'ConvertFrom-StringData', 'Import-LocalizedData'
CIM = 'Get-CimInstance', 'New-CimSession', 'Get-CimSession', 'Remove-CimSession', 'Invoke-CimMethod', 'Set-CimInstance', 'New-CimInstance', 'Get-CimClass', 'Get-CimAssociatedInstance'
WMI = 'Get-WmiObject', 'Invoke-WmiMethod', 'Set-WmiInstance', 'Register-WmiEvent', 'Remove-WmiObject'
Remoting = 'Enter-PSSession', 'Exit-PSSession', 'New-PSSession', 'Get-PSSession', 'Remove-PSSession', 'Test-WSman'
Files = 'Get-ChildItem', 'Add-Content', 'Set-Content', 'Remove-Content', 'Out-File'
Eventing = 'Register-WmiEvent', 'Get-Event', 'Register-CimIndicationEvent', 'Register-EngineEvent', 'Register-ObjectEvent', 'Unregister-Event', 'Get-EventSubscriber'
Web = 'ConvertTo-Html','Invoke-WebRequest', 'Invoke-RestMethod', 'WebClient'
Jobs = 'Start-Job', 'Get-Job', 'Stop-Job', 'Remove-Job', 'Wait-Job'
ScheduledJobs = 'Add-JobTrigger', 'Disable-JobTrigger', 'Disable-ScheduledJob', 'Enable-JobTrigger', 'Enable-ScheduledJob', 'Get-JobTrigger', 'Get-ScheduledJob', 'Get-ScheduledJobOption', 'New-JobTrigger', 'New-ScheduledJobOption', 'Register-ScheduledJob', 'Remove-JobTrigger', 'Set-JobTrigger', 'Set-ScheduledJob', 'Set-ScheduledJobOption', 'Unregister-ScheduledJob'
Scripting = 'Add-Member', 'Update-TypeData', 'Update-FormatData', 'New-ModuleManifest', 'Test-ModuleManifest', 'Trace-Command', 'Set-PSBreakpoint', 'Remove-PSBreakpoint', 'Get-PSBreakpoint', 'Remove-TypeData', 'Disable-PSBreakpoint', 'Get-Member', 'New-Guid'
Management = 'Get-Process', 'Stop-Process','Start-Process','Get-Service', 'Start-Service', 'Stop-Service', 'Restart-Service', 'Set-Service', 'New-Service', 'Remove-Service', 'Get-EventLog', 'Get-WinEvent', 'Get-ACL', 'Get-SmbShare', 'New-SmbShare', 'Set-SmbShare', 'Remove-SmbShare', 'Invoke-Command'
Critical = 'Invoke-Expression', 'Add-Type', 'Suspend-BitLocker'
Security = 'Get-ACL', 'Set-ACL','Get-CmsMessage','Protect-CmsMessage', 'Unprotect-CmsMessage','ConvertFrom-SecureString','ConvertTo-SecureString','Get-Credential','Set-ExecutionPolicy',' Get-PfxCertificate','Set-AuthenticodeSignature','Get-AuthenticodeSignature'
}
I can load this data file into a variable in my script like this:
$categories = Import-PowerShellDataFile C:\scripts\script-category.psd1
Each key is a category name, and each value is an array of command names.
PS C:\> $categories.web
ConvertTo-Html
Invoke-WebRequest
Invoke-RestMethod
WebClient
Parsing Commands
For now, I'm going to test with an array of command names. Eventually I'll build this list from the script file using the AST.
$cmds = 'Get-WinEvent', 'Invoke-Command', 'ConvertTo-Json', 'Write-Host', 'Out-Null'
My plan is to create another hashtable where the keys are the category names and the values are arrays of commands from the script that belong to that category. I'll start with an empty hashtable.
#initialize a hashtable
$categoryHash = @{}
Next, I can iterate through the command names and find the category for each command. If the category already exists in the hashtable, I can add the command to the array. If not, I can create a new array with that command.
foreach ($item in $cmds) {
$category = $categories.GetEnumerator().Where({ $_.Value -contains $item }).Key
#add commands as explicit arrays
if ($category -and $categoryHash.ContainsKey($category)) {
$categoryHash[$category] += $item
}
elseif ($category) {
#add the value as an array
$categoryHash.Add($category, @(,$item))
}
}
At the end I have a hashtable that shows which commands from the script belong to which categories.
PS C:\> $categoryHash
Name Value
---- -----
Management {Get-WinEvent, Invoke-Command}
Data {ConvertTo-Json}
What you do with this information is up to you. Because I'm going to store it in a data stream, I need to convert this to text. JSON is a good choice for this.
PS C:\> [PSCustomObject]$categoryHash | ConvertTo-Json
{
"Management": [
"Get-WinEvent",
"Invoke-Command"
],
"Data": [
"ConvertTo-Json"
]
}
If you were going to save this information externally, you might want to include the script path as well.
$categoryHash.Add("Path","c:\scripts\foo.ps1")
Parsing Scripts with the AST
Now that I have the categorization mechanics worked out, the next step is to parse a script file and extract the command names. This is done using the Abstract Syntax Tree (AST). The AST can be created using the static ParseFile() method of the System.Management.Automation.Language.Parser class.
I'll define the path to my sample script.
$path = 'C:\scripts\sample-script.ps1'
