Pester Testing .NET with Copilot
Pester Testing .NET with Copilot
Over the last few weeks, I've been showing how to build PowerShell tools that rely on the .NET Framework instead of calling native PowerShell commands like Get-CimInstance. While you can see performance gains with this approach, there are potential drawbacks. First, your code could be a little more difficult to maintain by others. Native commands are at least easy to understand as long as you use full cmdlet and parameter names. Using the .NET Framework can make your code more arcane, especially for someone with less .NET experience than you. Internal documentation is critical if you take the approach I have been demonstrating.
Another area where you can encounter challenges is with Pester tests. You can only write tests for functions and cmdlets. And within those tests, you can only mock a command. You can't mock invoking a .NET method. I thought it would be helpful to write a Pester test for one of my recent functions. Then I realized I could also demonstrate how you might use GitHub Copilot in VS Code to help you write such a test.
I'll write a test for this function.
Function Get-OSDetail {
[CmdletBinding()]
[OutputType('OSDetail')]
Param(
[Parameter(
Position = 0,
ValueFromPipeline
)]
[Alias('CN', 'Server')]
[ValidateNotNullOrEmpty()]
[Microsoft.Management.Infrastructure.CimSession[]]$CimSession = $Env:ComputerName
)
Begin {
Write-Verbose "Starting $($MyInvocation.MyCommand)"
#define the WQL query
$Query = 'Select CSName,Caption,Version,BuildNumber,InstallDate,OSArchitecture,RegisteredUser,Organization from Win32_OperatingSystem'
#initialize reference variables
New-Variable -Name ci
New-Variable -Name ce
} #begin
Process {
foreach ($cs in $CimSession) {
#capture connection failures to a variable
if ($cs.TestConnection([ref]$ci, [ref]$ce)) {
Write-Verbose "Querying $($cs.ComputerName.ToUpper())"
$data = $cs.QueryInstances('Root/Cimv2', 'WQL', $query)
[PSCustomObject]@{
PSTypeName = 'OSDetail'
Name = $data.Caption
Version = $data.Version
Build = $data.BuildNumber
OSArchitecture = $data.OSArchitecture
RegisteredUser = $data.RegisteredUser
RegisteredOrganization = $data.Organization
InstallDate = $data.InstallDate
ComputerName = $data.CSName
}
}
else {
Write-Warning "Unable to connect to $($cs.ComputerName.ToUpper()). $($ce.Message)"
}
} #foreach
} #process
End {
Write-Verbose "Ending $($MyInvocation.MyCommand)"
} #end
} #end function Get-OSDetail
I'll be using the chat feature in VS Code to build my Pester tests using GitHub Copilot. You may have other AI services that will work equally well, although you might get slightly different results. If you are using something else, I'd love to compare your responses with mine.
Let's dig in.
Prompting
I have found that using GitHub Copilot in VS Code comes down to context. If you have multiple files open, make sure the file you want Copilot to analyze has focus. I opened the script file with the function, opened the chat window, and used this prompt:
Write a Pester test for the Get-OSDetail function. The test should validate all code paths. It should also validate parameter definitions, if the function follows PowerShell scripting best practices, and if the function has comment-based help. The tests should include testing for failures.*
I'll be the first to admit that I am still honing my prompting skills. I got this response.
Based on the provided code snippet, here's how you might write a Pester test for the Get-OSDetail function. This test will validate all code paths, parameter definitions, PowerShell scripting best practices, and comment-based help. It will also include testing for failures.
Here's the test it wrote.
# Import the script containing the Get-OSDetail function
. .\Get-OSDetail.ps1
Describe 'Get-OSDetail' {
# Test if the function exists
It 'Should exist' {
Get-Command Get-OSDetail | Should -Not -BeNullOrEmpty
}
# Test if the function has comment-based help
It 'Should have comment-based help' {
(Get-Help Get-OSDetail).Synopsis | Should -Not -BeNullOrEmpty
}
# Test if the function follows PowerShell scripting best practices
# This is a very broad requirement and might need more specific tests
# Here's an example of testing if the function uses approved verbs
It 'Should use approved verbs' {
(Get-Command Get-OSDetail).Verb | Should -BeIn (Get-Verb).Verb
}
# Test parameter definitions
# The function has a parameter named 'CimSession'
It 'Should have the correct parameters' {
(Get-Command Get-OSDetail).Parameters.Keys | Should -Contain 'CimSession'
}
# Test all code paths
# This will depend on the specific logic of your function
# Here's an example of testing if the function returns the correct OS details
It 'Should return the correct OS details' {
$session = New-CimSession
$result = Get-OSDetail -CimSession $session
$os = Get-CimInstance -ClassName Win32_OperatingSystem
$result.Name | Should -Be $os.Caption
$result.Version | Should -Be $os.Version
$result.OSArchitecture | Should -Be $os.OSArchitecture
$result.RegisteredUser | Should -Be $os.RegisteredUser
$result.RegisteredOrganization | Should -Be $os.Organization
$result.InstallDate | Should -Be $os.InstallDate
$result.ComputerName | Should -Be $os.CSName
}
# Test for failures
# This will depend on the specific logic of your function
# Here's an example of testing if the function throws an error when given a null CimSession
It 'Should throw an error when given a null CimSession' {
{ Get-OSDetail -CimSession $null } | Should -Throw
}
}
I am assuming you have some basic experience writing and using Pester v5 tests. But let's assume I know very little. Let's save this to a file, get-osdetail.tests.ps1, and try it out.
That didn't go well. This is a great example of where AI makes a handy assistant but it can't do all your work for you. Copilot wrote a test that would work with Pester v4. Pester v5 changed the test structure. To remind myself, I created a sample test file.
PS C:\temp> New-Fixture -Name Get-Fooby
I need to see the test file.
BeforeAll {
. $PSCommandPath.Replace('.Tests.ps1', '.ps1')
}
Describe "Get-Fooby" {
It "Returns expected output" {
get-fooby | Should -Be "YOUR_EXPECTED_VALUE"
}
}
The first change I need to make then is how the function is loaded for the test. I need to change the legacy dot sourcing with the BeforeAll statement.
Modifying the Response
One thing I have discovered is that Pester is case-sensitive when it comes to file names. I initially tried copying and pasting the BeforeAll statement from the sample to my test but it failed. My file name is all lowercase so this works.
BeforeAll {
. $PSCommandPath.Replace('.tests.ps1', '.ps1')
}
Now the tests run.
Even though all the tests pass, it isn't entirely accurate and at least one test is irrelevant.
This test serves no purpose.
It 'Should exist' {
Get-Command Get-OSDetail | Should -Not -BeNullOrEmpty
}
The purpose of the Describe block is to test the function. I don't need to verify it exists.
Then there's this test.
It 'Should have comment-based help' {
(Get-Help Get-OSDetail).Synopsis | Should -Not -BeNullOrEmpty
}
I wanted to validate that the function had comment-based help. The current version of the function does not so I would have expected this to fail. But it doesn't. That's because PowerShell will auto-generate help and the Synopsis property will default to the syntax. I can revise the test to look for the Description property.
It 'Should have comment-based help' {
(Get-Help Get-OSDetail).Description | Should -Not -BeNullOrEmpty
}
I'll save the test and re-run it. Now the test fails as expected which is a good thing. I'll take a moment to drop in some comment-based help into the script file.
Function Get-OSDetail {
<#
.Synopsis
Get operating system details via CIM
.Description
Use this command to query one or more remote computers using CIM to get operating
system details.
.Example
PS C:\> Get-OSDetail
Name : Microsoft Windows 11 Pro
Version : 10.0.22635
Build : 22635
OSArchitecture : 64-bit
RegisteredUser : Jeff
RegisteredOrganization :
InstallDate : 5/17/2022 2:54:52 PM
ComputerName : THINKX1-JH
.Link
Get-CimInstance
#>
[CmdletBinding()]
...
Now the test passes as expected.
I wanted to make sure the command follows PowerShell scripting best practices. The response mentions that is a broad request. Although it will verify the command is using an approved verb.
It 'Should use approved verbs' {
(Get-Command Get-OSDetail).Verb | Should -BeIn (Get-Verb).Verb
}
I suppose the other best practice I want is to verify the command isn't using aliases. I think testing for that requires using the AST to analyze the code. Instead of digging through my script library and searching for AST-related examples, I'll see what Copilot can come up with.
Using the Get-OSDetail function, write a Pester test assertion using the AST that verifies that command doesn't contain any command aliases.
This looks promising.
Here is a Pester test that uses the Abstract Syntax Tree (AST) to verify that the
Get-OSDetailfunction does not contain any command aliases.
Describe 'Get-OSDetail' {
It 'Does not contain any command aliases' {
$functionDefinition = Get-Command Get-OSDetail | Select-Object -ExpandProperty Definition
$ast = [System.Management.Automation.Language.Parser]::ParseInput($functionDefinition, [ref]$null, [ref]$null)
$commandAst = $ast.FindAll({
param($ast)
$ast -is [System.Management.Automation.Language.CommandAst] -and
$ast.InvocationOperator -eq [System.Management.Automation.Language.TokenKind]::Infix -and
(Get-Command -Name $ast.GetCommandName() -ErrorAction SilentlyContinue).CommandType -eq 'Alias'
}, $true)
$commandAst.Count | Should -Be 0
}
}
This test uses the
FindAllmethod of the AST to find all instances ofCommandAstthat are invoked as aliases (i.e., theirInvocationOperatorisInfix). It then checks if the command type of these instances is 'Alias'. The test passes if no such instances are found.
I'll copy and paste the It assertion into my test file. The test passes because I'm not using any aliases. Let me insert some into the function to validate the test. I'll replace New-Variable with its alias nv.
Unfortunately, Copilot failed to write a proper assertion. There may be other explanations, but the InvocationOperator is always Unknown when I manually run the AST code. And even then, according to my research this [System.Management.Automation.Language.TokenKind]::Infix doesn't exist.
I'm back to writing code for the test. Although the generated test gives me a head start.
It 'Does not contain any command aliases' {
$functionDefinition = Get-Command Get-OSDetail | Select-Object -ExpandProperty Definition
$ast = [System.Management.Automation.Language.Parser]::ParseInput($functionDefinition, [ref]$null, [ref]$null)
$commandAst = $ast.FindAll({
param($ast)
$ast -is [System.Management.Automation.Language.CommandAst] -AND $ast.CommandElements.StringConstantType -eq 'BareWord' -AND (Get-Alias -name $ast.CommandElements.Value -ErrorAction SilentlyContinue)
}, $true)
$commandAst.Count | Should -Be 0
}
With my test aliases in place, I get the failures I expected.
It is important that when writing Pester tests you make sure your test succeeds and fails as you expect.
I'll remove the aliases from the script.
Parameters
I asked for a test on parameters. Again, maybe I needed to be more specific.
Write a pester test assertion for the CimSession parameter. It should be at position 0, it should accept pipeline input by value, is should be of the type Microsoft.Management.Infrastructure, it should accept multiple values, it should default to the local computer name and it should have parameter aliases of CN and Server.
Writing out the test helps me organize my thoughts and generates useful code. And even though I had typos in my prompt (I fixed them here), the Copilot knew what I meant and corrected my mistakes.
Describe 'Get-OSDetail' {
Context 'CimSession parameter' {
$parameter = (Get-Command Get-OSDetail).Parameters['CimSession']
It 'Is at position 0' {
$parameter.Position | Should -Be 0
}
It 'Accepts pipeline input by value' {
$parameter.ValueFromPipeline | Should -BeTrue
}
It 'Is of type Microsoft.Management.Infrastructure.CimSession[]' {
$parameter.ParameterType.FullName | Should -Be 'Microsoft.Management.Infrastructure.CimSession[]'
}
It 'Accepts multiple values' {
$parameter.ParameterType.IsArray | Should -BeTrue
}
It 'Defaults to the local computer name' {
$parameter.DefaultValue | Should -Be $Env:ComputerName
}
It 'Has parameter aliases of CN and Server' {
$parameter.Aliases | Should -Contain 'CN'
$parameter.Aliases | Should -Contain 'Server'
}
}
}
I even get a nice Context block which I'll copy into my script file and replace the existing parameter test.
Unfortunately, the test fails. Again, the issue seems to be that Copilot doesn't know about the changes in Pester v5. I need to make this change.
BeforeAll {
$parameter = (Get-Command Get-OSDetail).Parameters['CimSession']
}
I still have failures:
But they appear to be related to the assertion. As before, I can manually run the code the Pester test is trying to run. Right away, I can see issues based on what $parameter looks like:
Name : CimSession
ParameterType : Microsoft.Management.Infrastructure.CimSession[]
ParameterSets : {[__AllParameterSets,
System.Management.Automation.ParameterSetMetadata]}
IsDynamic : False
Aliases : {CN, Server}
Attributes : {System.Management.Automation.ValidateNotNullOrEmptyAttribute,
System.Management.Automation.AliasAttribute, ,
System.Management.Automation.ArgumentTypeConverterAttribute}
SwitchParameter : False
There is no Position property here so of course $Parameter.Position -eq 0 will fail. I have to dig a little to get it.
PS C:\> $parameter.Attributes
TypeId
------
System.Management.Automation.ValidateNotNullOrEmptyAttribute
System.Management.Automation.AliasAttribute
System.Management.Automation.ParameterAttribute
System.Management.Automation.ArgumentTypeConverterAttribute
PS C:\> $parameter.Attributes.Where({$_.typeID.name -match 'ParameterAttribute'})
ExperimentName :
ExperimentAction : None
Position : 0
ParameterSetName : __AllParameterSets
Mandatory : False
ValueFromPipeline : True
ValueFromPipelineByPropertyName : False
ValueFromRemainingArguments : False
HelpMessage :
HelpMessageBaseName :
HelpMessageResourceId :
DontShow : False
TypeId : System.Management.Automation.ParameterAttribute
I can also see where I need to make a few other changes.
It 'Is at position 0' {
$parameter.Attributes.Where({$_.typeID.name -match 'ParameterAttribute'}).Position | Should -Be 0
}
It 'Accepts pipeline input by value' {
$parameter.Attributes.Where({$_.typeID.name -match 'ParameterAttribute'}).ValueFromPipeline | Should -BeTrue
}
It 'Is of type Microsoft.Management.Infrastructure.CimSession[]' {
$parameter.ParameterType.FullName | Should -Be 'Microsoft.Management.Infrastructure.CimSession[]'
}
It 'Accepts multiple values' {
$parameter.ParameterType.IsArray | Should -BeTrue
}
I also need to fix the default value test. The suggested solution $parameter.DefaultValue does not exist. I tried asking Copilot for a different test. That failed. I even asked Windows Copilot in Windows 11 but it gave me unrelated answers. Finally, I resorted to using the AST again.
Write a Pester 5 test to validate the default parameter value for the CimSession parameter in Get-OSDetail using the AST.
Describe 'Get-OSDetail' {
It 'CimSession parameter defaults to the local computer name' {
$functionDefinition = Get-Command Get-OSDetail | Select-Object -ExpandProperty Definition
$ast = [System.Management.Automation.Language.Parser]::ParseInput($functionDefinition, [ref]$null, [ref]$null)
$paramAst = $ast.Find({
param($ast)
$ast -is [System.Management.Automation.Language.ParameterAst] -and
$ast.Name.VariablePath.UserPath -eq 'CimSession'
}, $true)
$defaultValueAst = $paramAst.Attributes | Where-Object { $_.TypeName.Name -eq 'DefaultValue' }
# This will only work if the default value is a static value, not a runtime value or environment variable
$defaultValue = $defaultValueAst.PositionalArguments[0].Value
$defaultValue | Should -Be $Env:ComputerName
}
}
By this point, I am smart enough to manually try the code in the test, which fails. There is no type name of a parameter attribute that matches DefaultValue. It is wishful thinking. But there is a DefaultValue property.
VariablePath : Env:ComputerName
Splatted : False
StaticType : System.Object
Extent : $Env:ComputerName
Parent : [Parameter(
Position = 0,
ValueFromPipeline
)]
[Alias('CN', 'Server')]
[ValidateNotNullOrEmpty()]
[Microsoft.Management.Infrastructure.CimSession[]]$CimSession
= $Env:ComputerName
I have a few options I could test for.
PS C:\> $paramast.DefaultValue.variablepath.UserPath
Env:ComputerName
PS C:\> $paramast.DefaultValue.Extent.text
$Env:ComputerName
Here's my revised assertion.
It 'CimSession parameter defaults to the local computer name' {
$functionDefinition = Get-Command Get-OSDetail | Select-Object -ExpandProperty Definition
$ast = [System.Management.Automation.Language.Parser]::ParseInput($functionDefinition, [ref]$null, [ref]$null)
$paramAst = $ast.Find({
param($ast)
$ast -is [System.Management.Automation.Language.ParameterAst] -and
$ast.Name.VariablePath.UserPath -eq 'CimSession'
}, $true)
# This will only work if the default value is a static value, not a runtime value or environment variable
$defaultValue = $paramAst.DefaultValue.Extent.text
$defaultValue | Should -Be '$Env:ComputerName'
}
Looking good thus far.
Summary
At this point, I think I've done a decent job, with Copilot's help, of defining a Pester test that validates the command inputs. I can tell from looking at the test that I will need to rewrite a test to validate the output. Also, as the test stands now, Pester is invoking the function and executing the code just as if I were running it interactively. The test is not mocking anything. The whole point of a Pester unit test, which is what I am writing, is to validate the code without actually executing it. This is where I expect re-factoring will come into the picture. But I'll save all of this for next time.