Toolmaking Toolmaking
At first glance, you might think I made a mistake with the title of today's article. But I didn't. I want to continue sharing my process in building an archive searching or summary solution for this newsletter. I expect that when I am finished I will have a set of PowerShell scripts or functions that I can use to generate the archive summary. However, I am finding that I need tools to help me build my tools.
This is not that unusual. A carpenter needs a hammer to build a house. A developer needs a text editor to write code. Someone has to create those tools. Typically, a carpenter doesn't make their own hammer. But PowerShell professionals can certainly create the tools they need to make their tools.
I thought I'd share my toolmaking toolmaking experiences with this project.
Email Objects
In the last several articles, I've shared code that I'm using to get published email newsletters using the Buttondown API. I've been saving them to an XML file using Export-Clixml so that I don't have to keep hitting the API. I can load the XML file and work with the data.
Today, I'm going to import that data and save it in a generic list.
$entries = [System.Collections.Generic.List[object]]::new()
Import-Clixml C:\scripts\behind-archive-entries.xml | Sort-Object -Property Published -Descending | Foreach-Object { $entries.Add($_) }
A list is easier to manipulate than a static array. You'll see why shortly.
The first thing I have to address is adding new items to this list. The XML file only contains entries based on when I generated the initial data set. Since then, I have published new content, so I need to add them.
I showed in a previous article how I created custom objects from the raw JSON data returned by the API. Here's a function that encapsulates that process.
Function New-BehindEmailItem {
[cmdletbinding()]
Param(
[Parameter(Mandatory, ValueFromPipeline)]
[object]$InputObject
)
Begin {
Write-Verbose "[$((Get-Date).TimeOfDay) BEGIN ] Starting $($MyInvocation.MyCommand)"
$out = [System.Collections.Generic.List[object]]::new()
} #begin
Process {
Write-Verbose "[$((Get-Date).TimeOfDay) PROCESS] Processing $($InputObject.Subject)"
foreach ($item in $InputObject) {
#create a custom typed object
#the object adds an array property for content tags
$obj = [PSCustomObject]@{
PSTYpeName = 'BehindEmailItem'
Title = $item.Subject
Type = $item.email_type
Published = $item.publish_date
Preview = parsePreview -Body $item.body
Tags = @()
Link = $item.absolute_url
}
$out.Add($obj)
} #ForEach
} #process
End {
$out | Sort-Object -Property Published -Descending
Write-Verbose "[$((Get-Date).TimeOfDay) END ] Ending $($MyInvocation.MyCommand)"
} #end
}
Using a type name,
BehindEmailItem, makes it possible to create custom format files and further extend the type.
The function relies on the private parsePreview function I showed in a previous article. This function assumes that parsePreview is available. The New-BehindEmailItem function needs input based on the API data. I have a separate function to get that data.
This is the complete script file.
#requires -version 5.1
Function Get-BehindArchiveData {
[cmdletbinding()]
Param(
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[ValidateLength(36, 36)]
[string]$APIKey,
[Parameter(HelpMessage = 'Specify a date to filter for emails published after this date')]
[ValidateNotNullOrEmpty()]
[Alias("After")]
[DateTime]$PublishedAfter
)
Write-Verbose "Starting $($MyInvocation.MyCommand)"
$head = @{Authorization = "token $bdapi" }
$base = 'https://api.buttondown.email'
$uri = "$base/v1/emails?status=imported&status=sent"
If ($PublishedAfter) {
$dt = Get-Date $PublishedAfter -Format 'yyyy-MM-dd'
$uri += "&publish_date__start=$dt"
}
$all = @()
Try {
do {
Write-Verbose "Processing $uri"
$all += (Invoke-RestMethod -Uri $uri -Headers $head -OutVariable get -ErrorAction Stop).results
$uri = $get.next
} Until ($null -eq $get.next)
}
Catch {
Throw $_
}
$All | New-BehindEmailItem
Write-Verbose "Ending $($MyInvocation.MyCommand)"
}
#dot source required files
. $PSScriptRoot\New-BehindEmailItem.ps1
. $PSScriptRoot\parsePreview.ps1
#Set the default API key parameter value
C:\scripts\unlock-vault.ps1
$bdAPI = Get-Secret -Name buttondown-api -Vault $myVault -AsPlainText
$PSDefaultParameterValues['Get-BehindArchiveData:APIKey'] = $bdAPI
You will see that I am dot-sourcing additional script files, and I'm getting my API key from the secrets vault. Each item from the API call is piped to the New-BehindEmailItem function to create my typed output.
$All | New-BehindEmailItem
I also added a parameter to limit the results to emails published after a certain date. This will be useful when I need to update my archive data as there's no reason to download stuff I already have.
After dot-sourcing this script, I can get new emails.
$new = Get-BehindArchiveData -PublishedAfter ($entries[0].Published.AddHours(12)) | Sort-Object -Property Published -Descending
I'm using the Published property of the first item in my list and adding 12 hours to it. The API parameter is inclusive, so if I use $entries[0].Published I will get results including the item I already have. I'm assuming that I won't publish more than one item in a 12-hour period. I'm also sorting the results in descending order so that the newest items are at the top of the list.
Now I can add the new items to my list.
$entries.Insert(0, $new)
This is why I wanted the collection object instead of a list. It is much easier to insert the new items at the beginning of the list. I want my list to be in descending order based on the Published property.
PS C:\> $entries[0..4] | Select-Object Title,Published
Title Published
----- ---------
Documents and Objects 3/5/2024 6:10:00 PM
Ask Jeff - February 2024 2/29/2024 6:13:29 PM
More Archive Toolmaking 2/27/2024 6:06:00 PM
Creating Buttondown Tooling 2/22/2024 6:35:43 PM
Making Progress 2/20/2024 8:01:05 PM
The approach I'm taking is the $entries is a data set that I will use to generate my archive summary.
In looking through the data, I know I need to manually add content tags. I also need to manually clean-up the preview text. As much as I've tried to parse the text to get a nice preview, it's not perfect and tedious. I think it will be just as easy to manually edit the preview text. As long as I save my data, this will be a one-time task for all current content. As I publish new content, I should only have to manually edit a few items. How can I make this easier?
Building a WPF Editor
I could manually edit the XML file. I'm avoiding using JSON because that will format and escape text to make it JSON compatible. That makes manually editing a bit more difficult. What I want is an editable display. So I built a tool to help me with my toolmaking.
This script will create a Windows Presentation Foundation (WPF) form that will display the data and allow me to edit the preview text and add tags. I am using a simple stack panel which makes this code easier to write.
#requires -version 5.1
#buttondown-archive-edit.ps1
Param([object[]]$InputObject)
Add-Type -AssemblyName PresentationFramework
Add-Type -AssemblyName PresentationCore
$form = New-Object System.Windows.Window
#define what it looks like
$form.Title = 'Newsletter Archive Summary'
$form.Height = 350
$form.Width = 500
$form.WindowStartupLocation = 'CenterScreen'
$stack = New-Object System.Windows.Controls.StackPanel
$label2 = New-Object System.Windows.Controls.Label
$label2.HorizontalAlignment = 'Left'
$label2.Content = 'Preview'
#add to the stack
$stack.AddChild($label2)
$Preview = New-Object System.Windows.Controls.TextBox
$Preview.Width = ($form.Width - 50)
$Preview.Height = ($form.Height - 150)
$Preview.AcceptsReturn = $True
$Preview.AcceptsTab = $True
$preview.TextWrapping = 'Wrap'
$preview.Text = 'Entry Title Here'
$previewTip = New-Object System.Windows.Controls.ToolTip
$previewTip.Content = 'Enter a preview of the newsletter here then click Save'
$Preview.ToolTip = $previewTip
$stack.AddChild($Preview)
#add tags
$label3 = New-Object System.Windows.Controls.Label
$label3.HorizontalAlignment = 'Left'
$label3.Content = 'Tags'
$stack.AddChild($label3)
$txtTags = New-Object System.Windows.Controls.TextBox
$txtTags.Width = $form.Width - 50
##add a tooltip on mouseover
$tt = New-Object System.Windows.Controls.ToolTip
$tt.Content = 'Enter tags separated by commas'
$txtTags.ToolTip = $tt
$stack.AddChild($txtTags)
#create a grid control for the buttons
$grid = New-Object System.Windows.Controls.Grid
#create a button
$btn = New-Object System.Windows.Controls.Button
$btn.Content = '_Save'
$btn.Width = 75
$btn.HorizontalAlignment = 'Center'
#System.Windows.Thickness new(double left, double top, double right, double bottom)
$btn.Margin = '0,10,0,0'
#this will now work
$Save = {
$InputObject[$i].Preview = $Preview.Text
$InputObject[$i].Tags = $txtTags.Text -split ','
}
#add an event handler
$btn.Add_click($Save)
#add to the stack
$grid.AddChild($btn)
$btnNext = New-Object System.Windows.Controls.Button
$btnNext.Content = '_Next'
$btnNext.Width = 75
$btnNext.HorizontalAlignment = 'Right'
$btnNext.Margin = '0,10,20,0'
#this will now work
$Next = {
$script:i++
$item = $InputObject[$script:i]
$form.Title = $item.Title
$Preview.Text = $item.Preview
$txtTags.Text = ($item.Tags -join ',')
}
#add an event handler
$btnNext.Add_click($Next)
#add to the stack
$grid.AddChild($btnNext)
$btnPrev = New-Object System.Windows.Controls.Button
$btnPrev.Content = '_Previous'
$btnPrev.Width = 75
$btnPrev.HorizontalAlignment = 'Left'
$btnPrev.Margin = '20,10,0,0'
#this will now work
$Previous = {
$script:i--
$item = $InputObject[$script:i]
$form.Title = $item.Title
$Preview.Text = $item.Preview
$txtTags.Text = ($item.Tags -join ',')
}
#add an event handler
$btnPrev.Add_click($Previous)
#add to the stack
$grid.AddChild($btnPrev)
#add the grid to the stack
$stack.AddChild($grid)
#add the stack to the form
$form.AddChild($stack)
$script:i = 0
$item = $InputObject[$script:i]
$form.Title = $item.Title
$Preview.Text = $item.Preview
$txtTags.Text = ($item.Tags -join ',')
#show the form
[void]($form.ShowDialog())
I'm not going to dive into how this code works today. I think I need to cover WPF-scripting in a separate set of articles.
I can use this script to "edit" my entry data.
. .\buttondown-archive-edit.ps1 $entries
I can edit the preview text. Not only can I trim it, I can also insert markdown syntax as needed. I can also add tags. After making a change, I can click the Save button to update the underlying data. The Next and Previous buttons will move to the next or previous item in the list.
The form doesn't have an explicit close button, but I can click the X in the upper right corner to close the form.
As long as I saved my changes, $Entries is updated. I have over 180 articles that will need to be updated so this will take some time. I'm not going to get it all done today. I need to make sure I re-export the data to the XML file so that I don't lose my changes.
$entries | Export-Clixml C:\scripts\behind-archive-entries.xml
Building from Data
At this point, I have a set of tools for getting, editing, and updating the email data for this newsletter. I need some time to finish updating the data, but I already know how I can use it.
I know I want to create a summary markdown document. I went through this process in previous articles. I took that code and created a script that will consume my data and generate the markdown file.
#requires -version 5.1
#new-behindarchivesummary.ps1
#create a markdown document
Param(
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[object[]]$InputObject,
[string]$Title = 'Behind the PowerShell Pipeline Archive Summary'
)
$doc = [System.Collections.Generic.List[string]]::new()
$doc.Add("# $Title`n")
foreach ($item in $InputObject) {
if ($item.Type -eq 'premium') {
$emoji = ':heavy_dollar_sign:'
}
else {
$emoji = ':globe_with_meridians:'
}
$doc.Add("## [$($item.Title)]($($item.Link)) $emoji `n")
$doc.Add("Published: $($item.Published)`n")
$doc.Add("$($item.Preview)`n")
$doc.Add("Tags: $($item.Tags -join ', ')`n")
}
#insert a subscribe button
$button = ''
$doc.Add($button)
$doc.Add("`n*Summary Created $(Get-Date -Format f )*`n")
#write the markdown document to the pipeline
$doc
The script writes the markdown to the pipeline, not a file. This give me flexibility. I can copy it to the clipboard so I can paste it into VSCode and look at.
.\new-behindarchivesummary.ps1 $entries[0..20] | Set-Clipboard
Or I can save it to a file.
.\new-behindarchivesummary.ps1 $entries[0..20] | Out-File .\behind-the-powershell-pipeline-archive-summary.md
Right now, my thought is to publish the summary and data as a GitHub gist. I have yet more tools in mind that I will write later. For now, I'm going to publish my data to a private gist using the GitHub API. Another option you could use is to write a script using the gh.exe command-line tool to accomplish the same task.
My GitHub API token is stored in my secrets vault. I need it to create the authorization header.
$token =Get-Secret gitToken -AsPlainText
$head = @{
Accept = 'application/vnd.github.v3+json'
Authorization = "token $Token"
}
$base = "https://api.github.com"
$owner = "jdhitsolutions"
Documentation on GitHub gists is at https://developer.github.com/v3/gists
In the gist, I want to publish the markdown document and the entry data as JSON. This will be easier to read on GitHub than cliXML. The body of the request will be a JSON object.
$data = @{
files = @{
"behind-the-powershell-pipeline-archive.json" = @{content = $($entries | ConvertTo-Json) }
"behind-the-powershell-pipeline-archive-summary.md" = @{content = $(Get-Content .\behind-the-powershell-pipeline-archive-summary.md | Out-String)}
}
description = 'Behind the PowerShell Pipeline Archive Data'
public = $False
} | ConvertTo-Json
Make sure your hashtable keys are all lower-case.
A gist can have multiple files. I'm creating a hash table where the key is the file name and the value is another hash table with a content key. I'm also setting the gist to private.
The last step is to push the data to GitHub.
$r = Invoke-RestMethod -Method Post -Uri "$base/gists" -Headers $head -Body $data -ContentType application/json
The response will contain information about the gist.
I will want to keep track of the ID.
$gistID = '2db0579dea00c96540b9d09b2bcaab93'
After I've updated the data, I will want to update the gist.
$data = @{
files = @{
"behind-the-powershell-pipeline-archive.json" = @{content = $($entries[0..30] | Sort-Object Published -descending | ConvertTo-Json | Out-String) }
}
} | ConvertTo-Json
This will only affect the files specified. To update the gist, use the Patch method.
$u =Invoke-RestMethod -Method Patch -Uri "$base/gists/$gistID" -Headers $head -Body $data -ContentType application/json
I will eventually turn this code into a script file to simplify the process.
Summary
Today was a mix of different scripting techniques. If you have skipped the earlier articles, you might find today's content a bit confusing or hard to follow, so take a little time to read through the previous articles.
For now, I'll keep updating my archive data and finalize my solution. At that point, I'll share what I've come up with. In the mean time, please leave comments or questions.