When developing automated processes, you will often have to consider what happens if you end up with more than one thread running at the same time. Sometimes you want to be reading and writing to a shared resource and your code relies on an output based on an input. If an input is given from process A and an unexpected result is read back by process B, perhaps stuff will break, or worse, it won’t and will just continue on its merry way creating and modifying resources in an unexpected way. You may also come up against a more common scenario requiring sequential processing such as writing to a common text-based logfile and coming up against resource locks.
This problem is nothing new and there are solutions for it, however the goalposts shift a little when you use Azure Automation runbooks. In this post, we’ll go over some of the options to avoid concurrency and present what I think is a pretty good solution.
Using Mutexes
In PowerShell and other languages, you can use a Mutex (Mutual Exclusion Object) to avoid running concurrent processes. A Mutex, aside from being a fun word to say, is essentially applying the logic of sitting in a circle in primary school and passing around a talking stick. The teacher enforces the rules that only the holder of the stick is allowed to talk. If you are running all your processes on a common server, using a Mutex is a simple and effective solution. You can use a Mutex with PowerShell like this:
# Create and initialise the Mutex object
$Mutex = [System.Threading.Mutex]::new($false, "MyMutex")
# Wait for/get the Mutex
$Mutex.WaitOne()
# Do some things
New-ImportantThing -importance maximum
# Release the Mutex
$Mutex.ReleaseMutex()This works well to avoid concurrency when you are running on a common system. This probably would work quite well in the Azure Automation world if you were running a Hybrid Runbook Worker, because you would likely be executing all your runbooks on the same system. If however you are not using Hybrid Workers and you are executing in an Azure sandbox (as in, cloud only), your job is simply allocated a system from a pool of workers, with no assurance that any two processes are actually running on the same system. It’s for this reason you cannot use Mutexes in Azure Automation accounts without a dedicated Hybrid Worker. Mutex, get in the bin.
Check If Other Jobs Are Running
OK, so a Mutex isn’t going to work so well, so let’s look at another approach. If you look at Microsoft docs for preventing concurrent jobs in Azure Automation, there is an article including some sample code showing you how you can do it. The crux of it is: use the Azure Automation PowerShell cmdlets to determine if there are any other jobs currently queued or running, and if so exit the script. This works pretty well, but it doesn’t offer proper protection against concurrency as if you had had a job fire up in the moment after the check, you still end up with concurrently running jobs. It also doesn’t allow you to concern yourself only with a portion of a script, because it will simply look for any job which is queued or running. If you have long-running scripts, this can cause a queue with no particular order and your job which you expect to complete in 15 minutes may take 30, 45, or if you’re unlucky it might just never find its timing right and just get stuck. Sorry Mutex, shift over and make some room in the bin for this idea.
Pass The Token
What you really want, is what exists. You want Mutex. But due to the nature of the environment, you can’t have Mutex, and dedicating a hybrid worker to solve this one problem is a bit of overkill (unless you already have a system set aside for this, in which case go ahead). What I wanted though was a Mutex-like solution that I could make portable to take to my customers, without the requirement of any additional infrastructure while keeping the complexity relatively low. So I brewed up my own Mutex at home in my bathtub, which works by writing values back and forth to shared Azure Automation Account variables to determine which process currently holds the token. Using this method, I was able to approximately replicate the functionality of Mutex. I did this by writing the following PowerShell class.
class TokenPass {
[string]$TokenName
[string]$TokenValue
[string] $TempToken = $null
[string] $TempReservationToken = $null
[string] $ReservationTokenName
hidden [int] $CountUp = 0
[int] $RetryNum = 2400
[string] $TokenSessionGuid = (New-Guid).Guid
[string] $TokenHolder
[string]GetToken() {
if ((Get-AutomationVariable -Name $this.TokenName) -eq $this.TokenSessionGuid)
{
$this.TokenHolder = $this.TokenSessionGuid
return "[TokenPass] You already hold the token. (01)"
}
if (($this.TokenName -or $this.TokenValue) -eq $null)
{
return "[TokenPass] Either TokenName or TokenValue is not set. Set these prior to calling GetToken. (02)"
}
else {
if ((Get-AutomationVariable -Name $this.ReservationTokenName) -ne $this.TokenValue)
{
do {
sleep 2
Write-Output "[TokenPass] Waiting to reserve token. (03)"
$this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
if ($this.TempReservationToken -eq $this.TokenValue)
{
Set-AutomationVariable -Name $this.ReservationTokenName -Value $this.TokenSessionGuid
}
sleep (Get-Random -Maximum 10 -Minimum 2)
$this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
if ($this.TempReservationToken -eq $this.TokenSessionGuid)
{
#Continue
$this.CountUp = ($this.RetryNum)+1
}
$this.CountUp = ($this.CountUp)+1
$this.TempReservationToken = $null
}
While ($this.CountUp -le $this.RetryNum)
$this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
$this.TempToken = (Get-AutomationVariable -Name $this.TokenName)
if ($this.TempReservationToken -eq $this.TokenSessionGuid)
{
Set-AutomationVariable -Name $this.TokenName -Value $this.TokenSessionGuid
}
sleep 2
$this.TempToken = (Get-AutomationVariable -Name $this.TokenName)
if ($this.TempToken -ne $this.TokenSessionGuid)
{
$this.TokenHolder = $this.TempToken
return "[TokenPass] Timed out attempting to reserve token. (05)"
}
return "[TokenPass] Got Token. (06)"
}
}
Set-AutomationVariable -Name $this.ReservationTokenName -Value $this.TokenSessionGuid
if ((Get-AutomationVariable -Name $this.ReservationTokenName) -ne $this.TokenSessionGuid)
{
do {
sleep 2
Write-Output "[TokenPass] Waiting to reserve token. (06)"
$this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
if ($this.TempReservationToken -eq $this.TokenValue)
{
Set-AutomationVariable -Name $this.ReservationTokenName -Value $this.TokenSessionGuid
}
sleep 2
$this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
if ($this.TempReservationToken -eq $this.TokenSessionGuid)
{
#Continue
$this.CountUp = ($this.RetryNum)+1
}
$this.CountUp = ($this.CountUp)+1
$this.TempReservationToken = $null
}
While ($this.CountUp -le $this.RetryNum)
$this.CountUp = 0
sleep 1
$this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
if ($this.TempReservationToken -eq $this.TokenSessionGuid)
{
Set-AutomationVariable -Name $this.TokenName -Value $this.TokenSessionGuid
}
sleep 2
$this.TempToken = (Get-AutomationVariable -Name $this.TokenName)
if (($this.TempToken -ne $this.TokenValue) -and ($this.TempToken -ne $this.TokenSessionGuid))
{
$this.TokenHolder = $this.TempToken
return "[TokenPass] Timed out attempting to get token. (09)"
}
return "[TokenPass] Got Token. (10)"
}
if ((Get-AutomationVariable -Name $this.ReservationTokenName) -eq $this.TokenSessionGuid)
{
Set-AutomationVariable -Name $this.TokenName -Value $this.TokenSessionGuid
}
return "[TokenPass] Got Token. (11)"
}
[string]ReturnToken() {
sleep 1
if (((Get-AutomationVariable -Name $this.TokenName) -eq $this.TokenSessionGuid) -and ((Get-AutomationVariable -Name $this.ReservationTokenName) -eq $this.TokenSessionGuid))
{
Set-AutomationVariable -Name $this.TokenName -Value $this.TokenValue
Set-AutomationVariable -Name $this.ReservationTokenName -Value $this.TokenValue
return "[TokenPass] Returned the token. (12)"
}
if ((Get-AutomationVariable -Name $this.TokenName) -eq $this.TokenValue)
{
return "[TokenPass] The token had already been returned either by this, or another process. (13)"
}
else {
return "[TokenPass] The token is checked out to another process and must be returned by that process. (14)"
}
}
[void]SetTokenProperties([string]$TokenName,[string]$TokenValue,[string]$TokenSessionGuid,[string]$ReservationTokenName) {
$this.TokenName = $TokenName
$this.ReservationTokenName = $ReservationTokenName
$this.TokenValue = $TokenValue
$this.TokenSessionGuid = $TokenSessionGuid
}
}
I include the code above in a PowerShell module which I import into my Azure Automation account, but you can of course simply paste it at the top of your code. Then, we can use it like so:
# Create and initialise the TokenPass object
$ConcurrencyToken = [TokenPass]::new()
# Set the names of the two Azure Automation Variables we will use
$ConcurrencyToken.TokenName = "TokenPass"
$ConcurrencyToken.ReservationTokenName = "TokenPassReservation"
# Set a value for the token which will be passed back and forth
$ConcurrencyToken.TokenValue = "TOKENPASS”
# Wait for/get the token
$ConcurrencyToken.GetToken()
# Return/release the token
$ConcurrencyToken.ReturnToken()
So How Does This Actually Work?
Here’s a step-by-step breakdown of what happens
- The code checks the value of the variable defined in “ReservationTokenName” if that value matches the value of the defined “TokenValue” it writes its own GUID to the variable
- The code then checks that the value of “ReservationTokenName” matches its own GUID, and if it does it writes that GUID to the second variable defined in “TokenName”
- The process which successfully wrote its own GUID to both variables now holds the token. Any other processes running “GetToken” will loop and wait until this process returns the token.
- Once you’re past the code block which needs to avoid concurrency, the token is returned with “ReturnToken”, which checks that the value of both tokens matches the value of the processes GUID. Then sets both variables back to the value defined in “TokenValue” allowing another process to take the token.
If you’re wondering why we are setting the same values against two different variables one right after the other, it’s because during the initial testing it was found there was a 1-2 second delay between writing a variable and having that same value read back. This led to a situation where multiple processes would check if the token was available, and more than one process would think it owned the token. Adding a second variable setting gate which depending on already having written to the first (and let’s be honest, a cheeky random sleep) solved this problem.
Don’t Forget To Return The Token!
If you go ahead and implement this idea, it’s going to of course be very important that you always return the token your process held when it no longer needs to hold it. Failure to return the token will cause other processes that rely on it to never be able to grab it, and eventually fail. When writing code that includes this class, always consider where code may break or fail and result in an un-returned token. Using common PowerShell tropes like try, catch, finally and including a return of the token in the finally block can be a good way to ensure the token gets returned.
Conclusion
Using this solution allows me to only enforce serial processing for smaller chunks of code, enabling faster execution of concurrently running jobs, and avoiding the situation of having to wait for another entire job to finish and then having to compete in a completely un-ordered queue. I’m not sure this is a completely bulletproof solution, but in volume testing for what would be considered an extraordinary load for my requirements, it performed without failure. It’s not quite Mutex, but I reckon it’s pretty good for a bathtub Mutex.
Hopefully this helps you solve some issues with concurrent processing in your environments, and as always, feel free to reach out to us to discuss any of your Azure infrastructure or automation challenges.
