Blogs

Avoid Concurrent Processes in Azure Automation

When developing automated processes, you will often have to consider what happens if you end up with more than one thread running at the same time. Sometimes you want to be reading and writing to a shared resource and your code relies on an output based on an input. If an input is given from process A and an unexpected result is read back by process B, perhaps stuff will break, or worse, it won’t and will just continue on its merry way creating and modifying resources in an unexpected way. You may also come up against a more common scenario requiring sequential processing such as writing to a common text-based logfile and coming up against resource locks.

This problem is nothing new and there are solutions for it, however the goalposts shift a little when you use Azure Automation runbooks. In this post, we’ll go over some of the options to avoid concurrency and present what I think is a pretty good solution.

Using Mutexes

In PowerShell and other languages, you can use a Mutex (Mutual Exclusion Object) to avoid running concurrent processes. A Mutex, aside from being a fun word to say, is essentially applying the logic of sitting in a circle in primary school and passing around a talking stick. The teacher enforces the rules that only the holder of the stick is allowed to talk. If you are running all your processes on a common server, using a Mutex is a simple and effective solution. You can use a Mutex with PowerShell like this:

# Create and initialise the Mutex object
$Mutex = [System.Threading.Mutex]::new($false, "MyMutex")

# Wait for/get the Mutex
$Mutex.WaitOne()

# Do some things
New-ImportantThing -importance maximum

# Release the Mutex
$Mutex.ReleaseMutex()

This works well to avoid concurrency when you are running on a common system. This probably would work quite well in the Azure Automation world if you were running a Hybrid Runbook Worker, because you would likely be executing all your runbooks on the same system. If however you are not using Hybrid Workers and you are executing in an Azure sandbox (as in, cloud only), your job is simply allocated a system from a pool of workers, with no assurance that any two processes are actually running on the same system. It’s for this reason you cannot use Mutexes in Azure Automation accounts without a dedicated Hybrid Worker. Mutex, get in the bin.

Check If Other Jobs Are Running

OK, so a Mutex isn’t going to work so well, so let’s look at another approach. If you look at Microsoft docs for preventing concurrent jobs in Azure Automation, there is an article including some sample code showing you how you can do it. The crux of it is: use the Azure Automation PowerShell cmdlets to determine if there are any other jobs currently queued or running, and if so exit the script. This works pretty well, but it doesn’t offer proper protection against concurrency as if you had had a job fire up in the moment after the check, you still end up with concurrently running jobs. It also doesn’t allow you to concern yourself only with a portion of a script, because it will simply look for any job which is queued or running. If you have long-running scripts, this can cause a queue with no particular order and your job which you expect to complete in 15 minutes may take 30, 45, or if you’re unlucky it might just never find its timing right and just get stuck. Sorry Mutex, shift over and make some room in the bin for this idea.

Pass The Token

What you really want, is what exists. You want Mutex. But due to the nature of the environment, you can’t have Mutex, and dedicating a hybrid worker to solve this one problem is a bit of overkill (unless you already have a system set aside for this, in which case go ahead). What I wanted though was a Mutex-like solution that I could make portable to take to my customers, without the requirement of any additional infrastructure while keeping the complexity relatively low. So I brewed up my own Mutex at home in my bathtub, which works by writing values back and forth to shared Azure Automation Account variables to determine which process currently holds the token. Using this method, I was able to approximately replicate the functionality of Mutex. I did this by writing the following PowerShell class.

class TokenPass {
    [string]$TokenName
    [string]$TokenValue
    [string] $TempToken = $null
    [string] $TempReservationToken = $null
    [string] $ReservationTokenName
    hidden [int] $CountUp = 0
    [int] $RetryNum = 2400
    [string] $TokenSessionGuid = (New-Guid).Guid
    [string] $TokenHolder
    [string]GetToken() {
        if ((Get-AutomationVariable -Name $this.TokenName) -eq $this.TokenSessionGuid)
            {
               $this.TokenHolder = $this.TokenSessionGuid
               return "[TokenPass] You already hold the token. (01)"
            }
        if (($this.TokenName -or $this.TokenValue) -eq $null)
            {
               return "[TokenPass] Either TokenName or TokenValue is not set. Set these prior to calling GetToken. (02)"
            }
        else {
            if ((Get-AutomationVariable -Name $this.ReservationTokenName) -ne $this.TokenValue)
            {
                do {
                    sleep 2
                    Write-Output "[TokenPass] Waiting to reserve token. (03)"
                    $this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
                    if ($this.TempReservationToken -eq $this.TokenValue)
                        {
                           Set-AutomationVariable -Name $this.ReservationTokenName -Value $this.TokenSessionGuid 
                        }
                    sleep (Get-Random -Maximum 10 -Minimum 2)
                    $this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
                    if ($this.TempReservationToken -eq $this.TokenSessionGuid)
                        {
                           #Continue
                           $this.CountUp = ($this.RetryNum)+1
                        }
                    $this.CountUp = ($this.CountUp)+1
                    $this.TempReservationToken = $null
                   }
                While ($this.CountUp -le $this.RetryNum)
                $this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
                $this.TempToken = (Get-AutomationVariable -Name $this.TokenName)
                if ($this.TempReservationToken -eq $this.TokenSessionGuid)
                    {
                        Set-AutomationVariable -Name $this.TokenName -Value $this.TokenSessionGuid 
                    }
                sleep 2
                $this.TempToken = (Get-AutomationVariable -Name $this.TokenName)
                if ($this.TempToken -ne $this.TokenSessionGuid)
                    {
                        $this.TokenHolder = $this.TempToken
                        return "[TokenPass] Timed out attempting to reserve token. (05)"
                    }
                return "[TokenPass] Got Token. (06)"
            }
        }
        Set-AutomationVariable -Name $this.ReservationTokenName -Value $this.TokenSessionGuid 
        if ((Get-AutomationVariable -Name $this.ReservationTokenName) -ne $this.TokenSessionGuid)
            {
                do {
                    sleep 2
                    Write-Output "[TokenPass] Waiting to reserve token. (06)"
                    $this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
                    if ($this.TempReservationToken -eq $this.TokenValue)
                        {
                           Set-AutomationVariable -Name $this.ReservationTokenName -Value $this.TokenSessionGuid 
                        }
                    sleep 2
                    $this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
                    if ($this.TempReservationToken -eq $this.TokenSessionGuid)
                        {
                           #Continue
                           $this.CountUp = ($this.RetryNum)+1
                        }
                    $this.CountUp = ($this.CountUp)+1
                    $this.TempReservationToken = $null
                   }
                While ($this.CountUp -le $this.RetryNum)
                $this.CountUp = 0
                sleep 1
                $this.TempReservationToken = (Get-AutomationVariable -Name $this.ReservationTokenName)
                if ($this.TempReservationToken -eq $this.TokenSessionGuid)
                    {
                        Set-AutomationVariable -Name $this.TokenName -Value $this.TokenSessionGuid 
                    }
                sleep 2
                $this.TempToken = (Get-AutomationVariable -Name $this.TokenName)
                if (($this.TempToken -ne $this.TokenValue) -and ($this.TempToken -ne $this.TokenSessionGuid))
                    {
                        $this.TokenHolder = $this.TempToken
                        return "[TokenPass] Timed out attempting to get token. (09)"
                    }
                return "[TokenPass] Got Token. (10)"
            }
        if ((Get-AutomationVariable -Name $this.ReservationTokenName) -eq $this.TokenSessionGuid)
        {
           Set-AutomationVariable -Name $this.TokenName -Value $this.TokenSessionGuid  
        }
        return "[TokenPass] Got Token. (11)"
    }
    [string]ReturnToken() {
        sleep 1
        if (((Get-AutomationVariable -Name $this.TokenName) -eq $this.TokenSessionGuid) -and ((Get-AutomationVariable -Name $this.ReservationTokenName) -eq $this.TokenSessionGuid))
            {
               Set-AutomationVariable -Name $this.TokenName -Value $this.TokenValue 
               Set-AutomationVariable -Name $this.ReservationTokenName -Value $this.TokenValue 
               return "[TokenPass] Returned the token. (12)"
            }
        if ((Get-AutomationVariable -Name $this.TokenName) -eq $this.TokenValue)
            {
               return "[TokenPass] The token had already been returned either by this, or another process. (13)"
            }
        else {
            return "[TokenPass] The token is checked out to another process and must be returned by that process. (14)"
        }
    }
    [void]SetTokenProperties([string]$TokenName,[string]$TokenValue,[string]$TokenSessionGuid,[string]$ReservationTokenName) {
        $this.TokenName = $TokenName
        $this.ReservationTokenName = $ReservationTokenName
        $this.TokenValue = $TokenValue
        $this.TokenSessionGuid = $TokenSessionGuid
    }
}

I include the code above in a PowerShell module which I import into my Azure Automation account, but you can of course simply paste it at the top of your code. Then, we can use it like so:

# Create and initialise the TokenPass object
$ConcurrencyToken = [TokenPass]::new()

# Set the names of the two Azure Automation Variables we will use
$ConcurrencyToken.TokenName = "TokenPass"
$ConcurrencyToken.ReservationTokenName = "TokenPassReservation"

# Set a value for the token which will be passed back and forth
$ConcurrencyToken.TokenValue = "TOKENPASS”


# Wait for/get the token
$ConcurrencyToken.GetToken()

# Return/release the token
$ConcurrencyToken.ReturnToken()

So How Does This Actually Work?

Here’s a step-by-step breakdown of what happens

  1. The code checks the value of the variable defined in “ReservationTokenName” if that value matches the value of the defined “TokenValue” it writes its own GUID to the variable
  2. The code then checks that the value of “ReservationTokenName” matches its own GUID, and if it does it writes that GUID to the second variable defined in “TokenName”
  3. The process which successfully wrote its own GUID to both variables now holds the token. Any other processes running “GetToken” will loop and wait until this process returns the token.
  4. Once you’re past the code block which needs to avoid concurrency, the token is returned with “ReturnToken”, which checks that the value of both tokens matches the value of the processes GUID. Then sets both variables back to the value defined in “TokenValue” allowing another process to take the token.

If you’re wondering why we are setting the same values against two different variables one right after the other, it’s because during the initial testing it was found there was a 1-2 second delay between writing a variable and having that same value read back. This led to a situation where multiple processes would check if the token was available, and more than one process would think it owned the token. Adding a second variable setting gate which depending on already having written to the first (and let’s be honest, a cheeky random sleep) solved this problem.

Don’t Forget To Return The Token!

If you go ahead and implement this idea, it’s going to of course be very important that you always return the token your process held when it no longer needs to hold it. Failure to return the token will cause other processes that rely on it to never be able to grab it, and eventually fail. When writing code that includes this class, always consider where code may break or fail and result in an un-returned token. Using common PowerShell tropes like try, catch, finally and including a return of the token in the finally block can be a good way to ensure the token gets returned.

Conclusion

Using this solution allows me to only enforce serial processing for smaller chunks of code, enabling faster execution of concurrently running jobs, and avoiding the situation of having to wait for another entire job to finish and then having to compete in a completely un-ordered queue. I’m not sure this is a completely bulletproof solution, but in volume testing for what would be considered an extraordinary load for my requirements, it performed without failure. It’s not quite Mutex, but I reckon it’s pretty good for a bathtub Mutex.

Hopefully this helps you solve some issues with concurrent processing in your environments, and as always, feel free to reach out to us to discuss any of your Azure infrastructure or automation challenges.

[mailpoet_form id="1"]

Other Recent Blogs

Microsoft Teams IP Phones and Intune Enrollment

Microsoft Teams provides a growing portfolio of devices that can be used as desk and conference room phones. These IP phones run on Android 8.x or 9.x and are required to be enrolled in Intune. By default, these devices are enrolled as personal devices, which is not ideal as users should not be able to enrol their own personal Android devices.

Read More »

Level 9, 360 Collins Street, 
Melbourne VIC 3000

Level 2, 24 Campbell St,
Sydney NSW 2000

200 Adelaide St,
Brisbane QLD 4000

191 St Georges Terrace
Perth WA 6000

Level 10, 41 Shortland Street
Auckland

Part of

Arinco trades as Arinco (VIC) Pty Ltd and Arinco (NSW) Pty Ltd. © 2023 All Rights Reserved Arinco™ | Privacy Policy | Sustainability and Our Community
Arinco acknowledges the Traditional Owners of the land on which our offices are situated, and pay our respects to their Elders past, present and emerging.

Get started on the right path to cloud success today. Our Crew are standing by to answer your questions and get you up and running.