Powershell – Serial vs Parallel Processing

Whenever I look for improving the performances of a script or code there is sometimes an option for parallelising the processing/execution of some of the workflows. It generally means adding complexity and is not always the case that it will make necessary the whole process so much faster, but for long-running ones made of independent sub-tasks, it is probably a very effective strategy.

A recent comment gave me the idea for this article, even if the subject could get potentially very complex if we go deep, but I will do my best to keep it very simple and at the same time giving you some good examples of it.

Serial Processing

I’m sure that everybody is familiar with the Foreach-Object cmdlet that allows you to iterate a list of objects passed as input as in this example:

The benefit of the serial processing is that the code is very easy to read, the elements are iterated sequentially one after the other, so the behaviour is predictable.

Parallel Processing

With Powershell 7 it is been introduced Foreach-object -parallel . In the following script, we will measure its performances and see how it works.

This is the output:

One important thing to note is the throttlelimit which has a default value of 5, depending on your case you might find the need to modifying it as shown in my example above to improve the performance of your script (e.g. reducing the total execution time).

One article that I found very useful was written by Paul Higinbotham that I recommend you to read.

Parallel Execution using JOBS

Leveraging background Jobs is also another way of parallelising your workflows, but the code you will write could be less simple to read compared to the foreach -parallel of powershell 7,  but you don’t need powershell core to run it.

A background job has three possible states: RUNNING, FAILED, COMPLETED. Once the jobs are started we need to monitor the when they are completed.

In my script,  I’ve used a do/while looking for RUNNING state of jobs to wait using a 1-second interval sleep and a counter that will be compared against a Timeout value to avoid waiting forever. To make the script more robust in your case you might consider the case when the state is FAILED and threat that exception appropriately according to your needs.

The last important note when working with Receive-Job cmdlet is the -keep flag. By default without keep the job will get deleted once retrieved so if you want to use the -keep flag you need to remember to clean and not leaving the job result persistent on your system.

With this result:

As usual, if you like these scripts you can find them on my GitHub repository.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.