Thursday, November 08, 2007

Parallel Build Tasks in MSBuild 2008

One of the things I've been struggling with for some time is that my build process has a rather long running step (thanks to the chatty SQL Server 2005 SMO), and all the other steps wait for the long running step to complete. The long running step is fairly inconsequential to the steps following, and I've always wanted that step of the build to run in parallel to the rest of the tasks.

With the upcoming release of Visual Studio 2008, this will be built in, which is very handy. The MSBuild task now has a new boolean attribute called BuildInParallel. As far as I can tell, the MSBuild task is the only task that has this new attribute. Using that in conjunction with the new /m:<Number of CPUs> MSBuild command line argument, you can get the projects provided to the MSBuild task to run in parallel, based on the number of CPU's you specify.

Below are three MSBuild project files that show a simple example of this feature.

This is the long running project, which sleeps for 15 seconds:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
    <Target Name="Test">
        <Message Text="Long Running - Start" />
        <Exec Command="Sleep.exe 15" />
        <Message Text="Long Running - End" />
    </Target>
</Project>

This is the project I want to run in parallel to the long running project, which sleeps for 5 seconds:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
    <Target Name="Test">
        <Message Text="Fast - Start" />
        <Exec Command="Sleep.exe 5" />
        <Message Text="Fast - End" />
    </Target>
</Project>

And finally, this is the root project, the one I provide MSBuild at the command line (Note the BuildInParallel attribute on the MSBuild task). Any tasks that occur after the parallel MSBuild task will not start until all the projects in the parallel MSBuild have completed.

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
    <Target Name="Test">
        <Message Text="Begin Build" />
        <MSBuild Projects="longrunning.proj;fast.proj" BuildInParallel="true" />
        <Message Text="End Build" />
    </Target>
</Project>

This is the result when I run the build without passing the /m:<Number of CPUs> argument to MSBuild (click the image to see full size):

serial output

This is the result when I run the build telling MSBuild to use 2 CPUs (click the image to see full size).

parallel output

Nice, reduced my build by 5 seconds.

One thing that I find odd, is that if you look closely, the message from the root build project "End Build" occurs before the end long running task message "Long Running - End". I've done some checking to make sure that following tasks wait for both of the projects in that MSBuild task to complete, and they do. There must be something about the output of that message that's messed up.

Gotta build a rig like Hanselman and get my build running on 4 CPUs. That would be lovely.

5 comments:

David said...

The screenshot links appear broken?

Jim Fiorato said...

Thanks David,

I've reposted the entry and all is well again.

Thanks so much for letting me know about it. Much appreciated.

Jim

David said...

Hmmmm. I still get a 403 error that says "Your client does not have permission to get URL /1.1 from this server." I'm using Firefox. But I checked IE7 and it's the same.

Jim Fiorato said...

Ok, third time has to be a charm.

I reposted the images as well, Hopefully this fixes the issue.

I'm using Live Writer, and I've changed the design a few times since this original post. Who knows what happened to the images....

Thanks again. Jim

David said...

There you go -- you got it!

Thanks. Good post.