Intel lists several such surprises in Volume 3, Section 8.2.3 of their x86/64 Architecture Specification. Here’s one of the simplest examples. Suppose you have two integers X and Y somewhere in memory, both initially 0. Two processors, running in parallel, execute the following machine code:

Don’t be thrown off by the use of assembly language in this example. It’s really the best way to illustrate CPU ordering. Each processor stores 1 into one of the integer variables, then loads the other integer into a register. (r1 and r2 are just placeholder names for actual x86 registers, such as eax.)

Now, no matter which processor writes 1 to memory first, it’s natural to expect the other processor to read that value back, which means we should end up with either r1 = 1, r2 = 1, or perhaps both. But according to Intel’s specification, that won’t necessarily be the case. The specification says it’s legal for both r1 and r2 to equal 0 at the end of this example — a counterintuitive result, to say the least!

One way to understand this is that Intel x86/64 processors, like most processor families, are allowed to reorder the memory interactions of machine instructions according to certain rules, as long it never changes the execution of a single-threaded program. In particular, each processor is allowed to delay the effect of a store past any load from a different location. As a result, it might end up as though the instructions had executed in this order:

Let’s Make It Happen

It’s all well and good to be told this kind of thing might happen, but there’s nothing like seeing it with your own eyes. That’s why I’ve written a small sample program to show this type of reordering actually happening. You can download the source code here.

The sample comes in both a Win32 version and a POSIX version. It spawns two worker threads which repeat the above transaction indefinitely, while the main thread synchronizes their work and checks each result.

Here’s the source code for the first worker thread. X, Y, r1 and r2 are all globals, and POSIX semaphores are used to co-ordinate the beginning and end of each loop.

sem_t beginSema1;
sem_t endSema;
 
int X, Y;
int r1, r2;
 
void *thread1Func(void *param)
{
    MersenneTwister random(1);                // Initialize random number generator
    for (;;)                                  // Loop indefinitely
    {
        sem_wait(&beginSema1);                // Wait for signal from main thread
        while (random.integer() % 8 != 0) {}  // Add a short, random delay
 
        // ----- THE TRANSACTION! -----
        X = 1;
        asm volatile("" ::: "memory");        // Prevent compiler reordering
        r1 = Y;
 
        sem_post(&endSema);                   // Notify transaction complete
    }
    return NULL;  // Never returns
};

A short, random delay is added before each transaction in order to stagger the timing of the thread. Remember, there are two worker threads, and we’re trying to get their instructions to overlap. The random delay is achieved using the same MersenneTwister implementation I’ve used in previous posts, such as when measuring lock contention and when validating that the recursive Benaphore worked.

Don’t be spooked by the presence of the asm volatile line in the above code listing. This is just a directive telling the GCC compiler not to rearrange the store and the load when generating machine code, just in case it starts to get any funny ideas during optimization. We can verify this by checking the assembly code listing, as seen below. As expected, the store and the load occur in the desired order. The instruction after that writes the resulting register eax back to the global variable r1.

$ gcc -O2 -c -S -masm=intel ordering.cpp
$ cat ordering.s
	...
	mov	DWORD PTR _X, 1
	mov	eax, DWORD PTR _Y
	mov	DWORD PTR _r1, eax
	...

The main thread source code is shown below. It performs all the administrative work. After initialization, it loops indefinitely, resetting X and Y back to 0 before kicking off the worker threads on each iteration.

Pay particular attention to the way all writes to shared memory occur before sem_post, and all reads from shared memory occur after sem_wait. The same rules are followed in the worker threads when communicating with the main thread. Semaphores give us acquire and release semantics on every platform. That means we are guaranteed that the initial values of X = 0 and Y = 0 will propogate completely to the worker threads, and that the resulting values of r1 and r2 will propogate fully back here. In other words, the semaphores prevent memory reordering issues in the framework, allowing us to focus entirely on the experiment itself!

int main()
{
    // Initialize the semaphores
    sem_init(&beginSema1, 0, 0);
    sem_init(&beginSema2, 0, 0);
    sem_init(&endSema, 0, 0);
 
    // Spawn the threads
    pthread_t thread1, thread2;
    pthread_create(&thread1, NULL, thread1Func, NULL);
    pthread_create(&thread2, NULL, thread2Func, NULL);
 
    // Repeat the experiment ad infinitum
    int detected = 0;
    for (int iterations = 1; ; iterations++)
    {
        // Reset X and Y
        X = 0;
        Y = 0;
        // Signal both threads
        sem_post(&beginSema1);
        sem_post(&beginSema2);
        // Wait for both threads
        sem_wait(&endSema);
        sem_wait(&endSema);
        // Check if there was a simultaneous reorder
        if (r1 == 0 && r2 == 0)
        {
            detected++;
            printf("%d reorders detected after %d iterations\n", detected, iterations);
        }
    }
    return 0;  // Never returns
}

Finally, the moment of truth. Here’s some sample output while running in Cygwin on an Intel Xeon W3520.

And there you have it! During this run, a memory reordering was detected approximately once every 6600 iterations. When I tested in Ubuntu on a Core 2 Duo E6300, the occurrences were even more rare. One begins to appreciate how subtle timing bugs can creep undetected into lock-free code.

Now, suppose you wanted to eliminate those reorderings. There are at least two ways to do it. One way is to set thread affinities so that both worker threads run exclusively on the same CPU core. There’s no portable way to set affinities with Pthreads, but on Linux it can be accomplished as follows:

    cpu_set_t cpus;
    CPU_ZERO(&cpus);
    CPU_SET(0, &cpus);
    pthread_setaffinity_np(thread1, sizeof(cpu_set_t), &cpus);
    pthread_setaffinity_np(thread2, sizeof(cpu_set_t), &cpus);

After this change, the reordering disappears. That’s because a single processor never sees its own operations out of order, even when threads are pre-empted and rescheduled at arbitrary times. Of course, by locking both threads to a single core, we’ve left the other cores unused.

On a related note, I compiled and ran this sample on Playstation 3, and no memory reordering was detected. This suggests that the two hardware threads inside the PPU effectively act as a single processor, with very fine-grained hardware scheduling.

Preventing It With a StoreLoad Barrier

Another way to prevent memory reordering in this sample is to introduce a CPU barrier between the two instructions. Here, we’d like to prevent the reordering of a store followed by a load. In common barrier parlance, we need a StoreLoad barrier.

On x86/64 processors, there is no specific instruction which acts only as a StoreLoad barrier, but there are several instructions which do that and more. The mfence instruction is a full memory barrier, which prevents memory reordering of any kind. In GCC, it can be implemented as follows:

    for (;;)                                  // Loop indefinitely
    {
        sem_wait(&beginSema1);                // Wait for signal from main thread
        while (random.integer() % 8 != 0) {}  // Add a short, random delay

        // ----- THE TRANSACTION! -----
        X = 1;
        asm volatile("mfence" ::: "memory");  // Prevent CPU reordering
        r1 = Y;

        sem_post(&endSema);                   // Notify transaction complete
    }

Again, you can verify its presence by looking at the assembly code listing.

	...
	mov	DWORD PTR _X, 1
	mfence
	mov	eax, DWORD PTR _Y
	mov	DWORD PTR _r1, eax
	...

With this modification, the memory reordering disappears, and we’ve still allowed both threads to run on separate CPU cores.

Similar Instructions and Different Platforms

Interestingly, mfence isn’t the only instruction which acts as a full memory barrier on x86/64. On these processors, any locked instruction, such as xchg, also acts as a full memory barrier — provided you don’t use SSE instructions or write-combined memory, which this sample doesn’t. In fact, the Microsoft C++ compiler generates xchg when you use the MemoryBarrier intrinsic, at least in Visual Studio 2008.

The mfence instruction is specific to x86/64. If you want to make the code more portable, you could wrap this intrinsic in a preprocessor macro. The Linux kernel has wrapped it in a macro named smp_mb, along with related macros such as smp_rmb and smp_wmb, and provided alternate implementations on different architectures. For example, on PowerPC, smp_mb is implemented as sync.

All these different CPU families, each having unique instructions to enforce memory ordering, with each compiler exposing them through different instrincs, and each cross-platform project implementing its own portability layer… none of this helps simplify lock-free programming! This is partially why the C++11 atomic library was recently introduced. It’s an attempt to standardize things, and make it easier to write portable lock-free code.

By the end of the trip, I had shot hundreds of photos and 80 video clips — about 20 minutes of video in total. The iPhone 4S has a great camera, but I quickly learned that its built-in video stabilization feature is, well, not state of the art. It can perform some modest stabilization in cases where you attempt to hold the camera still and point it in a single direction. But other times, it doesn’t seem to help at all.

That’s OK, because with a little patience, and a free utility called Deshaker by Gunnar Thalin, you can remove the camera shake on your PC when you get home. For example, here’s some footage I shot of a coati at the top of Cerro Chato. The first part of the video shows the original, shaky iPhone 4S video — I blame the tasty Costa Rican coffee — and the second part shows the result after running it through Deshaker. While the result may not be flawless, I find it to be a significant improvement:

Deshaker is a plugin for VirtualDub, a popular video editor by Avery Lee. The Deshaker documentation contains instructions on how to use it within VirtualDub, and there are additional guides available online. However, I had a lot of videos to stabilize, and following such guides would have been very time-consuming, as each video requires several manual steps. Not to mention I had trouble convincing VirtualDub to open the .mov files saved by the iPhone.

Therefore, I decided to automate the process by writing a Python script, which you can download here. This script makes use of an assortment of free tools for Windows: ffmpeg, AviSynth, Deshaker and a few others. These tools are definitely for power users — ffmpeg even calls itself a very experimental and developer-driven project. They’re great tools, but I must admit: Every time I use them, I feel a little bit of dread. I know I’m about to lose a big chunk of my weekend googling for information, trying different combinations, experimenting with settings and working around bugs. And once I’ve finished, I usually forget everything I learned.

So this time, I decided to document the process. In this post, I’ll show you how to install the software needed to run this script yourself. I’ll also describe each step performed by the script, and explain why I chose to implement it the way I did. The script is currently customized for iPhone 4S videos, but I hope to provide enough information here so that you can adapt it to work with other video sources as well.

Required Software

You’ll need to install all of the following free software, if you haven’t already. It all works on any version of Windows (XP or higher).

Install ffmpeg

ffmpeg is an open-source, cross-platform command-line tool which knows how to encode and decode a ton of audio/video formats. We’ll use it to encode our stabilized video as .mp4. I chose MP4 as the target format because Windows, Mac and Ubuntu all seem to recognize it without too much fuss.

I installed the 32-bit Static build of version git-41a097a (April 3) from this page. The build is distributed as a 7-zip file, so you’ll need to install the 7-zip extractor to open it. Installation is a simple matter of extracting the complete contents to a directory somewhere. I extracted it to c:\util\ffmpeg.

A small warning: In the past, I’ve found that the command-line arguments of ffmpeg may change from one version to the next. Therefore, if you search for examples of ffmpeg command lines, they may not work with the version you have. Hopefully the arguments won’t change too much in future versions, so that this guide will remain intact.

Install AVISynth

AVISynth is a frameserver, designed to provide audio/video input to other tools. It’s based on a neat trick: Once you install AVISynth, Windows will believe that any file with the extension .avs is a video file. In reality, an .avs file is just a text file you write in a custom scripting language, telling AVISynth how to render the video. We’ll use AVISynth to open our .mov files, rotate them, pad and trim extra frames, resize them, and most importantly — to run Deshaker.

AVISynth comes with a regular Windows installer. I installed the 32-bit version 2.5.8 from here. The 32-bit version is required if you want to use 32-bit plugins, which we do. When you install it, pay attention to the exact installation path, because you’ll need to remember it later.

Install Quicktime

Install Quicktime from Apple. Odds are, you already have it, but in case you don’t, here’s the download link. It’s required for QTSource to work.

Install QTSource

QTSource is a plugin for AVISynth. It gives AVISynth the ability to open .mov files, but it requires you to have Quicktime installed. There are alternative ways to open .mov files in AVISynth, but I couldn’t get any of them to work reliably with the iPhone 4S videos on my computer. So QTSource it is.

I used QTSource version 0.1.4, which you can currently download from the author’s download page (direct link). Simply open the zip file and extract QTSource.dll to your AVISynth plugins folder. This is the only file needed. In my case, I extracted it to C:\Program Files (x86)\AviSynth 2.5\plugins.

Install Deshaker

As I mentioned, Deshaker is a plugin for VirtualDub — but you don’t actually need VirtualDub to use it. It turns out that AVISynth can use any plugin written for VirtualDub. I downloaded Deshaker 3.0 from the author’s page (direct link). It’s a zip file, so simply open it and extract Deshaker.vdf to the folder of your choice. I happen to have VirtualDub installed in C:\Util\VirtualDub, so I’ve extracted it to C:\Util\VirtualDub\plugins. But any folder will do.

Install MediaInfo

MediaInfo is small utility to extract metadata from video files. We’ll use it to determine the rotation of each .mov file. It’s especially important to know the rotation because of the rolling shutter, as I’ll explain later. We want the CLI (Command Line Interface) version of MediaInfo, so that Python can interact with it. I installed the 32-bit version 0.7.56 from this page. It’s distributed as a zip file — I extracted mine to C:\Util\MediaInfo.

Install Python

Of course, you need Python. I developed the script in Python 2.7, but it might work in earlier versions. I prefer installing ActiveState Python for Windows, because it comes with PythonWin, but the regular installer will work too.

How to Run It

First, copy the iPhone videos to a folder on your PC. Then save the Python script somewhere; for example, in the same folder as the videos to convert. Open the Python script in a text editor and modify the hardcoded path names to correctly reflect the installation paths on your machine:

MEDIAINFO_FOLDER = r'C:\Util\MediaInfo'
DESHAKER_FOLDER = r'C:\Util\VirtualDub\Plugins'
FFMPEG_FOLDER = r'C:\Util\ffmpeg\bin'

Next, open a command prompt, navigate to the folder containing the videos you want to convert, and run any command similar to the following. If the Python script is in a different folder, make sure to specify its full path. You can also specify the name of a single video file, or any wildcard pattern accepted by Python’s glob module, such as *.mov.

python stabilize.py *.mov

While the script is running, the Command Prompt title bar will change to reflect encoding progress. I recommend resizing the window to be 100 columns wide so that ffmpeg’s progress messages fit completely on one line. If they extend to two lines, your Command Prompt window will fill up with spam. To change the width, right-click on the window and choose Properties.

The script saves the processed video back to the same folder as the .mov files, with the extension _stabilized.mp4. During the conversion process, it also creates a bunch of temporary files in the current working directory, which are deleted afterwards.

Finally, be warned that the conversion process is very slow. On my underpowered Core 2 Duo E6300, every minute of video takes about an hour to process.

What it Does

Here are all the steps performed by the script.

Determining the Video Rotation

The iPhone 4S knows which way you are holding the phone while you shoot the video, but it handles it in a funny way: It always records the video from the point of view of a non-rotated phone, and stores some metadata in the file to remember which way the phone was held. On Windows, the QuickTime video player recognizes this metadata, and will play the video back with the correct rotation; but Media Player Classic, an alternative player installed by the K-Lite Codec Pack, does not.

By the way, I recommend against filming vertical videos. They’re terrible for sharing online. In any case, you can view the rotation metadata by running MediaInfo on the video:

c:\util\mediainfo\mediainfo img_0128.mov

If the phone was rotated during recording, it will be listed in the output as either 90, 180 or 270.

...
Bit rate                                 : 24.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Rotation                                 : 90°         
Frame rate mode                          : Variable
Frame rate                               : 29.904 fps
...

The Python script automatically runs MediaInfo and extracts the rotation info from the output.

Generate a Log File Containing Camera Movement Information

The Deshaker plugin works in two passes. In the first pass, it determines camera movement information on a frame-by-frame basis. On the second pass, it corrects each video frame to produce a smoother video. Normally, you would perform both passes entirely within VirtualDub, but in our case, the Python script automates both passes using temporary AVISynth scripts.

The first pass is taken care of by pass1.avs, a temporary AVISynth script with contents similar to the following:

LoadVirtualDubPlugin("C:\Util\VirtualDub\Plugins\Deshaker.vdf", "Deshaker")
QTInput("D:\Photos\Costa Rica 2012\Videos\img_0128.mov", color=1)
TurnLeft()
Deshaker("18|1|30|4|1|0|1|0|640|480|1|1|650|650|1000|650|4|0|6|2|8|30|300|4|
D:\Photos\Costa Rica 2012\Videos\img_0128_deshaker.log|0|0|0|0|0|0|0|0|0|0|0|
0|0|1|12|12|10|5|1|1|10|10|0|0|1|0|1|0|0|10|1000|1|88|1|1|20|400|90|20|1")

• LoadVirtualDubPlugin() is required to make the Deshaker plugin work.
• QTInput() is the function which opens the .mov file using the QTSource plugin. The color=1 option tells the plugin to return video frames in the RGB32 format. This is the only format which the Deshaker plugin accepts. (We could have converted the pixel format using a separate AVISynth function, ConvertToRGB32, but the color=1 option is faster.)
• Since QTInput() uses QuickTime, which knows about the rotation metadata mentioned above, it will automatically rotate the video correctly according to the way the phone was held. We don’t actually want this; we want to process the video from the non-rotated camera’s point of view, for reasons explained below. That’s the what the TurnLeft() call is for in this case. The exact function may differ depending on the rotation.
• Finally, we pass each video frame to Deshaker(), which will determine the camera movement information for each frame. In this case, the information gets saved to the temporary log file D:\Photos\Costa Rica 2012\Videos\img_0128_deshaker.log.

To actually run the video through the AVISynth script, and generate a log file, we use the following command line. I won’t describe what all the options do, but you can always look them up in the ffmpeg documentation.

C:\Util\ffmpeg\bin\ffmpeg.exe -y -i pass1.avs -vcodec copy temp.avi

In the AVISynth script, you’ll notice a huge option string passed to Deshaker(). According to the author, Deshaker accepts its arguments as a long string because there is a limit to the number of separate arguments that can be passed from VirtualDub. There is a complete reference in the documentation. I’ve customized a few options in particular, which are highlighted below. Personally, I prefer these values over the defaults; your mileage may vary.

"18|1|30|4|1|0|1|0|640|480|1|1|650|650|1000|650|4|0|6|2|8|30|300|4|
D:\Photos\Costa Rica 2012\Videos\img_0128_deshaker.log|0|0|0|0|0|0|0|0|0|0|0|
0|0|1|12|12|10|5|1|1|10|10|0|0|1|0|1|0|0|10|1000|1|88|1|1|20|400|90|20|1"

Correct Each Frame and Encode a New Video

The second pass in the process of running Deshaker is taken care of by pass2.avs, a temporary AVISynth script with contents similar to the following:

LoadVirtualDubPlugin("C:\Util\VirtualDub\Plugins\Deshaker.vdf", "Deshaker")
QTInput("D:\Photos\Costa Rica 2012\Videos\img_0128.mov", color=1)
clip = TurnLeft()
clip + BlankClip(clip, 10)
Deshaker("18|2|30|4|1|0|1|0|640|480|1|1|650|650|1000|650|4|0|6|2|8|30|300|4|
D:\Photos\Costa Rica 2012\Videos\img_0128_deshaker.log|0|0|0|0|0|0|0|0|0|0|0|
0|0|1|12|12|10|5|1|1|10|10|0|0|1|0|1|0|0|10|1000|1|88|1|1|20|400|90|20|1")
Trim(0, FrameCount - 3)
Width > Height ? Lanczos4Resize(960, 540) : Lanczos4Resize(540, 960)
TurnRight()

The Deshaker options string is identical to the previous one, except the second option changes to 2 to indicate this is the second pass. This time, Deshaker will output the corrected video frames, but we need to perform a few extra steps in the script:

• Deshaker outputs 10 bogus frames at the beginning of the clip — the same number we specified for options #46 – 47 (Previous and Future frames) in the options string. We’ll trim those later, from the ffmpeg command line. It also eats the final 10 frames. To avoid losing real frames, we append 10 blank frames at the end of the input video, using the expression clip + BlankClip(clip, 10).
• For some reason, Deshaker also adds two garbage frames at the end of the video. Those are removed using the Trim() function.
• I like to resize the video down to 960×540 using Lanczos4Resize(). This makes the file size smaller and the processing time shorter, while still retaining a lot of detail. Besides, 1920×1080 is way too big and slow to view on the average desktop.
• Finally, we rotate the video back to its correct orientation.

We run the video through the above AVISynth script using the following unwieldy command line:

c:\util\ffmpeg\ffmpeg.exe -y -i pass2.avs -itsoffset 0.33333 -i img_0128.mov -map 0 -map 1:1
-pix_fmt yuv420p -vcodec libx264 -preset veryslow -crf 15 -x264opts frameref=15:fast_pskip=0
-acodec copy -ss 0.33333 img_0128_inprogress.mp4

The idea behind this command line is to grab the processed video from AVISynth script, grab the audio from the original iPhone 4S video, splice them together while compensating for audio drift, and save the result as img_0128_inprogress.mp4 using the best compression. I chose to let ffmpeg handle the audio because when I tried to let AVISynth handle it (by adding audio=1 to QTInput), ffmpeg reported errors. You can look up each command-line argument in the ffmpeg documentation, but here’s a detailed breakdown:

• -i pass2.avs and -i img_0128.mov define the two input files for ffmpeg.
• -map 0 -map 1:1 tells ffmpeg to take the video from the first input file and the audio from the second input file.
• -itsoffset 0.33333 tells ffmpeg to delay the audio signal of img_0128.mov so that it matches the video frames coming from the pass2.avs. Remember, Deshaker added 10 bogus frames to the beginning of the video, and since iPhone videos are 30 frames/sec, 10 frames equals 0.33333 seconds.
• -ss 0.33333 tells it to discard the first 10 frames when writing the output, while still keeping the audio correctly aligned. This is the only way to correctly discard the bogus video frames. (I tried using the Trim command within AVISynth instead, but it doesn’t work: It only ends up trimming the input, and Deshaker still adds bogus frames at the start.)
• -pix_fmt yuv420p converts the video frames to a color space compatible with H.264. The command line will still work without this option, but you’ll get a warning if you remove it.
• -vcodec libx264 -preset veryslow -crf 15 -x264opts frameref=15:fast_pskip=0 tells ffmpeg to use the highest quality H.264 compression. I basically just copied these settings from here. I didn’t bother with 2-pass H.264 compression because I didn’t want to run Deshaker twice. These settings give excellent video quality, though the resulting bitrate tends to be very high and could probably be optimized. If anyone wants to share some tips here, I’m all ears.
• -acodec copy tells ffmpeg to copy the compressed AAC audio data directly from the input to the output file without converting it. This way, we save processing time without degrading audio quality.

And that’s it! I’m not saying this is the best approach, but it worked well for me. If I was willing to spend more time on it, I would investigate using MVTools2 to estimate the camera motion and generate the Deshaker log file in the first pass, as it seems to run faster based on past experience.

If you have any success using this script to remove camera shake from your own videos, I’d be interested to hear about it!

In my previous post, I showed how to implement a synchronization primitive known as the Benaphore in C++ on Win32. The Benaphore is not lock-free (being a lock itself), but it does serve as a simple yet instructive example of writing a synchronization primitive in user space. It also offers very low overhead when there’s no lock contention.

One limitation of the implementation I showed was that it was non-recursive. This means that if the same thread attempts to obtain the same lock twice, it will deadlock. In this post, I’ll show how to extend the implementation to support recursive locking.

Recursive locking is useful when you have a module which calls itself through its own public interface. For example, in a memory manager, you might encounter the following code:

void* MemoryManager::Realloc(void* ptr, size_t size)
{
    AUTO_LOCK_MACRO(m_lock);
 
    if (ptr == NULL)
    {
        return Alloc(size);
    }
    else if (size == 0)
    {
        Free(size);
        return NULL;
    }
    else
        ...
}
 
void* MemoryManager::Alloc(size_t size)
{
    AUTO_LOCK_MACRO(m_lock);
 
    ...
}

AUTO_LOCK_MACRO is, of course, one of those funky C++ macros which obtains the lock and automatically unlocks it when we exit the function scope.

As you can see, if we pass NULL to Realloc, the lock will be obtained once by the Realloc function, and a second time (recursively) when Alloc is called. Obviously, it would be very easy to modify this particular example to avoid the recursive lock, but in large, multithreaded projects, you’re likely to find other examples.

We can extend our Win32 implementation of the Benaphore to support recursive locking as follows. I’ve added two new members to the class: m_owner, which stores the thread ID (TID) of the current owner, and m_recursion, which stores the recursion count.

Expert readers will note that this code does not use the new C++11 atomic library standard. As such, it’s destined to go out of style in the long run. However, this is the style we’ve been using in the game industry since the mid-2000′s. It will compile using any Microsoft compiler, and all Windows-specific calls have equivalents on other platforms.

// Define this to {} in a retail build:
#define LIGHT_ASSERT(x) { if (!(x)) DebugBreak(); }
 
class RecursiveBenaphore
{
private:
    LONG m_counter;
    DWORD m_owner;
    DWORD m_recursion;
    HANDLE m_semaphore;
 
public:
    RecursiveBenaphore::RecursiveBenaphore()
    {
        m_counter = 0;
        m_owner = 0;            // an invalid thread ID
        m_recursion = 0;
        m_semaphore = CreateSemaphore(NULL, 0, 1, NULL);
    }
 
    RecursiveBenaphore::~RecursiveBenaphore()
    {
        CloseHandle(m_semaphore);
    }
 
    void Lock()
    {
        DWORD tid = GetCurrentThreadId();
        if (_InterlockedIncrement(&m_counter) > 1) // x86/64 guarantees acquire semantics
        {
            if (tid != m_owner)
                WaitForSingleObject(m_semaphore, INFINITE);
        }
        //--- We are now inside the Lock ---
        m_owner = tid;
        m_recursion++;
    }
 
    void Unlock()
    {
        DWORD tid = GetCurrentThreadId();
        LIGHT_ASSERT(tid == m_owner);
        DWORD recur = --m_recursion;
        if (recur == 0)
            m_owner = 0;
        DWORD result = _InterlockedDecrement(&m_counter); // x86/64 guarantees release semantics
        if (result > 0)
        {
            if (recur == 0)
                ReleaseSemaphore(m_semaphore, 1, NULL);
        }
        //--- We are now outside the Lock ---
    }
};

As in the original Benaphore, the first thread to call Lock will take ownership without making any expensive kernel calls. It also performs some bookkeeping: It sets m_owner to its own TID, and m_recursion becomes 1. If the same thread calls Lock again, it will increment both m_counter and m_recursion.

Correspondingly, when the same thread calls Unlock, it will decrement both m_counter and m_recursion, but it will only call ReleaseSemaphore once m_recursion is decremented back down to 0. If m_recursion remains greater than 0, it means that the current thread is still holding the lock in an outer scope, so it’s not yet safe to relinquish ownership to other threads.

Now, if you scour the Internet, you’ll find it’s full of broken lock-free code and synchronization primitives. So why should you believe the code here is any different? For one thing, it’s been stress tested. In my opinion, that’s the most valuable thing. I’ve written a small test application which spawns various numbers of threads, each hammering on this lock at random times and with random recursion depths. It updates some shared data within each lock and performs various consistency checks. You can download the source code here.

And for good measure, here’s a TryLock method.

    bool TryLock()
    {
        DWORD tid = GetCurrentThreadId();
        if (m_owner == tid)
        {
            // Already inside the lock
            _InterlockedIncrement(&m_counter);
        }
        else
        {
            LONG result = _InterlockedCompareExchange(&m_counter, 1, 0);
            if (result != 0)
                return false;
            //--- We are now inside the Lock ---
            m_owner = tid;
        }
        m_recursion++;
        return true;
    }

For those interested in the fine details, here are a few in particular:

• In RecursiveBenaphore::Unlock, it’s important to set m_owner back to 0 before calling _InterlockedDecrement. Otherwise, data corruption is possible. For example, suppose there are two threads with TIDs 123 and 456. Thread 123 has just completed a call to Unlock, leaving m_owner set to 123. The following could happen next:
  1. Both threads simultaneously enter RecursiveBenaphore::Lock.
  2. Thread 456 performs _InterlockedIncrement, gets 1 as the result, and therefore skips the WaitForSingleObject.
  3. Thread 123 performs _InterlockedIncrement and gets 2 as the result.
  4. Thread 123 checks and sees that id == m_owner, because thread 456 hasn’t changed it yet. Therefore, it also skips over WaitForSingleObject.

  Shortly thereafter, both threads will return from Lock, each believing it owns the lock. The data protected by the lock will likely become corrupted. Indeed, if you download the test application and delete this part of RecursiveBenaphore::Unlock, it will fail pretty quickly.
   

• Also in RecursiveBenaphore::Unlock, the value of m_recursion is copied to a local variable exactly once, and used locally from that point on. We would not, for example, want to re-read the value of m_recursion again after _InterlockedDecrement. At that point, another thread could have changed it.
   
• You may notice that m_recursion is modified without using any atomic operations. That’s because between the call to _InterlockedIncrement in Lock and _InterlockedDecrement in Unlock, the thread owning the lock has exclusive access to both m_owner and m_recursion, with all the necessary acquire and release semantics. Using atomics on m_recursion would be unnecessary and wasteful.

How is the last point guaranteed? It’s guaranteed by the semaphore in the slow case, and the atomic instructions in the uncontended case. On x86 and x64, the _InterlockedIncrement call generates a lock xadd instruction, which acts as a full memory barrier, guaranteeing both acquire and release semantics. This property is unique to x86/64. If you port this code to a dual-core iOS device, like the iPad 2, it wouldn’t be enough to call OSAtomicIncrement32 in place of _InterlockedIncrement. You’d have to call OSAtomicIncrement32Barrier to have similar guarantees. Even on Xbox 360, which shares the Win32 API but runs on PowerPC, the correct function to call is actually InterlockedIncrementAcquire.

I may have lost a few readers by this paragraph. Hopefully, it’s an exciting kind of lost. I’ll talk more about memory access semantics in the next post.

For those who haven’t delved into writing synchronization primitives, perhaps the RecursiveBenaphore has offered a glimpse into how delicate such code can be. Every small detail is there for a reason, ordering is critical and hidden guarantees are at play.

Why would you do such a thing? Well, in the past, some platforms (like BeOS) didn’t provide a lightweight mutex in the native API. Today, that’s not really a concern. I’m mainly showing this because it’s an interesting look at implementing synchronization primitives in general. As a bonus, it just so happens this implementation shaves almost 50% off the overhead of the Windows Critical Section in the uncontended case.

For the record, there are numerous ways to write your own mutex — or lock — entirely in user space, each with its own tradeoffs:

• Spin locks. These employ a busy-wait strategy which has the potential to waste CPU time, and in the worst case, can lead to livelock when competing threads run on the same core. Still, some programmers have found measurable speed improvements switching to spin locks in certain cases.
• Peterson’s algorithm is like a spinlock for two threads. A neat trick, but seems useless on today’s platforms. I find it noteworthy because Bartosz Milewski used this algorithm as a case study while discussing the finer points of the x86 memory model.
• Charles Bloom has a long writeup describing various mutex implementations. Excellent information, but possibly greek to anyone unfamiliar with C++11′s atomics library and Relacy‘s ($) notation.

Some of those implementations are pretty advanced. Here’s a relatively simple technique, using a semaphore and some atomic operations. I came up with it while writing my post about lock contention, but soon afterwards learned it was already in use as far back as 1996, when some engineers referred to it as the Benaphore. Here’s a C++ implementation for Win32:

#include <windows.h>
#include <intrin.h>
 
class Benaphore
{
private:
    LONG m_counter;
    HANDLE m_semaphore;
 
public:
    Benaphore()
    {
        m_counter = 0;
        m_semaphore = CreateSemaphore(NULL, 0, 1, NULL);
    }
 
    ~Benaphore()
    {
        CloseHandle(m_semaphore);
    }
 
    void Lock()
    {
        if (_InterlockedIncrement(&m_counter) > 1)
        {
            WaitForSingleObject(m_semaphore, INFINITE);
        }
    }
 
    void Unlock()
    {
        if (_InterlockedDecrement(&m_counter) > 0)
        {
            ReleaseSemaphore(m_semaphore, 1, NULL);
        }
    }
};

This implementation also serves as a convenient introduction to atomics, which are at the heart of many lock-free algorithms.

_InterlockedIncrement is an atomic operation on Win32. If multiple threads attempt an atomic operation at the same time, on the same piece of data, they will all line up in a row and execute one-at-a-time. This makes it possible to reason about what happens, and ensure correctness. It even works on multicore and multiprocessor systems. (For more information about atomics, check out this post by JF Dubé.)

Every modern processor supports atomic operations, though the APIs may differ in the meaning of the return values. On Win32, _InterlockedIncrement adds 1 to the specified integer and returns the new value. Since m_counter is initialized to 0, the first thread to call Lock will receive a return value of 1 from _InterlockedIncrement. As such, it skips over the WaitForSingleObject call and returns immediately. The lock now belongs to this thread, and life is peachy.

If another thread calls Lock while the first thread is still holding it, it will receive a return value of 2 from _InterlockedIncrement. This is a clue that the lock is already busy. At this point, it’s not safe to continue, so we bite the bullet and jump into one of those expensive kernel calls: WaitForSingleObject. This performs a decrement on the semaphore. We specified an initial count of 0 in CreateSemaphore, so the thread is now forced to wait until someone else comes along and increments this semaphore before it can proceed.

Next, suppose the first thread calls Unlock. The return value of _InterlockedDecrement will be 1. This is a clue that another thread is waiting for the lock, and that we should increment the semaphore using ReleaseSemaphore. The second thread is then able to continue, and it effectively obtains ownership of the lock.

Even if the timing is very tight, and the first thread calls ReleaseSemaphore before the second calls WaitForSingleObject, everything will function normally. And if you add a third, fourth or any number of other threads into the picture, that’s fine too. For good measure, you can even add a TryLock function to the implementation:

    bool TryLock()
    {
        LONG result = _InterlockedCompareExchange(&m_counter, 1, 0);
        return (result != 0);
    }

Performance and Caveats

You might have noticed the underscore in front of _InterlockedIncrement. This is the compiler intrinsic version of InterlockedIncrement. It outputs a lock xadd instruction in place. And since Lock is defined right inside the class definition of Benaphore, the compiler treats it as an inline function. A call Benaphore::Lock compiles down to 10 instructions using default Release settings, and there are no function calls in the uncontended case. Here’s the disassembly:

In the uncontended case, the Benaphore even outperforms a Critical Section on Win32. I timed a pair of uncontended lock/unlock operations, just as I previously did for the Mutex and Critical Section, and found the following timings:

If you have an application which is hitting a lock millions of times per second on Win32, this Benaphore implementation just might boost your overall performance by a couple of percent. If you forego the intrinsics and just use regular, non-intrinsic version of the atomics, an indirect call into kernel32.dll is involved, so the Benaphore loses some of its performance edge: 49.8 ns on my Core 2 Duo.

Furthermore, with some (fairly heavy) code modifications, you could even share this Benaphore between processes — something Critical Section isn’t capable of. You’d have to put m_counter in shared memory, and use a named semaphore.

There are some limitations to be aware of. First, this implementation is non-recursive. If the same thread attempts to lock the same Benaphore twice, it will deadlock. In my next post, I’ll extend the implementation to allow recursion.

Finally, an even more subtle caveat: If you port this code directly to other platforms, such as MacOS X and Linux, then depending how you use it, your code may become susceptible to priority inversion. MacOS X and Linux avoid priority inversion by performing priority inheritance when you take a POSIX lock. If you use a Benaphore, you’ll bypass this OS mechanism and it won’t be able to help you. Things are different on Windows, though: Windows fights priority inversion by randomly boosting the priority of starving threads, which behaves the same regardless of your choice of synchronization primitive.

Perhaps the turning point came in May 2004, when Intel canceled its latest single-core development effort to focus on multicore designs. Later that year, Herb Sutter wrote his now-famous article, The Free Lunch Is Over. Not all software will run remarkably faster year-over-year anymore, he warned us. Concurrent software would continue its meteoric rise, but single-threaded software was about to get left in the dust.

So, what’s happened since 2004? Clearly, multicore computing has become mainstream. Everybody acknowledges that single-threaded CPU performance no longer increases as quickly as it previously did — but at what rate is it actually increasing?

It’s tough to find an answer. Bill Dally of nVidia threw out a few numbers in a recent presentation: He had predicted 19% per year, but says it’s turned out closer to 5%. Last year, Chuck Moore of AMD presented this graph, suggesting that single-threaded CPU performance recently started going backwards:

These figures aren’t really consistent, and both struck me as a little low. Moreover, I couldn’t find another source to corroborate them. So I decided to crunch the numbers myself. I turned to SPEC, an industry-standard benchmark that’s been going strong since 1989. It’s the same benchmark used to plot a few data points on the above graph.

SPEC licenses their benchmarking software to various companies, collects results back from those licensees, and makes those results available on their website. One of their benchmark series, SPECint, was designed to measure the single-threaded integer performance of a machine. That sounds perfect, except for one catch: many licensees use automatic parallelization. I took some pains to remove those results from the dataset. I’ll share the method at the end of this post, and you can let me know if you think it’s valid.

I fetched SPEC’s data on Feb. 7, grouped the results by CPU brand, and generated the following graph. It consists of 5052 test results from 715 different CPU models, all gathered over the last 17 years:

Each test result is plotted according its hardware availability date, and the vertical axis uses a logarithmic scale. The graph incorporates results from three different benchmark suites (CPU95, CPU2000 and CPU2006), but I’ve normalized the results in order to see historic trends.

The red line is meant to represent mainstream CPU performance. I drew it manually, using the less-than-scientific method of eyeballing the points for Pentium, PowerPC, Athlon and Core. If you’re willing to trust this line, it seems that in the eight years since January 2004, mainstream performance has increased by a factor of about 4.6x, which works out to 21% per year. Compare that to the 28x increase between 1996 and 2004! Things have really slowed down.

Here are a few machines located along the red line in the graph:

As you can see, Intel deserves credit for squeezing out the most single-threaded performance since 2004. If you remove all Intel CPUs from the data, a different picture emerges:

This is not too surprising, as AMD is pretty open about their stance on single-threaded performance. Bulldozer, their latest microarchitecture, is meant to shine in multithreaded workloads.

So far we’ve only looked at integer performance. SPEC also publishes SPECfp, an equivalent benchmark for floating-point performance. Floating-point performance has always been important for heavy-duty computation such as scientific simulation or 3D rendering. Here are the results, which I’ve also adjusted to eliminate autoparallelization:

Prior to 2004, it climbed even faster than integer performance, at 64% per year: a doubling period of 73 weeks. After that, it leveled off at the same 21% per year.

Up until 2002, we see a huge difference in floating-point performance between mainstream and workstation CPUs. The Alpha, SPARC and MIPS all ran up to 8x faster. Of course, you had to pay $10000 or more to get your hands on such a workstation. This is an interesting reminder that CPUs are, in fact, things created by businesses to make money! They don’t become faster entirely by technological forces. They become faster by economic forces.

Which brings us back to the present day. For reasons which others understand better than me, involving thermal design power and ILP, it’s now more cost-effective for manufacturers to pack additional cores onto a die than to push the single-threaded performance envelope much further.

Given the significance of this shift away from single-threaded performance, I was surprised to not find more information about the actual trajectory of performance since 2004. At the same time, I can’t guarantee that the data I’ve presented perfectly reflects single-threaded CPU performance. I think my conclusions are fair, but any feedback or criticism about the approach is more than welcome.

How These Graphs Were Generated

All Python scripts are available on github. These scripts will download, analyze and adjust SPEC’s data, and render the graphs. If you’d like to run them yourself, see the README file for exact instructions.

As already mentioned, recent compilers like Intel C++ and IBM XL feature automatic parallelization, and it greatly skews the results towards certain benchmarks. For example, check out the performance of 462.libquantum in this result! SPEC permits the use of autoparallelization as long as it’s clearly indicated. Unfortunately, this compiler feature is so widely enabled, I couldn’t simply exclude all such results. If I had done so, I would be left with zero results for Intel’s Core i3, i5 and i7 processor families.

The compromise I chose was to identify the top six benchmarks which seem to benefit from automatic parallelization, disqualify those benchmarks from the test suite, and take the geometric mean of the remaining ones. This approach assumes that automatic parallelization does not work on every benchmark. For the list of disqualified benchmarks, and the algorithm which identifies them, check the github files.

In the end, you’ll find that even if you leave the disqualified benchmarks in the results, it doesn’t significantly change the conclusions in this post. It shifts most of the CPU2006 results upwards — up to 25% — which simultaneously shifts the conversion ratios from CPU95 and CPU2000 upwards, keeping everything roughly in line.

In the future, it would be interesting for licensees to submit more results without automatic parallelization. That would help us more easily observe the performance trend of single CPU cores.

A profiling module is different from conventional profilers like xperf or VTune in that no third-party tools are required. You just drop the module into any C++ application, and the process collects and reports performance data by itself.

This particular profiling module is meant to act on one or more target modules in the application. A target module can be anything which exposes a well-defined API, such as a memory allocator. To make it work, you must insert a macro named API_PROFILER into every public function exposed by that API. Below, I’ve added it to dlmalloc, one of the functions in the Doug Lea Malloc API. The same macro should be added to dlrealloc, dlfree, and other public functions as well.

DEFINE_API_PROFILER(dlmalloc);

void* dlmalloc(size_t bytes)
{
    API_PROFILER(dlmalloc);

#if USE_LOCKS
    ensure_initialization();
#endif

    if (!PREACTION(gm))
    {
        void* mem;
        size_t nb;
        if (bytes <= MAX_SMALL_REQUEST)
        {
            ...

The macro takes a single argument, which is just an identifier for the target module being profiled. For this to be a valid identifier, you must place exactly one DEFINE_API_PROFILER macro at global scope, as seen above. You can also insert DECLARE_API_PROFILER anywhere at global scope, perhaps in a header file, in the same way that you'd forward declare a global variable or function.

When the application runs, each thread will automatically log performance statistics once per second, including the thread identifier (TID), time spent inside the target module, and the number of calls. Here, we see performance statistics across six different threads:

TID 0x13bc time spent in "dlmalloc": 7/1001 ms 0.7% 6481x
TID 0x1244 time spent in "dlmalloc": 6/1000 ms 0.6% 6166x
TID 0x198 time spent in "dlmalloc": 0/3072 ms 0.0% 2x
TID 0x11d0 time spent in "dlmalloc": 0/1113 ms 0.0% 6x
TID 0x12a4 time spent in "dlmalloc": 0/1000 ms 0.0% 20x
TID 0xc14 time spent in "dlmalloc": 4/1011 ms 0.4% 3243x

To identify each thread, simply break in the debugger and look for the TID in the Threads view.

Most of the profiling module is implemented in a single header file, as follows. For simplicity, I've only provided the Windows version, but you could easily port the code to other platforms.

#define ENABLE_API_PROFILER 1     // Comment this line to disable the profiler
 
#if ENABLE_API_PROFILER
 
//------------------------------------------------------------------
// A class for local variables created on the stack by the API_PROFILER macro:
//------------------------------------------------------------------
class APIProfiler
{
public:
    //------------------------------------------------------------------
    // A structure for each thread to store information about an API:
    //------------------------------------------------------------------
    struct ThreadInfo
    {
        INT64 lastReportTime;
        INT64 accumulator;   // total time spent in target module since the last report
        INT64 hitCount;      // number of times the target module was called since last report
        const char *name;    // the name of the target module
    };
 
private:
    INT64 m_start;
    ThreadInfo *m_threadInfo;
 
    static float s_ooFrequency;      // 1.0 divided by QueryPerformanceFrequency()
    static INT64 s_reportInterval;   // length of time between reports
    void Flush(INT64 end);
 
public:
    __forceinline APIProfiler(ThreadInfo *threadInfo)
    {
        LARGE_INTEGER start;
        QueryPerformanceCounter(&start);
        m_start = start.QuadPart;
        m_threadInfo = threadInfo;
    }
 
    __forceinline ~APIProfiler()
    {
        LARGE_INTEGER end;
        QueryPerformanceCounter(&end);
        m_threadInfo->accumulator += (end.QuadPart - m_start);
        m_threadInfo->hitCount++;
        if (end.QuadPart - m_threadInfo->lastReportTime > s_reportInterval)
            Flush(end.QuadPart);
    }
};
 
//----------------------
// Profiler is enabled
//----------------------
#define DECLARE_API_PROFILER(name) \
    extern __declspec(thread) APIProfiler::ThreadInfo __APIProfiler_##name;
 
#define DEFINE_API_PROFILER(name) \
    __declspec(thread) APIProfiler::ThreadInfo __APIProfiler_##name = { 0, 0, 0, #name };
 
#define TOKENPASTE2(x, y) x ## y
#define TOKENPASTE(x, y) TOKENPASTE2(x, y)
#define API_PROFILER(name) \
    APIProfiler TOKENPASTE(__APIProfiler_##name, __LINE__)(&__APIProfiler_##name)
 
#else
 
//----------------------
// Profiler is disabled
//----------------------
#define DECLARE_API_PROFILER(name)
#define DEFINE_API_PROFILER(name)
#define API_PROFILER(name)
 
#endif

The DEFINE_API_PROFILER macro defines a thread-local variable using the __declspec(thread) modifier. This gives each thread its own private data, independent of other threads, so the whole system works in a multithreaded environment with little performance penalty. In GCC, the equivalent storage class modifier would be __thread. The overhead for such storage is low, but on Windows, there's one catch: you can't use it across DLLs.

The API_PROFILER macro creates a C++ object on the stack, taking advantage of the constructor to signal the beginning and the destructor to signal the end of the section being measured. The macro uses a token-pasting trick, using the current line number, to create unique local variable names.

It's important not to call this macro recursively. In other words, don't insert API_PROFILER anywhere that might be called within the scope of another API_PROFILER marker, using the same identifier. If you do, you'll end up counting the time spent inside the target module twice! If absolutely necessary, you could modify the profiling module to circumvent this limitation, at the cost of a little extra overhead.

The destructor sometimes calls a function named Flush. It's a heavier function, so we define it in a separate .cpp file, and make sure it's only called once per second:

#if ENABLE_API_PROFILER
 
static const float APIProfiler_ReportIntervalSecs = 1.0f;
 
float APIProfiler::s_ooFrequency = 0;
INT64 APIProfiler::s_reportInterval = 0;
 
//------------------------------------------------------------------
// Flush is called at the rate determined by APIProfiler_ReportIntervalSecs
//------------------------------------------------------------------
void APIProfiler::Flush(INT64 end)
{
    // Auto-initialize globals based on timer frequency:
    if (s_reportInterval == 0)
    {
        LARGE_INTEGER freq;
        QueryPerformanceFrequency(&freq);
        s_ooFrequency = 1.0f / freq.QuadPart;
        MemoryBarrier();
        s_reportInterval = (INT64) (freq.QuadPart * APIProfiler_ReportIntervalSecs);
    }
 
    // Avoid garbage timing on first call by initializing a new interval:
    if (m_threadInfo->lastReportTime == 0)
    {
        m_threadInfo->lastReportTime = m_start;
        return;
    }
 
    // Enough time has elapsed. Print statistics to console:
    float interval = (end - m_threadInfo->lastReportTime) * s_ooFrequency;
    float measured = m_threadInfo->accumulator * s_ooFrequency;
    printf("TID 0x%x time spent in \"%s\": %.0f/%.0f ms %.1f%% %dx\n",
        GetCurrentThreadId(),
        m_threadInfo->name,
        measured * 1000,
        interval * 1000,
        100.f * measured / interval,
        m_threadInfo->hitCount);
 
    // Reset statistics and begin next timing interval:
    m_threadInfo->lastReportTime = end;
    m_threadInfo->accumulator = 0;
    m_threadInfo->hitCount = 0;
}
 
#endif

In the above code, printf is used for logging, but you could easily replace it with calls to sprintf and OutputDebugString, or anything else. The nice thing about logging to a console is that it works even when there is no graphical display, such as during the loading screen of a game, or when the application is starting up. Those are moments when you might be particularly interested in profiling a specific API.

Another convenient thing about this profiling module is that no explicit initialization is required. The very first time the macro is hit, it will call Flush. The first thread to enter Flush will see that s_reportInterval is not yet initialized, and will initialize itself. It doesn't matter if two threads end up trying to initialize the globals at the same time; they will both write the same result.

I measured the overhead introduced by the API_PROFILER macro on two processors: 99 ns on a 1.86 GHz Core 2 Duo, and 30.8 ns on a 2.66 GHz Xeon. That's just a little slower than an uncontended Windows Critical Section, making this a pretty good technique for fine-grained profiling. You could reduce the overhead further by calling __rdtsc instead of QueryPerformanceCounter, but the resulting numbers would be less reliable on multicore systems, so I chose not to mess with that.

Built-in profiling modules are nothing new — Jeff Everett describes another in-game profiler in Game Programming Gems 2. Hopefully, I've at least presented a few twists on the idea. I'd be interested to hear about any twists of your own. As far as I know, no third-party profiler is capable of profiling a multithreaded API as easily & accurately as the method I've described here — whether it's Valgrind, xperf, VTune, Shark, PIX, Tuner, Visual Studio Profiler, or any other. Readers, correct me if I'm wrong!

Such profilers can, on the other hand, show you when a particular module becomes heavy — the module's internal functions will appear near the top of PC sampling summaries, for example. Sometimes, even random breaking offers a similar clue. At that point, you might be compelled to use a built-in profiling module like this one, to drill deeper and to measure the impact of subsequent code changes.

The Windows SDK provides two lock implementations for C/C++: the Mutex and the Critical Section. (As Ned Batchelder points out, Critical Section is probably not the best name to give to the lock itself, but we’ll forgive that here.)

The Windows Critical Section is what we call a lightweight mutex. It’s optimized for the case when there are no other threads competing for the lock. To demonstrate using a simple example, here’s a single thread which locks and unlocks a Windows Mutex exactly one million times.

HANDLE mutex = CreateMutex(NULL, FALSE, NULL);
for (int i = 0; i < 1000000; i++)
{
    WaitForSingleObject(mutex, INFINITE);
    ReleaseMutex(mutex);
}
CloseHandle(mutex);

Here's the same experiment using a Windows Critical Section.

CRITICAL_SECTION critSec;
InitializeCriticalSection(&critSec);
for (int i = 0; i < 1000000; i++)
{
    EnterCriticalSection(&critSec);
    LeaveCriticalSection(&critSec);
}
DeleteCriticalSection(&critSec);

If you insert some timing code around the inner loop, and divide the result by one million, you'll find the average time required for a pair of lock/unlock operations in both cases. I did that, and ran the experiment on two different processors. The results:

The Critical Section is 25 times faster. As Larry Osterman explains, the Windows Mutex enters the kernel every time you use it, while the Critical Section does not. The tradeoff is that you can't share a Critical Section between processes. But who cares? Most of the time, you just want to protect some data within a single process. (It is actually possible to share a lightweight mutex between processes - just not using a Critical Section.)

Now, suppose you have a thread which acquires a Critical Section 100000 times per second, and there are no other threads competing for the lock. Based on the above figures, you can expect to pay between 0.2% and 0.6% in lock overhead. Not too bad! At lower frequencies, the overhead becomes negligible. I'm ignoring the hidden cost of synchronizing the processor's cache, which is something I'll write about in a future post, but it doesn't make a big difference.

Other Platforms

In MacOS 10.6.6, a lock implementation is provided using the POSIX Threads API. It's a lightweight mutex which doesn't enter the kernel unless there's contention. A pair of uncontended calls to pthread_mutex_lock and pthread_mutex_unlock takes about 92 ns on my 1.86 GHz Core 2 Duo. Interestingly, it detects when there's only one thread running, and in that case switches to a trivial codepath taking only 38 ns.

MacOS also offers NSLock, an Objective-C class, but this is really just a wrapper around the aforementioned POSIX mutex. Because each operation must wind its way through objc_msgSend, the overhead is a little higher: 155 ns on my Core 2 Duo, or 98 ns if there's only a single thread.

Naturally, Ubuntu 11.10 provides a lock implementation using the POSIX Threads API as well. It's another lightweight mutex, based on a Linux-specific construct known as a futex. A pair of pthread_mutex_lock/pthread_mutex_unlock calls takes about 66 ns on my Core 2 Duo. You can even share this implementation between processes, but I didn't test that.

Even the Playstation 3 SDK offers a choice between a lightweight mutex and a heavy one. Back in 2007, early in the development of a Playstation 3 game I worked on, we were using the heavy mutex. Switching to the lightweight mutex made the game start 17 seconds faster! For me, that's when the difference really hit home.

In my previous post, I argued against the misconception that locks are slow and provided some data to support the argument. At this point, it should be clear that if you aren't using a lightweight mutex, the entire argument goes out the window. I'm fairly sure that the existence of heavy lock implementations has only added to this misconception over the years.

Some of you old-timers may point out ancient platforms where a heavy lock was the only implementation available, or when a semaphore had to be used for the job. But it seems all modern platforms offer a lightweight mutex. And even if they didn't, you could write your own lightweight mutex at the application level, even sharing it between processes, provided you're willing to live with certain caveats. You'll find one example in my followup post, Roll Your Own Lightweight Mutex.

It’s true that locking is slow on some platforms, or when the lock is highly contended. And when you’re developing a multithreaded application, it’s very common to find a huge performance bottleneck caused by a single lock. But that doesn’t mean all locks are slow. As I’ll show in this post, sometimes a locking strategy achieves excellent performance.

Perhaps the most easily-overlooked source of this misconception: Not all programmers may be aware of the difference between a lightweight mutex and a “kernel mutex”. I’ll talk about that in my next post, Always Use a Lightweight Mutex. For now, let’s just say that if you’re programming in C/C++ on Windows, the Critical Section object is the one you want.

Other times, the conclusion that locks are slow is supported by a benchmark. For example, this post measures the performance of a lock under heavy conditions: each thread must hold the lock to do any work (high contention), and the lock is held for an extremely short interval of time (high frequency). It’s a good read, but in a real application, you generally want to avoid using locks in that way. To put things in context, I’ve devised a benchmark which includes both best-case and worst-case usage scenarios for locks.

Locks may be frowned upon for other reasons. There’s a whole other family of techniques out there known as lock-free (or lockless) programming. Lock-free programming is extremely challenging, but delivers huge performance gains in a lot of real-world scenarios. I know programmers who spent days, even weeks fine-tuning a lock-free algorithm, subjecting it to a battery of tests, only to discover hidden timing bugs several months later. The combination of danger and reward can be very enticing to a certain kind of programmer — and this includes me, as you’ll see in future posts! With lock-free techniques beckoning us to use them, locks can begin to feel boring, slow and busted.

But don’t disregard locks yet. One good example of a place where locks perform admirably, in real software, is when protecting the memory allocator. Doug Lea’s Malloc is a popular memory allocator in video game development, but it’s single threaded, so we need to protect it using a lock. During gameplay, it’s not uncommon to see multiple threads hammering the memory allocator, say around 15000 times per second. While loading, this figure can climb to 100000 times per second or more. It’s not a big problem, though. As you’ll see, locks handle the workload like a champ.

Lock Contention Benchmark

In this test, we spawn a thread which generates random numbers, using a custom Mersenne Twister implementation. Every once in a while, it acquires and releases a lock. The lengths of time between acquiring and releasing the lock are random, but they tend towards average values which we decide ahead of time. For example, suppose we want to acquire the lock 15000 times per second, and keep it held 50% of the time. Here’s what part of the timeline would look like. Red means the lock is held, grey means it’s released:

This is essentially a Poisson process. If we know the average amount of time to generate a single random number — 6.349 ns on a 2.66 GHz quad-core Xeon — we can measure time in work units, rather than seconds. We can then use the technique described in my previous post, How to Generate Random Timings for a Poisson Process, to decide how many work units to perform between acquiring and releasing the lock. Here’s the implementation in C++. I’ve left out a few details, but if you like, you can download the complete source code here.

QueryPerformanceCounter(&start);
for (;;)
{
    // Do some work without holding the lock
    workunits = (int) (random.poissonInterval(averageUnlockedCount) + 0.5f);
    for (int i = 1; i < workunits; i++)
        random.integer();       // Do one work unit
    workDone += workunits;
 
    QueryPerformanceCounter(&end);
    elapsedTime = (end.QuadPart - start.QuadPart) * ooFreq;
    if (elapsedTime >= timeLimit)
        break;
 
    // Do some work while holding the lock
    EnterCriticalSection(&criticalSection);
    workunits = (int) (random.poissonInterval(averageLockedCount) + 0.5f);
    for (int i = 1; i < workunits; i++)
        random.integer();       // Do one work unit
    workDone += workunits;
    LeaveCriticalSection(&criticalSection);
 
    QueryPerformanceCounter(&end);
    elapsedTime = (end.QuadPart - start.QuadPart) * ooFreq;
    if (elapsedTime >= timeLimit)
        break;
}

Now suppose we launch two such threads, each running on a different core. Each thread will hold the lock during 50% of the time when it can perform work, but if one thread tries to acquire the lock while the other thread is holding it, it will be forced to wait. This is known as lock contention.

In my opinion, this is a pretty good simulation of the way a lock might be used in a real application. When we run the above scenario, we find that each thread spends roughly 25% of its time waiting, and 75% of its time doing actual work. Together, both threads achieve a net performance of 1.5x compared to the single-threaded case.

I ran several variations of the test on a 2.66 GHz quad-core Xeon, from 1 thread, 2 threads, all the way up to 4 threads, each running on its own core. I also varied the duration of the lock, from the trivial case where the the lock is never held, all the way up to the maximum where each thread must hold the lock for 100% of its workload. In all cases, the lock frequency remained constant — threads acquired the lock 15000 times for each second of work performed.

The results were interesting. For short lock durations, up to say 10%, the system achieved very high parallelism. Not perfect parallelism, but close. Locks are fast!

To put the results in perspective, I analyzed the memory allocator lock in a multithreaded game engine. During gameplay, with 15000 locks per second coming from 3 threads, the lock duration was in the neighborhood of just 2%. That’s well within the comfort zone on the left side of the diagram.

These results also show that once the lock duration passes 90%, there’s no point using multiple threads anymore. A single thread performs better. Most surprising is the way the performance of 4 threads drops off a cliff around the 60% mark! This looked like an anomaly, so I re-ran the tests several additional times, even trying a different testing order. The same behavior happened consistently. My best hypothesis is that the experiment hits some kind of snag in the Windows scheduler, but I didn’t investigate further.

Lock Frequency Benchmark

Even a lightweight mutex has overhead. As my next post shows, a pair of lock/unlock operations on a Windows Critical Section takes about 23.5 ns on the CPU used in these tests. Therefore, 15000 locks per second is low enough that lock overhead does not significantly impact the results. But what happens as we turn up the dial on lock frequency?

The algorithm offers very fine control over the amount of work performed between one lock and the next, so I performed a new batch of tests using smaller amounts: from a very fine-grained 10 ns between locks, all the way up to 31 μs, which corresponds to roughly 32000 acquires per second. Each test used exactly two threads:

As you might expect, for very high lock frequencies, the overhead of the lock itself begins to dwarf the actual work being done. Several benchmarks you’ll find online, including the one linked earlier, fall into the bottom-right corner of this chart. At such frequencies, you’re talking about some seriously short lock times — on the scale of a few CPU instructions. The good news is that, when the work between locks is that simple, a lock-free implementation is more likely to be feasible.

At the same time, the results show that locking up to 320000 times per second (3.1 μs between successive locks) is not unreasonable. In game development, the memory allocator may flirt with this frequency during load times. You can still achieve more than 1.5x parallelism if the lock duration is short.

We’ve now seen a wide spectrum of lock performance: cases where it performs great, and cases where the application slows to a crawl. I’ve argued that the lock around the memory allocator in a game engine will often achieve excellent performance. Given this example from the real world, it cannot be said that all locks are slow. Admittedly, it’s very easy to abuse locks, but one shouldn’t live in too much fear — any resulting bottlenecks will show up during careful profiling. When you consider how reliable locks are, and the relative ease of understanding them (compared to lock-free techniques), locks are actually pretty awesome sometimes.

The goal of this post was to give locks a little respect where deserved — corrections are welcome. I also realize that locks are used in a wide variety of industries and applications, and it may not always be so easy to strike a good balance in lock performance. If you’ve found that to be the case in your own experience, I would love to hear from you in the comments.

Any time you have events which occur individually at random moments, but which tend to occur at an average rate when viewed as a group, you have a Poisson process.

For example, the USGS estimates that each year, there are approximately 13000 earthquakes of magnitude 4+ around the world. Those earthquakes are scattered randomly throughout the year, but there are more or less 13000 per year. That’s one example of a Poisson process. The Wikipedia page lists several others.

In statistics, there are a bunch of functions and equations to help model a Poisson process. I’ll present one of those functions in this post, and demonstrate its use in writing a simulation.

The Exponential Distribution

If 13000 such earthquakes happen every year, it means that, on average, one earthquake happens every 40 minutes. So, let’s define a variable λ = and call it the rate parameter. The rate parameter λ is a measure of frequency: the average rate of events (in this case, earthquakes) per unit of time (in this case, minutes).

Knowing this, we can ask questions like, what is the probability that an earthquake will happen within the next minute? What’s the probability within the next 10 minutes? There’s a well-known function to answer such questions. It’s called the cumulative distribution function for the exponential distribution, and it looks like this:

Basically, the more time passes, the more likely it is that, somewhere in the world, an earthquake will occur. The word “exponential”, in this context, actually refers to exponential decay. As time passes, the probability of having no earthquake decays towards zero — and correspondingly, the probability of having at least one earthquake increases towards one.

Plugging in a few values, we find that:

• The probability of having an earthquake within the next minute is . This value is pretty close to , our prescribed earthquake frequency, but it’s not equal.
• The probability of having an earthquake within the next 10 minutes is .

In particular, note that after 40 minutes — the prescribed average time between earthquakes — the probability is only . So, given any 40 minute interval of time, it’s pretty likely that we’ll have an earthquake within that time interval, but it won’t always happen.

Writing a Simulation

Now, suppose we want to simulate the occurrence of earthquakes in a game engine, or some other kind of program. First, we need to figure out when each earthquake should begin.

One approach is to loop, and after each interval of X minutes, sample a random floating-point value between 0 and 1. If this number is less than , then start an earthquake! X could even be a fractional value, so you could sample several times per minute, or even several times per second. This approach will probably work just fine, as long as your random number generator is uniform and offers enough numerical precision. However, if you intend to sample 60 times per second, with λ = , you’ll need at least 18 bits of precision from the random number generator, which the Standard C Runtime Library doesn’t always offer.

Another approach is to sidestep the whole sampling strategy, and simply write a function to determine the exact amount of time until the next earthquake. This function should return random numbers, but not the uniform kind of random number produced by most generators. We want to generate random numbers in a way that follows our exponential distribution.

Donald Knuth describes a way to generate such values in section 3.4.1 (D) of The Art of Computer Programming. Simply choose a random point on the y-axis between 0 and 1, distributed uniformly, and locate the corresponding time value on the x-axis. For example, if we choose the point 0.2 from the top of the graph, the time until our next earthquake would be 64.38 minutes.

Given that the inverse of the exponential function is ln, it’s pretty easy to write this analytically, where U is the random value between 0 and 1:

The Implementation

Here’s one way to implement it in Python. Note that you can’t pass zero to math.log, but we avoid that by subtracting the result of random.random, which is always less than one, from one.

import math
import random

def nextTime(rateParameter):
    return -math.log(1.0 - random.random()) / rateParameter

Here are a few sample calls. The values look pretty reasonable:

>>> nextTime(1/40.0)
91.074923814190498
>>> nextTime(1/40.0)
46.88573030224817
>>> nextTime(1/40.0)
14.965086245136733
>>> nextTime(1/40.0)
26.902965535881194

Let’s run some tests to make sure that the average time returned by this function really is 40. The following expression calculates the average of one million calls, and the results are pretty consistent. I’m always amazed to see randomness behaving the way we want!

>>> sum([nextTime(1/40.0) for i in xrange(1000000)]) / 1000000
39.985564565743751
>>> sum([nextTime(1/40.0) for i in xrange(1000000)]) / 1000000
40.029018385760551
>>> sum([nextTime(1/40.0) for i in xrange(1000000)]) / 1000000
40.016843319423266
>>> sum([nextTime(1/40.0) for i in xrange(1000000)]) / 1000000
39.965097296560664

Just for fun, here’s a series of points spaced according to the output of nextTime. This is basically what a Poisson process looks like when plotted along a timeline:

And here’s an implementation of nextTime in C, using the standard library’s random number generator. Again, we’re careful not to pass zero to logf.

#include <math.h>
#include <stdlib.h>

float nextTime(float rateParameter)
{
    return -logf(1.0f - (float) random() / (RAND_MAX + 1)) / rateParameter;
}

This technique could have various applications in a game engine, such as spawning particles from a particle emitter, or choosing moments when an AI could take a decision. I also use it in my next post, to measure the performance of threads which hold a lock for various intervals of time.

Any stats experts out there? If I’ve abused any terminology, or if you see any way to improve this post, I’d be interested in your comments.

The Mandelbrot set is a traditional favorite among authors of obfuscated code. You can find obfuscated code in C, Perl, Haskell, Python and many other languages. Nearly all examples render the Mandelbrot set as ASCII art.

The following Python script, on the other hand, begins as ASCII art:

_                                      =   (
                                        255,
                                      lambda
                               V       ,B,c
                             :c   and Y(V*V+B,B,  c
                               -1)if(abs(V)<6)else
               (              2+c-4*abs(V)**-0.4)/i
                 )  ;v,      x=1500,1000;C=range(v*x
                  );import  struct;P=struct.pack;M,\
            j  ='<QIIHHHH',open('M.bmp','wb').write
for X in j('BM'+P(M,v*x*3+26,26,12,v,x,1,24))or C:
            i  ,Y=_;j(P('BBB',*(lambda T:(T*80+T**9
                  *i-950*T  **99,T*70-880*T**18+701*
                 T  **9     ,T*i**(1-T**45*2)))(sum(
               [              Y(0,(A%3/3.+X%v+(X/v+
                               A/3/3.-x/2)/1j)*2.5
                             /x   -2.7,i)**2 for  \
                               A       in C
                                      [:9]])
                                        /9)
                                       )   )

It renders the Mandelbrot set as a full-color, anti-aliased, 1500×1000 image. Click to enlarge:

No third-party libraries are required — just pure Python. However, it will only run on Python 2.5 – 2.7; Python 3 is not supported. The output file is written to M.bmp, in Windows bitmap format.

It runs very slowly, taking about 18 minutes on my 1.86 GHz Core 2 Duo (or 9 minutes using PyPy). With some modifications, it’s possible to make this code run up to 20 times faster. However, doing so requires sacrificing either code size or image quality.

If you’re willing to leave the script running for a few hours, you can increase the image resolution on line 8. (Just make sure the width is divisible by 4.) The resulting detail is quite nice. Here are some 1:1 pixel excerpts from an image rendered at 7200×4800:

The entire 7200×4800 image is too large to share here, but it’s perfect for making prints. So that’s what I did! Notice the Python script superimposed in the lower-left corner. Is this the first poster to include its own source code?

If this kind of thing gives you kicks, you can order your own print (or a coffee mug) at CafePress.