View the documentation

View on Github

Mintomic (short for “minimal atomic”) draws lot of inspiration from the C/C++11 atomic library standards, with an important exception: In Mintomic, all atomic operations are “relaxed”. The only way to enforce memory ordering is with explicit fences. Here’s an example taken from my post about weak hardware ordering, rewritten using Mintomic:

// Define a shared atomic variable:
mint_atomic32_t flag;

void IncrementSharedValue10000000Times(TimeWaster& tw)
{
    int count = 0;
    while (count < 10000000)
    {
        tw.wasteRandomCycles();

        // Atomic read-modify-write operation:
        if (mint_compare_exchange_strong_32_relaxed(&flag, 0, 1) == 0)
        {
            mint_thread_fence_acquire();    // Acquire fence
            g_sharedValue++;
            mint_thread_fence_release();    // Release fence

            // Atomic store:
            mint_store_32_relaxed(&flag, 0);
            count++;
        }
    }
}

I started Mintomic out of the desire for a better way to share examples of lock-free programming on this blog. So why not just use C++11 atomics instead?

• Availability. Some readers are likely stuck using Visual Studio 2010, GCC 4.3, or an older compiler in which C++11 atomics are not available.
• Efficiency. The C++11 atomic library standard does not guarantee an efficient implementation, only a correct one. Technically, it does not even guarantee your code will be lock-free. Mintomic, on the other hand, always aims to generate optimal lock-free machine code on the platforms it supports.
• Lower learning curve. Mintomic asks you to keep fewer concepts in your head than C++11 atomics. I actually hope Mintomic will serve as a stepping stone towards learning C++11 atomics, since it maps to a subset of the same functionality and uses a similar naming convention.

In particular, because all of Mintomic’s atomic operations are relaxed, there are no sequentially consistent data types. This is different from C++11′s atomic library standard, where every atomic operation is sequentially consistent by default. To remind you of the difference, all atomic functions in Mintomic have the _relaxed suffix on their names.

Mintomic comes with a test suite which you can build and run yourself. The only requirement is CMake. This test suite helps ensure that Mintomic was implemented correctly on each platform, while bringing the library to life with a working set of examples. Here’s what its output looks like as a 64-bit Windows application:

And here it is on an iPhone 4S:

Every test case with a _fail suffix on its name contains an intentional bug. These tests are allowed to fail, and in general, designed to do so, in the same spirit as previous blog posts written here. You’ll notice that on 64-bit Windows, more of these test cases pass (10/11) than on the iPhone (6/11). The point is to show how incorrect lock-free code may succeed on certain platforms out of luck, depending on things like compiler ordering, machine word size and hardware memory model.

To support this test suite, Mintomic also comes with MintThreads, a portable C module to create and manipulate threads and semaphores, and MintPack, a collection of useful data structures in C++.

I’ll admit, the existence of a library like Mintomic is a bit funny, for several reasons. First, Mintomic is squarely focused on low-level lock-free programming, providing only relaxed atomics and standalone memory fences — two things which some C++ experts claim you should avoid using. Second, lock-free programming is definitely not something which every programmer will ever need to do, and in a real project, it typically accounts for a very tiny fraction of the codebase. And finally, some programmers believe that lock-free programming can always be encapsulated using patterns and techniques such as the actor model, goroutines, or functional programming.

Meanwhile, in the games industry at least, we use Mintomic-like primitives to achieve real-world performance gains on a regular basis. Not out of hubris, but because we have deadlines and performance targets, and there are (rare) occasions when it genuinely makes a difference. I’m certain that in the long term, low-level lock-free programming will continue to play a role in games, audio synthesis, financial trading software — anywhere that parallel, high-contention tasks must be optimized for latency. Therefore, it’s worth trying to stop messing up, which is why I keep blogging about it.

I think Mintomic turned out decent, though there are still things to improve. Feedback and suggestions are welcome! In the meantime, stay tuned for further examples using Mintomic.

For example, after installing some new software, you might want to know which files have changed on your hard drive. Or, if you’re a programmer getting started on a new project, you may need to follow a complex and unfamiliar build process. A list of recently modified files can reveal a lot about how that build process works.

Here’s a short Python script to create such a list. It lists the contents of a folder recursively, sorted by modification time.

As a simple example, I ran it after setting up a fresh copy of my random number sequence project. Here’s the output (with some lines deleted to save space):

2013-01-14 21:44:29       5564 .\build\Testing\Temporary\LastTest.log
2013-01-14 21:44:29         29 .\build\Testing\Temporary\CTestCostData.txt
------------------------------
2013-01-14 21:28:38         91 .\build\Win32\Release\ALL_BUILD\ALL_BUILD.lastbuildstate
2013-01-14 21:28:38       1560 .\build\Win32\Release\ALL_BUILD\custombuild.command.1.tlog
2013-01-14 21:28:38       6386 .\build\Win32\Release\ALL_BUILD\custombuild.read.1.tlog
2013-01-14 21:28:38        674 .\build\Win32\Release\ALL_BUILD\custombuild.write.1.tlog
2013-01-14 21:28:38         51 .\build\CMakeFiles\generate.stamp
2013-01-14 21:28:37         91 .\build\RandomSequence.dir\Release\RandomSequence.lastbuildstate
2013-01-14 21:28:37        678 .\build\RandomSequence.dir\Release\mt.command.1.tlog
2013-01-14 21:28:37        818 .\build\RandomSequence.dir\Release\mt.read.1.tlog
2013-01-14 21:28:37        446 .\build\RandomSequence.dir\Release\mt.write.1.tlog
2013-01-14 21:28:37       7680 .\build\Release\RandomSequence.exe
...
------------------------------
2013-01-14 21:28:21         86 .\build\CMakeFiles\cmake.check_cache
2013-01-14 21:28:21      12856 .\build\CMakeCache.txt
2013-01-14 21:28:21       3712 .\build\RandomSequence.sln
2013-01-14 21:28:21        270 .\build\CMakeFiles\TargetDirectories.txt
2013-01-14 21:28:21        391 .\build\CTestTestfile.cmake
2013-01-14 21:28:21       1586 .\build\cmake_install.cmake
2013-01-14 21:28:21       4204 .\build\CMakeFiles\generate.stamp.depend
2013-01-14 21:28:21      25207 .\build\ZERO_CHECK.vcxproj
2013-01-14 21:28:21        832 .\build\ZERO_CHECK.vcxproj.filters
...
------------------------------
2013-01-14 21:27:40        959 .\randomsequence.h
2013-01-14 21:27:40        416 .\.git\index
2013-01-14 21:27:40       1255 .\main.cpp
2013-01-14 21:27:40        714 .\README.md
2013-01-14 21:27:40        246 .\CMakeLists.txt
2013-01-14 21:27:40         12 .\.gitignore
2013-01-14 21:27:40        336 .\.git\config
2013-01-14 21:27:40        201 .\.git\logs\refs\heads\master
2013-01-14 21:27:40        201 .\.git\logs\HEAD
...

The horizontal dashes separate modifications greater than 10 seconds apart, which helps organize the files visually into groups. In reverse order, you can see the groups of files created by git clone, project files generated by cmake, the build output from cmake --build, and a couple of files written by ctest.

I’ve used this kind of script to help make sense of the filesystem on Ubuntu, and to figure out where files were written on MacOS X using the App Store.

Command-Line Options

Running with no options or with --help displays the following help message:

Usage: list_modifications.py [options] path [path2 ...]

Options:
  -h, --help    show this help message and exit
  -g SECS       set threshold for grouping files
  -f EXC_FILES  exclude files matching a wildcard pattern
  -d EXC_DIRS   exclude directories matching a wildcard pattern

You can filter the output using -f and -d. For example:

list_modifications.py -d obj* -f *.log -f *.bin -g 30 .git build\CMakeFiles

The above command lists the contents of the .git and build\CMakeFiles folders, excluding the objects subfolder and any files ending in .log or .bin. It also groups files modified within 30 seconds of each other, instead of the default 10.

A Quick Look at the Code

This script is a pretty good example of the kind of problem Python can solve quickly using very little code. Here’s a quick run-through.

parser = optparse.OptionParser(usage='Usage: %prog [options] path [path2 ...]')
parser.add_option('-g', action='store', type='long', dest='secs', default=10,
                  help='set threshold for grouping files')
parser.add_option('-f', action='append', type='string', dest='exc_files', default=[],
                  help='exclude files matching a wildcard pattern')
parser.add_option('-d', action='append', type='string', dest='exc_dirs', default=[],
                  help='exclude directories matching a wildcard pattern')
options, roots = parser.parse_args()

This block of code takes care of all command-line option parsing using the built-in optparse module. optparse is deprecated as of Python 2.7, but it’s handy and available since Python 2.5. The --help option is handled automatically.

The -f option uses the 'append' action with a default of [], which means the user can specify -f multiple times, creating a list. In the previous example, we end up with options.exc_files set to ['*.log', '*.bin']. Any leftover positional arguments are assigned to roots as another list; in the previous example, roots becomes ['.git', 'build\\CMakeFiles'].

def iterFiles(options, roots):
    """" A generator to enumerate the contents of directories recursively. """
    for root in roots:
        for dirpath, dirnames, filenames in os.walk(root):
            name = os.path.split(dirpath)[1]
            if any(fnmatch.fnmatch(name, w) for w in options.exc_dirs):
                del dirnames[:]  # Don't recurse here
                continue
            for fn in filenames:
                if any(fnmatch.fnmatch(fn, w) for w in options.exc_files):
                    continue
                path = os.path.join(dirpath, fn)
                mtime = os.path.getmtime(path)
                size = os.path.getsize(path)
                yield mtime, size, path

iterFiles looks like a function definition, but the presence of the yield statement in the body means it actually defines a generator. As such, calling iterFiles() does not actually execute the function. It returns an iterator, which you can then use in a for loop, as we’ll see later.

iterFiles uses the os.walk generator, which lets us modify the contents of dirnames in-place during iteration. In particular, we clear the contents of the list using del dirnames[:] to avoid descending into certain subdirectories.

In the above code, the expression any(fnmatch.fnmatch(name, w) for w in options.exc_dirs) is known as a generator expression. It’s a lot like a list comprehension, but we’re allowed to omit the square brackets since the list is fed to a single function. In this case, the any function will return True if fnmatch.fnmatch(name, w) returns True for any item in the list.

ptime = 0
for mtime, size, path in sorted(iterFiles(options, roots), reverse=True):
    if ptime - mtime >= options.secs:
        print('-' * 30)
    timeStr = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(mtime))
    print('%s %10d %s' % (timeStr, size, path))
    ptime = mtime

Here, we feed the iterFiles generator to sorted, resulting in a sorted list of 3-tuples. The list is sorted by the first item in the tuple — the modification time — which is exactly what we want. We loop through, writing one line of formatted output for each tuple. Since Python lets us multiply a string by an integer, '-' * 30 is used as a shortcut for drawing horizontal lines.

That’s all there is to it! Hopefully, some readers have managed pick up a few nuggets of Pythonic goodness along the way.

A lot of them fit the following usage pattern:

• The keys and values have simple integer or pointer types.
• The keys are effectively random.
• The basic operation is lookup-or-insert: Look for an existing item first, and add one if not found. Usually, the item is found.
• Deletes happen infrequently, and when they do, they tend to happen in bulk.

Occasionally, one of these maps is identified as a performance bottleneck. It raises the question: What’s the best data structure to use in this scenario?

I wrote a small C++ application to compare two top candidates, a Judy array and a finely-tuned hash table, using various experiments. All the code is available on GitHub.

A Judy array — specifically, the JudyL variant — is an efficient mapping of integer keys to integer values. It is optimized to avoid CPU cache misses as often as possible. Its memory consumption scales smoothly with number of entries, even when the keys are sparsely distributed. It’s a real feat of engineering — hats off to Doug Baskins.

A hash table is a relatively well-known data structure. For this post, I’ve written a custom hash table based on open addressing with linear probing. This particular hash table will dynamically resize itself as the population grows.

Now, there are already several existing shootouts between Judy arrays and hash tables on the web. Some of them are good, but in this post, I wanted to target a very specific usage pattern. Also, I wrote the single most efficient integer hash table I could, with the explicit goal of taking Judy down — no smorgasbord of lame hash tables here. Finally, I took considerable pains to ensure that the benchmarking method produced fair, precise and detailed results, as outlined in the README. Feedback is welcome.

Adding Items to the Map

In this experiment, more than 10000000 unique 32-bit random keys are inserted into each map, and the insertion times are measured. The timings were taken on my aging Core 2 Duo E6300.

Each point on the graph represents the time to insert the Nth element into a map containing N – 1 elements, averaged around small neighborhoods of N.

First, you’ll notice all the spikes in the hash table insertion times. These spikes are due to dynamic resizing of the hash table, which comes at a cost. Once the table becomes 75% full — for example, when the 12280th item is inserted into a hash table of size 16384 — we dynamically allocate a new hash table with double the size, copy all existing items to new hash slots, and delete the old table. As you’d expect, this operation is slow. For example, where most items take between 30 and 40 nanoseconds to insert, the 12280th item takes 0.4 milliseconds.

That’s expensive, but remember, resizing happens infrequently. In practice, these one-time costs are amortized against thousands of other insert operations which run very quickly. Another way to look at it is that the area under the red curve is much smaller than the area under the blue curve, meaning that the total CPU time spent inserting into the hash table is much less, even with the presence of spikes. And in cases where it really matters, you could always pre-allocate the hash table to a large enough size, eliminating those spikes completely.

Judy makes heavy use of the memory allocator, so to show it in its best light, I hardwired the Judy array to DLMalloc. DLMalloc is a great allocator, and helps avoid the possibility of hidden performance pitfalls on Windows, such as DLL runtime overhead and (God forbid) the debug system heap.

It should be acknowledged that Judy insertion times are faster in cases when the keys are not random; however, that’s not what we’re testing in this experiment.

Finding Existing Items in the Map

Usually, during the lookup-or-insert operation, an existing item is already found in the map. That makes lookup times even more significant than insert times.

In this next experiment, each map is filled to various populations. At each population, thousands of random lookups are performed, and the average time per lookup is plotted. Here, the lookup function is actually the same as the insert function used in the previous experiment — the only difference is that this time, we call it using keys which are known to already exist in the map.

At all populations, hash table lookups are twice as fast as Judy array lookups, give or take. I suspect part of the reason is because hash tables cause even fewer cache misses than Judy. With a good hash function, most lookups should be found in the first memory address checked; if not, linear probing ensures they are likely to be found in the same cache line. Notice that when the population reaches 98304 items, the hash table stops fitting entirely within the CPU’s 2 MB L2 cache, which is when the lookup times begin to climb.

Because the keys are random, the choice of hash function doesn’t matter too much for this experiment. Nonetheless, I borrowed MurmurHash3′s integer finalizer as the hash function. It consists of a few XORs, multiplies and shifts, and seems to distribute the hashes just as effectively when the keys are not random.

h ^= h >> 16;
h *= 0x85ebca6b;
h ^= h >> 13;
h *= 0xc2b2ae35;
h ^= h >> 16;

I chose not to benchmark the delete operation, because under the given usage pattern, deletes tend to be one-time operations which happen in bulk, such as when unloading a section of a game world. Still, one cool thing about hash tables with linear probing is that the Delete method needs only to shuffle existing entries around. This hash table also offers a Compact method, to optionally reclaim memory. Finally, you could write an even faster delete function if you require that Compact is always called at the end of each bulk delete.

Polluting the CPU Cache Between Operations

The above experiments are purely micro-benchmarks, which makes them somewhat unrealistic. A real application would run plenty of other code in between operations on the map. This would continuously evict parts of the map from the CPU cache, which would have a direct impact on the map’s performance.

With that in mind, I added some artificial cache pollution to both experiments, and ran them again. This time, random chunks of memory were manipulated between each operation. Using this strategy, I collected two extra datasets for each data structure, and plotted them below. The thin lines have an average of 1000 bytes of cache pollution between operations, while the thick lines have an average of 10000 bytes.

   

As you can see, cache pollution makes all of the timings slower, especially at larger populations. Regardless, the hash table still runs roughly twice as fast as the Judy array at nearly all populations and pollution levels.

Obviously, the most useful performance measurements would be taken in a real application using a consistently reproducible test case. For such measurements, an APIProfiler-like profiling technique comes in handy.

Memory Consumption

Judy is known for its memory-efficiency. So, how does its memory consumption compare to that of the hash table? To find the answer, I called dlmalloc_stats, a function which includes the overhead of the memory allocations themselves, an intrinsic cost which is all-too-easily overlooked. This graph plots the total bytes consumed by each map, divided by the total number of items in the map.

You can see that the hash table’s memory consumption follows a sawtooth pattern, doubling in size each time the 75% threshold is reached. Judy’s memory consumption, on the other hand, scales much more smoothly, and tends to hover around the lower end of the hash table’s amount. Still, there are some points where the two lines cross, which means that the hash table actually takes less memory at certain populations. Beyond 1000 items, though, those cases are generally the exception.

Judy also has the interesting ability to consume even less memory when the keys are not distributed randomly. In particular, when all the keys are clustered around a small range, its memory consumption tends towards that of a plain array. That’s impressive, but again, not the usage pattern we’re testing in this experiment.

Conclusion

For the given usage pattern, a Judy array is not the fastest associative map. But for general use, Judy arrays tend to be among the fastest associative maps while also being among the most memory-efficient, which is what made them a contender here in the first place. As a bonus, Judy arrays are always sorted. That means you can iterate sequentially through the map at any moment, something not possible using a hash table — though this property generally does not bring any benefit in the usage pattern studied here. Judy is also LGPL-licensed and patented, which may make it unsuitable for certain kinds of development.

The hash table tested here was specifically written for the job. In general, you can’t just grab any module with “hash table” in the name, slap it in place and expect the same performance. The addressing strategy, choice of hash function, handling of free/deleted slots, and reliance on inlining, templates and indirection each has the potential to bog down the CPU in various ways.

Another fun, useful fact about this type of hash table is that it lends itself surprisingly well to lock-free programming. But that’s a subject for another post.

I faced this problem recently, and considered several options before finally implementing a custom, non-repeating pseudo-random number generator which runs in O(1) time, requires just 8 bytes of storage, and has pretty good distribution. I thought I’d share the details here.

Approaches Considered

There are already several well-known pseudo-random number generators (PRNGs) such as the Mersenne Twister, an excellent PRNG which distributes integers uniformly across the entire 32-bit range. Unfortunately, calling this PRNG 10000000 times does not tend to generate a sequence of 10000000 unique values. According to Hash Collision Probabilities, the probability of all 10000000 random numbers being unique is just:

That’s astronomically unlikely. In fact, the expected number of unique values in such sequences is only about 9988367. You can try it for yourself using Python:

>>> len(set([random.randint(0, 0xffffffff) for i in xrange(10000000)]))
9988432

One obvious refinement is to reject random numbers which are already in the sequence, and continue iterating until we’ve reached 10000000 elements. To check whether a specific value is already in the sequence, we could search linearly, or we could keep a sorted copy of the sequence and use a binary search. We could even track the presence of each value explicitly, using a giant 512 MB bitfield or a sparse bitfield such as a Judy1 array.

Another refinement: Instead of generating an arbitrary 32-bit integer for each element and hoping it’s unique, we could generate a random index in the range [0, N) where N is the number of remaining unused values. The index would tell us which free slot to take next. We could probably locate each free slot in logarithmic time by implementing a trie suited for this purpose.

Brainstorming some more, an approach based on the Fisher-Yates Shuffle is also quite tempting. Using this approach, we could begin with an array containing all possible 32-bit integers, and shuffle the first 10000000 values out of the array to obtain our sequence. That would require 16 GB of memory. The footprint could be reduced by representing the array as a sparse associative map, such a JudyL array, storing only those x where A[x] ≠ x. Or, instead of starting with an array of all possible 32-bit integers, we could start with an initial sequence of any 10000000 sorted integers. In an attempt to span the available range of 32-bit values, we could even model the initial sequence as a Poisson process.

All of the above approaches either run in non-linear time, or require large amounts of storage. Several of them would be workable for a sequence of just 10000000 integers, but it got me thinking whether a more efficient approach, which scales up to any sequence length, is possible.

A Non-Repeating Pseudo-Random Number Generator

The ideal PRNG for this problem is one which would generate a unique, random integer the first 2³² times we call it, then repeat the same sequence the next 2³² times it is called, ad infinitum. In other words, a repeating cycle of 2³² values. That way, we could begin the PRNG at any point in the cycle, always having the guarantee that the next 2³² values are repeat-free.

One way to implement such a PRNG is to define a one-to-one function on the integers — a function which maps each 32-bit integer to another, uniquely. Let’s call such a function a permutation. If we come up with a good permutation, all we need is to call it with increasing inputs { 0, 1, 2, 3, … }. We could even begin the input sequence at any value.

For some reason, I remembered from first-year Finite Mathematics that when p is a prime number, has some interesting properties. Numbers produced this way are called quadratic resides, and we can compute them in C using the expression x * x % p. In particular, the quadratic reside of x is unique as long as . For example, when p = 11, the quadratic residues of 0, 1, 2, 3, 4, 5 are all unique:

As luck would have it, it also happens that for the remaining integers, the expression p - x * x % p fits perfectly into the remaining slots. This only works for primes p which satisfy .

This gives us a one-to-one permutation on the integers less than p, where p can be any prime satisying . Seems like a nice tool for building our custom PRNG.

In the case of our custom PRNG, we want a permutation which works on the entire range of 32-bit integers. However, 2³² is not a prime number. The closest prime number less than 2³² is 4294967291, which happens to satisfy . As a compromise, we can write a C++ function which permutes all integers below this prime, and simply maps the 5 remaining integers to themselves.

unsigned int permuteQPR(unsigned int x)
{
    static const unsigned int prime = 4294967291;
    if (x >= prime)
        return x;  // The 5 integers out of range are mapped to themselves.
    unsigned int residue = ((unsigned long long) x * x) % prime;
    return (x <= prime / 2) ? residue : prime - residue;
}

This function, on its own, is not the world's best permutation -- it tends to cluster output values for certain ranges of input -- but it is one-to-one. As such, we can combine it with other one-to-one functions, such as addition and XOR, to achieve a much better permutation. I found the following expression works reasonably well. The intermediateOffset variable acts as a seed, putting a variety of different sequences at our disposal.

permuteQPR((permuteQPR(x) + intermediateOffset) ^ 0x5bf03635);

On GitHub, I've posted a C++ class which implements a pseudo-random number generator based on this expression.

0xc2ab6929
0xa0e2502c
0x95b7c9eb
0x2fb14c01
0x5d983e09
0xba14e8c1
0x90994968
0x07058db1
0x061acc1f
0x969860fe
0x92f5a666
0xfa1b08af
0x18a3354f
...

I've also posted a working project to verify that this PRNG really does output a cycle of 2³² unique integers.

So, how does the randomness of this generator stack up? I'm not a PRNG expert, so I put my trust in TestU01, a library for testing the quality of PRNGs, published by the University of Montreal. Here's some test code to put our newly conceived PRNG through its paces. It passes all 15 tests in TestU01's SmallCrush test suite, which I guess is pretty decent. It also passes 140/144 tests in the more stringent Crush suite.

========= Summary results of SmallCrush =========

 Version:          TestU01 1.2.3
 Generator:        ursu_CreateRSU
 Number of statistics:  15
 Total CPU time:   00:00:49.95

 All tests were passed

Perhaps this approach for generating a sequence of unique random numbers is already known, or perhaps it shares attributes with existing PRNGs. If so, I'd be interested to find out. If you wanted, you could probably adapt it to work on ranges of integers other than 2³² as well. Surfing around, I noticed that OpenBSD implements another non-repeating PRNG, though I'm not sure their implementation is cyclical or covers the entire number space.

Incidentally, this PRNG is used in my next post, This Hash Table Is Faster Than a Judy Array.

Do you know any other way to solve this problem?

The answer is that we don’t have to. Let’s work with 32-bit fixed-point math instead, and see how far that gets us.

A 32-bit fixed-point number can represent fractions in the interval [0, 1) using 32 bits of precision. In other words, it can encode the first 32 bits of any binary fraction. To define such a fixed-point number, we simply take a regular 32-bit unsigned integer, and imagine it as the top of a fraction over 2³².

As you can see, 0x0288df0d represents a fixed-point number approximately equal to D, which in the previous post, we estimated as the probability of encountering a delta value of 0 in one of our sorted sequences. It's actually not such a bad approximation, either: The error is within 0.00000023%.

If you recall the way we partitioned the real number line in the previous post, 0x0288df0d would represent the boundary between the first and second partitions. In a similar way, we can compute the boundaries around all of the first 63 partitions

and compile them into a lookup table:

static const u32 LUTsize = 64;
const u32 LUT[LUTsize] = {
    0×00000000, 0x0288df0d, 0x050b5170, 0×07876772,
    0x09fd3131, 0x0c6cbea6, 0x0ed61f9d, 0x113963bd,
    0x13969a84, 0x15edd348, 0x183f1d3b, 0x1a8a8766,
    0x1cd020ad, 0x1f0ff7cc, 0x214a1b5e, 0x237e99d4,
    0x25ad817f, 0x27d6e088, 0x29fac4f7, 0x2c193cad,
    0x2e32556d, 0x30461cd1, 0x3254a056, 0x345ded53,
    0x366210ff, 0x3861186f, 0x3a5b1096, 0x3c500649,
    0x3e40063a, 0x402b1cfa, 0x421156fd, 0x43f2c095,
    0x45cf65f7, 0x47a75337, 0x497a944b, 0x4b49350b,
    0x4d134131, 0x4ed8c45a, 0x5099ca03, 0x52565d8e,
    0x540e8a41, 0x55c25b43, 0x5771dba1, 0x591d1649,
    0x5ac41611, 0x5c66e5b1, 0x5e058fc6, 0x5fa01ed4,
    0x61369d41, 0x62c9155d, 0x6457915a, 0x65e21b51,
    0x6768bd44, 0x68eb8119, 0x6a6a709d, 0x6be59584,
    0x6d5cf96c, 0x6ed0a5d9, 0x7040a435, 0x71acfdd4,
    0x7315bbf3, 0x747ae7b7, 0x75dc8a2c, 0x773aac4a
};

You'll recognize this lookup table as the same one found in the source code listing.

The Encoder

We begin the encoding process by setting up a writeInterval structure having two fixed-point members: lo and range. Initially, lo = 0 and range = 0xffffffff.

This represents an interval along the real number line. Obviously, range is slightly less than 1.0, so we aren't quite taking advantage of the full number line, but it's very close, and it fits inside 32 bits.

Next, we're going to subdivide this interval. Subdividing is a straightforward matter of linear interpolation. Given a fixed-point value x, we locate the corresponding point within our interval by performing a lerp:

To implement this formula using 32-bit fixed-point math, we must cast to 64 bits before multiplying, then right-shift the result by 32.

u32 lerp(u64 x)        { return lo + (u32) ((range * x) >> 32); }

Now, suppose we wish to encode an initial delta value of 29. First, we need to find the boundaries of the partition which corresponds to 29. To find them, we consult LUT[29] and LUT[30] in our lookup table, lerp those values within the writeInterval, and assign the results to A and B. In this case, we end up with A = 0x402b1cf9 and B = 0x421156fc. Since this is the very first partition along the number line, we immediately know that our final encoded binary fraction will lie somewhere within the interval [A, B).

You'll notice that A and B share the same first 6 bits. We can go ahead and write those 6 bits to the output stream, because they'll never change, no matter what the final encoded binary fraction ends up being. So let's write them, then shift both A and B to the left by 6 bits each.

We can now think of A and B as trailing digits on the encoded output. At this point, we've written the binary digits we're sure about, and A and B represent the digits we're not sure about yet. We need to continue subdividing to drill deeper and gain more certainty about subsequent digits.

Before handling the next delta value in the input sequence, we set up a new writeInterval, setting lo to A, and range to B - A. We then process the next delta value in the same way we did the first, interpolating new A and B values within the new writeInterval. Once again, we output all of the most significant bits which A and B agree on, shifting those values to compensate. We repeat these steps until the entire input sequence is exhausted, at which point we encode a dummy value somewhere within in the final interval (I chose 32), then flush the contents of A to the bit stream.

It turns out that shifting the values of A and B to the left is critical: By shifting them, we maximize the resulting value of range, so there's always enough numerical precision to subdivide further. This trick lets us get away with using 32-bit fixed point math at every step during the encoding process.

Breaking Up Large Deltas

The lookup table we defined, LUT, contains only 64 entries. Obviously, we can't use it to look up partition boundaries for delta values greater than 62. We could, perhaps, extend it to hold all 100000001 possible boundary values, but if we did, our fixed-point representation would quickly run out of numerical precision. Check the partitioned number line again: the partition for delta value 10000 is already very close to 1.0. In fact, it would require more than 143 fractional binary digits to represent the boundaries of this partition; way more than 32. And the partitions to the right of that would require even more.

We need a better way. If you do the math, the partition boundaries are actually described by the following function:

And lucky for us, this function satisfies a useful relation:

This relation is basically just another lerp like the one we saw earlier. It tells us that if we want to encode any delta value that is greater than or equal to 63, we can simply reduce the delta value by 63 and encode it within the subinterval [LUT(63), 1) instead.

If the delta value is very large, we simply repeat the reduction step until we have a delta value less than 63. You'll notice this technique bears some similarity to the Golomb encoding strategy I described earlier, except that it doesn't encode bits explicitly; only implicitly, as the result of continuous subdivision of the writeInterval.

    void pushDelta(u32 delta)
    {
        while (delta >= LUTsize - 1)
        {
            encode(LUTsize - 1); // Use the [LUT(63), 1) subinterval
            delta -= LUTsize - 1;
        }
        encode(delta);
    }

The Decoder

As you'd expect, the decoder reverses the arithmetic encoding process, also using 32-bit fixed-point math. It begins by setting up a readInterval structure with lo = 0 and range = 0xffffffff, then reads the first 32 binary digits of the encoded bit stream into a fixed-point value readSeq.

That's all the information the decoder needs to correctly determine the first delta value. It finds this delta value by performing a binary search through the lookup table, lerping every LUT entry through the readInterval until it identifies the partition containing readSeq. If readSeq is greater than every lerped entry, it means the delta value was greater than or equal to 63, in which case the actual delta value is reconstructed by iterating the decoding step.

Either way, once the decoder identifies the correct partition, it knows exactly how many bits the encoder pushed to the bitstream, and it sets up a new readInterval structure which exactly matches the one used by the encoder for the next delta value.

Carrying the One

After implementing all of the above, one problematic case remains. Once in a while, when several nested partitions align perfectly, we may end up with a partition like this:

The encoder has not yet shifted the upper bits of A and B to the output, since they don't match. Unfortunately, if we subtract A from B to determine the range of the next writeInterval, it will be too small to divide uniquely into further partitions. We've run out of numerical precision. The whole algorithm breaks.

To solve this problem, we must allow some bit shifting even in cases where the leading binary digits don't match, such as in the above case. It ends up boiling down to a simple rule: As long as the second bit of A is 1 and the second bit of B is 0, keep shifting A to the output.

In doing so, we end up having to account for some funky scenarios: A becomes greater than B; we have to perform some lerps which wrap around the range of an integer; and there are cases where we need to "carry a one" back into previous bits we've already written to the bit stream. I won't drill into all the details here, because it would be too boring, and I assume the handful of readers who are interested enough will study the source code anyway. In short, once everything is taken care of, range never dips below 0x40000000, providing plenty of precision.

Evidence of Memory Efficiency

I modified the source code so that it outputs some statistics to stderr, and implemented a Python function to validate the program using whatever input sequence you provide. Here's the output from a few trial runs:

$ python -i validate.py
>>> from random import *
>>> validate([randint(0, 99999999) for i in xrange(1000000)])
Result OK after 34.537 secs
minRange=0x40000012 maxCircularBytes=1011728
>>> validate([99999999] * 1000000)
Result OK after 25.832 secs
minRange=0x40000231 maxCircularBytes=1011732
>>> validate([62 * i for i in xrange(999999)] + [99999999])
Result OK after 15.247 secs
minRange=0x40000631 maxCircularBytes=1011728
>>> validate([int(gauss(99999999, 1000000)) % 100000000 for i in xrange(1000000)])
Result OK after 29.216 secs
minRange=0x40000006 maxCircularBytes=1011728

The highlighted number tells us the maximum number of bytes in use by the circular buffer during each trial run. No matter what I tried, I couldn't make it consume more than 1011732 bytes. That's well within the limit of 1013000 bytes reserved for the buffer.

It's also remarkably close to the fundamental limit of 1011717 bytes which we derived in the last post, especially considering a few bytes are wasted at the end of the sequence when flushing the BitWriter.

Pseudo-Mathematical Proof of Efficiency

It should be possible to construct a mathematical proof giving an upper bound on the amount of memory needed to encode a sorted sequence using this method. I'm totally going to handwave through this part, but if you accept that the encoding process works the same even if you "break up large deltas" using a lookup table of just 2 elements, 0 and D, it follows that it takes

to output a single value in the sorted sequence, which happens 1000000 times, and

to increment the value by 1, which happens at most 99999999 times. Therefore, in pure theory, it shouldn't take more than 6.65821147 × 1000000 + .01435529 × 99999999 ≈ 8093740 bits to encode any sequence, which is pretty close to the fundamental limit we derived earlier. The proof would also have to add some small epsilon value to account for the loss of precision that happens at each step using fixed-point math, but I haven't figured that out, and I'm not really interested in going that far.

In wrapping up this subject, I learned something interesting while reading up on arithmetic coding. Dozens of US patents related to the technique have been granted in recent years. One side effect from these patents is that, apparently, most JPEG images in use today take up to 25% more space than they would without the existence of such patents. It's rather funny (and a bit sad) when you consider that the intended goal of the patent system is to advance science and technology.

In that post, I gave a high-level outline of the approach, and showed an encoding scheme — which I’ve since learned is Golomb coding — which comes close to meeting the memory requirements, but doesn’t quite fit. Arithmetic coding, on the other hand, does fit. It’s interesting, because this problem, as it was phrased, almost seems designed to force you into arithmetic coding (though Nick Cleaton’s solution manages to avoid it).

I had read about arithmetic coding before, but I never had any reason to sit down to implement it. It always struck me as kind of mystical: How the heck you encode information in a fraction of a bit, anyway? The 1MB Sorting Problem turned out to be a great excuse to learn how arithmetic coding works.

How Many Sorted Sequences Even Exist?

It’s important to note that the whole reason why we are able to represent a sorted sequence of one million 8-digit numbers in less than 1 MB of memory is because mathematically, there simply aren’t that many different sorted sequences which exist.

To see this, consider the following method of encoding a sorted sequence as a row of boxes, read from left to right.

An empty box tells us to increment the current value (initially 0), and a box with a dot inside it tells us to return the current value as part of the sorted sequence. For example, the above row of boxes corresponds to the sorted sequence { 3, 7, 7, 10, 15, 16, … }.

Now, to encode one million 8-digit numbers, we’d need exactly 1000000 boxes containing dots, plus a maximum of 99999999 empty boxes. In fact, when there are exactly 99999999 + 1000000 boxes in total, we can encode every possible sorted sequence using a unique distribution of dots. The question then becomes: How many different ways are there to distribute those dots? This is a straightforward number of combinations problem:

That’s a lot of combinations. Now, think of the contents of memory as one giant label which represents exactly one of those combinations. The exponent on the 2, above, gives us a lower limit for how many bits of memory would be required to come up with a unique label for every possible combination. In this case, it can’t be done using fewer than 8093730 bits, or 1011717 bytes.

That’s the fundamental limit. No encoding scheme can ever do better than that; it would be like trying to uniquely label every state in the USA using fewer than 6 bits. On the bright side, 1011717 bytes is comfortably less than our 1048576 byte limit, which is encouraging.

The Probability of Encountering Each Delta Value

In the last post, we saw the potential of encoding delta values — the differences between numbers in a sorted sequence. Thinking in terms of the above rows of boxes, let’s take a look at the probability of encountering each delta value.

Since we know that there are 99999999 empty boxes, and 1000000 boxes containing dots, the probability of any particular box containing a dot is just:

For simplicity, let’s now imagine an infinite row of boxes, with dots occurring at the same frequency as this. The probability of encountering a delta value of 0 is, then, the same as the probability of a box containing a dot, which is just D.

How about a delta value of 1? Well, the probability of the first box being empty is (1 – D), while the probability of the second box containing a dot is still just D. Since each outcome is an independent event, we can multiply those probabilities together. The probability of encountering a delta value of 1, then, is

And in general, the probability encountering a delta value of N is

Now, let’s draw the real number line in the interval [0, 1), and let’s subdivide it into partitions according to the probabilities of each delta value. They begin quite small — you can see the first three partitions for delta values 0, 1 and 2 squished all the way to the left — and they get infintessimally smaller as we proceed to the right, as larger delta values are exponentially less likely to occur.

If you were to throw a dart at this number line, the likelihood of hitting each partition is about the same as the likelihood of encountering each delta value in one of our sorted sequences.

That’s exactly the kind of information that’s useful for arithmetic coding.

The Idea Behind Arithmetic Encoding

Arithmetic encoding is able to encode a sequence of elements — in this case, delta values — by progressively subdividing the real number line into finer and finer partitions. At each step, the relative width of each partition is determined by the probability of encountering each element.

Suppose the first delta value in the sequence is 27. We begin by locating the corresponding partition in the original number line, and zooming into it. This gives us a new interval of the real number line to work with — in this case, roughly from .236 to .243 — which we can then subdivide further. Let’s use the same proportions we used for the first element.

Suppose the next element in the sequence is 39. Again, we locate the corresponding partition and zoom in, subdividing the interval into even finer partitions.

In this way, the interval gets progressively smaller and smaller. We repeat these steps one million times: once for each element in the sequence. After that, all we need to store is a single real value which lies somewhere within the final partition. This value will unambiguously identify the entire one-million-element sequence.

As you can imagine, to represent this value, it’s going to take a lot of precision. Hundreds of thousands times more precision than you’ll find in any single- or even double-precision floating-point value. What we need is a way to represent a fractional value having millions of significant digits. And in arithmetic coding, that’s exactly what the final encoded bit stream is. It’s one giant binary fraction having millions of binary digits, pinpointing a specific value somewhere within the interval [0, 1) with laser precision.

That’s great, you might be thinking, but how the heck do we even work with numbers that precise?

This post has already become quite long, so I’ll answer that question in a separate post. You don’t even have to wait, because it’s already published: See Arithmetic Encoding Using Fixed-Point Math for the thrilling conclusion!

I had promised to explain it in a followup post, and in the meantime, there’s been a flurry of discussion in the comments and on Reddit. In particular, commenter Ben Wilhelm (aka ZorbaTHut) already managed to explain most of it (Nice work!), and by now, I think quite a few people already get it. Nonetheless, I’ll write up another explanation as promised.

Data Structures

In this implementation, there is a staging area and a circular buffer. The staging area is just big enough to hold 8000 plain 32-bit integers. The circular buffer always holds a sorted sequence of numbers, using a compact representation which we’ll get to shortly.

We read numbers into the staging area until it’s full. Once it’s full, we sort it in-place, then merge it with the sorted contents of the circular buffer. The merge is basically the same merge operation performed in Mergesort, but we conserve memory using a trick: The new sequence is written to the buffer immediately following the previous one, and the previous sequence is erased while it’s read. This being a circular buffer, we wrap around to the beginning of the buffer once we’ve reached the end.

These steps are repeated until all of the input is read and the final staging area is merged. At that point, we have the entire sorted sequence stored in our circular buffer.

The staging area is really just an optimization. Theoretically, we don’t need it — we could merge each input number directly into the circular buffer as it comes in. The only problem is that as the sorted sequence grows, the merge operation becomes very slow, mainly due to the decoding and re-encoding process that’s involved. In fact, if you increase the staging area to hold 20000 integers instead of just 8000, the program runs twice as fast. That’s why I made the staging area as large as possible in the available memory.

A Sequence of Deltas

Next, we need a way to fit the sorted number sequence into a circular buffer significantly less than 1MB (1048576 bytes) in size. Many people on Stack Overflow and Reddit came up with this idea, which is indeed where I got it: Instead of storing the actual numbers, we store a delta sequence — the differences from one number to the next. Because the sequence is already sorted, these delta values tend to be quite small. In fact, given a sequence of N 8-digit numbers, the average delta value is just

By the time we’ve reached a million numbers, the average delta shrinks to just 100. Deltas around this size can be represented using 7 bits, since 2⁷ = 128, so intuitively, you can begin to see the potential to save memory. It’s also OK to use a lot more bits to represent larger delta values, since those appear far less frequently. The bigger the delta, the sooner we start to reach the end of the list!

Encoding Each Delta

All that’s left to explain is how we actually encode each delta value. Since each one will use a variable number of bits, it helps to look at the circular buffer as a bit stream, which is what BitReader and BitWriter are for.

There are many ways to encode them, and it’s tough to find a strategy which respects our extremely limited memory budget in every case. For example, this comment describes an encoding strategy which takes 1000000 bytes in the case where every delta is less than 128, but up to 1564452 bytes when other delta values are mixed in.

It turns out that the optimal encoding strategy is arithmetic coding. But before jumping into that, let’s first take a look at an alternate encoding strategy which requires a bit more memory than we’re allowed, but is much easier to understand. This alternate coding also has a few similarities to the arithmetic coding implementation I wrote, so it’ll serve as a pretty good warm-up to that one.

In this alternate strategy, we decode the next delta value as follows. Define an integer variable accumulator, initialize it to 0, then look at the incoming bit stream:

• If the next bit is 1, add 64 to accumulator and repeat.
• If the next bit is 0, read a 6-bit integer from the bitstream, add it to accumulator and return accumulator as the delta.

How much memory does this strategy need to represent an entire number sequence? Well, each delta value will require at least one 0 followed by 6 bits, and we’re going to have a million of those. On top of that, each leading 1 causes our sequence value to increase by 64, which obviously can’t happen more than times. Using this strategy, every possible sequence should fit within

Indeed, a sequence where all deltas are zero except for one 99999999 requires exactly that many bits. That’s more than 21KB over budget, so we have to do better.

Effectively, the above strategy follows a path determined by the bit stream to arrive at the next delta value. Here’s a graph to illustrate several possible bit paths:

For those who don’t already recognize it, this turns out to be a Huffman encoding tree. Huffman coding is really just a special case of arithmetic coding. (And as commenter Stergios Stergiou points out, this particular style of Huffman coding is known as Golomb coding.)

When arithmetic coding is used, the entire sequence fits safely within < 1013000 bytes, all the time. This strategy proved kind of challenging to implement, so it took the most time to get right. In the next two posts, I’ll describe how it was implemented using all that fancy bit magic you see in Decoder::decode and Encoder::encode.

You have a computer with 1M of RAM and no other local storage. You must use it to accept 1 million 8-digit decimal numbers over a TCP connection, sort them, and then send the sorted list out over another TCP connection. The list of numbers may contain duplicates, which you may not discard. Your code will be placed in ROM, so you need not subtract the size of your code from the 1M. You have been given code to drive the Ethernet port and handle TCP/IP connections, and it requires 2k for its state data, including a 1k buffer via which your code will read and write data.

The challenge here, of course, is that you can’t fit all the data in memory as raw integers. There are 100 million possible 8-digit decimal numbers, so even if you pack each number into 27 bits (2²⁷ = ~134 million), that would take 3375000 bytes of storage. The hypothetical machine available to you has only 1048576 bytes of storage. At first glance, it seems to defy the laws of mathematics.

The question received enough attention that a lot of people proposed solutions on Stack Overflow and in the Reddit discussion, but so far, nothing that runs. There’s some code for a solution which doesn’t work, many proposals without code, and some hilarious out-of-the-box answers. A lot of answers come close, but won’t work on every possible input.

In thinking about this problem, it occurred to me that I’ve already written a blog post containing a clue which leads to the ideal solution. I won’t give away which post, but after that, I couldn’t resist cobbling together a working implementation in C++.

The source code listing is just 339 lines with very few comments. I thought I’d post it here to see if anybody can figure out which algorithm was used:

Proof that the memory constraints are satisified:

typedef unsigned int u32;

namespace WorkArea
{
    static const u32 circularSize = 253250;
    u32 circular[circularSize] = { 0 };         // consumes 1013000 bytes

    static const u32 stageSize = 8000;
    u32 stage[stageSize];                       // consumes 32000 bytes

    ...

Together, these two arrays take 1045000 bytes of storage. Tack on the hypothetical 2KB overhead for the TCP connection, and you’re left with a healthy margin of 1048576 – 1045000 – 2×1024 = 1528 bytes for remaining variables and stack space.

It runs in about 23 seconds on my Xeon W3520. You can verify that the program works using the following Python script, assuming a program name of sort1mb.exe. See if you can find any inputs which break it!

from subprocess import *
import random

sequence = [random.randint(0, 99999999) for i in xrange(1000000)]

sorter = Popen('sort1mb.exe', stdin=PIPE, stdout=PIPE)
for value in sequence:
    sorter.stdin.write('%08d\n' % value)
sorter.stdin.close()

result = [int(line) for line in sorter.stdout]
print('OK!' if result == sorted(sequence) else 'Error!')

I submitted this answer to the Stack Overflow question, but it’s pretty late to the party. Update: You’ll find a detailed explanation in my next post, 1MB Sorting Explained.

Nick Cleaton originally proposed the currently accepted solution. His approach is different, and not in any runnable form, but the idea also looks valid.

If there’s one thing that characterizes a weakly-ordered CPU, it’s that one CPU core can see values change in shared memory in a different order than another core wrote them. That’s what I’d like to demonstrate in this post using pure C++11.

For normal applications, the x86/64 processor families from Intel and AMD do not have this characteristic. So we can forget about demonstrating this phenomenon on pretty much every modern desktop or notebook computer in the world. What we really need is a weakly-ordered multicore device. Fortunately, I happen to have one right here in my pocket:

The iPhone 4S fits the bill. It runs on a dual-core ARM-based processor, and the ARM architecture is, in fact, weakly-ordered.

The Experiment

Our experiment will consist of an single integer, sharedValue, protected by a mutex. We’ll spawn two threads, and each thread will run until it has incremented sharedValue 10000000 times.

We won’t let our threads block waiting on the mutex. Instead, each thread will loop repeatedly doing busy work (ie. just wasting CPU time) and attempting to lock the mutex at random moments. If the lock succeeds, the thread will increment sharedValue, then unlock. If the lock fails, it will just go back to doing busy work. Here’s some pseudocode:

count = 0
while count < 10000000:
    doRandomAmountOfBusyWork()
    if tryLockMutex():
        // The lock succeeded
        sharedValue++
        unlockMutex()
        count++
    endif
endwhile

With each thread running on a separate CPU core, the timeline should look something like this. Each red section represents a successful lock and increment, while the dark blue ticks represent lock attempts which failed because the other thread was already holding the mutex.

It bears repeating that a mutex is just a concept, and there are many ways to implement one. We could use the implementation provided by std::mutex, and of course, everything will function correctly. But then I'd have nothing to show you. Instead, let's implement a custom mutex — then let's break it to demonstrate the consequences of weak hardware ordering. Intuitively, the potential for memory reordering will be highest at those moments when there is a "close shave" between threads — for example, at the moment circled in the above diagram, when one thread acquires the lock just as the other thread releases it.

The latest version of Xcode has terrific support for C++11 threads and atomic types, so let's use those. All C++11 identifiers are defined in the std namespace, so let's assume using namespace std; was placed somewhere earlier in the code.

A Ridiculously Simple Mutex

Our mutex will consist of a single integer flag, where 1 indicates that the mutex is held, and 0 means it isn't. To ensure mutual exclusivity, a thread can only set flag to 1 if the previous value was 0, and it must do so atomically. To achieve this, we'll define flag as a C++11 atomic type, atomic<int>, and use a read-modify-write operation:

int expected = 0;
if (flag.compare_exchange_strong(expected, 1, memory_order_acquire))
{
    // The lock succeeded
}

The memory_order_acquire argument used above is considered an ordering constraint. We're placing acquire semantics on the operation, to help guarantee that we receive the latest shared values from the previous thread which held the lock.

To release the lock, we perform the following:

flag.store(0, memory_order_release);

This sets flag back to 0 using the memory_order_release ordering constraint, which applies release semantics. Acquire and release semantics must be used as a pair to ensure that shared values propagate completely from one thread to the next.

If We Don't Use Acquire and Release Semantics...

Now, let's write the experiment in C++11, but instead of specifying the correct ordering constraints, let's put memory_order_relaxed in both places. This means no particular memory ordering will be enforced by the C++11 compiler, and any kind of reordering is permitted.

void IncrementSharedValue10000000Times(RandomDelay& randomDelay)
{
    int count = 0;
    while (count < 10000000)
    {
        randomDelay.doBusyWork();
        int expected = 0;
        if (flag.compare_exchange_strong(expected, 1, memory_order_relaxed))
        {
            // Lock was successful
            sharedValue++;
            flag.store(0, memory_order_relaxed);
            count++;
        }
    }
}

At this point, it's informative to look at the resulting ARM assembly code generated by the compiler, in Release, using the Disassembly view in Xcode:

If you aren't very familiar with assembly language, don't worry. All we want to know is whether the compiler has reordered any operations on shared variables. This would include the two operations on flag, and the increment of sharedValue in between. Above, I've annotated the corresponding sections of assembly code. As you can see, we got lucky: The compiler chose not to reorder those operations, even though the memory_order_relaxed argument means that, in all fairness, it could have.

I've put together a sample application which repeats this experiment indefinitely, printing the final value of sharedValue at the end of each trial run. It's available on GitHub if you'd like to view the source code or run it yourself.

Here's the iPhone, hard at work, running the experiment:

And here's the output from the Output panel in Xcode:

Check it out! The final value of sharedValue is consistently less than 20000000, even though both threads perform exactly 10000000 increments, and the order of assembly instructions exactly matches the order of operations on shared variables as specified in C++.

As you might have guessed, these results are entirely due to memory reordering on the CPU. To point out just one possible reordering — and there are several — the memory interaction of str.w r0, [r11] (the store to sharedValue) could be reordered with that of str r5, [r6] (the store of 0 to flag). In other words, the mutex could be effectively unlocked before we're finished with it! As a result, the other thread would be free to wipe out the change made by this one, resulting in a mismatched sharedValue count at the end of the experiment, just as we're seeing here.

Using Acquire and Release Semantics Correctly

Fixing our sample application, of course, means putting the correct C++11 memory ordering constraints back in place:

void IncrementSharedValue10000000Times(RandomDelay& randomDelay)
{
    int count = 0;
    while (count < 10000000)
    {
        randomDelay.doBusyWork();
        int expected = 0;
        if (flag.compare_exchange_strong(expected, 1, memory_order_acquire))
        {
            // Lock was successful
            sharedValue++;
            flag.store(0, memory_order_release);
            count++;
        }
    }
}

As a result, the compiler now inserts a couple of dmb ish instructions, which act as memory barriers in the ARMv7 instruction set. I'm not an ARM expert — comments are welcome — but it's safe to assume this instruction, much like lwsync on PowerPC, provides all the memory barrier types needed for acquire semantics on compare_exchange_strong, and release semantics on store.

This time, our little home-grown mutex really does protect sharedValue, ensuring all modifications are passed safely from one thread to the next each time the mutex is locked.

If you still don't grasp intuitively what's going on in this experiment, I'd suggest a review of my source control analogy post. In terms of that analogy, you can imagine two workstations each having local copies of sharedValue and flag, with some effort required to keep them in sync. Personally, I find visualizing it this way very helpful.

I'd just like to reiterate that the memory reordering we saw here can only be observed on a multicore or multiprocessor device. If you take the same compiled application and run it on an iPhone 3GS or first-generation iPad, which use the same ARMv7 architecture but have only a single CPU core, you won't see any mismatch in the final count of sharedValue.

Interesting Notes

You can build and run this sample application on any Windows, MacOS or Linux machine with a x86/64 CPU, but unless your compiler performs reordering on specific instructions, you won't witness any memory reordering at runtime — even on a multicore system! Indeed, when I tested it using Visual Studio 2012, no memory reordering occurred. That's because x86/64 processors are what is usually considered strongly-ordered: When one CPU core performs a sequence of writes, every other CPU core sees those values change in the same order that they were written.

This goes to show how easy it is to use C++11 atomics incorrectly without knowing it, simply because it appears to work correctly on a specific processor and toolchain.

Incidentally, the release candidate of Visual Studio 2012 generates rather poor x86 machine code for this sample. It's nowhere near as efficient as the ARM code generated by Xcode. Meanwhile, performance is the main reason to use lock-free programming on multicore in the first place! It's enough to turn me off using C++11 atomics on Windows for the time being. [Update Feb. 2013: As mentioned in the comments, the latest version of VS2012 Professional now generates much better machine code.]

This post is a followup to an earlier post where I demonstrated StoreLoad reordering on x86/64. In my experience, however, the need for #StoreLoad does not come up quite as often in practice as the ordering constraints demonstrated here.

Finally, I'm not the first person to demonstrate weak hardware ordering in practice, though I might be the first to demonstrate it using C++11. There are earlier posts by Pierre Lebeaupin and ridiculousfish which use different experiments to demonstrate the same phenomenon.

A memory model tells you, for a given processor or toolchain, exactly what types of memory reordering to expect at runtime relative to a given source code listing. Keep in mind that the effects of memory reordering can only be observed when lock-free programming techniques are used.

After studying memory models for a while — mostly by reading various online sources and verifying through experimentation — I’ve gone ahead and organized them into the following four categories. Below, each memory model makes all the guarantees of the ones to the left, plus some additional ones. I’ve drawn a clear line between weak memory models and strong ones, to capture the way most people appear to use these terms. Read on for my justification for doing so.

Each physical device pictured above represents a hardware memory model. A hardware memory model tells you what kind of memory ordering to expect at runtime relative to an assembly (or machine) code listing.

Every processor family has different habits when it comes to memory reordering, and those habits can only be observed in multicore or multiprocessor configurations. Given that multicore is now mainstream, it’s worth having some familiarity with them.

There are software memory models as well. Technically, once you’ve written (and debugged) portable lock-free code in C11, C++11 or Java, only the software memory model is supposed to matter. Nonetheless, a general understanding of hardware memory models may come in handy. It can help you explain unexpected behavior while debugging, and — perhaps just as importantly — appreciate how incorrect code may function correctly on a specific processor and toolchain out of luck.

Weak Memory Models

In the weakest memory model, it’s possible to experience all four types of memory reordering I described using a source control analogy in a previous post. Any load or store operation can effectively be reordered with any other load or store operation, as long as it would never modify the behavior of a single, isolated thread. In reality, the reordering may be due to either compiler reordering of instructions, or memory reordering on the processor itself.

When a processor has a weak hardware memory model, we tend to say it’s weakly-ordered or that it has weak ordering. We may also say it has a relaxed memory model. The venerable DEC Alpha is everybody’s favorite example of a weakly-ordered processor. There’s really no mainstream processor with weaker ordering.

The C11 and C++11 programming languages expose a weak software memory model which was in many ways influenced by the Alpha. When using low-level atomic operations in these languages, it doesn’t matter if you’re actually targeting a strong processor family such as x86/64. As I demonstrated previously, you must still specify the correct memory ordering constraints, if only to prevent compiler reordering.

Weak With Data Dependency Ordering

Though the Alpha has become less relevant with time, we still have several modern CPU families which carry on in the same tradition of weak hardware ordering:

• ARM, which is currently found in hundreds of millions of smartphones and tablets, and is increasingly popular in multicore configurations.
• PowerPC, which the Xbox 360 in particular has already delivered to 70 million living rooms in a multicore configuration.
• Itanium, which Microsoft no longer supports in Windows, but which is still supported in Linux and found in HP servers.

These families have memory models which are, in various ways, almost as weak as the Alpha’s, except for one common detail of particular interest to programmers: they maintain data dependency ordering. What does that mean? It means that if you write A->B in C/C++, you are always guaranteed to load a value of B which is at least as new as the value of A. The Alpha doesn’t guarantee that. I won’t dwell on data dependency ordering too much here, except to mention that the Linux RCU mechanism relies on it heavily.

Strong Memory Models

Let’s look at hardware memory models first. What, exactly, is the difference between a strong one and a weak one? There is actually a little disagreement over this question, but my feeling is that in 80% of the cases, most people mean the same thing. Therefore, I’d like to propose the following definition:

A strong hardware memory model is one in which every machine instruction comes implicitly with acquire and release semantics. As a result, when one CPU core performs a sequence of writes, every other CPU core sees those values change in the same order that they were written.

It’s not too hard to visualize. Just imagine a refinement of the source control analogy where all modifications are committed to shared memory in-order (no StoreStore reordering), pulled from shared memory in-order (no LoadLoad reordering), and instructions are always executed in-order (no LoadStore reordering). StoreLoad reordering, however, still remains possible.

Under the above definition, the x86/64 family of processors is usually strongly-ordered. There are certain cases in which some of x86/64′s strong ordering guarantees are lost, but for the most part, as application programmers, we can ignore those cases. It’s true that a x86/64 processor can execute instructions out-of-order, but that’s a hardware implementation detail — what matters is that it still keeps its memory interactions in-order, so in a multicore environment, we can still consider it strongly-ordered. Historically, there has also been a little confusion due to evolving specs.

Apparently SPARC processors, when running in TSO mode, are another example of a strong hardware ordering. TSO stands for “total store order”, which in a subtle way, is different from the definition I gave above. It means that there is always a single, global order of writes to shared memory from all cores. The x86/64 has this property too: See Volume 3, sections 8.2.3.6-8 of Intel’s x86/64 Architecture Specification for some examples. From what I can tell, the TSO property isn’t usually of direct interest to low-level lock-free programmers, but it is a step towards sequential consistency.

Sequential Consistency

In a sequentially consistent memory model, there is no memory reordering. It’s as if the entire program execution is reduced to a sequential interleaving of instructions from each thread. In particular, the result r1 = r2 = 0 from Memory Reordering Caught in the Act becomes impossible.

These days, you won’t easily find a modern multicore device which guarantees sequential consistency at the hardware level. However, it seems at least one sequentially consistent, dual-processor machine existed back in 1989: The 386-based Compaq SystemPro. According to Intel’s docs, the 386 wasn’t advanced enough to perform any memory reordering at runtime.

In any case, sequential consistency only really becomes interesting as a software memory model, when working in higher-level programming languages. In Java 5 and higher, you can declare shared variables as volatile. In C++11, you can use the default ordering constraint, memory_order_seq_cst, when performing operations on atomic library types. If you do those things, the toolchain will restrict compiler reordering and emit CPU-specific instructions which act as the appropriate memory barrier types. In this way, a sequentially consistent memory model can be “emulated” even on weakly-ordered multicore devices. If you read Herlihy & Shavit’s The Art of Multiprocessor Programming, be aware that most of their examples assume a sequentially consistent software memory model.

Further Details

There are many other subtle details filling out the spectrum of memory models, but in my experience, they haven’t proved quite as interesting when writing lock-free code at the application level. There are things like control dependencies, causal consistency, and different memory types. Still, most discussions come back the four main categories I’ve outlined here.

If you really want to nitpick the fine details of processor memory models, and you enjoy eating formal logic for breakfast, you can check out the admirably detailed work done at the University of Cambridge. Paul McKinney has written an accessible overview of some of their work and its associated tools.