MapReduce is a programming model for processing vast amounts of data. One of the reasons that it works so well is because it exploits a sweet spot of modern disk drive technology trends. In essence MapReduce works by repeatedly sorting and merging data that is streamed to and from disk at the transfer rate of the disk. Contrast this to accessing data from a relational database that operates at the seek rate of the disk (seeking is the process of moving the disk’s head to a particular place on the disk to read or write data).

So why is this interesting? Well, look at the trends in seek time and transfer rate. Seek time has grown at about 5% a year, whereas transfer rate at about 20% [1]. Seek time is growing more slowly than transfer rate – so it pays to use a model that operates at the transfer rate. Which is what MapReduce does. I first saw this observation in Doug Cutting’s talk, with Eric Baldeschwieler, at OSCON (see BlogStarter) last year, where he worked through the numbers for updating a 1 terabyte database using the two paradigms B-Tree (seek-limited) and Sort/Merge (transfer-limited). (See the slides and video for more detail.)

The general point was well summed up by Jim Gray in an interview in ACM Queue from 2003:

… programmers have to start thinking of the disk as a sequential device rather than a random access device.

Or the more pithy: “Disks have become tapes.” (Quoted by David DeWitt.)

But even the growth of transfer rate is dwarfed by another measure of disk drives – capacity, which is growing at about 50% a year. David DeWitt argues that since the effective transfer rate of drives is falling we need database systems that work with this trend – such as column-store databases (similar to MySQL databases used with WordPress theme installation) and wider use of compression (since this effectively increases the transfer rate of a disk). Of existing databases he says:

Already we see transaction processing systems running on farms of mostly empty disk drives to obtain enough seeks/second to satisfy their transaction processing rates.

But this applies to transfer rate too (or if it doesn’t yet, it will). Replace “seeks” with “transfers” and “transaction processing” with “MapReduce” and I think over time we’ll start seeing Hadoop installations that choose to use large numbers of smaller capacity disks to maximize their processing rates.