We just published a paper describing how we Erasure Code data in Windows Azure Storage that won a Best Paper Award at the June 2012 USENIX Annual Technical Conference. This was joint work between Microsoft Research and the Windows Azure Storage team.

The paper can be found here.

Windows Azure Storage is a cloud storage system that provides customers the ability to store seemingly limitless amounts of data for any duration of time that is highly available and durable. When using Windows Azure Storage, you have access to your data from anywhere, at any time, and only pay for what you use and store.

The internal details for how Windows Azure Storage works is described in our SOSP paper here. One of the areas only briefly touched on in the SOSP paper was the fact that in the background we lazily erasure code data to reduce its storage overhead while keeping your data durable and highly available.

In our USENIX paper we describe how we do erasure coding in Windows Azure Storage. For erasure coding, we introduce a new set of codes we call Local Reconstruction Codes (LRC). LRC reduces the number of erasure coding fragments that need to be read when reconstructing data fragments that are offline, while still keeping the storage overhead low. The important benefits of LRC are that it reduces the bandwidth and I/Os required for reconstruction reads over prior codes, while still allowing a significant reduction in storage overhead. It is optimized to efficiently reconstruct fragments in the face of (a) single fragment failures (e.g., failed disk, node, or rack), (b) when fragments are offline due to an upgrade, or (c) when access to a fragment is slow. In the paper we describe how LRC is used in Windows Azure Storage to provide low overhead durable storage with consistently low read latencies. In addition, we describe our erasure coding implementation and important design decisions that went into it.

Brad Calder