Highly Scalable, Ultra-Fast and Lots of Choices

Key/Value Store

Your domain model or an important part of it is rather simple, i.e. there are few interdependencies and constraints. But the application's data changes often or your application generates a huge amount of it.

There are many cases when you need to store data that is not critical to the application, but you've got plenty of it. Think about statistical data, status information of an online game or similar scenarios where small chunks of data are written often and read more or less often. Using a relational database in such cases does not scale well because of transactional overhead or database locks.

Speed is your main concern when reading and writing data. Your application must react quickly even at peak time. The ability to scale well is more important than data integrity and the ability to comfortably query data.

Typical features of relational database systems (such as transactional support and referential integrity) are not important to you. At least, they are less important than throughput. But your data still needs to be stored safely and high availability is a key issue.

Your domain model is not highly complex and your application's data has little structure, but you need at least support for some data types.

Your application's data might change rapidly, triggered by many simultaneous users, but read operations also need to execute quickly.

* * *

Therefore:

Choose a Key/Value Store, which stores little-structured data associated with unique keys.

The general idea of a Key/Value Store is to dispose of the concept of highly structured tables, columns and values in favor of a very simple programing interface: You associate simple data values such as numbers or texts with unique keys, and the database very efficiently stores and retrieves your data. Some Key/Value Stores provide operations to manipulate structures such as lists and sets whereas other Key/Value Stores only support basic or no explicit data types.

This simplicity provides the basis for supreme scalability. Most Key/Value Stores keep all data in memory all the time to achieve high performance. Their strategies to write data to disk and to distribute data within a cluster differ significantly. Keeping all data in memory limits the amount of data that every node can hold. Unless you are able to set up large clusters, this may also limit the maximum size of the stored data.

Key/Value Stores typically transfer the responsibility for the creation of unique keys to the application. It can be difficult to find a good strategy to choose appropriate keys (i.e. both meaningful and unique).

If you need to search for values, Key/Value Stores typically provide little comfort. The programming interface to a Key/Value Store is much more limited than what you might be used to from object-relational mapping tools, for example.

There are several variants of Key/Value Stores that focus on specific application development use cases and provide more sophisticated functionality. If you need to store large binary data, consider using a Blob Store. If you need to analyze large amounts of data at the same time as it is gathered, a Column Family Store might be a better option. If your application consists of a rich domain model, a Document Store might be appropriate.

Examples of Key/Value Stores are Memcached, Couchbase, Redis and Riak.

Back to the pattern overview