The internet is a deep mine of data. Data about people using things, recommending things, talking about things. Data about data, generated by machines which sample the data to produce more data. To do almost anything of interest with data it needs to be stored somewhere, perhaps just for a short time, processed, and moved from one system to another.
During a discussion with a friend who works on messaging system, the question arose of why there is so much hype among application developers around technologies that help with the storage of data, such as NoSQL databases, but so little about ideas that help with the movement. He had a straightforward opinion:
“I think people can easily see they have to store data somewhere but not that they need to move data around.”
This makes sense. Much of the time the datastore is actually seen to be responsible for the movement as well. When a developer writes some code to query a database, the database driver and network service that carry the query across to the service and bring the results back to the application are regarded as part of the database. The way that the two communicate can have a profound effect on how easy it is to use and scale such technology, but it is rarely regarded as a discrete component in the system.
Distributed systems are, almost by their nature, difficult. The growth of the internet, mobile devices, and near ubiquitous WiFi have all lead to an explosion in the scale and complexity of the code and systems needed to power and enable these networks of devices.
Architectures based on the idea of messaging are some of the best we’ve created so far for taming complexity, through composing simple elements into systems that can withstand the demands placed upon them.
Messaging has grown from being a powerful tool in finance to delivering thousands of social updates per second, managing millions of instant messages, connecting critical systems in businesses, governments, hospitals and more, and being at the heart of smart grids and sensor networks. Messaging systems are used to process terabytes of data and millions of updates per second, and are only expected to handle more, with more flexibility, and ever more complex requirements.
But there’s a catch to building a system around messaging. Messaging requires thinking about flows of information, about failure and reliability, about conversations, and quanta of data. As an analogy, while many concepts from one data store can be efficiently applied to another, if you try to approach a key value database, such as Riak, the same way you’d use a relational database, like MySQL, you’re going to eventually run into trouble. The same is true with messaging based systems - without understanding the glue being used, it is difficult to build a model that holds together under stress.
Messaging as an idea is built on a history of engineers addressing immediate problems, and my hope is that by looking at some of those, their challenges, and the solutions they arrived at, we can begin to approach the fundamentals that run through all messaging systems.
What Is Messaging?
Messaging is not as easy to define as might be expected. Messaging systems pass messages from process to process. Messages are generally discrete chunks of data, usually with some control or header information, and some contents or body. They can contain arbitrarily structured data, or nothing. They can be sent to one recipient, or to many. They may be reliably ordered, or not, and they may be reliably delivered, or not. The processes involved can be servers on two different machines separated by a continent, two processes of the same application within a single machine, or sometimes even simply two different threads of the same process.
The first useful ideas worth pulling in are message passing and message queueing. Message passing is the (fairly simple) idea that the only allowed communication between two discrete units of computation is via sending messages back and forth. This may seem pretty obvious, but rules out ideas like two processes accessing the same piece of shared storage, or reading the same fields in a database to communicate. Both communicating processes (or code blocks) must be “active” at the same time to allow the communication to occur, and the sender is targeting a specific receiver. In some cases message passing is an operation that requires synchronising both sender and receiver, but in many cases sending is asynchronous, and receiving is synchronous - so the receiver decides to read the message, and waits until it is completely accessed. This type of read is referred to as blocking, where the application pauses until the task has completed to some level, as opposed to non-blocking where the application can continue immediately.
Message queueing is the idea of one application pushing a message onto a queue - a holding buffer of some kind - which can then be consumed from the queue by a receiving application. The queue allows the two application to be unaware of each other, just aware (or at least hopeful) that something will eventually service or process the messages on the queue. The queue provides a shared space both participants can address, usually with the addition of ordering and transactional guarantees. This promotes the decoupling of the applications, making them less dependent on each other. There are several dimensions of coupling that can be addressed.
Technology coupling is reduced as the two applications can be written in different programming languages and run on different platforms, as long as they can read or write to the queue. Location is decoupled, as the two applications are just required to be in locations with access to the queue, but not necessarily be in the same location as each other, or to be aware of that location. The applications may be decoupled in temporal terms as well, in that the writer and reader never need to be run at the same time. The rate of message publishing and consumption is decoupled somewhat due to the buffering effects of the queue. The final type of coupling is in the format of the data that is passed between the applications, which is a slightly more complex issue, but may be addressed by using a standard or “canonical” format or message translators which modify message bodies from the format used by one client to the format of another.
These ideas are the initial building blocks with which people have constructed increasingly complex messaging systems. Before we talk about those messaging systems themselves though, it’s worth going even further back, to the beginnings of networks as we know them, and the invention of packet switching.