sbt 1.0 is now released, and everyone in the sbt community is hard at work upgrading their plugins. Because many sbt plugins depend on each other, there will be a short period of time (that we’re in now) where people won’t be able to upgrade their builds to sbt 1.0 because the plugins their builds use aren’t yet upgraded and released. However, that doesn’t mean you can’t cross build your plugin for sbt 1.0 now, simply upgrade to sbt 0.13.16 and use its sbt plugin cross building support.

I had a small problem yesterday though when working on the sbt-web upgrade, part of my plugin needed to be substantially rewritten for sbt 1.0 (sbt 1.0’s caching API now uses sjson-new rather than sbinary, so all the formats needed to be rewrittien). I didn’t want to rewrite this without an IDE because I knew nothing about sjson-new and needed to be able to easily browse and navigate its source code to discover how to use it, and I wanted the immediate feedback that IDEs give you on whether something will compile or not. The problem with doing this is that my build was still using sbt 0.13.16, and I couldn’t upgrade it because not all the plugins I depended on supported sbt 1.0. So, I came up with this small work around that I’m posting here for anyone that might find it useful, before reimporting the project into IntelliJ, I added the following configuration to my build.sbt:

sbtVersion in pluginCrossBuild := "1.0.0"
scalaVersion := "2.12.2"

Unfortunately it seems that you can’t leave this in the build file to ensure that sbt 1.0 is always the default, it seems that the sbt cross building support doesn’t override that setting (this is possibly a bug). But if you add that to your build.sbt right before you import into IntelliJ, then remove it later when you’re done developing for sbt 1.0, it’s a nice work around.

I have recently finished an implementation of socket.io server-side support for Play Framework. Naturally, I’ve become intimately familiar with the protocol, and I’ve formed a few opinions of it during the course of my efforts, which I’m going to share.

All reviews of technology are relative, relative to the things that the reviewer values in technology. Not everyone values the same things, and so this means that a review by someone who values different things to you is probably irrelevant to you. My reason for publishing this review is that there are a good number of people out there who share similar technology values to me, and so they will find this review useful as they evaluate whether socket.io is right for them. For people that don’t share the same values as I do, there’s not really much point in reading this blog post, you’ll probably disagree with me, but the disagreement will most likely be a disagreement in values, not on socket.io’s ability to meet those values. We can debate all day what set of values is right for software development, but this is not the blog post for that.

So what are my values? Here are a selection of values that are relevant to this review:

• I am a strong proponent of reactive systems, that is, systems that are responsive, resilient, scalable and message driven. In this context, streaming with backpressure is something that I see as important - a resilient system needs to propagate backpressure to ensure parts of the system don’t get overloaded.
• I’m strongly for tools, libraries and frameworks that enable high productivity software development.
• I think it’s important to have well defined standards and interfaces to maximise compatibility between decoupled implementations.

§The good

Why should you use socket.io? I was initially sceptical that socket.io offered much value at all, but over the course of implementing it, I’ve changed my view on that.

A lot of people are saying that with all major browsers supporting WebSockets, there’s no need for socket.io anymore. This is based on the assumption that all socket.io offers is a fallback mechanism to long polling when WebSockets are not available. This is completely false, most pertinently because socket.io doesn’t even provide that, the fallback to long polling is provided by a protocol that socket.io sits on top of, called engine.io. Here’s basically what engine.io provides:

1. Multiple underlying transports (WebSockets and long polling), able to deal with disparate browser capabilities and also able to detect and deal with disparate proxy capabilities, with seamless switching between transports.
2. Liveness detection and maintenance.
3. Reconnection in the case of errors.

The split between engine.io and socket.io is actually a great thing, engine.io implements layer 5 of the OSI model (the session layer), while socket.io implements layer 6, the presentation layer. I don’t know if the authors of engine.io/socket.io were aware of how closely their split mapped to the OSI model, but they did a great job here.

Now, even if you say that wide support of WebSockets makes engine.io redundant, it only makes half of the first point above redundant - it doesn’t address proxies that don’t support WebSockets (although this can be somewhat worked around by using TLS), and then it doesn’t have any mechanism for liveness detection (yes, WebSockets support ping/pong but there’s no way to force a browser to send pings or monitor for liveness, so the feature is useless), and browser WebSocket APIs have no in built reconnection features.

So that’s engine.io, back to my earlier point, transparent fallback of transports is not a socket.io feature. So what exactly does socket.io provide then? Socket.io provides three main features:

1. Multiplexing of multiple namespaced streams down a single engine.io session.
2. Encoding of messages as named JSON and/or binary events.
3. Acknowledgement callbacks per event.

The first two of these I think are important features, the third will feature in what I think is bad about socket.io.

§Namespaces

Multiple namespaces I think is a great feature of socket.io. If you have a rich application that does a lot of push communication back and forth with the server, then you will likely have more than one concern that you’re communicating about. For example, I’ve been working on a monitoring console, this console dynamically subscribes to many different streams based on what is currently in view, sometimes it needs to view logs, sometimes it needs to view state changes, some times it needs to view events. These different concerns are rendered in different components, and should be kept separate, their lifecycles are separate, the backend implementations of the streams are separate, etc. What we don’t want is one WebSocket connection to the server for each of these streams, as that is going to balloon out the number of WebSocket connections. Instead, we want to multiplex them down the one connection, but have both the client and the server treat them as if they were separate connections, able to start and stop independently. This is what socket.io allows. socket.io namespaces can be thought of (and look like) RESTful URL paths. A chat application may encode a connection to a chat room as /chat/rooms/<room-name>, for example. And then a client can connect to multiple rooms simultaneously, disconnect them independently, and handle their events independently.

Without this feature of socket.io, if you did want to multiplex multiple streams down one connection, you would have to encode your multiplexing protocol, implementing join room and leave room messages for example, and then on the server side you would have to carefully manage these messages, ensuring that subscriptions are cleanly cleaned up. There is often a lot more to making this work cleanly than you might think. socket.io pushes all this hard work into the client/server libraries, so that you as the application developer can just focus on your business problem.

§Event encoding

There are two important advantages to event encoding, one I think is less important, and the other is more important.

The less important one is that when the protocol itself encodes the naming of events, libraries that implement the protocol can then understand more about your events, and provide features on top of this. The JavaScript socket.io implementation does just that, you register handlers per event, passing the library a function to handle each event of a particular name. Without this, you’d have to implement that subscription mechanism yourself, you’d probably encode the name inside a JSON object that each message sent down the wire would have to conform to, and then you’d have to provide a mechanism for registering callbacks for that event.

The reason why I think this is the less important advantage is because I think callbacks are a bad way to deal with streams of events communicated over the network. Callbacks are great where the list of possible events is far greater than the number of events that you’re actually interested in, such as in UIs, because they allow you to subscribe to events on an ad hoc basis. But when events are expensive, such as when they’re communicated over a network, then usually you are interested in all events. In those cases, you also often need higher level mechanisms like backpressure, delivery tracking and guarantees, and lifecycle management, callbacks don’t allow this, but many streaming approaches like reactive streams do.

So, what’s the more important advantage? It allows the event name to be independent of the actual encoding mechanism, and this is of particular important for binary messages. It’s easy to put the event name inside a JSON object, but what happens when you want to send a binary message? You could encode it inside the JSON object, but that requires using base 64 and is inefficient. You could send it as a binary message, but then how do you attach meta data to it like what the name of the event is? You’d have to come up with your own encoding to encode the name into the binary, along with the binary payload. socket.io does this for you (it actually splits binary messages into multiple messages, a header text message that contains the name, and then a linked binary message). Without this feature of socket.io, it’s I think it’s impractical to use binary messages over WebSockets unless you’re streaming nothing but binary messages.

§The bad

So, we’ve covered the good, next is the bad.

§Callback centric

I’ve already touched on this, but in my opinion the callback centric nature of socket.io is real downside. As I said earlier, one of my values is reactive streaming, as this allows resilience properties such as backpressure and message guarantees to be implemented. Callbacks offer no such guarantees, if a callback needs to do a whole bunch of asynchronous tasks, there’s no way for it go back to the caller and say “hey, can you wait a minute before you invoke me again, I just have to go and store this to a database”. And so, an over zealous client can overwhelm a server as it sends events at a higher rate than it can process, causing the server to run out of memory or CPU. Likewise, when sending events, there’s no way for the emitter to say to the caller “hey, this consumer has a slow network, can you hold off on emitting more events?”, and so if the server is producing events too quickly for the consumer, it can run out of memory as it buffers them.

Of course, whether callbacks or streams are used to implement socket.io servers and clients is completely an implementation concern, and has nothing to do with socket.io. The implementation of socket.io that I wrote is completely streams based, using Akka streams to send events, and so backpressure is supported. On the client side, in the systems I’ve worked on, we use ngrx to handle events, which once again is stream based. And the authors of socket.io cannot be entirely faulted for implementing a callback based library, the browser WebSocket APIs only support a callback based mechanism with no support for backpressure.

Nevertheless, the callback centric design of socket.io manifests itself in the socket.io protocol - socket.io events are not single messages, but lists of messages, akin to argument lists that get passed to a callback, which is an awkward format to work with when streaming. As a socket.io library implementer that wants to provide full support for the socket.io protocol, this makes defining encoders/decoders for events awkward, because you can’t just supply an API for encoding/decoding a simple JSON structure, you have to allow decoding/encoding a list of JSON structures. For end users though, this doesn’t have to be a major concern - simply design your protocols to only use single argument socket.io events. So I wouldn’t treat this as a reason not to use socket.io, it’s just a feature (ie, supplying multiple messages rather than single messages in a single event) that I think you shouldn’t use.

§Acknowledgements

socket.io allows each event to carry an acknowledgement, which is essentially a callback attached to the event. It can be invoked by the remote side, which will result in the callback on local side being invoked with the arguments passed by the remote side. They’re convenient because they allow you to essentially attach local context to an event that doesn’t get sent over the wire (typically, this context is closed over by the callback function), and then when it’s invoked you have that context to handle the invocation.

Acknowledgements are wrong for all the same reasons that socket.io’s callback centric approach is wrong, acknowledgements subvert back pressure and provide no guarantees (with a stream you can have causal ordering and use that for tracking to implement at least once delivery guarantees, but acknowledgements have no such ordering and hence no such guarantees can be implemented).

Once again, this isn’t a reason not to use socket.io, you can simply not use this feature of socket.io, it doesn’t get in the way if you don’t use it.

§No message guarantees

Again, I’ve touched on this already, but I think it needs to be stated, socket.io offers no mechanism for message guarantees. The acknowledgements feature can be used to acknowledge that a message was received, but this requires the server to track what messages have been received. A better approach would be a more Kafka like approach, where the client is responsible for its own tracking, and it does this by tracking message offsets, and then when a stream is resumed, after a disconnect for whatever reason, the open message sent to the namespace would include the offset of the last message it received, allowing the server to resume sending messages from that point. This feature incidentally is built into Server Sent Events, so it would be nice to see it in socket.io too.

Of course, this can be built on top of socket.io, but I think it’s something that the transport should provide.

§The ugly

At a high level, I don’t think anything about socket.io is really that ugly. My ugly concerns about socket.io mostly exist at a low level, they’re things that only an implementer of the protocol will see, but some of them can impact end users.

§No specification

There is no specification for the socket.io wire protocol. This page seems to think it’s a specification, but all it describes is some of the abstract concepts, it says nothing about how socket.io packets get mapped down onto the wire.

Specifications are important for interoperability. The only way I could implement socket.io support is by reverse engineering the protocol, sometimes I had to do this with wireshark, since browser debugging tools don’t show the contents of binary WebSocket frames or binary request payloads.

Now, I’ve reversed engineered it and implemented tests, that’s a one time problem right? Wrong. Unless the socket.io developers never release another version of socket.io again, there will be incompatibilities in future. New features may be added. The developers might do it in a way that is backwards compatible with their own implementation, but because there’s no specification, other implementations may have implemented their parsing differently, which will cause them to break with the new feature. There may be edge cases that I didn’t come across. And so on.

Although the lack of specification primarily impacts me now, it will negatively impact users of socket.io in future, and this needs to be considered when deciding whether socket.io is right for your project or not.

§Weird encodings

The way socket.io and engine.io are encoded, especially to binary, is very weird in places. For example, there are about 5 different ways that integers get encoded, including one very odd one where each decimal digit is encoded as an octet of value 0 to 9, with a value of 255 used as a terminator for the number. Which might make sort of make sense (but not really) if you had a use case for arbitrary precision integers, but this is for encoding the length of a payload, where a fixed length word, like 32 bit unsigned network byte order, would have done just fine. I mean, it’s not the end of the world to have all these different ways to encode integers, but it really doesn’t inspire a lot of confidence in the designers ability to design network protocols that will be tolerant to future evolution.

People much smarter than us have, over many years, come up with standard ways to encode things, which overcome many gotchas of communicating over a network, including performance concerns, efficiency concerns and compatibility concerns. There are then many libraries that implement these standard ways of encoding things to facilitate writing compatible implementations. Why forgo all that knowledge, experience and available technology, and instead come up with new ways to encode integers? It just seems very odd to me.

§Unnecessary binary overhead

There’s not a lot of use cases for sending binary messages, but many of the use cases I can think of I would want to have as little overhead as possible, such as streaming voice/video. Binary messages in socket.io require sending two messages, one text message that looks like a regular text message, including the name and namespace of the event, and a JSON encoded placeholder for the binary message (it literally looks something like {"_placeholder":true,"num":1}), and then the binary message gets sent immediately after. This seems to me to be a lot of overhead, it would have been better to encode the entire event into one message, using a separate binary encoding for encoding the namespace/name and then placing the binary message in that.

I can understand a little why it is the way it is - because events contain multiple messages, you can mix binary and text messages. Having one reference the other with placeholders is a sensible way to encode that. But, this all comes back to the callback centric nature of socket.io, the reason events contain multiple messages is that callbacks can have multiple arguments, if each event only contained one message then this wouldn’t be an issue.

§Conclusion

I think socket.io is a very useful piece of technology, and is incredibly relevant today in spite of the popular view that widespread support for WebSockets makes it redundant. I would recommend that it be used for highly interactive applications, its namespacing in particular is its strongest point.

When using it, I recommend not taking advantage of the multi-argument and acknowledgement features, rather, simply use plain single argument events. This allows it to integrate well with any reactive streaming technology that you want to use, and allows you to use causal ordering based tracking of events to implement messaging guarantees.

The underlying protocol is a bit of a mess, and this is compounded by the lack of a specification. That particular point can be fixed, and I hope will be, but it will require more than just someone like myself writing a spec for it, it requires the maintainers of the socket.io reference implementations to be committed to working within the spec, and ensuring compatibility of new features with all implementations going forward. There’s no point in having a spec if it’s not followed. This will introduce friction into how quickly new features can be added, but this is a natural consequence of more collaboration, and more collaboration is a good thing.

This post is a little off topic compared to the rest of my blog, but it’s a topic that is close to my heart, and I feel I have to share it.

I am disheartened in the leadership of the churches and church groups that I am part of, as I fail to see our church leaders make any definitive statement over whether abuse (either emotional or physical) is grounds for divorce. Over the past weeks in Australia, there has been a big discussion over domestic abuse amongst Christians. This was triggered by a TV program and accompanying article published by the ABC.

In the Christian circles that I mix in, reactions to this have been mostly positive, rather than getting defensive, church leaders and attendees alike have seen it as an important wake up call, to make changes to the way we address abuse in our churches, particularly in our public messaging. As the ABC pointed out, one of the major problems was that abuse victims often never hear from churches that their husbands abuse is unacceptable.

In spite of this positive discussion, I am yet to hear a leader in the conservative church circles that I mix in come out and say “Abuse is grounds for divorce”. And I don’t know why that is. I don’t know if it’s because they think abuse is not grounds for divorce, I don’t know if it’s because they think it is and everyone knows it, I don’t know if it’s because they think it is but they want to be careful in how they word it. But, I do think it is terrible that there is no clear message on this, and so I am disheartened.

Matthew 5:32 says:

But I tell you that anyone who divorces his wife, except for sexual immorality, makes her the victim of adultery, and anyone who marries a divorced woman commits adultery.

Taken at face value, this verse says that abuse is not grounds for divorce. But, is that a timeless argument applying throughout the ages, or does it speak into a certain culture, and need to be adapted for different cultures? I am no theologian and I have never gone to Bible college so what I say now does not come with any authority, but to me it seems to be cultural. And the lack of clear teaching from church leaders here compels me to talk about this.

Firstly, Jesus is speaking to husbands in this passage, not wives. And I don’t think that that’s just because it was a male dominated culture and we can just assume that whatever Jesus said to men he was also saying to women. Marriage was very different back then. Wives were bought with a dowry. Their only chance at success in life was to marry. To not marry often meant a life of poverty, and to be divorced? A life of shame and poverty. The topic of wives divorcing their husbands in Jesus day didn’t make sense, there was no precedent for it, how can property say to its owner “I am no longer your property?” And it didn’t make sense because why would a woman choose a life of shame and poverty over the security of having a husband? In the time that Jesus spoke into, the only context in which divorce was understood and made sense is men divorcing their wives, and not the other way around.

Today, marriage is very different. A wife is not considered her husbands property, she is not bought, she enters into and stays in a marriage of her own volition. Divorce does not mean the end of her security, it does not equate to a life of poverty and shame. We must not let the fact that today marriage is on equal terms confuse us into thinking that therefore whatever Jesus said about marriage 2000 years ago applies equally to men and women.

In the context of Jesus’ time, for a man to divorce his wife was to sentence her to a life of shame and poverty. This makes Jesus’ command an incredibly practical and loving command, he is instructing men to not sentence their wives to a life of shame and poverty - with the exception being if they willingly leave the relationship for another man. The exception was merely an out for a husband to allow him, the only person that could, to initiate the divorce when the wife refused to honour her wedding vows. Does that same command make sense in today’s context? I don’t think so. I think in today’s context, the exception goes away. Why? Because in today’s society, women can freely divorce their husbands. So, if a woman doesn’t want to be with her husband, and wants to be with someone else instead, she can just do that. She couldn’t before. And so husbands no longer need an out, they no longer need to divorce their wives when their wife refuses to honour her wedding vows, because the wife is able to do that herself. As is modelled throughout the Bible, with God’s relationship to Israel, I believe we are to be forgiving, if our spouse commits adultery, but turns back to us in genuine repentance, we are to forgive them. So unfaithfulness is no longer the excuse it used to be for divorce.

So what about abuse. In the context of a husband being the only person that can initiate a divorce, and a wife for whom divorce is the worst thing that could possibly happen to her (better to be abused by your husband than face constant abuse living on the streets), and especially in a context where society saw it as a mans role to own and control his wife, it would have made no sense for Jesus to have mentioned abuse as grounds for divorce, it just doesn’t fit into that picture. But in today’s context, where divorce is not the end of the world for a woman, everything changes. Abuse is a direct contradiction to a couples wedding vows, it makes the vows a lie, rendering them null and void. I believe a woman can divorce a man who abuses her, because he has already rendered their wedding vows void. I also believe a man can divorce his wife if she abuses him, though as I understand it that is a far less common occurrence.

Is there a place for repentance and forgiveness? Absolutely. But I believe such repentance is a new statement of the wedding vows - a second marriage has taken place, one in which a woman is free to enter or not, just as she was in her first marriage to the man. The decision to forgive a husband is completely hers, and counsellors should not try to push her in that direction, rather, counsellors should express extreme scepticism, for the sake of the woman’s safety, at any repentance that the husband claims, and, if the woman decides to proceed with remarriage, a counsellor needs to temper the reunion with extreme caution. Counsellors must never forget that they themselves can easily fall trap to a husbands manipulation, just because a husband recites the biblical principles behind self sacrificial love in marriage perfectly does not at all mean that he is repentant.

I wish my church leaders would come out and make a clear statement that abuse is grounds for divorce. I wish they would make it loudly, clearly, and often, so that women in abusive relationships are able to hear that there is a way out. I am greatly disheartened that this has not happened already.

The problem with CQRS is that it makes things more complex because it decreases consistency.

I hear this argument a lot. As the author of a framework that strongly encourages CQRS, I obviously am biased against this opinion, and disgaree with it.

Of course, the statement and my disagreement needs context and qualification. If you have a traditional monolithic system, where all data operations are done on a single database, and that single database supports ACID transactions, then yes, switching to CQRS will decrease consistency. However, that’s not the context in which CQRS usually comes up, and it’s not the context that I recommend using it in. The context is microservices, and more and more industry experts are recommending that people move away from monoliths and move to microservices.

§Microservices decrease consistency

A core principle of microservices is that they should own all their own data. Another core principle is that they should be able to act autonomously. This means they shouldn’t need to depend on other services - in particular making synchrconous calls to other services - in order to implement the functions they are responsible for.

If a service is to be autonomous then, it needs to store all the data necessary for it to achieve its functions. Sometimes this data may be owned by other services - and so in order to function, those services need to publish that data for this service to consume and update its own store. This means in a microservices system, the same data is going to be stored in many places. Will this data be strongly consistent across the entire system? Unless every time you do an operation, you stop all other requests to the entire system, and process the operation until it succeeds, the answer is no, it will often be inconsistent.

So, a switch to microservices is by nature a switch to decreased consistency. Decreased consistency is a consequence of microservices.

§Inconsistency can be very bad

Because we are using microservices, our system can become inconsistent, and we need to deal with that. How inconsistent our system can get depends on how we deal with it. The typical, very high level hand wavy approach to dealing with inconsistency in a distributed system is to use eventual consistency. But how is that actually achieved?

Let’s imagine service B needs some data owned by service A to implement its responsibility. In this scenario, A is responsible for writing the data, it’s the thing that handles the command, while B is responsible for reading the data, it does the query. In order for B to act autonomously though, it can’t query A directly, it needs to have A push it the data so that it can update its own query store, and then when the time comes to use it, query it locally.

Service A could do this synchronously, when an operation is done on service A, it will make a call on service B to update it with the results. Now this works fine on paper, but what happens when service B is down at that time? Does service A have to keep retrying until service B comes back up? What if service A then goes down?

As you can see we now have a consistency problem, one that will not eventually become consistent. Service A has been updated, but service B failed to update. It will stay inconsistent forever, we could say that it is terminally inconsistent, and will require manual intervention in order to fix. The reason we have this problem is that we combined our command responsibility and query responsibilities in the one operation, and since that operation wasn’t atomic, a partial failure of the operation will lead to terminal inconsistency.

§CQRS gives you consistency back

CQRS means separating command responsibilities from query responsibilities. In our scenario, using CQRS, service A handles the command, and updates its state, and is done. Then, in a separate operation, another process will take the result of the operation on A, and asynchronously push it to service B.

Since the operation is asynchronous, service B doesn’t need to be up at the time - for example it can use an at least once messaging queue to handle the message. If service B isn’t up at the time, then the system will be inconsistent for a period of time. However, when service B comes back up, and then processes the message, the system will become consistent again, it will be eventually consistent.

So by employing CQRS, we are able to get back some of the consistency guarantees that we lost when we moved to microservices. We don’t have a globally consistent system, but we can guarantee an eventually consistent system.

§CQRS is a necessary evil

CQRS is complex, far more complex than handling both the command and query responsibilities of data in single operations on a strongly consistent database. However, in a microservices world, we don’t have the luxury of relying on a single strongly consistent database. In that world, inconsistency is a given. If someone says using CQRS in microservices means you lose consistency - they have failed to acknowledge that they lost consistency the moment they started using microservices, it was not CQRS that lost them that consistency. Rather, CQRS is a very powerful tool that allows us to address the inherent inconsistency of microservices to give us eventual consistency instead of terminal inconsistency.

In architecting a system, a question that you often have to ask is how much information should be included in an event? The more information you include, the more the services that receive the event will be to be able to process it without looking up information from other services, which will make your system more resilient and scalable. For example, if a user submits an order, the system might publish an order submitted event. But should it include the list of items that were ordered in that event? If it does, then services consuming that event will not need to go back to the order service to find out what items the order has, they’ll be able to process that event in isolation from the order service and in doing so be more resilient.

On the other hand, if you include too much information in your event, your event becomes very large, which will impact the throughput of your messaging infrastructure and the cost of processing events. While this is true, it’s tempting to consider the problem as primarily a resilience vs throughput problem, and this is how I used to think of it. However, I no longer think of that as the major trade off.

One way to view events is to see them as facts. The words event and fact have very similar meanings, and can often be used interchangeably. An event is something that happened (or will happen, but in the context of systems architecture we always use it to refer to something that has happened). A fact is something that happened. However they have a different focus, and consequently when we think of events vs thinking of facts, we apply slightly different thought processes and constraints. When referring to something as an event, the emphasis is placed on what happened. When referring to something as a fact, the emphasis is placed on the truth of the thing that happened.

This subtle difference can change the way we think about the messages that convey this information. A fact is indisputable, so if an event is a fact, then it should only contain information that is indisputable. The user submitted the order at this time. That is indisputable, it is a fact that the user did that. The order contains these items. That is disputable, the order may have been changed since it was submitted due to the availability of items in it changing.

The property of our system that is in question here is consistency. If we treat all events as facts, containing only indisputable information, then we improve the consistency of our system. A service can’t make the mistake of trusting information in an event that might later become false, since the event won’t contain that information. And this consistency issue is, in general, a bigger issue than the throughput of your message broker. There are many things that can be tuned to increase throughput, but addressing inconsistency requires a lot more than just tuning.

And so I’ve realised that the biggest concerns when deciding how much information to include in events are resilience and consistency. To include more information beyond the fact that happened is to increase resilience at the cost of consistency.

Of course, it is a trade-off, and one that only the business requirements can decide where the appropriate balance lies. If I have an email notification service that is subscribing to order events, and it needs the list of order items in order to render an email notification, it is fine for it to simply read the items from the event to generate the email to send.

On the other hand this depending on the items from the event will cause a problem for the inventory service, which needs to adjust its inventory levels according to what the order contains when it’s eventually dispatched. There are two ways this could be addressed while still maintaining resilience - the inventory items could receive all events regarding changes to items in the order from the start, through to when the order is submitted, and right through to dispatch. Or it may consider the order submitted event with its list the starting state of the order, and subscribe to subsequent change events for the order.