Designing Network Protocols
4 stars based on
It frustrates me when people use ASCII instead of packed bitmaps for things like this packet transmitted once a second from potentially hundreds or thousands of nodes, that each frontend proxy has to parse into a binary form binary protocol design before using it. Maybe it's a really small amount of CPU but it's just one of many things which could easily be more efficient.
This thread on HN continued with dozens of other posts from many authors, with peterwwillis holding his ground on his original point. Compared to the 2. However, one of the few features I binary protocol design write for 2. The primary use case is for use by the mod lbmethod heartbeat moduleto direct traffic to the least loaded server in a reverse proxy pool. After stopping development of our thick client and peer to peer systems, we were moving to a HTTP based distribution of binary protocol design content.
We had a pool of super cheap storage nodes, which liked to die far too binary protocol design. We built a system to have the storage nodes heartbeat with what content they had available, and a reverse proxy that would send clients to the correct storage server.
This enabled a low operational overhead around configuration of both our storage nodes and of the reverse proxy. Operations would just bring on a new storage node, put content on it, and it would automatically begin serving traffic.
If the storage node died, traffic would be directed to other nodes still online. Enable flexible load balancing for reverse proxy servers. For Joost we had good switches since we were previously setup for high packet rate peer to peer traffic. We also had previously used multicast for other projects. We choose to use a simple UDP multicast heartbeat as our server communication medium. If you only have 10 nodes, you can do everything by hand.
If you have hundreds of nodes, you are most likely building a hierarchical distribution of load. In my experience it is not a common configuration to have 10, application servers behind a single load balancer. I believe the sweet spot for this automatic configuration via multicast is pools between 10 and servers. The Apache HTTP server is all about being the flexible centerpiece of internet architectures, with many binary protocol design producers, consumers, and interfaces.
We must have a network protocol that is easily implemented in any programing language or binary protocol design, without adding additional dependencies. At Joost we embedded the available video content catalogs into the heartbeat advertisements. We needed a protocol that would be open to proprietary extensions without causing pain.
In a clustered systems you do not want the overhead of the cluster communication to negatively affect your application. It binary protocol design important here to understand binary protocol design many binary protocol design will actually hit packet-per-second limits before raw bandwidth limits.
We also assumed at this point in time all binary protocol design have gigabit internal networking. In my experience the difference between a 20 byte packet and an 8 byte packet that is being multicasted once a second is not a relevant issue on modern LANs.
Even with servers emitting packets, this is How efficient this network flow is will depend on your exact multicast configuration and your specific switches, but in most configurations it is a non-issue.
Wireshark and packet dumps are the best friend of a Network Admin. When people are doing packet dumps, they are looking for problems. A simple ASCII encoding of data will be easy for these people to see when they are in times of stress.
Decoding a more complex binary encoding might get added as a feature to Wireshark someday, but it is yet another barrier. Design for the long term: Design all public network protocols to be around for 10 years or longer. Include a versioning scheme. Don't assume that 10 years from now your encoding system will still be around.
I love msgpack for internal applications, but on these binary protocol design scales for a public protocol, nothing beats straight up ASCII bytes. Given the above considerations in at Joost, I started sketching out the possible formats for the multicast packet.
I considered using a binary format, but the immediate problem was having extendable fields. This meant we would need more than a few simple bytes. To create an extensible binary format, I started looking at serialization frameworks like Apache Thrift.
At this time in Thrift had only been open sourced a few monthsand it really wasn't a stable project. Since the number of binary object formats like BSONGoogle Protocol BuffersApache Avroand Msgpack have exploded, but just 4 years ago there really weren't any good standardized choices or formats for a pure-C project.
The only existing choice would be to use ASN. I decided that because of this and the other goals around debug-ability to peruse an ASCII based encoding of the content. I made the decision to use query string style parameters as the best compromise for the content of the multicast packet's content. If I were to need to implement the same system today, there are a few things I might change, but I don't think any of them are critical mistakes given the original design constraints:.
Binary protocol design encodings of information can be both smaller and faster, but sometimes a simple ASCII encoding is sufficient, and should not be overlooked. The decision should consider the real world impact of the choice. In the last few years we have seen the emergence of Thrift or Protocol Buffers which are great for internal systems communication, but are still questionable when considering protocols implemented by many binary protocol design and consumers.
For products like the Apache HTTP server, we also do not want to be encumbered by large dependencies, which rules out many of these projects. When designing the content of this heartbeat packet, I was binary protocol design about the following issues: Decoding a more complex binary binary protocol design might get added as a feature to Wireshark someday, but it is yet another barrier Design for the long term: What I did in Given the above considerations in at Joost, I started sketching out the possible formats for the multicast packet.
The choices for non-binary formats were: While XML is everywhere, and almost all languages have good bindings, it would be the most verbose choice. I also felt that it is too extendable.
Binary protocol design later would add namespaces and other features that would make binary protocol design a consumer much more difficult.
Easier to consume, and today there are libraries for all languages. This format is understood by every component in a web server stack, and Apache already included examples of parsing this type of format. The format binary protocol design also easy to build without external libraries, meaning reimplementation in any language is very easy.
The use binary protocol design a key and value system also means implementers binary protocol design use simple data structures like a linked list or hash for interacting with their representation. The number of worker processes that are ready to accept new connections. The number of binary protocol design processes that currently servicing requests.