Efficient Data Transfer with Protocol Buffers

In the highly-connected environment of IoT technology, efficient communication and exchange of data holds an elevated level of importance. There is probably nothing more frustrating in UX than loading spinners, but while they are a necessary evil, it's important to strive for the lowest possible delay at all times.

Data Transfer: A solved problem (?)

When visualizing two devices as physical locations in the real world, there are different options for how to build the roads connecting A to B. Furthermore, the vehicles transporting the messages from one end to the other are important as well.

In this metaphor, the roads are built out of WiFi concrete, Bluetooth asphalt or plain-old cables. They express the transmission medium over which the exchange of messages is happening. The vehicles also come in different shapes and sizes, enclosing the data within them. They express the transmission protocol, the common language spoken by the two endpoints. Like with media, there are several options to choose from for the protocol: XML, JSON and YAML only show the tip of the iceberg.

With a flourishing ecosystem like this in place already, couldn't all IoT communication simply use XML or JSON? Is there even a need to introduce new protocols? The answers are Yes and Yes: Text-based serialization formats like JSON will allow devices to talk to each other, but their overhead may introduce significant delays in the constrained world of IoT devices, with their limited processing power. Therefore, let's explore a viable competitor to these existing solutions in the form of Protocol Buffers.

Protocol Buffers

The Protocol Buffer specification (protobuf for short) describes a binary serialization format developed at Google. It's claims are bold, stating that protobuf is a "language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler". It allows for highly compressed and performant message exchange between devices and also incorporates versionized data structures.

The Protocol Buffer workflow is divided into three phases: Definition, Compilation, Integration. The specification defines the format of a "message" (in programmer's terms, think of these as classes) and a compiler to convert these messages into workable code in a target language, protoc. The final step is the integration of the generated source code into our existing program alongside some helper libraries, and the flow is complete. This generated code will understand how to serialize an object into protobuf's binary byte stream, as well as how to parse this byte stream and convert it back into an object.

Protobuf Message definition

Note: All messages described in this post use the proto2 syntax over the more recent version, proto3. Both versions are not fully compatible with each other! Despite the existence of a newer version, proto2 is not deprecated and proved to be slightly more useful and flexible for our use case.

syntax = "proto2";
package com.example.models;

import "Tag.proto";

message Post {
    required int32 id = 1;
    required string title = 2;
    optional Type type = 3;
    optional string content = 4 [default = "No content."];
    repeated com.example.models.Tag tags = 5;

    enum Type {
        BLOG = 0;
        GALLERY = 1;
    }
}

The format for protobuf types will look familiar to most programmers working in a modern programming language, as it resembles a class structure. In fact, when compiling the above message using protoc, the generated source code will most likely contain a class in whatever target language the message is compiled into. The properties of the class are expressed as fields in protobuf and come in different flavors: required, optional and repeated (for typed arrays). Note how in proto2, optional fields may provide a default value if the sender did not specify any value for this field (this is the case for the content field above).

Messages can also reference each other and include namespaces. For the above example, consider a second file Tag.proto with a corresponding message in the same folder as Post.proto, which is imported at the top of the file. Finally, nested types are possible as demonstrated, with the embedded Type enumeration.

Finally, all fields inside of a message have a unique numerical identifier: The title of a Post message is associated with the field number 2, for instance. It is important to always obey the uniqueness of these values! Consider a change to the Post structure where the type is completely scrapped, but a new field for the post's author is added. To achieve backwards compatibility with older clients, the author's field must not re-use field number 3, previously used by the type. General convention is to comment out removed fields in the message, or maintain a list of "old fields" in a comment block at the top. Either way, the new field must receive a unique identifier, in this case the number 6.

syntax = "proto2";
package com.example.models;

import "Tag.proto";

message Post {
    required int32 id = 1;
    required string title = 2;
    // optional Type type = 3;
    optional string content = 4 [default = "No content."];
    repeated com.example.models.Tag tags = 5;
    required string author = 6;
}

syntax = "proto2";
package com.example.models;

import "Tag.proto";

/**
 * Removed fields:
 * - type = 3
 */
message Post {
    required int32 id = 1;
    required string title = 2;
    optional string content = 4 [default = "No content."];
    repeated com.example.models.Tag tags = 5;
    required string author = 6;
}

Using protoc to compile a message

One of the biggest advantages of protobuf is its property of being a language-agnostic format. The same message can be used as the foundation for compiling into different target languages. In our example, the IoT team could compile our Post into C++, while the Mobile team could compile the same message into Java or Swift. Since the serialization code is generated automatically, the error-prone way of maintaining the same structure for communication on both ends disappears.

The magic glue between the consumers of Protocol Buffers is its compiler, protoc. It will receive some .proto files as its input and write out source code in whatever target language it is asked to generate. Out of the box, there is support for several common languages. On top of that, a powerful plug-in system allows third parties to write custom bindings for otherwise unsupported languages.

Going back to our example, let's compile the messages into Java and Swift using protoc. Note that Swift support is added to the compiler through a plug-in maintained by Apple.

> ls -l
-rw-r--r--@ 1 marcel  staff  352 Feb 14 07:54 Post.proto
-rw-r--r--  1 marcel  staff  157 Feb 14 07:54 Tag.proto

> protoc --java_out=java/ --swift_out=swift/ Post.proto Tag.proto

The above command sent to protoc will instruct it to compile both Post.proto and Tag.proto, and dump the results into a folder named java and swift, respectively. Of course, the appearance of the generated code varies between platforms. In any case however, the corresponding platform will be able to utilize it to communicate with the other side using a common language, the protobuf.

So how small is it then?

Ultimately, the most pressing question after all this talk about protobuf comes down to this. Since there is undoubtedly more effort required to get Protocol Buffers up and running compared to, let's say, writing up a JSON document, the gains would need to be substantial in order to justify this initial overhead.

For our benchmark, the messages from our example above are used to compress the following Post object into protobuf's byte stream. Then, the size of this payload is compared to equivalent structures using XML and JSON. All inputs are accessible through the Gist below.

post = Post()
post.id = 100
post.title = "Efficient Data Transfer with Protocol Buffers"
post.type = Type.BLOG
post.content = "In the highly-connected environment of IoT technology..."

Effective Data Transfer with Protocol Buffers

Effective Data Transfer with Protocol Buffers. GitHub Gist: instantly share code, notes, and snippets.

262588213843476Gist

We can observe that the Protocol Buffer variant of the object clocks in at just over 100 bytes, which is about half the size of the XML version. Note that the optimization of the message actually causes an increase in size when gzip compression is applied to the protobuf content, however it's still the smallest of the bunch.

A new challenger?

In recent years, a new competitor has entered the transmission protocol scene, and it's one with an attitude. The self-esteem of the Cap'n Proto project is admirable, as it boasts its qualities with sassy taglines such as "Infinitely faster!" and many more stabs at the achievements of Protocol Buffers.

It's not an unfair case of foul-mouthing the competition, however, but rather a cheeky self-reflection: The creator behind the Cap'n Proto format is one of the original creators of Google's protobuf format, taking their years of experience with efficient data transfer and evolving it into this new form.

Our example is certainly not smaller with Cap'n Proto, but that does not invalidate the outrageous speed improvements claimed by the tool.  Is this a potential leap-off point for a sequel to this post? 🤔

Resources

https://developers.google.com/protocol-buffers
https://github.com/apple/swift-protobuf
https://capnproto.org/

Cover Photo by Holden Baxter / Unsplash