Project for Computer Networks - INF142, Spring 2010

Author: Eirik Schwenke

Foreword

Many thanks to all the other slackers, that managed to get an extension for all of us that were too lazy to even ask — and to the professor for granting the extension.

Without your help, this paper would not have been accepted.

Also many apologies for the rough quality of this document; I only found the time to drag together a quick draft — hopefully it is still coherent enough to be readable.

Introduction

I have chosen to write a bit about Google protocol buffers8, and how they relate to the design of custom network protcols. I describe how Protocol Buffers fits into a modern stack of network tools, and show some examples that aim to highlight some of the strenghts and weaknesses vis-a-vis XML, JSON and plain text protocols.

Background

Network Protocols

According to Tan2712:

"(…) a protocol is an agreement between the communicating parties on how communication is to proceed."

Google protocol buffers8 is a tool for developing and using application level protocols. It allows for serialization and marshalling of complex data structures, and so allows programs to communicate through persistent storage on disk, or over a network.

Originally protocol buffers was created to help with writing client-server libraries and programs. Traditionally, when creating a new client/server system, there have been a few choices on how to proceed.

There is good old RPC/XDR dating back to RFC101413, but more recently defined in RFC450614. XDR is in many ways similar to protocol buffers. XDR also defines an on the wire format for data, and abstracts message description to a seperate file (an ".x"-file").

Another older competetior in the same niche is ICE10: the Internet Communication Engine. A comprehensive set of librariers, standards and languages for building complex, hetrogenous systems.

What most people most assosiate protocol buffers with, especially with it's focus on serializing and marshalling data, would be to more recent technolgies: XML and JSON and XMLRPC.

XML or Extensible Markup Language, have long been held up as the ultimate tool for data exchange on the internet. But the Document Object Model, or DOM for short, along with the backwards compatible and extendible syntax of XML, have lead to very real performance issues.

XML promises to be machine verifiable, human readable, and extentible — and it delivers on all these promises, but at a steep price in complexity, and indirectly, perfomance. Writing a complete XML parser is no small undertaking — and the resuliting program is more complex, more difficult to test — and quite a lot slower — than most software authors like.

Leveraging the more-or-less linear Simple Api for XML Processing (SAX) allivietes some of the issues — but if all that is needed is some simple type-safety for moving data — XML is often the wrong tool for the job.

Many developers feeling that XML (and XMLRPC) is too bloated, have turned to JavaScript Object Notation (JSON). JSON is more compact than XML, but lacks many tools for type checking that XML provides. Both JSON and XML serialize data as text, and neither have any inherent support for encoding numbers directly; both use decimal notation that is both space consuming to store, and time consuming to convert and process.

Motivation — According to Google

According to the protocol buffers [homepage] [gpbh]:

Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data — think XML, but smaller, faster, and simpler.

and:

Protocol buffers were initially developed at Google to deal with an index server request/response protocol. Prior to protocol buffers, there was a format for requests and responses that used hand marshalling/unmarshalling of requests and responses, and that supported a number of versions of the protocol (…) [This] complicated the rollout of new protocol versions, because developers had to make sure that all servers between the originator of the request and the actual server handling the request understood the new protocol before they could flip a switch to start using the new protocol.

Further, according to Google:

Protocol buffers were designed to solve many of these problems:

(…)

As the system evolved, it acquired a number of other features and uses:

Implementation

Encoding and Decoding

Like XDR, protocol buffers defines a binary encoding scheme for data, along with a separate way to define metadata. A message may be blindly decoded without knowledge of the concrete protocol meta-data, without complete loss of information.

The basic datatypes are defined, as are hierarchies of structures — but the basic idea is to use metadata (.proto-files) when making programs that encode and decode messages.

To better illustrate the difference between XML and protocol buffers for serializing data, we can consdier the following 100-line C program9 that is able to decode any protocol buffer message (Note the main method is removed — see below):

/* pb.c -- decode a protobuf into its field numbers and wire values.
 *
 * This decoder does not require a .proto file.  It is equivalent to a
 * generic XML parser, in that it can parse the file, but doesn't know
 * semantically what any of it means.  Also like XML, it doesn't know whether
 * a 32-bit value is an integer or a float, signed or unsigned.  Again, this
 * is much like XML, where you are similarly in the dark if you see a fragment
 * like <foo>332</foo>.
 *
 * Joshua Haberman <joshua@reverberate.org>
 */

#include <stdint.h>
#include <stdio.h>

enum wire_type {
    WT_VARINT = 0,
    WT_64BIT  = 1,
    WT_STRING = 2,
    WT_32BIT  = 5
};

/* Structure representing a tag number and its corresponding wire value.
 * With a .proto file we can refine this data by:
 * - translating field_number -> field_name
 * - translating the wire value into the specific value (ex. 32-bit -> float)
 */
struct key_value_pair
{
    int field_number;
    enum wire_type wire_type;
    union {
      uint64_t varint;
      uint64_t _64_bit;
      struct {
          char *start;
          int len;
      } string;
      uint32_t _32_bit;
    } value;
};

/* A callback that is called every time we parse a key/value pair */
typedef void (*yield_t)(struct key_value_pair*);

uint64_t decode_varint(char *buf, char **end)
{
    uint64_t ret = 0;
    int bitpos = 0;
    for(int bitpos = 0; *buf & 0x80 && bitpos < 64; bitpos += 7, buf++)
        ret |= (*buf & 0x7F) << bitpos;
    ret |= (*buf & 0x7F) << bitpos;
    *end = buf+1;
    return ret;
}

uint32_t get_32_le(char *buf, char **end)
{
    *end = buf+4;
    return buf[0] | (buf[1] << 8) | (buf[2] << 16) | (buf[3] << 24);
}

void decode_protobuf(char *buf, char *buf_end, yield_t yield)
{
    while(buf < buf_end)
    {
        uint64_t key = decode_varint(buf, &buf);
        struct key_value_pair pair = {
            .field_number = key >> 3,
            .wire_type = key & 0x07
        };
        switch(pair.wire_type)
        {
            case WT_VARINT:  /* varint */
                pair.value.varint = decode_varint(buf, &buf);
                break;

            case WT_64BIT:  /* 64 bit */
                pair.value._64_bit = get_32_le(buf, &buf);
                buf += 4;
                pair.value._64_bit |= (uint64_t)get_32_le(buf, &buf) << 32;
                buf += 4;
                break;

            case WT_STRING:  /* string */
                pair.value.string.len = decode_varint(buf, &buf);
                pair.value.string.start = buf;
                buf += pair.value.string.len;
                break;

            case WT_32BIT:  /* 32 bit */
                pair.value._32_bit = get_32_le(buf, &buf);
                buf += 4;
                break;
        }

        yield(&pair);
    }
}

void yield_cb(struct key_value_pair *pair)
{
    printf("Field number: %d, Wire type: %d\n", pair->field_number,
                                                pair->wire_type);
}

The important part about this small program is that do do something similar (in a low-level language like C) for XML, is significantly more complicated, due to the nature of how XML is decoded.

The other important thing note note, is although this program knows nothing of the partiuclars of a given protocol, it is still able to recognize strings and integers.

On-the-wire — Differences From XML/JSON

As we've seen in the exampe above, the binary encoding is quite simple. The full encoding (and decoding) details are well documented at http://code.google.com/apis/protocolbuffers/docs/encoding.html.

Basically the messages are serialized as a series of integers, encoded as varints. Each message consists of one or more fields, that have a fixed or variable length. This data is then binary encoded, with a field number, optional length (for variable length fields) and the data.

For example, given the protocol definition:

// person.proto
message Person {
  required uint64 id = 1;
  required string name = 2;
}
// Java code generation requires a different
// name for the protocol (now PersonProt) and the message class (Person)
option java_outer_classname = "PersonProt";

encoding the data Eirik Schwenke as the string name and 12 as id, would yield the binary representation (represented here in hex, with comments):

Field number 1 and sire type 0 (in high and low bits)
0x08

 12 (id)
0x0c

Field number 2 and type 2 (in high and low bits)
0x12

14  (Number of symbols in the string "Eirik Schwenke")
0x0e

  E   i     r    i    k         S    c   h    w    e    n    k    e
0x45 0x69 0x72 0x69 0x6b 0x20 0x53 0x63 0x68 0x77 0x65 0x6e 0x6b 0x65

One byte holds type-information (for the id: varint(0) and 1, for the string: lenght delimited(2) and tag:2). The integer is encoded using varints, and because of it fits in 7 bits, is simply represented as itself. The string has a field representing the length, followed by the character codes encoded as ints.

Contrast this with JSON:

{"person": {"id": 12, "name": "Eirik Schwenke"}}

  {   "
0x7b 0x22
  P    e   r    s    o    n     "    :        { 
0x50 0x65 0x72 0x73 0x6f 0x6e 0x22 0x3a 0x20 0x7b

 "    i    d    "    :    1    2 
0x22 0x69 0x64 0x22 0x3a 0x31 0x32

 ,    "    n    a    m    e     "   :          "
0x2c 0x22 0x6e 0x61 0x6d 0x65 0x22 0x3a 0x20 0x22

  E   i     r    i    k         S    c   h    w    e    n    k    e
0x45 0x69 0x72 0x69 0x6b 0x20 0x53 0x63 0x68 0x77 0x65 0x6e 0x6b 0x65
  "    }    }
0x22 0x7d 0x7d

While the actual string is encoded the same, we now have 24 bytes of metadata and lost the ability to skip over the name-field (as we don't know the length of the field). In addition the integer is encoded rather innefficiently as a unicode string, that requires rather complex parsing (first from string to decimal, then from decimal to integer).

With XML the situation is somewhat better, if we leave out the XML header, and assume a best-case encoding of the data:

<Person name="Eirik Schwenke" id="12" />

A more realistic example would be much worse, however:

<?xml version="1.0" encoding='UTF-8'?>
<Person xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
        xsi:noNamespaceSchemaLocation="SimplePerson.xsd">
    <name>Eirik Schwenke</name>
    <id>12</id>
</Person>

Just looking at the hex, we see the difference in space-requirements:

0x3c 0x3f 0x78 0x6d 0x6c 0x20 0x76 0x65 0x72 0x73 0x69 0x6f
0x6e 0x3d 0x22 0x31 0x2e 0x30 0x22 0x20 0x65 0x6e 0x63 0x6f
0x64 0x69 0x6e 0x67 0x3d 0x27 0x55 0x54 0x46 0x2d 0x38 0x27
0x20 0x3f 0x3e 0x0a 0x3c 0x50 0x65 0x72 0x73 0x6f 0x6e 0x20
0x78 0x6d 0x6c 0x6e 0x73 0x3a 0x78 0x73 0x69 0x3d 0x22 0x68
0x74 0x74 0x70 0x3a 0x2f 0x2f 0x77 0x77 0x77 0x2e 0x77 0x33
0x2e 0x6f 0x72 0x67 0x2f 0x32 0x30 0x30 0x31 0x2f 0x58 0x4d
0x4c 0x53 0x63 0x68 0x65 0x6d 0x61 0x2d 0x69 0x6e 0x73 0x74
0x61 0x6e 0x63 0x65 0x22 0x20 0x0a 0x78 0x73 0x69 0x3a 0x6e
0x6f 0x4e 0x61 0x6d 0x65 0x73 0x70 0x61 0x63 0x65 0x53 0x63
0x68 0x65 0x6d 0x61 0x4c 0x6f 0x63 0x61 0x74 0x69 0x6f 0x6e
0x3d 0x22 0x53 0x69 0x6d 0x70 0x6c 0x65 0x50 0x65 0x72 0x73
0x6f 0x6e 0x2e 0x78 0x73 0x64 0x22 0x3e 0x0a 0x3c 0x6e 0x61
0x6d 0x65 0x3e 0x45 0x69 0x72 0x69 0x6b 0x20 0x53 0x63 0x68
0x77 0x65 0x6e 0x6b 0x65 0x3c 0x2f 0x6e 0x61 0x6d 0x65 0x3e
0x0a 0x3c 0x69 0x64 0x3e 0x31 0x32 0x3c 0x2f 0x69 0x64 0x3e
0x0a 0x3c 0x2f 0x50 0x65 0x72 0x73 0x6f 0x6e 0x3e

The most important difference is still the inability to seek in the stream easily. This is important if want to do processing on only select fields in a complex message, get all the id's of a message consisting of a list of person-messages, for instance.

On the other hand, if the data one whish to transfer is something for which XML fits well, the overhead ceases to be as significant. Particularily if all parsing of the data will need to build a data structure similar to the DOM — then XML (and JSON) is a good fit.

The chief difference between JSON and XML is that there is no formal, and well defined way to check the integrety of JSON-encoded data — the notion of well formedness applies to any XML-document, and if we have a schema, we can also validate from a data type perspective, as well as a structural perspective.

Both JSON and XML are more like documents, or trees/lists — while protocol buffers is more like a C struct or an array; data is indexed, and we can easily skip over the parts we aren't interested in.

Building a Protocol

From Protocol Definition to Serialized Messages

We have now looked at what protocol buffers provide: A means to describe metadata (.proto files) and a means to encode and decode data. In addition the toolkit includes a code generator that generates basic code from the protocol defintions.

For example, the sample person protocol above, leads to the following generated python class:

# person_pb2.py
# Generated by the protocol buffer compiler.  DO NOT EDIT!

from google.protobuf import descriptor
from google.protobuf import message
from google.protobuf import reflection
from google.protobuf import service
from google.protobuf import service_reflection
from google.protobuf import descriptor_pb2

_PERSON = descriptor.Descriptor(
  name='Person',
  full_name='Person',
  filename='person.proto',
  containing_type=None,
  fields=[
    descriptor.FieldDescriptor(
      name='id', full_name='Person.id', index=0,
      number=1, type=4, cpp_type=4, label=2,
      default_value=0,
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
    descriptor.FieldDescriptor(
      name='name', full_name='Person.name', index=1,
      number=2, type=9, cpp_type=9, label=2,
      default_value=unicode("", "utf-8"),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
  ],
  extensions=[
  ],
  nested_types=[],  # TODO(robinson): Implement.
  enum_types=[
  ],
  options=None)

class Person(message.Message):
  __metaclass__ = reflection.GeneratedProtocolMessageType
  DESCRIPTOR = _PERSON

Using this from python code is as simple as:

#person_test.py
import person_pb2
p = person_pb2.Person()
p.name = "Eirik Schwenke"
p.id = 12
p.SerializeToString() #returns: '\x08\x0c\x12\x0eEirik Schwenke'

#We can now write our instance to disk:
df = open('person.bin','w')
df.write(p.SerializeToString())
df.close()

Using the same functions from above, we can also read our data from C:

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

/* Inspired by the manual page for mmap on Debian GNU/Linux (man 2 mmap)
 * Note: errorhandling has been removed in order to save space. */
int main(int argc, char *argv[])
{
  char *addr;
  int fd;
  struct stat sb;
  off_t pa_offset;
  size_t length;
  ssize_t s;

  /* No error handling! Don't try this at home */
  fd = open("person.bin", O_RDONLY);
  fstat(fd, &sb);
  length = sb.st_size;
  pa_offset = 0 & ~(sysconf(_SC_PAGE_SIZE) - 1);
  addr = mmap(NULL, length, PROT_READ, MAP_PRIVATE, fd, pa_offset);

  decode_protobuf(addr, addr+length, yield_cb);
}

If we compile and run this code, we get:

gcc pb.c --std=c99
./a.out 
Field number: 1, Wire type: 0
Field number: 2, Wire type: 2

This isn't very exciting.

The normal mode of working with protcol buffers is to use code generation — and if we generate a simple java class based on our PersonProt.proto we can easily verify that we can read the file generated by python (Note: the java backend needs a seperate name for the message type class (Person), and the protocol (PersonProt) — see the person.proto snippet above. This is why the code below refers to a PersonProt class).

#Generate java code:
protoc -I. --java_out=. person.proto

We can now whip up a simple reader:

// PersonTest.java
import java.io.*;

public class PersonTest
{
  public static void main(String[] args) throws Exception
  {
    PersonProt.Person.Builder p = PersonProt.Person.newBuilder();
    // Read the existing person:
    p.mergeFrom(new FileInputStream("person.bin"));
    System.out.println(p.getId());
    System.out.println(p.getName());
  }
}
/* Compile with (on Debian):
   javac PersonTest.java -cp ".:/usr/share/java/protobuf.jar"

   Run with:
   java -cp ".:/usr/share/java/protobuf.jar" PersonTest
*/

This correctly outputs:

12
Eirik Schwenke

Testing: Text or Binary?

We have not yet looked at how protocol buffers differs from plain text protocols, such as SMTP (Send Mail Transfer Protocol2), and HTTP/1.1 (Hyper Text Transfer Protocol 1.1 4).

The major advantage of the various plain text protocols such as SMTP or HTTP, is the ease by which they may be informally tested, using simple tools such as telnet3. The major downside of these protocol is the difficulty in dealing with various character encodings, and transmission of binary data.

For example, if we want to check that the web server operating on port 80 at www.debian.org, all we need to do, is fire up telnet:

telnet www.debian.org 80
Trying 213.129.232.18&hellip;
Connected to www.debian.org.
Escape character is '^]'.

GET / HTTP/1.1
host: www.debian.org

HTTP/1.1 200 OK
Date: Mon, 01 Mar 2010 19:51:33 GMT
Server: Apache/2.2.9 (Debian) mod_perl/2.0.4 Perl/v5.10.0
Content-Location: index.en.html
Vary: negotiate,accept-language,Accept-Encoding
TCN: choice
Last-Modified: Tue, 23 Feb 2010 11:29:01 GMT
ETag: "2acc83a-36a8-48042db393940"
Accept-Ranges: bytes
Content-Length: 13992
Content-Type: text/html
Content-Language: en

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
     "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">

(The rest of the index page follows)

Or we may try the web server at tools.ietf.org:

telnet tools.ietf.org 80
Trying 194.146.105.14
Connected to tools.ietf.org
Escape character is '^]'.
GET / HTTP/1.1
host: tools.ietf.org

HTTP/1.1 200 OK
Date: Mon, 01 Mar 2010 19:33:39 GMT
Server: Apache/2.2.14 (Debian)
Accept-Ranges: bytes
Transfer-Encoding: chunked
Content-Type: text/html; charset=ISO-8859-1

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html >
(The rest of the index page follows)

Note that we can see that the server has added a redundant space in the opening html-tag — demonstrating the ease with which some parts of a plaintext protocol implementation can be inspected by hand.

The fact that any web browser (a HTTP client) will mostly likely render the page without complaint highlights another important aspect of plain text protocols; one is required to follow the principle of "Be strict in what you send, but lenient about what you accept".

Frequently simply terminating a session due to badly formed input won't do — these protocols have to account for the possibility that an actual human — or a very poorly constructed program — is on the other end of the connection.

Also note that the Content-length field in the first session is difficult to verify by hand (although it is quite easily readable to an unassisted human). It lists the number of bytes in the response (the html-document in this case), and we can't hope to count the bytes without some form of program, like wc (the unix word count program) and a transcript of the session.

So even when the protocol is built using plain text, we would need automated/programmable tools to test the implementation.

Another interesting point is that the documents (RFCs or otherwise) that describe these protocols are almost formal enough that they could be used as a basis for generating a basic client or server for the protocol. That is, the text leave little room for ambiguity, but even where Backus-Naur11 form is used to document the syntax of messages, the semantics is wholly left to formal English — not a language any program generator understands.

Protocol buffers is geared towards program generation, and can therefore readily be used to prototype an implementation, as well as serving as a basis for documenting the protocol.

RPC, HTTP — Binary or Bust?

Ph.D. Fielding's seminal essay on Representional State Transfer (REST6) addresses many of the needs for a protcol that supports distributed document sytems and applications, and in particular the World Wide Web (WWW).

Fielding has been closely involved with the development of HTTP4 and the Apache Web server. It is therfore noteworthy that he has been working towards enhanchments to HTTP that involve, among other things, a binary repsentation of tokens. This work can be seen in his presentation of the Waka5 protocol.

The Apache Jserv protocol (AJP7) is another example of reengineering HTTP with (among other things) a binary encoding, in order to achieve higher point-to-point performance (in this case between a tightly coupled client-facing HTTP proxy, and a (java) application server).

Replacing HTTP for general deployment on the WWW involves a lot more than simply changing the way messages are encoded, however. As outlined in REST6, a key to the success of the Web, is it's reliance of many layers of caching.

While the Waka-proposal suggest a way to upgrade HTTP in a backwards compatible manner — any such undertaking is far from trivial.

In the smaller case of services that primarily are geared towards programs, rather than human users, micro protocols make more sense. This is similar to how the Apache Web server and Apache Tomcat can communicate over AJP, in addition to using HTTP (via eg: mod_proxy15).

Protocol Buffers and Forwards Compatability

Protocol buffers deals with forward compatability by allowing optional and repeated fields to be added to a protocol after initial deployments. Old

implementations will ignore the new fields, while newer versions should assume sane default values.

There is also a special feature called extensions that allow a protocol designer to assign a range of labels (Indexes into messages), for future use — the protocol buffer framwork then exposes extentions in a way that allows the programmer to check for available extensions in a given message.

If we want to add an optional field email to our person message, we can either simply add the field as a new optional field — or we can alter our definition of person to allow for extensions:

// In person.proto
message Person {
   required uint64 id = 1;
   required string name = 2;

   // We could also simply add another optional field
   // like: optional string email = 3; and retain
   // backwards compatability
   extensions 1000 to 2000;
}
option java_outer_classname = "PersonProt";

We then extend the protocol:

// person2.proto
import "person.proto"

extend Person {
  optional string email = 1000;
}

Using the new extention is now as simple as (from python):

import person2_pb2
from person2_pb2 import person_pb2

f = open("person.bin", "r")

p = person_pb2.Person.FromString(f.read())
f.close()
p.name
# Outputs: u'Eirik Schwenke'
p.HasExtension(person2_pb2.email)
# returns: false
p.Extensions[person2_pb2.email] = "eirik.schwenke@student.uib.no"
p.HasExtension(person2_pb2.email)
# Now returns: true, and:
p.Extensions(person2_pb2.email)
# returns
'eirik.schwenke@student.uib.no'

If we had simply added an optional field, we would still have been able to work with the the data — and that might have been appropriate for new core features of our protocol.

If we wanted to add optional messages/values that we saw as remaining optional (we might not assume that every program/service cares wether a person has a (known) email address, for instance) — the extention framework might be more suitable.

Messages Alone do not Make a Protocol

Having a portable, somewhat typesafe way to serialize and marshal data is a great help towards designing interoperable services and clients. But it does not make a full protocol — for that we need some way to define behaviour too.

While it is quite feasible to simply leave the rest to the code, it would be nice to define some interfaces for remote procedure calls (RPCs) as well.

For this type of use, Google protocol buffers8 defines a feature called sevices, that extends GPB to provide an almost complete RPC framework. Why almost complete ? There's actually no implementation available in the framework itself.

Protocol buffers can generate stub-implementations, but leave it up to the programmer to implement the transport part, and to specify authentication or authorization.

One way to achieve authentication would be to wrap an (unauthenticated) protobuf based protocol in an authenticated SSL16 session, or require a VPN — essentially delegating the problem down to the transport layer (from the protobuf point of view).

"Trust all data" isn't a very solid way to engineer security however — the only alternative protobuf provides, is to add the required session handling to the protocol (or a protocol, as protbufs may be encapsulated in other protobuf protocols quite easily).

Remote Procedure Calls

There exists a library to help bootstrap services based on protobuf; Protobuf Socket RPC.

Installing PBS-RPC for python is as easy as:

easy_install protobuf-socket-rpc

With that done, we can create a simple service-definition, using our earlier, extended person message:

// person-rpc.proto
// package protobuf.socketrpc;
import "person.proto";
import "person2.proto";

service PersonRpc {
   // Takes a Person2 message with an optionally defined
   // email-field, and returns a Person with an occourences of an
   // ampersand ("@") replaced by the string "(at)".
   rpc simpleObfuscate(Person) returns(Person);
}

We now need to implement the service-class, and set up a script to run the client and the server:

#PersonRpcImpl.py
import person_rpc_pb2

class PersonRpcImpl(person_rpc_pb2.PersonRpc):
  def simpleObfuscate(self,controller,request,done):
    # Print the request (a Person)
    print request

    # Extract email from the person-message received (if any)
    if (request.HasExtension(person_rpc_pb2.person2_pb2.email)):
      email = request.Extensions[person_rpc_pb2.person2_pb2.email]
      print "Got email: %s" % (email)
      email = email.replace("@","(at)")
    else:
      email = ""
      print "No email set"

    # Create a reply
    response = person_rpc_pb2.person_pb2.Person()
    response.name = request.name
    response.id = request.id
    response.Extensions[person_rpc_pb2.person2_pb2.email] = email

    # We're done, call the run method of the done callback
    done.run(response)

Once the service has been defined, the script for running the server is short and sweet:

#ServerRun.py
import protobuf.server as server
import PersonRpcImpl as impl

# Create and register the service 
service = impl.PersonRpcImpl()
server = server.SocketRpcServer(8090)
server.registerService(service)

# Start the server
print 'Serving on port 8090'
server.run()

And finally the client:

#ClientRun.py
import protobuf
import person_rpc_pb2

port = 8090
host = "127.0.0.1"
r1 = person_rpc_pb2.person_pb2.Person()
r1.name = "Eirik Schwenke"
r1.id = 12
r1.Extensions[person_rpc_pb2.person2_pb2.email] = "eirik@schwenke.info"

r2 = person_rpc_pb2.person_pb2.Person()
r2.name = "DJ Ango"
r2.id = 13

service = protobuf.RpcService(person_rpc_pb2.PersonRpc_Stub, port, host)
print "preforming synchronous rpc, sending:", r1
response = service.simpleObfuscate(r1, timeout=10000)
print "got back:", response
print "preforming synchronous rpc, sending:", r2
response = service.simpleObfuscate(r2, timeout=10000)
print "got back:", response

Run the server, and the client, and we get the following output — on the server side:

python ServerRun.py
Serving on port 8090
id: 12
name: "Eirik Schwenke"
[email]: "eirik@schwenke.info"

Got email: eirik@schwenke.info
id: 13
name: "DJ Ango"

No email set

And on the client side:

python ClientRun.py
preforming synchronous rpc, sending: id: 12
name: "Eirik Schwenke"
[email]: "eirik@schwenke.info"

got back: id: 12
name: "Eirik Schwenke"
[email]: "eirik(at)schwenke.info"

preforming synchronous rpc, sending: id: 13
name: "DJ Ango"

got back: id: 13
name: "DJ Ango"
[email]: ""

Conclusion

Protocol buffers allows one to easily describe data that is to be communicated between processes, either via files, or over the network.

It's chief advantage over XDR, is that some type information is always included; exactly enough to allow for efficient parsing of data. One could achieve similar results with XDR — but it would require the definition of a "base" message type, listing a message type number (or magic constant) of a fixed length, along with a mapping of fields.

It would still require quite a bit of ingenuity with XDR to get the benefit of simple, yet powerful structures like:

 // addressbook.proto
 message Person {
   optional string name = 2;
   repeated string email = 3;
   repeated string phone = 4;

   extensions 1000 to 2000;
 }
 message Addressbook {
   repeated Person listing = 5;
 }

That easily allows one to work with a variable length list of Persons, that may, or may not, have a name, one or more email addresses or phone numbers — in addition to any number of further extensions, such as PNG-images or homepage-urls.

Protocol buffers are not limited to single implementation either, there is for instance the μpb1 project that implements decoding of protocol buffers in C, based on the data along with the .proto-source files — without any code generation.

The main benefit of protocol buffers over XML and XMLRPC is it's simplicity — it does not deal with different character encodings. It also has the potential to be a lot more efficient at handling small amounts of data, or several small records of data.

XML is superior(or more robust) when dealing with document-like objects — but for smaller amounts of data protocol buffers seem a natural fit.

The other big difference with XML and JSON and protocol buffers is ofcourse that webbrowsers can parse both JSON and XML — and both forms therefore work nicely when using various forms of RPC over the web, with the browser as the user interface.

For more "back-end" tasks, like server to server communication, protocol buffers are more attractive. One avoids the complexity of inventing an entire protocol, and at the same time avoid the overhead of JSON or XML processing.

It is quite easy to imagine a simple database query protocol (similar to the many binary interfaces to SQL database systems, such as those used between the MySQL client and MySQL server, or Microsoft Access and Microsoft SQL Server) — that would combine efficiency and portability. Not only portability accross platforms by way of a C library, but accross implementations, with easy access to python or java implementation as the need arose.

As a fullblown alternative to RPC or ICE10, protocol buffers leave a bit to be desired. The service/RPC part of the platform doesn't feel as mature or as well tested as the more fundamental features of data marshalling/serialization and unmarshalling.

Considering the amount of support the platform already has, both within Google, and among third party developers, it would be surprising if protocol buffers will not thrive as an RPC alternative as well — even if there might be no actual benefit to using protocol buffers over ICE at present.

A high degree of visibility in the marketplace (aka hype) coupled with a growing and maturing number of implentations should help protcol buffers succeed. After all, being able to write a semi-reliable, extensible and future proof (if enteriely useless) email obfuscator service in a handful lines of code shows just how protocol buffers can help kickstart client/server projects.

Appendix

Some Technical Notes

This document is authored using Markdown, but will be supplied in HTML, as requested. For information on Markdown, and the version I'm using. please see the Python Markdown Homepage.

In order to generate html-output from the source document, the following commands were used:

sed 's/\.\.\./\&hellip;/g' protocol-buffers.mkd \
| sed -re 's/([ \t\r\n]|^)(-{3})([ \t\r\n]|$)/\1\&mdash;\3/g;' \
> temp.mkd ;\
markdown temp.mkd -x meta -x footnotes -x toc > protocol-buffers.html; \
echo '</body></html>' >> protocol-buffers.html

  1. μpb - Micro Protocol Buffers Joshua Haberman http://wiki.github.com/haberman/upb/ 

  2. SMTP: Simple Mail Transfer Protocol Request for Comments: 5321 http://tools.ietf.org/html/rfc5321 

  3. Telnet Protocol Specification Request for Comments: 854 http://tools.ietf.org/html/rfc854 See also man 1 telnet: BSD General Commands Manual telnet - user interface to the TELNET protocol 

  4. Hypertext Transfer Protocol — HTTP/1.1 Request for Comments: 2616 http://tools.ietf.org/html/rfc2616 

  5. Waka: A replacement for HTTP Roy T. Fielding, Ph.D. 2002 ApacheCon Presentation http://gbiv.com/protocols/waka/200211_fielding_apachecon.ppt 

  6. Architectural Styles and the Design of Network-based Software Architectures Roy T. Fielding, Ph.D. 2000 Ph.D Dissertation http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm 

  7. Apache JServ Protocol http://httpd.apache.org/docs/2.2/mod/mod_proxy_ajp.html 

  8. http://code.google.com/apis/protocolbuffers  

  9. Josh Haberman 100 lines of C that can parse any Protocol Buffer (c) 2008 http://blog.reverberate.org/2008/07/12/100-lines-of-c-that-can-parse-any-protocol-buffer/ 

  10. Internet Communication Engine http://www.zeroc.com/overview.html 

  11. http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form  

  12. Andrew S. Tanenbaum Computer Networks 4th ed. Pearson Education International (c) 2003 page 27 

  13. XDR: External Data Representation Standard Request for Comments: 1014 http://tools.ietf.org/html/rfc1014 

  14. XDR: External Data Representation Standard Request for Comments: 4506 http://tools.ietf.org/html/rfc4506 

  15. Apache Module mod_proxy http://httpd.apache.org/docs/2.2/mod/mod_proxy.html 

  16. Secure Socket Layer / Transport Layer Security http://en.wikipedia.org/wiki/Transport_Layer_Security