Introduction to MessagePack-RPC

The official MessagePack blog is here! Today I want to introduce recent progress at the MessagePack project. It’s a extremely efficient object serialization library like JSON, but very fast and small.

Why MessagePack?

There’re some projects in the same area. Google’s protocol buffer, Facebook’s thrift, Apache avro, etc. Why another one?

Two important points. For speed, and tight integration with each language.

(1) Speed

For speed, MessagePack has zero copy serialization + streaming deserialization. Please look at Introduction to MessagePack. We also compared with other libraries in the view point of speed and size. Please look at here.

(2) Tight Integration with the language

Actually “MessagePack” itself is the name of the format, like JSON. And many libraries handling it were written by many languages. Currently, C, C++, Java, Ruby, Python, Perl, PHP, Haskell, Lua, D implementations are available.

Some implementations provide the easy packaging. For example, Ruby version is provided as Ruby gem.

$ gem install msgpack

Same for other languages (Maven2 for Java, PyPI for Python, CPAN for Perl, Hackage for Haskell).

The library API is basically the same across languages but provides the convenient way for the language. Here’s the deselialization with for loop in Python version. Really cool!

>>> unpacker = msgpack.Unpacker()
>>> buf = b'¥x93¥x01¥x02¥x03' * 5
>>> len(buf)
20
>>> unpacker.feed(buf[:9])
>>> for o in unpacker:
...     print o
...
[1, 2, 3]
[1, 2, 3]
>>> unpacker.feed(buf[9:])
>>> for o in unpacker:
...     print o
...
[1, 2, 3]
[1, 2, 3]
[1, 2, 3]

MessagePack-RPC

The next move of our project is developing MessagePack-RPC, RPC library using MessagePack for object serialization. It’s fast, and achieves cross language object exchange.

The repository is available here.

• github:msgpack-rpc

Currently, only C++/Java/Ruby implementations are available. The nice thing is that, each implementation depends on nice event loop libraries for the languages. Ruby version uses Rev, C++ version uses mpio, Java version uses JBoss netty.

And also, please remember MessagePack supports “Streaming Deserializer”. When you got the data from the network, the server/client are able to deserialize in a streaming way.

By using them, MessagePack-RPC provides high-performance RPC. And also provide asynchronous call at the client-side.

Let’s look at the example. Most users use script language at the client-side, and C++/Java program at the server side.

Ruby client

It’s provided as ruby gem. Please install with “gem install msgpack-rpc”. Following is the example of calling “hello” function with some arguments.

require 'rubygems'
gem 'msgpack'
gem 'msgpack-rpc'
require 'msgpack'
require 'msgpack/rpc'

begin
  cli = MessagePack::RPC::Client.new("127.0.0.1", 1985)
  cli.timeout = 5
  v = cli.call(:hello0)
  v = cli.call(:hello1, 1)
  v = cli.call(:hello2, 1, 2)
  cli.close
rescue MessagePack::RPC::TimeoutError
  p $!
end

It also supports asynchronous call. It’s really useful when you call multiple services at the same time.

client = new Client(...);
future = client.send(...) # asynchronous call returns
future.join() # waiting the completion
result = future.result();

futures = []
futures << client.send(...)
futures << client.send(...)
futures.each { |f| f.join() }

Of course, you can use custom data structure. Example code is here.

Java server

The following code implements high-performance Java server. The Java version uses netty, Java NIO Socket Framework by JBoss community. netty provides multithread + event model. It’s really FAST, and extensible.

package org.msgpack.rpc;

import java.io.IOException;

import org.msgpack.rpc.client.Client;
import org.msgpack.rpc.server.*;

public class App 
{
    public int hello0() {
    	System.out.println("hello0");
        return 0;
    }
    public int hello1(int a) {
    	System.out.println("hello1");
        return 1;
    }
    public int hello2(int a, int b) {
    	System.out.println("hello2");
        return 2;
    }
    
    public static void main( String[] args )
    {
        Server s = new TCPServer();
        try {
        	System.out.println("listen!");
        	s.listen(1985, new App());
        } catch (IOException e) {
        	e.printStackTrace();
        }
    }
}

Difference between MessagePack-RPC and Thrift

Way faster. The following graph shows the performance difference between C++ MessagePack-RPC v.s C++ Thrift v.s. Sun-RPC.

Tight integration with each language.

Release management per language bindings, not by whole project. Formerly, I submitted to the bug fix patch to the PHP binding. But Thrift release cycle is decided by all language bindings. Then, they couldn’t release as soon as possible.

We’re developing actively. Many languages don’t support RPC yet, but ongoing.

And also note that, we’re now developing Thrift compatible IDL (Interface Definition Language) compiler for MessagePack. If you use the Thrift, the move to MessagePack-RPC will be really easy

Question

If you have any question, please contact with @frsyuki or @kzk_mover with #msgpack hash tag, or #msgpack on freenode.net, or project mailing list.

About these ads