Introduction to MessagePack-RPC

The official MessagePack blog is here! Today I want to introduce recent progress at the MessagePack project. It’s a extremely efficient object serialization library like JSON, but very fast and small.

Why MessagePack?

There’re some projects in the same area. Google’s protocol buffer, Facebook’s thrift, Apache avro, etc. Why another one?

Two important points. For speed, and tight integration with each language.

(1) Speed

For speed, MessagePack has zero copy serialization + streaming deserialization. Please look at Introduction to MessagePack. We also compared with other libraries in the view point of speed and size. Please look at here.

(2) Tight Integration with the language

Actually “MessagePack” itself is the name of the format, like JSON. And many libraries handling it were written by many languages. Currently, C, C++, Java, Ruby, Python, Perl, PHP, Haskell, Lua, D implementations are available.

Some implementations provide the easy packaging. For example, Ruby version is provided as Ruby gem.

$ gem install msgpack

Same for other languages (Maven2 for Java, PyPI for Python, CPAN for Perl, Hackage for Haskell).

The library API is basically the same across languages but provides the convenient way for the language. Here’s the deselialization with for loop in Python version. Really cool!

>>> unpacker = msgpack.Unpacker()
>>> buf = b'¥x93¥x01¥x02¥x03' * 5
>>> len(buf)
20
>>> unpacker.feed(buf[:9])
>>> for o in unpacker:
...     print o
...
[1, 2, 3]
[1, 2, 3]
>>> unpacker.feed(buf[9:])
>>> for o in unpacker:
...     print o
...
[1, 2, 3]
[1, 2, 3]
[1, 2, 3]

MessagePack-RPC

The next move of our project is developing MessagePack-RPC, RPC library using MessagePack for object serialization. It’s fast, and achieves cross language object exchange.

The repository is available here.

Currently, only C++/Java/Ruby implementations are available. The nice thing is that, each implementation depends on nice event loop libraries for the languages. Ruby version uses Rev, C++ version uses mpio, Java version uses JBoss netty.

And also, please remember MessagePack supports “Streaming Deserializer”. When you got the data from the network, the server/client are able to deserialize in a streaming way.

By using them, MessagePack-RPC provides high-performance RPC. And also provide asynchronous call at the client-side.

Let’s look at the example. Most users use script language at the client-side, and C++/Java program at the server side.

Ruby client

It’s provided as ruby gem. Please install with “gem install msgpack-rpc”. Following is the example of calling “hello” function with some arguments.

require 'rubygems'
gem 'msgpack'
gem 'msgpack-rpc'
require 'msgpack'
require 'msgpack/rpc'

begin
  cli = MessagePack::RPC::Client.new("127.0.0.1", 1985)
  cli.timeout = 5
  v = cli.call(:hello0)
  v = cli.call(:hello1, 1)
  v = cli.call(:hello2, 1, 2)
  cli.close
rescue MessagePack::RPC::TimeoutError
  p $!
end

It also supports asynchronous call. It’s really useful when you call multiple services at the same time.

client = new Client(...);
future = client.send(...) # asynchronous call returns
future.join() # waiting the completion
result = future.result();

futures = []
futures << client.send(...)
futures << client.send(...)
futures.each { |f| f.join() }

Of course, you can use custom data structure. Example code is here.

Java server

The following code implements high-performance Java server. The Java version uses netty, Java NIO Socket Framework by JBoss community. netty provides multithread + event model. It’s really FAST, and extensible.

package org.msgpack.rpc;

import java.io.IOException;

import org.msgpack.rpc.client.Client;
import org.msgpack.rpc.server.*;

public class App 
{
    public int hello0() {
    	System.out.println("hello0");
        return 0;
    }
    public int hello1(int a) {
    	System.out.println("hello1");
        return 1;
    }
    public int hello2(int a, int b) {
    	System.out.println("hello2");
        return 2;
    }
    
    public static void main( String[] args )
    {
        Server s = new TCPServer();
        try {
        	System.out.println("listen!");
        	s.listen(1985, new App());
        } catch (IOException e) {
        	e.printStackTrace();
        }
    }
}

Difference between MessagePack-RPC and Thrift

Way faster. The following graph shows the performance difference between C++ MessagePack-RPC v.s C++ Thrift v.s. Sun-RPC.

Tight integration with each language.

Release management per language bindings, not by whole project. Formerly, I submitted to the bug fix patch to the PHP binding. But Thrift release cycle is decided by all language bindings. Then, they couldn’t release as soon as possible.

We’re developing actively. Many languages don’t support RPC yet, but ongoing.

And also note that, we’re now developing Thrift compatible IDL (Interface Definition Language) compiler for MessagePack. If you use the Thrift, the move to MessagePack-RPC will be really easy :-)

Question

If you have any question, please contact with @frsyuki or @kzk_mover with #msgpack hash tag, or #msgpack on freenode.net, or project mailing list.

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Yahoo BuzzAdd to Newsvine Like This!

About these ads

About kazukiohta

My Twitter account is @kzk_mover.
This entry was posted in MessagePack-RPC. Bookmark the permalink.

9 Responses to Introduction to MessagePack-RPC

  1. Pingback: Tweets that mention Introduction to MessagePack-RPC | MessagePack Blog -- Topsy.com

  2. bill says:

    This may seem like a silly question, but is there any reason why Javascript is also not a targeted language?

  3. Pingback: MessagePack-RPC for Java/Pythonを実装しました。 - moratorium

  4. Pingback: Streaming Serialization/Deserialization with MessagePack | MessagePack Blog

  5. Navigateur says:

    Will it also support C#, other .NET languages, Cocoa, ActionScript (for e.g. Adobe Air apps)? I need to support as many platforms as imaginable, because I’m making a connector for 3rd-party apps. Also, can anybody become a MessagePack contributor? YOU GUYS ARE AWESOME – Many thanks, N.

  6. Pingback: Tomato Blog

  7. Hi there,

    We are trying to develop an application which will use c# for client, the client wil need to communicate with server so we need an rpc to do that,

    My question is : will there be an rpc implementation of c# for Msgpack? if so, when will it be published?

    Thank you for the concern,

  8. Pingback: Работа с MessagePack и MessagePack RPC в Haskell | Записки программиста

  9. Gloqq says:

    Hi, the last in section “(1) Speed” (here == http://msgpack.sourceforge.net/benchmark) is broken.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s