A library and Demo application for Kiva oAuth

8 Mar

This week Kiva Engineering team did a beta release of its protected API set. Through these APIs applications can fetch more in depth information about a specific lender. This ability in turn will help consuming applications present more accurate and meaningful analysis. Coincidentally this was exactly the API set I was waiting for. On my KivaSearch tool I was in the middle of implementing some ‘Me’ filters. What ‘Me’ filters were to do was to let the users choose search criteria like ‘Find loans that match my previous lending habits’, ‘Find Loans that are to countries other than those I have lent to’, ‘Find loans from my preferred MFIs’, etc. As you can imagine the pivotal data for these criteria is “Who is ‘I’”?. In the absence of the protected APIs and its oAuth implementation all I could do was to help the user find her own lender_id. This was error prone, easy to spoof and most importantly frustrating. With the oAuth implementation available from Kiva, all my user needs to do is to click on a ‘Login’ link and login to Kiva using her regular credentials. Just as in any oAuth implementation, she should then authorize KivaSearch to access some information on her behalf. Very easy and straight forward.

As the first step to implementing this feature I put together a wrapper NodeJS library for Kiva oAuth APIs. I am only following the route to allow a web application to authorize it with Kiva. For devices it might be a slightly different route. If that is what you need please feel free to fork the code.To demonstrate the usefulness of this library I also have written a demo application and it can be accessed online at KivaOauth Demo. The demo lets you login to kiva and and fetch back and display your full name and current account balance. You need a Kiva account to use it anywhere beyond the landing page. So, if you don’t have a Kiva account yet this is a good time to get one as well :) .

  • To use this library these are the steps.Decide on the access rights your application needs on behalf of the user. Currently the access rights are the following

“access” ==> Basic information about the user account, Name, lender id
“user_balance” ==> Current account balance at Kiva in USD.
“user_email” ==> User’s registered email.
“user_anon_lender_data” ==> Ability to fetch loan information for this lender even for the loans she lent anonymously.
“user_loan_balances” ==> Current payback status and balances of each of the user’s loans.

exports.scopes = [
    "access",
    "user_balance",
    "user_email",
    "user_anon_lender_data",
    "user_loan_balances"
  ];

myapp.js

  • Define two HTTP routes for your application
  1.  authorize url (eg. /auth/kiva) : This is where the user lands when he is starting the authorization process. From this point user will be redirected to Kiva and on successful authorization she will be redirected back to your applications ‘callback’ url, which is..
  2. callback url (eg. /auth/kiva/callback) When the user lands here the response will include sufficient information for you know the user and the rights she has granted for your application. As a responsible developer on this step you should securely save this information and redirect user to the resource you wish her to access.
var kivaoAuth = new (require('./kivaoAuth.js')).$();
app.get('/auth/kiva', function(req,resp){
	if(!isAccessSet(req)){
		kivaoAuth.getOAuthRequestToken(req,resp);
	}
	else{
		resp.sendfile(__dirname  + '/public/checkstuff.html')
	}

});
app.get('/auth/kiva/callback',
	function (req, resp){
		kivaoAuth.getOAuthAccessToken(req,resp,function(data){
			var id = data.user_account.lender_id;
			access_tokens[id] = data;
			var crypted = kivacrypt.encrypt(id);
			resp.cookie('_appversion55667654', crypted, {maxAge: 60 * 60 * 24 * 365 * 1000, httpOnly: true});
			resp.sendfile(__dirname  + '/public/checkstuff.html')
		});
	}
);

snippet from app.js

In my callback execution for the demo application saves  ’access_token’ in an application level data structure  I also push an encrypted cookie into the client end with lender_id information. I am pushing cookie to the client to know the user on return. I am encrypting it to make sure that it is not tampered with especially since I am setting a very large expiration window for the cookie. Saving ‘access_token’ to the local store will help me fetch back user specific ‘access_token’ when she returns.

  • Register your application along with a valid callback url.

Remember that it is not Kiva Server, but your client browser doing the ‘callback’. So the callback url need not be a public one. You can make it look like a public one easily by modifying your hosts file. I have a small description on it within the README. For eg. My Callback url looks like http://test.bonigopalan.com:3000/auth/kiva/callback , it exists only on my localhost and test.bonigopalan.com points to 127.0.0.1.

kiva-app-account

  • Input your application’s identity information into myapp.js
exports.appid = "com.bonigopalan.test";
exports.clientid="com.bonigopalan.test";
exports.clientsecret = "pxIvokqHsvogHdvjDCArwyqvdxrMEkrC";

exports.scopes = [
    "access",
    "user_balance",
    "user_email",
    "user_anon_lender_data",
    "user_loan_balances"
  ];

exports.callbackURL = "http://test.bonigopalan.com:3000/auth/kiva/callback";

  • Create a file named cryptkey.js as per the provided template cryptkey.template.js

You are good to go now. All basic scaffolding is done.

kivacrypt.js is my rudimentary encryption library. You are encouraged to modify it to match your paranoia level. I ain’t checking in my cryptkey.js either. You have to create that file manually using cryptkey.template.js. See, I know a thing or two about paranoia. At least a few friends from my past are sure to chuckle here.

Within the demo application access tokens are managed in a very rudimentary way. It is only representing a persistent data store. You should decide on either an encrypted database table or a redis cache for keeping this safe but easily accessible.

var access_tokens = [];
function isAccessSet(req){
if(!getUser(req)){
return false;
}
return true;
}

function getUser(req){
var key = req.cookies._appversion55667654;
if(!key) return null;
return access_tokens[kivacrypt.decrypt(key)];
}

function deleteUser(req){
var key = req.cookies._appversion55667654;
if(!key) return;
delete(access_tokens[kivacrypt.decrypt(key)]);
}

Snippet from app.js

Now, you are all set to use some protected APIs on Kiva end. For the demo purposes I am using the one to find current user balance.

app.get('/balance',function(req,resp){
		if(!isAccessSet(req)){
			resp.send("Not authorized",401);
			return;
		}
		kivaoAuth.getBalance(getUser(req).access,function(data){
			//console.log(data);
			resp.send(data);
		});
})

The corresponding client call is from checkstuff.html and it looks something like this. It uses Jquery to make the AJAX call and do STUFF.

    	function checkBalance(){
            $.ajax({
              url: "/balance",
              processData: false,
              contentType: "application/json",
              method:"GET",
              success:function(data) {
              	$('#kiva_balance').text(JSON.parse(data).user_balance.balance);
              	$('#balancemodal').modal('show');

              },
              error:function(error) {
              	$(location).attr('href','/');
              }
            });
    	}

Action behind the scenes

Plenty of work happens behind the scenes, wrapped cozily within kivaoAuth.js. It uses values from myapp.js to identify application credentials. It then uses a great node module, node-oauth to first request a ‘request_token’. The received request_token is short lived. So it uses the request_token and exchanges it with kiva for a long lived ‘access_token’. At this stage your application is authorized at Kiva to fetch data using credentials in ‘access_token’ (along with access_secret). When you query kiva using an access_token, specific to an user, to kiva your application represents the user. Your token is valid till the time user revokes her decision to allow your application rights to her information at Kiva.

You can experiment will all these features at the demo app. But first you need a Kiva account. Go for it!

The code discussed here is here on Github as well.

How fast is SPDY Anyways?

2 Mar

This post summarizes the results of my initial experimentation with SPDY protocol.  In summary the results did not show any dramatic improvement in speeding up loading of my test web page.  My test scenario was with the server on my server grade laptop (NodeJS server, SPDY over SSL)  in Pune, India and the client Google Chrome browser running on a machine in Sunnyvale, CA, USA.  The test page loaded the same 1million pixel image side by side, one side from a SPDY server and the other side from a NON-SPDY server.  Everything except for the transport was equivalent for both servers.  Test results are visual.  So here you go, my test screen capture video (all thanks to VLC Media Player.)

The test showed only marginal increase in speed for SPDY based fetch.  Now there are some factors that would make this test con conclusive.  Foremost of which is that it at the most only shows that NodeJS’s SPDY wrapping is not high performing.  There are other servers (Notably HTTPD)  which supports SPDY and ‘might’ give better results.  I will test it and report back.  On extremely slow connections however there was dramatic speed gains with SPDY with it performing three times as fast as plain HTTP connection.  But with a bandwidth as little as 1Mbps, the difference in speed gains were to the magnitude of 10%.

SPDY (pronounced SP-EE-DY) has promise of speeding up our internet experience.  It is a protocol actively developed and promoted by Google over the last few years as part of chromium.  It has already found serious early adopters in heavy weights such as facebook, twitter and wordpress.  Gmail and various applications from Google are also served over SPDY whenever possible.   If you are using any of these websites on Chrome or on recent versions of Firefox you are already accessing data over SPDY.  On the server side Apchae HTTPD, Tomcat, Jetty, Python, Ruby and NodeJS has implementations that can stream data over SPDY as an alternative.  Libraries in almost all popular languages and platforms are also out there – such as JAva Client., iOS client, C,  Android Client and the usual suspects such as PHP, Ruby and JavaScript.  The observed speedup is in the tune of 60% when compared with HTTP.  This is huge.  The basic premise of implementation strategy is that by using HTTP a lot of time is wasted in Round Trips between client and Server.  While some round trips cannot be avoided, SPDY gives developers a chance to minimize the impact of these round trips on end user experience.  This minimization is ensured through some clever practical solutions.

  1. Ability to PUSH content preemptively to client.  For example while loading index.html that has four <IMG> tags, HTTP will first load and parse the HTML and then make four separate server requests to fetch back the four <IMG> images.  Now, if the server can start pushing the image content right when index.html is requested, and that through the more efficient protocol such as SPDY, then over all load time of index.html should reduce considerably.
  2. Ability to HINT to the client to request for certain resources before an HTML page mandates its need.  This is a more conservative take on point 1.
  3. Provision to set the frame sizes for data pushes to client.  This is great since now depending on the bandwidth server can decide on sending data across in as few pushes as possible.
  4. Multiplexed Streams : SPDY does not restrictions on number of parallel HTTP requests to the servers.  In Asynchronous web this make a lot of sense.  Even on the most modern web browsers today the number of concurrent server connections is capped at 6.  Not too good.
  5. SPDY uses TCP more efficiently.

The server I am using for this test is once again written in JavaScript and is running with NodeJS (v0.8.16) and Node-SPDY (1.4.5).  It uses a 1 million pixel image generated using a simple Java code.  Running it locally might not be most ideal.  If you have access to geographically separated machines please feel free to deploy test and report back.  Source code is published here.

Code to create large test image.

	public void creareLargeJpeg() throws Exception{
		int width =  1000;
		int height = 1000;

		BufferedImage bi = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
		Graphics g = bi.getGraphics();
		Random r = new Random(System.currentTimeMillis());
		for (int x = 0 ; x < width; x+=10){
			for (int y = 0 ; y < height; y+=10){
				Color c = new Color(r.nextInt(255),r.nextInt(255),r.nextInt(255));
				g.setColor(c);
				g.fillRect(x, y, 10, 10);
			}
		}
		g.dispose();
		File output = new File("target/output.jpg");
		ImageIO.write(bi, "JPEG", new FileOutputStream(output));
	}

Code for the SPDY and plain HTTP server (server.js)

var express = require('express');
var app = express();
var app1 = express();
var spdy = require('spdy'),
	fs = require('fs');

app.use(express.static(__dirname + '/public'));
app.listen(3000);

var options = {
  key: fs.readFileSync('keys/spdy-key.pem'),
  cert: fs.readFileSync('keys/spdy-cert.pem'),
  ca: fs.readFileSync('keys/spdy-csr.pem'),
  windowSize: 1024, // Server's window size
};

app1.use(express.static(__dirname + '/public'));
var server = spdy.createServer(options, app1);
server.listen(3333);

console.log("HTTP Server at PORT: 3000 and SPDY Server on 3333");

The client HTML Code (index.html)

<head>
</head>
<body>
	<table>
		<tr>
			<th>WITH SPDY</th>
			<th>NO SPDY</th>
		</tr>
		<tr>
			<td>
				<img src="https://loclhost:3333/images/galaxy.jpg" width="500"/>
			</td>
			<td>
				<img src="http://localhost:3000/images/galaxy.jpg" width="500"/>
			</td>
		</tr>
	</table>
</body>

Revelation.js : 30days Progress Report

18 Feb

A month back I set out on a goal of restarting writing JavaScript.  I am still hanging in there learning more.  The first  application that came out of this goal chain is up and running without any bugs – actively used by around 120 users from around the world (as of yesterday).  A few loans have trickled out through my application, through Kiva into different parts of the world – not counting the 14 high quality loans I identified using the same.  There are a few feature requests as well as shout outs with offers of help with coding.  Healthy signs and I am very proud of what I could learn and accomplish in this time.

The however code is bulging!  While server side code is very organized, the single page client side application is already pushing 1000 lines.  It is not good.  So I am going to pull a code cleanup sprint to refactor the client part into Jade templates.  Jade is a great template engine that takes away verbosity of HTML.   This will also help me logically split the application into different files while keeping the overall theme of single page application intact.  Something more to learn this week then!  Nice!

Blasphemy on Browser aka Full NodeJS on Browser

15 Feb

https://github.com/devasur/node_webkit_tutorials

Today let us talk about blasphemy. Direct data inserts into databases, listening for messages on message queues, Streaming data between random sockets over custom protocols. None of this is blasphemy. However coded in an HTML page and run on a browser it is nothing less than the demonstration of darkest of dark arts. And I endorse it.

Few days back my colleague and I were discussing the direction client side computing should take. While we did not agree on everything, we had a general consensus that there is a lot more that needs to be done on the client side considering the computing power each device carries. Differing computations back to another machine should only be a last resort – a well deserved bonus for an application. This aligns very well in enterprise products. Here users don’t really care whether application is cross browser compatible in all aspects or whether it is not breaking any browser imposed security bullshit. The application is developed in a controlled environment, there is no rogue code and users want most bang for their buck. From this argument base ditching browser based applications and moving to native, compiled executable seemed the right path to take. However in the engineering world more people are choosing to master only web technologies. So it is a Catch-22 situation when more complex code gets written in native technologies targeting a particular OS – the maintenance will become a nightmare. Moreover, the visible trends from Microsoft and Google with their OS strategies is to use HTML5 predominantly for its own UI rendering. Those signals should not be ignored. Also HTML5 can create some of the most beautiful and responsive user interfaces. This is a plus. But the clincher remains that only so much can be done from the browser!

My devious mind suggested – “Why not we fork chromium and mod it to provide a bunch of hooks to do everything we want?”

The silence ensued was overwhelming. It was not the first time my devious mind had suggested things that sounded simple and straight forward but had proved to be an utter PITA to implement. But in theory it sounded like a possible solution. So we agreed to do some hacking on that front.

The first task of any complex task is to ask Google what it thinks of it. And voilà! It answered in the affirmative that there are other radicals out there who not only are preaching but putting some productive work into such solutions. I was in luck! One project stood out as the perfect match for my quest. So I decided to zero in on it.

Roger Wang’s Node-Webkit :https://github.com/rogerwang/node-webkit

What this gentleman has done is remarkable. He recognized that both node on the server side and chromium on the browser front are using the same Google V8 JavaScript engine to do all the heavy lifting. So, it is not very difficult to make nodejs do its heavy work on a slightly modified chromium. What it means is that everything that you can do from nodejs on the server side, now you can do on the client side. At the same time you have a full access to the browser DOM as well. Hence you can render UI right there where nodejs is executing. That is a great hacker mind.

And what can nodejs do? As of now a lot of things.

- Threads [Check]
- Socket connections to ports talking stuff other than HTTP [Check]
- Database connectivity [Check]
- Communing cating with message Queues [Check]
- Image Processing [Check]

You should checkout the available modules at https://npmjs.org/. All those are easily usable from the browser. Some of those are irrelevant. For example ‘express’ – a great tool to setup a web framework for server side request processing but quite unlikely that you would want to run that on the browser. But if you ever want to, you know you can!

Now with the theory in check I wanted to verify that it actually worked.

Problem 1:

From the browser (don’t cringe I know I am using that term quite loosely here) connect to an image server (written in java) running on my local box and fetch back a series of PNG images and render it on a grid. The clincher here of course is that my server does not talk HTTP. It streams plain png image bytes. Not a problem said node. Let us see.

First I wrote the image Server. That I thought would be the easy part. It is simple server that replies back to any client connection with a random generated image. It is well behaved. You should be able to compile it easily with ‘javac’ alone without any external dependencies.

javac src\org\entelligentsia\image\*.java -d .\bin
java -cp .\bin\  org.entelligentsia.image.Server
Server started and listening on PORT:4444
^C
Initiating orderly shutdown
No pending requests in queue.  Closing socket
Server socket is closed.  Good Bye!
Server Socket is no more accepting connections :socket closed
package org.entelligensia.image;

import java.awt.Graphics2D;
import java.awt.image.BufferedImage;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.UUID;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import javax.imageio.ImageIO;

public class Server {
	ServerSocket serverSocket = null;
	private static final ExecutorService clientThreadExecutors = Executors.newCachedThreadPool();
	private static final ExecutorService serverThreadExecutor = Executors.newSingleThreadExecutor();

	public Server(final int port) throws Exception{
		serverThreadExecutor.submit(new Runnable(){
			public void run() {
				try {
					serverSocket = new ServerSocket(port);
					System.out.println("Server started and listening on PORT:" + port);
					while(!serverSocket.isClosed()){
						try {
							Socket clientSocket = serverSocket.accept();
							System.out.println("New Client connection");
							clientThreadExecutors.submit(new ImageSpitter(clientSocket));
						} catch (java.net.SocketException e) {
							System.out.println("Server Socket is no more accepting connections :" + e.getMessage());
						}
					}
				} catch (IOException e) {
					e.printStackTrace();
				}
			}
		});
	}

	public void shutdown() throws Exception{
		System.out.println("Initiating orderly shutdown");
		clientThreadExecutors.shutdown();
		while(!clientThreadExecutors.isShutdown()){
			System.out.println("Waiting for pending tasks to finish");
		}
		System.out.println("No pending requests in queue.  Closing socket");
		serverSocket.close();
		System.out.println("Server socket is closed.  Good Bye!");
		serverThreadExecutor.shutdown();

	}

	class ImageSpitter implements Runnable{
		Socket clientSocket = null;
		public ImageSpitter(Socket socket){
			this.clientSocket = socket;
		}
		public void run() {
			try {
				OutputStream os = clientSocket.getOutputStream();
				BufferedImage bi = getRandomImage();
				ByteArrayOutputStream bos = new ByteArrayOutputStream();
				ImageIO.write(bi, "PNG", bos);
				byte[] image = bos.toByteArray();
				System.out.println("Writing " + image.length + " bytes to output");
				os.write(image);
				os.flush();
				os.close();
			} catch (IOException e) {
				e.printStackTrace();
			}
		}

		private BufferedImage getRandomImage(){
			BufferedImage off_Image = new BufferedImage(250, 20,BufferedImage.TYPE_INT_RGB);
			Graphics2D g2 = off_Image.createGraphics();
			g2.drawString(UUID.randomUUID().toString(), 10, 15);
			return off_Image;
		}
	}

	public static void main(String[] args) throws Exception{
		final Server s = new Server(4444);
		Runtime.getRuntime().addShutdownHook(new Thread(){
			@Override
			public void run() {
				try {
					s.shutdown();
				} catch (Exception e) {
					System.out.println("Unable to shutdown server:" + e.getMessage());
				}
			}
		});
		while(true){
		}
	}
}

Now the exciting client part. Here I needed something that can make a socket connection to localhost:4444, and then read the returned byte[] into a local variable. This is my image. I also needed this little something to have the ability to put it as an image on the browser UI.

<!DOCTYPE html>
<html>
<head>
    <title>Image fetcher</title>
</head>
<body>
    <script type="text/javascript">
      var net = require('net');
      var images = []
      for (var i =0 ; i < 5 ; i++){
        var socket = net.createConnection(4444,"localhost");
        socket.on('data',function(data){
            document.write('<img alt="Embedded Image" src="data:image/png;base64,' + data.toString('base64') + '" /> <br>');
        });
        socket.on('error',function(err){
          console.log(err);
        })
      }
    </script>
</body >
</html>

That is it. Here I am using the nodejs base library ‘net’. It has ‘createConnection’ that connects to a ‘HOST’ at a ‘PORT’. It can also send data in but my image server does not need that. As soon it is connected in it starts streaming back an image. Now I connected a listener for ‘data’ event on the ‘socket’. Data comes back as a ‘Buffer’, a native JavaScript object in nodejs world. Converting it into a base64 encoded image data stream was easy once again with a call to ‘Buffer.toString(‘base64′) ‘. Then I am using the omnipresent ‘document.write()’ to put the image on the browser.

To run this code you need node-webkit binary for your platform. Currently it comes in Win32, Linux and MacOS flavors. I could test it only on a Windows 7 (64bit) machine as that was the machine I was using today. Basically you pass either a zip of your client code or the directory containing your client code itself to nw executable.

nw socket_image_fetch

And it should open up your nw browser, which looks and feel just the normal chromium, and assuming the image server is running will show you 5 random images pasted one below the other.

demo

Kiva Search got hosted

5 Feb

Few clicks and some typing later, Kiva Search went into public beta on AppFog (Nodester) cloud. Very cool!

http://kivasearch.aws.af.cm/ , Currently supported on IE10>, Chrome and Safari.

Kiva Search : NodeJS, MongoDB, JQuery, handlebarjs and a bit of Bootstrap.

5 Feb

Screenshot from 2013-02-05 00:06:21    Screenshot_2013-02-05-09-52-53

https://github.com/devasur/kivasearch

Currently Hosted @ Kiva Search on Amazon Cloud

Just finished writing my full blown JavaScript search tool for Kiva and I was too excited to keep it a secret anymore :) .  I have been an involved kiva lender for many years now.  For the last couple of years I have been using the nifty tool kivalens to search and locate suitable loans to lend to.  It has worked great for me – extremely satisfied.  The only challenge had been that it is Silverlight based.  So does not work easily on Linux machines.  I took kivalens’s  functionality as a base to learn NodeJS by converting it into a pure HTML5 single page client with a NodeJS backend.  It currently searches using almost all the filtering options the mother application has.  It still lacks some features but those are not really important to me.  But if time permits I would love to make it full fledged and publish open sourced.

The application works like this.  It uses the public kiva RESTful APIs to get newest loans from Kiva in JSON format.  To it it adds in Field Partner information.  Some of the field partner information is not available through public APIs.  So it uses server side webpage scrapping to get this additional information.  It saves it all into a database.  Couple of background jobs keeps this information updated.  These jobs also purged fully funded loan information from the database so that its size is in check.

The client UI is fairly simple.  It has some draggable sliders to mark min-max for some key lending parameters.  When the parameters are changed matching loans are fetched and rendered as a list.  User can click through to complete lending.  It is simple and it works.

The serve side is NodeJS.  I have used ‘express‘ as a web framework to manage the listening thread.  MongoJS is used to connect and manage data stored in MongoDB.  Three setTimeout() background jobs synchronizes Kiva datasets with local database.   For webpage scrapping to collect additional partner information for more search options I used node-htmlparser and node-soupselect two brilliant node modules.

For the client side I went with Twitter Bootstrap for basic scaffolding.  JQuery is used for managing various AJAX calls.  Handlebarjs is used as the templating framework for rendering loan data.

One great side benefit of this learning was that I got introduced to a super editor called Sublime Text.  It supports a huge number of programming languages and has a very flexible extension system to enhance its features.  Writing NodeJS JavaScript on this IDE was like silk.  Very impressed.

For a JavaScript expert this would be child’s play.  But for me, someone who 90% time writes no-ui server code in Java, this whole project was a small scale Mt. Everest.  I learnt so much of JavaScript and about the amazing collaborative development that is happening on the NodeJS front,  The simplicity of NodeJS and the completeness of its support system in terms of all necessary tools and libraries is just amazing.  I will continue to learn NodeJS by improving on what I have now.

An ambitious product roadmap for the next 30 days.

  • Ability to save and retrieve search criteria based on some user credentials.
  • A notification system that can push loan availability information to user from the backend even if user is not using the client.
  • Repackage the client JS as a PhoneGap project for iOS and Android.
  • More search criteria options.
  • More visualization options for the loan and portfolio demographics.

Exciting times ahead!

Messaging between Java and Node.js through RabbitMQ

22 Jan

The simplicity of the whole integration blew me away.  Under load the program performed marvelously.  If it is load enough for you, you can decide for yourself.  I wrote the same code twice today.  First on a Ubuntu 12.04 machine and later on a Windows 7 machine.  All components running locally.  Code – identical.  Setup – identical.

Problem:

Send a million Strings from a java program running in its own process to a Node.js program written in Java Script.

Step 1:

Install and configure RabbitMQ server.  This is straight forward.  Documentation is clear on this at http://www.rabbitmq.com/configure.html.  I did not set any specific configuration parameters other than defining ERLANG_HOME as a system variable in Windows 7 machine.  On the Linux environment apt-get install took care of configuring Erlang correctly.  If you are wondering why Erlang is required – Rabbit MQ needs Erlang runtime for execution.  It also needs Python for running its very useful web console.

Once installed and configured you will be able to navigate to http://localhost:15672 to access the management console.  http://www.rabbitmq.com/management.html

Start RabbitMQ.

Install Node.js if you do not have it installed.

Java Program

I used the latest AMQP client dependency through maven to get things going.

<dependency>
    <groupId>com.rabbitmq</groupId>
    <artifactId>amqp-client</artifactId>
    <version>3.0.1</version>
</dependency>

Java code involved creating a ConnectionFactory pointing at the server running RabbitMQ,  Creating a Channel to publish messages to the queue I want and lastly steps to release resources.  These are exact steps that one would use with JMS also.

ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection con = factory.newConnection();
Channel channel = con.createChannel();
final String message = "Hello Rabbit";
for (int i = 0 ; i < 1000 * 1000 ; i++){
    channel.basicPublish("", "java2nodeQueue", false,false,null, (message + i).getBytes());
}
channel.close();
con.close();
JavaScript (Node.js) part

var express = require("express");
var amqp = require("amqp");
var app = express();
var PORT = 3000;
var connection = amqp.createConnection({host: 'localhost'});
app.listen(PORT);
connection.addListener('ready', function () {
      console.log("connected to " + connection.serverProperties.product);
      var q = connection.queue('java2nodeQueue');
      q.subscribe(function (m) {
          console.log(m.data.toString());
      });
    });
console.log("HTTP Server started at PORT:" + PORT + "...");
In JavaScript I am using a popular node module ‘express’ to create a web server wait loop.  It is much more powerful than the trivial use I am drawing here.  Commonly express helps to assemble RESTful APIs on the node side.  The guy doing majority of the work in the background is another node module ‘amqp’. It creates a connection to RabbitMQ server and creates a listener for our queue – ‘java2nodeQueue’.

JavaScript (Node.js) part

var express = require("express");
var amqp = require("amqp");
var app = express();
var PORT = 3000;
var connection = amqp.createConnection({host: 'localhost'});
app.listen(PORT);
connection.addListener('ready', function () {
      console.log("connected to " + connection.serverProperties.product);
      var q = connection.queue('java2nodeQueue');
      q.subscribe(function (m) {
          console.log(m.data.toString());
      });
    });
console.log("HTTP Server started at PORT:" + PORT + "...");

It is a good idea to use npm to install required modules into the Node.js project.

npm install express
npm install amqp

Once that is done you can run the listener using command

rabbitmq-poc\js>node server.js
HTTP Server started at PORT:3000...
connected to RabbitMQ

Here you have both connected to RabbitMQ and have created a non-durable queue ‘java2nodeQueue’. Very cool!

Next run the java code to pump in messages (here for simplicity sake I am sending only 10 messages but will share the million string result at the end)

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSendReceiveMessages
Sent message :Hello Rabbit0
Sent message :Hello Rabbit1
Sent message :Hello Rabbit2
Sent message :Hello Rabbit3
Sent message :Hello Rabbit4
Sent message :Hello Rabbit5
Sent message :Hello Rabbit6
Sent message :Hello Rabbit7
Sent message :Hello Rabbit8
Sent message :Hello Rabbit9
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.147 sec

On the other console where the node is running server.js, the received messages are print out simultaneously.

On the RabbitMQ management console I see some good stats on my on-durable queue also.  Note that this queue will disappear the moment I disconnect my Node server.  That is pretty neat too!

image

When I pumped the million messages, I am only interested in the message through put rates.  These are the statistics.

image

While Java Pumped in messages at around 27500 messages per second , RabbitMQ was able to manage deliveries into Node.js at around 7000 messages per second.  These statistics are from the Windows 7 machine.  All around it is pretty impressive for both node.js and RabbitMQ.  Performance comparison for the same program on Linux machine with identical configurations proved that RabbitMQ and Node together works about three times as fast compared to Windows 7.  On Linux RabbitMQ was able to deliver around 20,000 messages / second to the the Node.js receiver.  Even the Java message pump program performed considerably better (41000 messages / s) on Linux machine.

Follow

Get every new post delivered to your Inbox.