Source code discussed in this post can be freely downloaded and used in whatever way or form you want.  SVN URL is, svn checkout http://whiteboardjunkie.googlecode.com/svn/trunk/parallelly parallelly-read-only

JDK7 provides a library solution through java.nio.file.WatchService  to the classic folder watching dilemma Java developers went through for a decade.  It defines an interface closely matching the Observable.  JDK7 also provides concrete implementation classes implementing this interface to watch for folder changes.  The key performance boosts from this implementation is drawn from it using the OS interrupts to watch a folder (where ever those are available for consumption).  This approach frees up unnecessary cycle time and complexity in code for a ‘Watcher’ thread.  I liked it.  A good discussion of how to go about implementing a FolderWatcher is discussed here by Venish Joe in his post titled Monitor a Directory for Changes using Java.

When experimenting with this feature for one of the products we are building it annoyed the hell out of me that there is no listener paradigm out of the box for this feature.  I would have loved to register a  listener for particular events on a folder and then execute code only when these events occur.  That way my code will be free from the boilerplate instructions of polling for events and more annoyingly that infinite loop to poll for these events.   How I wished something like this was available.

		FolderWatchers.getInstance().addFolderListener(Paths.get("c:/boni/"),
				new ChangeListener(ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY) {
					@Override
					public void onEvent(WatchEvent<Path> anEvent) {
						System.out.println("LISTENER 1 " + anEvent.kind().name().toString()
								+ " " + anEvent.context());
					}
				});

Here I am registering a ChangeListener to be notified whenever there is any CREATE, DELETE or modify operations on path “c:/boni” (or it’s subfolders).  The meat of execution I am interested in is coded inside the ‘onEvent’.   This is a working code. What is done here is to take away all the mundane tasks of creating a watch service for a ‘Path’ and then polling through the events away.  The underlying constructs are completely thread safe.  Furthermore, ‘addFolderListener’ returns a ‘FolderWatcherFuture’ through which you can cancel the task.   Simple and elegant.  Without much further ado I present to you FolderWatchers (and its test case)

FolderWatchers.java

package org.boni.parallely.service;

import java.io.IOException;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

public class FolderWatchers {

	private static FolderWatchers watchers = null;
	private static WatchService watchService = null;
	private static final ExecutorService executor = Executors.newCachedThreadPool();
	private FolderWatchers() throws IOException{
		watchService = FileSystems.getDefault().newWatchService();
	}
	public static FolderWatchers getInstance() throws IOException {
		if (null == watchers){
			watchers = new FolderWatchers();
		}
		return watchers;
	}
	public FolderWatcherFuture addFolderListener(Path path, ChangeListener changeListener) throws IOException {
		FolderWatcher aWatcher = new FolderWatcher(path,changeListener,watchService);
		Future f = executor.submit(aWatcher);
		FolderWatcherFuture future = new FolderWatcherFuture(aWatcher, f);
		return future;
	}

	public void cancelFolderWatching(FolderWatcherFuture folderWatcherFuture) {

	}
}
class FolderWatcher implements Runnable{
	private WatchKey watchKey = null;
	private ChangeListener listener = null;
	protected FolderWatcher(Path path, ChangeListener changeListener, WatchService watchService) throws IOException {
		watchKey = path.register(watchService, changeListener.getEventTypes());
		listener = changeListener;
	}
	@Override
	public void run() {
		while(!cancel){
			for (WatchEvent anEvent : watchKey.pollEvents()){
				listener.onEvent((WatchEvent<Path>)anEvent);
			}
		}
	}
	private boolean cancel = false;
	protected void cancel(){
		this.cancel = true;
	}
}

class FolderWatcherFuture implements Future{
	private FolderWatcher watcher = null;
	private Future future = null;
	public FolderWatcherFuture(FolderWatcher watcher, Future future){
		this.watcher = watcher;
		this.future = future;
	}
	@Override
	public boolean cancel(boolean mayInterruptIfRunning) {
		watcher.cancel();
		return future.cancel(mayInterruptIfRunning);
	}
	@Override
	public Object get() throws InterruptedException, ExecutionException {
		return future.get();
	}
	@Override
	public Object get(long timeout, TimeUnit unit) throws InterruptedException,
			ExecutionException, TimeoutException {
		return future.get(timeout, unit);
	}
	@Override
	public boolean isCancelled() {
		return future.isCancelled();
	}
	@Override
	public boolean isDone() {
		return future.isDone();
	}
}

TestFolderWatcher.java

package org.boni.parallely.service;




import static java.nio.file.StandardWatchEventKinds.ENTRY_CREATE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_DELETE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY;




import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.WatchEvent;




import org.junit.Test;




public class TestFolderWatcher {
	@Test
	public void testFutureWatchService() throws IOException {
		FolderWatchers.getInstance().addFolderListener(Paths.get("c:/boni/"),
				new ChangeListener(ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY) {
					@Override
					public void onEvent(WatchEvent<Path> anEvent) {
						System.out.println("LISTENER 1 " + anEvent.kind().name().toString()
								+ " " + anEvent.context());
					}
				});
		FolderWatchers.getInstance().addFolderListener(Paths.get("C:/boni/installs"),
				new ChangeListener(ENTRY_CREATE, ENTRY_DELETE, ENTRY_MODIFY) {
					@Override
					public void onEvent(WatchEvent<Path> anEvent) {
						System.out.println("LISTENER 2 " + anEvent.kind().name().toString()
								+ " " + anEvent.context());
					}
				});
		while (true) {
			;
		}
	}
}

Source code discussed in this post can be freely downloaded and used in whatever way or form you want.  SVN URL is, svn checkout http://whiteboardjunkie.googlecode.com/svn/trunk/parallelly parallelly-read-only

I put together couple of classes to ease testing pieces of code under concurrency load.  It is generic enough and accommodates a certain level of flexibility.  It is reusable for your specific purpose and completely eliminates the need to write plumbing code to manage concurrent execution. I found it way more simpler than some of the nonstandard ‘frameworks’ solving the same problem.   What it does not do currently is to watch for thread locks.  A bit more hacking will add that feature.  Another day.

	@Test
	public void testServerUnderLoad(){
		final JumbleServer jumbleServer = new JumbleServer();
		CompletionServiceForConcurrencyTest<String> resultService = new CompletionServiceForConcurrencyTest<String>(Executors.newCachedThreadPool());
		for (int i = 0 ; i < MAX_THREAD_COUNT; i++){
			resultService.submit(
				new ConcurrentTestCallable<String>(new Object[]{
														UUID.randomUUID().toString()
													}) {
					@Override
					String businessLogic() throws Exception{
						return jumbleServer.jumbleString((String)(parameters[0]));
					}
					@Override
					boolean isResultValid(String output) throws Exception {
						String input = (String)(parameters[0]);
						char[] inputChars = input.toCharArray();
						char[] outputChars = output.toCharArray();
						Arrays.sort(inputChars);
						Arrays.sort(outputChars);
						//Once sorted both char[] should be the same :)
						if (new String(inputChars).equals(new String(outputChars)))
							return true;
						return false;
					}
			});
		}
		Assert.assertTrue(resultService.test());
	}

That is all.  This tests, instantiates the class to test and calls the method it wants to test in  MAX_THREAD_COUNT number of threads.  Execution is managed through a cached thread pool.  It collects back results from all threads and validates each of the threads worked as expected.  You need to implement two methods specific to your test case viz.

abstract T businessLogic() throws Exception;
abstract boolean isResultValid(T output) throws Exception;

Finally it prints out a summary of test execution.  The results look something like this.  For ease of verification I have added random failure as a feature to simulate Exception conditions for some threads.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.boni.parallely.service.ConcurrencyTestJig
Jan 12, 2012 5:53:46 PM org.boni.parallely.service.CompletionServiceForConcurrencyTest test
SEVERE: Exception while executing thread
java.lang.Exception: This is not a good time to Jumble
        at org.boni.parallely.service.ConcurrencyTestJig$JumbleServer.jumbleString(ConcurrencyTestJig.java:28)
        at org.boni.parallely.service.ConcurrencyTestJig$1.businessLogic(ConcurrencyTestJig.java:63)
        at org.boni.parallely.service.ConcurrencyTestJig$1.businessLogic(ConcurrencyTestJig.java:60)
        at org.boni.parallely.service.ConcurrentTestCallable.call(ConcurrentTestCallable.java:14)
        at org.boni.parallely.service.ConcurrentTestCallable.call(ConcurrentTestCallable.java:5)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
Concurrency test summary : TOTAL_THREADS:26, SUCCESS:24, FAILURE:2
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Jan 12, 2012 5:53:46 PM org.boni.parallely.service.CompletionServiceForConcurrencyTest test
SEVERE: Exception while executing thread
java.lang.Exception: This is not a good time to Jumble
        at org.boni.parallely.service.ConcurrencyTestJig$JumbleServer.jumbleString(ConcurrencyTestJig.java:28)
        at org.boni.parallely.service.ConcurrencyTestJig$1.businessLogic(ConcurrencyTestJig.java:63)
        at org.boni.parallely.service.ConcurrencyTestJig$1.businessLogic(ConcurrencyTestJig.java:60)
        at org.boni.parallely.service.ConcurrentTestCallable.call(ConcurrentTestCallable.java:14)
        at org.boni.parallely.service.ConcurrentTestCallable.call(ConcurrentTestCallable.java:5)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.207 sec <<< FAILURE!

Results :

Failed tests:
  testServerUnderLoad(org.boni.parallely.service.ConcurrencyTestJig)

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

This test uses a simple Jig (it will be laughable if I call it a framework) to make such tests look trivial by eliminating thread management code completely.  Let us take a look.

Main outer container of execution of the test jig is a custom ExecutorCompletionService  that neatly wraps the executor of your choice.  It has a mildly overidden submit(Callable<T>) and a completely deprecated submit(Runnable).  Feel free to modify the code to implement submit(Runnable) if that fits your specific test scenario better.  After submitting all your Callables (it is a special type of Callable and we will get to it in a bit), you call test().  This will use its ‘ExecutorCompletionService ’ capabilities to collect all the results from threads and generate a test result summary.  How cool is that!

CompletionServiceForConcurrencyTest.java

class CompletionServiceForConcurrencyTest<T> extends ExecutorCompletionService<Result<T>>{
	private final Log logger = LogFactory.getLog(CompletionServiceForConcurrencyTest.class);
	private int count = 0;
	public CompletionServiceForConcurrencyTest(Executor executor){
		super(executor);
	}
	@Override
	public Future<Result<T>> submit(Callable<Result<T>> task) {
		count = count+1;
		return super.submit(task);
	}
	@Override
	public Future<Result<T>> submit(Runnable task, Result<T> result) {
		throw new RuntimeException("Testing through runnable submission not supported currently");
	}
	/**
	 * Packs all the ran test results neatly by providing summary information
	 * and exception logs if any.
	 * @return boolean : TRUE -> All threads ran as expected.
	 */
	public boolean test(){
		List<Result<T>> results = new ArrayList<Result<T>>();
		for (int i = 0 ; i < count; i++){
			try {
				//waits on the CompletionService to make available results
				//as and when a thread completes.  The results are added
				//to a List for verification in the next step.
				results.add(take().get());
			} catch (InterruptedException e) {
			} catch (ExecutionException e) {
			}
		}
		int failedRsults = 0;
		for (Result<T> aResult : results){
			if (aResult.result == null){
				logger.error(aResult, aResult.e);
				failedRsults = failedRsults + 1;
			}
		}
		System.out.println("Concurrency test summary : TOTAL_THREADS:" + results.size() + ", SUCCESS:" + (results.size() - failedRsults) + ", FAILURE:" + failedRsults );
		return (failedRsults == 0) ? true : false;
	}
}

Next let me introduce you to my Callable.  It is based on the abstract class ConcurrentTestCallable.  It is Generic to accommodate different return types from Callable execution.  It is overiding the method ‘call’  and provides a Result Value.  The Result data object wraps result, exception or a message back from the callable.  Unfortunately in my jig I am making ‘call’ final.  Feel free to change it.

ConcurrentTestCallable.java

abstract class ConcurrentTestCallable<T> implements Callable<Result<T>>{
	Object[] parameters = null;
	public ConcurrentTestCallable(Object[] testInputParameters){
		this.parameters = testInputParameters;
	}
	/**
	 * Implement your execution logic here.  It is supposed to either return
	 * a result of type T or throw an appropriate exception in case of error
	 * scenario.
	 * @return T result of exection
	 * @throws Exception appropriate exception wrapping error condition.
	 */
	abstract T businessLogic() throws Exception;
	/**
	 * Logic to check whether the returned result is as expected.  You have access to the
	 * Input parameters through Object[] parameters.
	 * @param output Result from {@link businessLogic}
	 * @return boolean confirming accuracy of the result or not.
	 * @throws Exception
	 */
	abstract boolean isResultValid(T output) throws Exception;
	public final Result<T> call() throws Exception {
		try {
			T result = businessLogic();
			if (isResultValid(result)){
				return new Result<T>(result, null, "PERFECT MATCH",parameters);
			}
			return new Result<T>(result,null,"UNEXPECTED RESULTS",parameters);
		} catch (Exception e) {
			return new Result<T>(null,e,"Exception while executing thread",parameters);
		}
	}
}

Input parameters for execution of thread is passed in through an Object[] on the constructor.  There are two abstract methods that needs implementation.  The inputParameters are stored in a ‘protected’ local variables.  So those can be accessed from either of these methods.

1. abstract T businessLogic() throws Exception
2. abstract boolean isResultValid(T output) throws Exception

When we submit the thread, method ‘call’ will be invoked.  call simply delegates the execution first to ‘T businessLogic()’.  It uses ‘isResultValid(T)’ to validate.  Depending on the results an appropriate ‘Result’ object is returned.  If Result.result is null your thread did not execute as planned.

Result.java

//Generic result carrier to return both
//results and any possible exceptions from Callable (Thread)
class Result<T>{
	T result = null;
	Exception e = null;
	String message = null;
	Object[] inputParameters = null;
	Result(T aResult, Exception e, String message, Object[] inputParameters){
		this.result = aResult;
		this.e = e;
		this.message = message;
		this.inputParameters = inputParameters;
	}
	@Override
	public String toString() {
		return message;
	}
}

In the example discussed in this post I am using a server JumbleServer.jumble() that randomly throws an exception to simulate problem scenarios.  You can access the complete source code from subversion @

svn checkout http://whiteboardjunkie.googlecode.com/svn/trunk/parallelly parallelly-read-only

Edit[1/17/2012]:

To check whether the test jig is flexible as I make it sound like I got down to testing it a bit.  What I did is to write a different piece of code with a different set of responsibilities.  This one is a ‘SweatShopFactory’ that churns out nothing less than ‘Persons’.  Wow! Now that is sinister.

	class Person implements Serializable{
		String name;
		char sex;
		int age;
	}
	class SweatShopServer{
		private final Random random = new Random(System.currentTimeMillis());
		public List<Person> getPersons(char sex, int ageFrom, int ageTo, int count){
			List<Person> persons = new ArrayList<Person>();
			while(persons.size() < count){
				Person aPerson = new Person();
				aPerson.name = UUID.randomUUID().toString();
				aPerson.sex = random.nextBoolean() ? 'F' : 'M';
				aPerson.age = random.nextInt(ageTo);
				if (aPerson.age > ageFrom && aPerson.age < ageTo && aPerson.sex == sex){
					//This is where I am rigging the tests to create
					//some random failures;
					if (aPerson.age == 17) aPerson.age = ageTo + 1;
					persons.add(aPerson);
				}
			}
			return persons;
		}
	}
[sourcecode]

To test this class under load I wrote this (and only this) code.

[sourcecode language="java"]
	@Test
	public void testPersonsServerUnderLoad(){
		final SweatShopServer sweatshopServer = new SweatShopServer();
		CompletionServiceForConcurrencyTest<List<Person>> resultService = new CompletionServiceForConcurrencyTest<List<Person>>(Executors.newCachedThreadPool());
		final Random random = new Random(System.currentTimeMillis());
		for (int i = 0 ; i < MAX_THREAD_COUNT; i++){
			resultService.submit(
				new ConcurrentTestCallable<List<Person>>(new Object[]{random.nextBoolean() ? 'M' : 'F', 0, random.nextInt(100), random.nextInt(100)}) {
					@Override
					List<Person> businessLogic() throws Exception{
						char sex = ((Character)(parameters[0])).charValue();
						int ageFrom = ((Integer)(parameters[1])).intValue();
						int ageTo = ((Integer)(parameters[2])).intValue();
						int count = ((Integer)(parameters[3])).intValue();
						return
						sweatshopServer.getPersons(	sex, 
													ageFrom, 
													ageTo, 
													count);
					}
					@Override
					boolean isResultValid(List<Person> output) throws Exception {
						char sex = ((Character)(parameters[0])).charValue();
						int ageFrom = ((Integer)(parameters[1])).intValue();
						int ageTo = ((Integer)(parameters[2])).intValue();
						int count = ((Integer)(parameters[3])).intValue();
						Assert.assertNotNull(output);
						Assert.assertTrue("Expected size:" + count + ", Actual size:" + output.size(), count == output.size());
						for (Person aPerson : output){
							Assert.assertEquals(aPerson.sex,sex);
							Assert.assertTrue("Expected age > " + ageFrom , aPerson.age > ageFrom);
							Assert.assertTrue("Expected age < " + ageTo, aPerson.age < ageTo);
						}
						return true;
					}
					
			});
		}
		Assert.assertTrue(resultService.test());
	}

And it worked just the way I thought it would. Perfectly!

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.boni.parallely.service.ConcurrencyTestJig
Jan 17, 2012 10:50:38 AM org.boni.parallely.service.CompletionServiceForConcurrencyTest test
SEVERE: Exception while executing thread
java.lang.IllegalArgumentException: n must be positive
        at java.util.Random.nextInt(Random.java:250)
Concurrency test summary : TOTAL_THREADS:16, SUCCESS:15, FAILURE:1
        at org.boni.parallely.service.ConcurrencyTestJig$SweatShopServer.getPersons(ConcurrencyTestJig.java:61)
        at org.boni.parallely.service.ConcurrencyTestJig$1.businessLogic(ConcurrencyTestJig.java:87)
        at org.boni.parallely.service.ConcurrencyTestJig$1.businessLogic(ConcurrencyTestJig.java:80)
        at org.boni.parallely.service.ConcurrentTestCallable.call(ConcurrentTestCallable.java:28)
        at org.boni.parallely.service.ConcurrentTestCallable.call(ConcurrentTestCallable.java:5)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.457 sec <<< FAILURE!
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

Results :

Failed tests:
  testPersonsServerUnderLoad(org.boni.parallely.service.ConcurrencyTestJig)

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

Ok, now that is some minimal validation. Now it is time to use it some real code. May be to load test our Image Server

Say I have a string ‘abc-def’ to be indexed.  The say Lucene works (well actually the StandardAnalyzer that delegates the tokenizing through StandardTokenizer works) is by splitting ‘abc-def’ into ‘abc’, ‘def’ to index the two words separately.  The problem now is that when someone searched with “abc-def” Lucene says there are no matching records.  Well this, even to us the defenders of engineering dignity, was plain unacceptable.  Not much digging later I saw that it was exactly the way in which StandardAnalyzer is designed to work.  No problem I said.  One of the perks of being a heavily open source shop with awareness of GPL Vs. LGPL pitfalls is that we know we could tweak the platforms we use to make it work the way we want.  So to me the plan was to modify the StandardAnalyzer to not to tokenize strings containing hyphens (‘-’).

Now the challenge was that bug was reported at sharp 6PM, right after I tucked in my laptop and told my wife that I will pickup the pasta on my way home.   The techie hubris got the better of me.  Out came the laptop but not the power chord since I figured it is below my dignity to work beyond whatever power is left on the machine for such a trivial fix. Big mistake – never under estimate a problem, never assume while debugging.

Shortly after I started to dig in I realized that it need more than Java to fix this tiny bug.  StandardAnalyzer under its hoods delegates most of the heavy lifting to StandardTokenizer - which to my shock was a generated java file.  The source was in flex; written specifically for a generator unknown to me yesterday – JFlex.  As we speak we are best of buddies.  Alright then out came the power chord and I got down to RTFM of JFlex.  An hour and a few test cases a later I was reasonably confident to perform an invasive surgery on the StandardTokenizerImpl.jflex.  Or so I thought.

Though my micro test cases around JFlex worked well my main testcase on the product code base refused to budge.  It was adamant that the tokenizing happened at hyphens irrespective of what I wrote in JFlex.  By 11PM I accepted defeat and drove home.

There is wisdom in whoever said that sometimes the best solution to a hard problem is sleep.  Today morning I sat down to tackle the gorilla and I saw that the plain problem was hiding in plain sight.  Nice guys at Lucene development team had left a note in the same package where they bundled StandardTokenizerImpl.jflex.  It said, for the love of goo use JDK1.4 for to run Jflex to generate code.  Wait! 1.4?? 1.4?? A decade old JDK?  ’Yes’; it said.  So I went and got hold of JDK 1.4 (That took the JDK count on my notebook 5).

WARNING: if you change StandardTokenizerImpl.jflex and need to regenerate
the tokenizer, only use Java 1.4 !!!
This grammar currently uses constructs (eg :digit:, :letter:) whose
meaning can vary according to the JRE used to run jflex. See
https://issues.apache.org/jira/browse/LUCENE-1126 for details.
For current backwards compatibility it is needed to support
only Java 1.4 – this will change in Lucene 3.1.

// floating point, serial, model numbers, ip addresses, etc.
// every other segment must have at least one digit
NUM        = (
		   {HAS_DIGIT} ({P} {ALPHANUM})+
		   | {ALPHANUM} {P} {HAS_DIGIT}
           | {HAS_DIGIT} {P} {ALPHANUM}
           | {ALPHANUM} ({P} {HAS_DIGIT} {P} {ALPHANUM})+
           | {HAS_DIGIT} ({P} {ALPHANUM} {P} {HAS_DIGIT})+
           | {ALPHANUM} {P} {HAS_DIGIT} ({P} {ALPHANUM} {P} {HAS_DIGIT})+
           | {HAS_DIGIT} {P} {ALPHANUM} ({P} {HAS_DIGIT} {P} {ALPHANUM})+)

// floating point, serial, model numbers, ip addresses, etc.
// every other segment must have at least one digit
NUM        = (
			 {ALPHANUM} {P} {ALPHANUM}
		   | {ALPHANUM} ({P} {ALPHANUM})+
		   | {HAS_DIGIT} ({P} {ALPHANUM})+
		   | {ALPHANUM} {P} {HAS_DIGIT}
           | {HAS_DIGIT} {P} {ALPHANUM}
           | {ALPHANUM} ({P} {HAS_DIGIT} {P} {ALPHANUM})+
           | {HAS_DIGIT} ({P} {ALPHANUM} {P} {HAS_DIGIT})+
           | {ALPHANUM} {P} {HAS_DIGIT} ({P} {ALPHANUM} {P} {HAS_DIGIT})+
           | {HAS_DIGIT} {P} {ALPHANUM} ({P} {HAS_DIGIT} {P} {ALPHANUM})+)

The custom analyzer code is here. The MyTokenizer.java is a generated java file when we compile the modified .jflex grammar file with JFlex.

public final class MyCustomAnalyzer extends org.apache.lucene.analysis.Analyzer {
	static Version luceneVersion = null;
	static StandardAnalyzer STANDARD = null;
	public MyCustomAnalyzer(Version version){
		super();
		this.luceneVersion = version;
		STANDARD = new StandardAnalyzer(luceneVersion){
			@Override
			public TokenStream tokenStream(String fieldName, Reader reader) {
			    MyTokenizer tokenStream = new MyTokenizer(luceneVersion, reader);
			    tokenStream.setMaxTokenLength(STANDARD.getMaxTokenLength());
			    TokenStream result = new StandardFilter(tokenStream);
			    result = new LowerCaseFilter(result);
			    result = new StopFilter(StopFilter.getEnablePositionIncrementsVersionDefault(luceneVersion), result, StandardAnalyzer.STOP_WORDS_SET);
			    return result;
			}
		};
	}
	@Override
	public TokenStream tokenStream(String fieldName, Reader reader) {
		return STANDARD.tokenStream(fieldName, reader);
	}
	@Override
	public TokenStream reusableTokenStream(String fieldName, Reader reader)
			throws IOException {
		return STANDARD.reusableTokenStream(fieldName, reader);
	}
}

Miracle!  It worked as I wanted.  Exactly as I wanted.  My test cases passed with flying colors and I was laughing ear to ear.  Great!

So, for the unlucky or the intrigued who is left to write a custom analyzer to introduce grammar changes in tokenizing – be prepared to befriend JFlex.  Believe me though it is a nice tool and I can already see a few places in parsing where my team would love to put those muscles into work.

Ciao!

Step 1:

Create a folder ./externals

Step 2:

Add the new folder into repository using command,

svn add ./externals

Step 3:

Create a file “externals.txt” with these two lines

tomcat/bin http://svn.apache.org/repos/asf/tomcat/tc6.0.x/trunk/bin
apacheds/resources http://svn.apache.org/repos/asf/directory/apacheds/tags/1.5.7/resources

Step 4:

From command line run the following command.

svn propset svn:externals ./externals -F externals.txt

You are all set.  Any updates on externals folder (after committing the newly created folder) will fetch you the latest from tomcat/trunk and apacheds/tags/1.5.7/resources.  What the command has done is to set a property names ‘svn:externals’ into the folder metadata of folder ‘externals’ that tells svn client that some of its child folders are to be fetched/updated/committed through a different repository location.

  If you change any parts of the “externals” code, you will be able to check it in right into the correct code-base.  This solution avoids potential mess ups involving code base mash-ups.

One flipside of this solution is that while branching or tagging you will need to be extra careful in case your symbolic links also needs to shift.  To this effect it is worth noting that there are other useful commands for managing svn folder properties.  Check out the documentation here.

There is subverison documentation discussing this feature.  But I did not find it very easy to understand.

Note

Currently, MySQL does not implement these functions according to the specification. Those that are implemented return the same result as the corresponding MBR-based functions.

Then the obvious question is “What is complete?” In this context complete is what is specified by the guys at Open Geospatial Consortium (OGC).  Hibernate spatial acknowledges that none of these databases are fully OGC compliant.  So it uses a clever mechanism to still provide a uniform API to the underlying spatial databases.  It uses a Provider that lets each databse specific provider to define its own ‘HibernateSpatialXXXXDialect’.  HibernateSpatialDialect invariably extends one of the Dialects provided by Hibernate.  So to a large extent it is hugely dependent on the twists and turns versions of Hibernate goes through.  At the same time it has always tried to keep itself as close to the curve as possible and have updated its versions matching to Hibernate’s.   Recently we shifted the spatial database from MySQL to Oracle.  This shift was painless because we used this spatial layer in between.  90% of the spatial code worked out of the box when I changed the provided from mysql to oracle.  The 10% of the code that did not work well was pertaining to data fetch back.   Investigation into these unearthed problems revealed some serious shortcomings of MySQL as a spatial data store.

It is interesting to note that spatial data management can be creatively used beyond GeoSpatial systems.  Geo spatial systems can be thought of as a specific use case of a family of problems dealing with annotated images with spatial relationship between various annotations.  These images can be of any type – for example a portrait, a scan of an architectural drawing, a digitized high resolution image of the solar system, images from molecular analysis of samples etc.  In one of the future posts I will discuss some ideas on how to creatively use spatial database for innovative solutions.

Select A From Magazine a Where a.month = :month and a.year= :year and a.type= :magazineType

Now consider the scenario where the API writer expected the following behavior when one or more of the parameters in the WHERE CLAUSE is null.

1. year == null; return all records where YEAR IS NULL.

2. type == null; return all records where TYPE IS NULL.

It is not uncommon to come across such situations when the WHERE clause includes some NULLable fields.  The problem here is that SQL would expect the Query to read the following for Scenario 1 & 2

1. Select A From Magazine a Where a.month = :month and a.year is NULL and a.type= :magazineType

2. Select A From Magazine a Where a.month = :month and a.year= :year and a.type IS NULL

Since we have established that either ‘year’ or ‘type’ can be null it is quite possible that both YEAR and TYPE are NULL.  Then the correct Query is supposed to read:

3. Select A From Magazine a Where a.month = :month and a.year IS and a.type IS NULL

This is going to create a problem for application developer since he is expected now to either maintain a bunch of named Queries or construct these EJBQLs dynamically so that the appropriate IS NULL substitutions takes the place of ‘= null’ locations.  Ideally I would have preferred developer handling such scenarios in code.  However this issue missed our thorough testing routines since the hibernate implementation of JPA Handler for MySQL seemed intelligent enough to handle the IS NULL conversions internally when one of the parameters passed was null.  However this is not the case with corresponding Oracle Dialect class.  In our product there were close to 200 NamedQueries and we did not have enough time to make all those code changes and run the product through a complete test cycle.  Therefore we chose to introduce a Generic routine to check and handle this scenario.  This way the changes are central and will be well testable.

Expected behavior:

Whenever system is asked to execute a named Query an internal routine will check whether any of the passed parameters is a null value.  If it is a null then then an appropriate Query conversion will be done to convert = null or !=null usage if any, to IS NULL or IS NOT NULL.  The resulting Query will then be executed as a dynamic Query.

The design had ensured there were only two places in the system through which a @NamedQuery can go for execution.  This part of the code I modified to check for null values in any incoming ‘Named Parameter’.  If one or more found the Query is then translated into a dynamic query with Equivalent IS NULL / IS NOT NULL Conversions.

Problem 1 : Given a NamedQuery’s name, Get the corresponding Ejbql  query string.

	public NamedQueries getNamedQueries(){
		NamedQueries queries = persistentClass.getAnnotation(NamedQueries.class);
		return queries;
	}
	
	protected String getQuery(String queryName){
		NamedQueries queries = getNamedQueries();
		if (null == queries){
			return null;
		}
		NamedQuery[] namedQueries = queries.value();
		if (null == namedQueries || namedQueries.length <= 0) return null;
		for (NamedQuery aQuery : namedQueries){
			if (queryName.equals(aQuery.name()))
				return aQuery.query();
		}
		return null;
	}

In this code, instance nariable ‘persistentClass’ points to the Type (Class) of the main Entity.  All my Named Queries for an Entity bean are defined within that Entity bean (as annotations).  This design consistency coupled with a bit of Java Reflection gets me the Query String fairly easily.  Afterwards a risky but effective code scanning ensures correct Query translation.

	protected String convertEqualToNullToIsNull(String query, String paramemeterIdentifier) {
		if (StringUtils.indexOf(query, "!= :" + paramemeterIdentifier) != StringUtils.INDEX_NOT_FOUND){
			return StringUtils.replaceOnce(query, "!= :" + paramemeterIdentifier, " IS NOT NULL ");
		}
		else if (StringUtils.indexOf(query, "= :" + paramemeterIdentifier) != StringUtils.INDEX_NOT_FOUND){
			return StringUtils.replaceOnce(query, "= :" + paramemeterIdentifier, " IS NULL ");
		} 
		return query;
		
	}

The wrapper API call that ties together all these together is this:

	protected Query handleNullParameterConversions(final String name, Map<String , ? extends Object> params){
		logger.debug("IS NULL Conversion For Query : " + name);
		String parameterizedQuery = getQuery(name);
		logger.debug(parameterizedQuery);
		//correct query with necessary IS NULL / IS NOT NULL
		if (null != params){
			Set<String> keys = params.keySet();
			for (String aKey : keys){
				Object value = params.get(aKey);
				if (null == value){
					logger.debug("IS NULL CONVERSION Required For:" + aKey);
					parameterizedQuery = convertEqualToNullToIsNull(parameterizedQuery, aKey);
					logger.debug(parameterizedQuery);
				}
			}
		}
		logger.debug("Final Query:"+ parameterizedQuery);
		javax.persistence.Query query = getEntityManager().createQuery(parameterizedQuery);
		
		for (final Map.Entry<String, ? extends Object> param : params
				.entrySet()) {
			if (null == param.getValue()) continue;
			query.setParameter(param.getKey(), param.getValue());
		}
		
		return query;
	}

javax.persistence.Query query = getEntityManager().createQuery(parameterizedQuery);

For such hack codes it will be criminal to commit to a large codebase without adequate test case.  I wanted to keep the test case a bit future proof.  So what I needed was a test that can scan and identify all @NamedQueries, then pick all the named parameters in the where clause and then test for each and every parameter combination.  For example in our  original Query,

Select A From Magazine a Where a.month = :month and a.year= :year and a.type= :magazineType

The three named parameters creates 7 potential combinations (Any 3 IS NULL [3], Any 2 IS NULL [3], All 3 IS NULL [1]).  For n parameter Query the number of test cases are

N = nC1 + nC2 + nC3 + … + nCn

So first I wrote a trivial package scanner to identify all the DaoImpls that need to be tested.  Then I passed each to the following method to test validity of resulting query for each parameter combinations.

	private void testNamedQueriesOfDao(GenericJpaDaoImpl dao){
		logger.debug("TESTING:" + dao.getClass().getName());
		dao.setEntityManager(getEntityManager());
		NamedQueries queries = dao.getNamedQueries();
		if (null == queries){
			logger.debug("No Named Queries found in:" + dao.getClass().getName());
			return;
		}
		NamedQuery[] namedQueries = queries.value();
		if (null == namedQueries) fail("Named Queries are null");
		for (NamedQuery aQuery : namedQueries){
			String query = aQuery.query();
			logger.debug("Original Query:" + query);
			String[] variables = dao.getNamedParametersInSequence(query);
			List<String[]> allCombinations = getallCombinationsFor(variables);
			for (String[] aCombination : allCombinations){
				combinations = combinations + 1;
				logger.info("Parameter(s):" + toString(aCombination));
				Map<String, Object> parameters = new HashMap<String, Object>(); 
				for (String paramemeterIdentifier : aCombination){
					parameters.put(paramemeterIdentifier, null);
				}
				Query convertedQuery = dao.handleNullParameterConversions(aQuery.name(), parameters) ;
			}
			queryCount = queryCount + 1;
		}
		
	}

I also wrote not so elegant a method to fetch all combinations (sub sets actually) of a given sub array.

	private List<String[]> getallCombinationsFor(String[] variables) {
		int length = variables.length;
		List<String[]> stringCombinations = new ArrayList<String[]>();
		if (length <= 1){
			stringCombinations.add(variables);
			return stringCombinations;
		}
		String maxValue = "";
		for (int i = (length - 1) ; i >=0 ; i--){
			maxValue = maxValue + i;
		}
		logger.debug("Max Value for Base :" + length + " = " + maxValue);
		int maxIntValue = Integer.parseInt(maxValue,length);
		logger.debug("Equivalent Max Value in decimal : = " + maxIntValue);
		List<String> combinations = new ArrayList<String>();
		for (int i = 0 ; i <= maxIntValue; i++){
			String aCombination = Integer.toString(i, length);
			if (isUnique(aCombination,maxValue.toCharArray())){
				combinations.add(aCombination);
				if (aCombination.indexOf("0") == -1){
					combinations.add("0" + aCombination);
				}
			}
		}
		Collections.sort(combinations);
		for (String aCombination : combinations){
			logger.debug(aCombination);
			char[] positions = aCombination.toCharArray();
			List<String> aStringCombination = new ArrayList<String>();
			for (char aPosition : positions){
				aStringCombination.add(variables[Integer.valueOf(String.valueOf(aPosition))]);
			}
			String [] justAdded = null;
			stringCombinations.add(justAdded = aStringCombination.toArray(new String[positions.length]));
			String newlyAdded = "";
			for (String aString : justAdded){
				newlyAdded = newlyAdded + aString + ",";
			}
			logger.debug(newlyAdded);
		}
		return stringCombinations;
	}

All these together gave me enough confidence that the code base is sufficiently fool proof.  However as part of the next minor version release we are going to clean up the code base and make appropriate Query handling to incorporate IS NULL / IS NOT NULL deliberately.  Automatic interpretations of logic is trouble in long term maintenance.

Yes! Google made its Extension development so easy for an UI layman like me that exactly 4 hours after the idea I had version 1.0 out.  I was able to easily distribute it to fellow chrome users in the dailymile community and get their valuable feedback.  As we speak the version that we see on the left is 1.0.4, with some feature additions and bugs fixed.  Since I was brimming with ideas to make this extension better but not having enough time at hand to sit down and code it through I have decided to open source it through my regular Google Code channel and see whether someone else can help me out.  Are you listening?

A chrome extension is a powerful application that runs within the security sandbox of the browser.  As part of every chrome browser distribution, Google has packaged in all tools one needs to develop and distribute extensions.  These tools include a powerful packaging mechanism, rich set of extensions APIs, a debugger, a resource tracker through which you can inspect the DOM and various HTTP requests.

Chrome is the most comprehensive HTML 5 reference implementation out there with supports for local storage, audio, video, Canvas and many more.  The extensions API empowers you to take advantage of all these services to make your software as useful and intuitive to the end user as you want.  Clearly imagination is the limit in what an extension can do.

In this specific dailymile extension’s case all I am doing it to make an XMLHttpRequest to a hosted dailymile API that returns the data I want in a JSON format.  Through a JavaScript callback I then parse and dynamically paint the user interface.  If my user wants to see more details of the workout (or a note or an image), a click will open the data in its full glory on a chrome tab.  To achieve all this all I had to do was to write a manifest.json file that defines which are all the web resources I am going to access, what all browser permission I would need and finally what is the starting page of my extension.  While developing the extension browser allows me to load, test and debug from my development directory.  Once fully satisfied with the product I can package the extension into a .crx distributable using browser tools itself.  Once I have a CRX file I could either choose to host it on google chrome extensions library (to sign up as chrome extensions developer there is a nominal USD5 fee) or host on your own.  I decided to host it on my own through my Google Code page.  The packaging has a powerful addition of auto updating.  Once enabled all I need to do to get a newer version of my software to my current users is to change a file on an HTTP location that announces a new version is ready for download.  Chrome will check this file periodically and auto update.  I setup an auto updating for this extension as well.  Overall a very neat development, packaging and distribution system.  Very impressed.

Source code available @
svn checkout https://whiteboardjunkie.googlecode.com/svn/trunk/dailymile
Browse Source code here

To everyone who runs online campaigns to ‘Save’ stuff I have a word for advice, Ctrl-S.

World comprises of only two types of events; those that are shared on facebook and those that are unworthy of.