Java Generics Presentation (Part 2)

Our team holds weekly Lunch & Learns on various technologies, so today I presented Part 2 on Java Generics.

Java Generics Presentation (Part 1)

Until recently, the majority of our project code had to run on WebLogic 8.1, which constrained most of the code-base to 1.4 language features (WebLogic 8.1 will not run on a 1.5+ JVM). A little over a year ago our client decided to migrate to the JBoss 4.3.0 JEE container. This meant that the code would now be running on a Java 1.6 JVM, and we are now taking advantage of the new 1.6 language features.

Our team holds weekly Lunch & Learns on various technologies, so today I gave the first of several presentations on Java Generics.

On Dissent

Iran sentences Blogger to 19 years in prison!

I like to think I'm open-minded about world politics (I read from the left and the right). This is just indefensible. A society that won't tolerate dissent just isn't built to last. You need a feedback loop to realize progress.

Visualvm

If you need to profile your Java application and you don't want shell out $ for a proprietary tool, you might be interested in visualvm. It's licensed under the GPL and includes a number of cool features:

1. It automatically connects to any JVM on your system (including the visualvm jvm itself) as they start up.
2. It includes high level monitors of CPU usage, Heap, Classes and Threads
3. Detailed views of Threads
4. CPU and Memory profiling.

I haven't had much time to play with it, but it seems to just work!

JBPM Migrator (Part 2)

My previous post discussed the JBPM migrator, a small project we wrote to handle the migration of JBPM process instances. Since that post, we had several requests to open-source the code. We managed to get the code to the JBPM team in late August.

The JBoss team have taken our code and implemented it in the 4.2 version of the JBPM release. The developer responsible for doing this made a number of significant changes (this is to be expected because we developed the migrator for version 3.2.2 of JBPM), so much so that it really doesn't resemble the work that we originally did. Neverthesless, JBPM 4.2 is now capable of performing migrations!

A number of organizations (ourselves included) are still using version 3.2.2 of JBPM and are asking to get their hands on the original version of the migrator. To address this, I have renamed the project as jbpm-instance-migrator and published it on Google Code under the LGPL license.

At this point, I would ask anyone who is interested in this project to download the code and give it a try. Please submit any issues, requests, or questions you may have to the project (not to this blog). Your feedback is welcome.

JBPM Migrator

Overview

We are using the JBPM workflow library (version 3.2.2) on a project here at Intelliware. After some analysis, we chose JBPM as our process modelling tool because it was open source and it was easy to integrate into our technology stack (Java 4, Hibernate for persistence).

Over time, a process definition needs to change. Usually these changes reflect new business requirements, but they can also be related to a bug fix or an improvement in the existing process. So when we release a new version of a software product, we may need to update the older process instances in the database. Rather than provide a mechanism to migrate process instances, the JBPM library supports multiple versions of a process definition simultaneously:

Process instances always execute to the process definition that they are started in. But JBPM allows for multiple process definitions of the same name to coexist in the database. So typically, a process instance is started in the latest version available at that time and it will keep on executing in that same process definition for its complete lifetime. When a newer version is deployed, newly created instances will be started in the newest version, while older process instances keep on executing in the older process definitions.¹

This wasn't very appealing to us. Our application has processes that can be resumed at any point in the future (potentially years later) so by following the JBPM prescribed approach our developers would have to support - and the QA folks would have to test - outdated process instances for years to come.

What we wanted was the ability to migrate outdated process instances to the current process definition. Indeed, the JBPM documentation addresses this approach:

An alternative approach to changing process definitions might be to convert the executions to a new process definition. Please take into account that this is not trivial due to the long-lived nature of business processes. Currently, this is an experimental area so for which there are not yet much out-of-the-box support.

As you know there is a clear distinction between process definition data, process instance data (the runtime data) and the logging data. With this approach, you create a separate new process definition in the JBPM database (by e.g. deploying a new version of the same process). Then the runtime information is converted to the new process definition. This might involve a translation cause tokens in the old process might be pointing to nodes that have been removed in the new version. So only new data is created in the database. But one execution of a process is spread over two process instance objects...

JBPM doesn't provide a tool to do this, but the code is open source and well documented, so we build a jbpm-migrator ourselves.

The Migrator

Given an old process instance, the migrator is responsible for transferring data to the latest process instance. The migrator transfers:

1. 
   All of the tokens. This is facilitated through the use of mappings .
   


2. 
   All persistent and transient variables.


3. 
   It adds a migration memo (a String in the persistent variables map) to the new process instance which records info about the migration (the current date, the old process definition version#, the old process instance id , etc).

The migration is performed recursively, so each sub-process is migrated according to the above steps.

Mapping Token Nodes

One of the key challenges when migrating a process instance is the renaming or removal of wait state nodes. Wait state nodes (the green boxes in the diagram below) are where tokens reside when a process instance is persisted. Determining where a token should be placed is facilitated through a migration. A Migration contains a map that tells the Migrator where to put a token from a deprecated wait state node in the current process. Consider the following three versions of a Process called 'Application':

For the Application Process, two migrations would be written²:

1. 
   Migration #1 (maps tokens from version #1 to version #2):

   {'init' => 'start', 'invalid' => 'Requires Review', 'end' => 'application completed'}



2. 
   Migration #2 (maps tokens from version #2 to version #3):

   {'start' => 'application received'}

Our migrator takes these individual migration maps and creates a composite map. With the composite map, the migrator can map a token from any outdated wait-state node directly to the appropriate node in the current process. The composite map of these two migrations would be written as:

{'init' => 'application received', 'start' => 'application received', 'invalid' => 'Requires Review', 'end' => 'application completed'}

Note that the map only needs to explain what to do with tokens on deprecated nodes (e.g. init, start, invalid, and end). No mapping is required for non-deprecated nodes (e.g. managerial audit, application completed, and Requires review). By default, if no mapping exists for a wait state node (i.e. it is not deprecated) the migrator will attempt to move the token to a node with the same name in the new version.

The example I am using includes a discrete migration for each version of the process but this is not always required. Depending on the changes being made to the Process Definition, it is possible that the developer will not be required to include a migration at all. This is a good thing. It means that we don't have to write a migration for every single process definition change and there is almost no configuration required on behalf of the developer.

But there is a cost. Deprecated nodes can never be used in future definitions of your process. So with our 'Application' Process, the init, start, invalid, and end nodes can never be used again in the process definition (as wait states). Doing so would break the migrator.

Defining a Migration

Migrations are written as Java classes. The class must implement the Migration interface, it must not be abstract, and it must contain a default constructor. The Migration interface declares one method that must be implemented:

public StateNodeMap createNodeMap();

Here is how you would express the first migration for 'Application' Process Definition example:

public class ApplicationProcessMigration001 implements Migration{

public StateNodeMap createNodeMap() {

return new StateNodeMap(new String[][]{

{"init", "start"}, {"invalid", "Requires Review"}, {"end", "application completed"}

});

}

}

Defining a Migrator

How do we create the migrator and use it to perform a migration? Like this:

Migrator migrator = new Migrator(“ApplicationProcess”, jbpmContext, “com.foobar.ApplicationProcessMigration”);

ProcessInstance newProcess = migrator.migrate(oldProcess);

The parameters used to create the Migrator instance are:

1. 
   The name of the Process Definition that it will be migrating.
   


2. 
   A JbpmContext instance. The migrator requires this to look up the latest Process Definition.
   


3. 
   The Migration base class name. The migrator assumes that your migrations use the pattern package.ClassName{migration#}. For the base Class name “com.foobar.ApplicationProcessMigration”, the migrator will attempt to load and instantiate classes named “com.foobar.ApplicationProcessMigration001”, “com.foobar.ApplicationProcessMigration002”, etc, until it can’t find any valid classes.

The third point is another example of convention over configuration, resulting in less maintenance and tedium on behalf of the developer.

Unit and Integration Testing

I'm putting this section last, but it was one of our top concerns when considering an approach to migrations. We debated a number of approaches to testing and most of them were deemed to be too complex and error prone.

We already unit tested our JBPM process definitions to make sure that transitions point to valid nodes and that all actions declared in the process were available on the Classpath. With regards to the migrations, we have a base test that asserts that:

1. 
   A developer has not introduced a deprecated node into the current process definition.


2. 
   All current nodes in the composite map exist in the process definition.


3. 
   All current nodes in the composite map are valid wait state nodes.

Our application is deployed several times a week to a test environment where QA folks test functionality and validate stories. This provides a great opportunity to discover problems with our application early, and the migration code is no exception. We write migrations throughout the entire development cycle (not just when we release) to make sure, so if a QA person loads up an outdated process from from last week, the migrator will be run. This gives us a chance to find runtime bugs that the unit tests can't locate.

---

1. http://docs.jboss.com/jbpm/v3.2/userguide/html/jpdl.html#processversioning
2. I'm using the Ruby literal syntax for a Hash because there isn't one in Java.

The Pitfalls of Dynamic Proxy Serialization

Recently, we ran into problems de-serializing proxy objects. The first pitfall involved a bug in Weblogic that prevented us from de-serializing a java.lang.reflect.Proxy instance. The second pitfall occurred as a result of using a Cglib proxy. In our efforts to deal with these pitfalls we managed to deepen our understanding of how these libraries create dynamic proxies.

Pitfall #1

We wanted to use the java.lang.reflect.Proxy class to make instances of one (very simple) class assignable to different types of interfaces. Everything was going just swimingly and we were close to checking in our code. When it came time to run our integration tests, Weblogic began throwing a ClassNotFoundException whenever it attempted to de-serialize a proxy instance in a MessageDrivenBean. Once we confirmed that the interfaces were indeed on the Classpath, we fired up the debugger and dug a little deeper (but not until we consumed a a copious amount of Coca-Cola and Marlboros... thanks for the tip Mike). We discovered that - in the process of de-serializing our proxy object - the java.io.ObjectInputStream was trying to load our interfaces with the wrong classloader. It was using the parent of the Weblogic application classloader which had no knowledge of the classpath in our EAR file. We were were pretty certain that the reason was due to a bug in the way Weblogic was managing classloaders in it's JMS implementation (of course, Weblogic is not open source so we can't be 100% certain of this).

We couldn't de-serialize the object ourselves inside the message bean and we couldn't modify the way Weblogic managed classloaders, so we tried using the Cglib library instead. The Cglib proxy worked? The reason is because the ObjectInputStream de-serializes Cglib instances differently than it does a java.lang.reflect.Proxy instance. Why? Because of a key difference between the two types of proxy objects;

• When you create an instance using the JDK Proxy, it generates a subclass of the java.lang.reflect.Proxy class for you (the subclass is assignable to any interfaces you provide). Furthermore, the class generated by the java.lang.reflect.Proxy will have a different class descriptor (in fact, a Proxy class descriptor).

• On the other hand, the Cglib library generates a subclass of an arbitrary class that you provide (as with the JDK proxy, the Cglib generated subclass will also be assignable to any interfaces you choose to provide). The class generated by the Cglib library will have a regular class descriptor.

Whenever the ObjectInputStream de-serializes an object, it peeks at the Class descriptor. If it has a Proxy class descriptor, it invokes the ObjectInputStream.resolveProxyClass() method which creates a new version of the proxy subclass in the JVM (if it didn't already exist) based on the declared interfaces. Because the Cglib class has a normal class descriptor, the ObjectInputStream.resolveClass() method is invoked instead. For reasons we're not sure of, the classloader that is retrieved in the ObjectInputStream.resolveClass() method is the correct one and can resolve our interfaces/classes.

Pitfall #2

Now our Cglib proxy was being de-serialized correctly when sent to a JMS MessageDrivenBean. But we we were also sending these proxies to a rich client, and the Cglib proxies were not being de-serialized there. We were stunned, but only temporarily. This result was to be expected. The Cglib generated objects were instances of a subclass generated dynamically on a different JVM. Our rich client JVM didn't recognize the class and was quite justified in throwing a ClassNotFoundException.

It turns out there is a pretty simple solution to this problem. Basically, you need to implement the writeReplace() and readResolve() methods on the object that your proxy delegates methods to. The delegate uses these methods to de-proxy and re-proxy itself depending on whether it is being serialized or de-serialized. In our case, our delegate object is the super-class of our Proxy instance, so we implemented the writeReplace() and readResolve() methods on it. Here is an example using a class called DataHandler;

public static class DataHandler implements Serializable {
private static final long serialVersionUID = 1L;
private final String someData;

public DataHandler(String someData){
this.someData = someData;
}
public String getSomeData() {
return someData;
}

public boolean equals(Object obj) {
return obj instanceof DataHandler ?      
((DataHandler)obj).getSomeData().equals(this.someData)
: false;
}

public Object writeReplace() throws ObjectStreamException{
return new DataHandler(this.someData);
}
public Object readResolve() throws ObjectStreamException{
return ProxyFactory.createCglibProxy(DataHandler.class,
                   new Class[]{IFoo.class},
                   new Object[] {this.someData});
}
}

public interface IFoo extends java.io.Serializable{
}

We create a proxy with the Cglib library like so;

Enhancer enhancer = new Enhancer();
enhancer.setSuperclass(DataHandler.class);
enhancer.setInterfaces(new Class[]{IFoo.class});
enhancer.setCallback(new NoOpCallback());//a NoOp callback. All methods will be invoked on the superclass
IFoo foo = (IFoo) enhancer.create(
new Class[]{String.class},
new Object[] {"someData"});//generates a subclass of DataHandler that assignable to the IFoo interface.

When the foo instance is serialized, the super-class (DataHandler) method writeReplace() is invoked. The writeReplace method returns a new instance of the DataHandler class, thereby discarding the Proxy wrapper subclass. It's this new instance of DataHandler that gets serialized, not the original subclass. When the DataHandler instance is eventually de-serialized, the readResolve() method is invoked on it. The readResolve() will generate and return a new proxy subclass equivalent to the original . We learned this little trick by studying the code in the Hibernate project which uses Cglib for proxies and is open-source.

Antwrap

Recently, I wrote a library called Antwrap that can be used to invoke Ant tasks from a Ruby script. My original motivation for writing it was to use Rake as a build tool on a Java project. While Ant provides an abundance of useful tasks, I've always found its XML syntax makes it hard to write concise, fine-grained tasks (the original author of Ant states as much). Rake is an internal DSL written in Ruby(which is concise), so you can define your own abstractions in the build script and you have a huge selection of libraries at your disposal.

I wanted to use Rake as a build tool but there was an obvious hurdle to clear; all those Java specific tasks that we take for granted in Ant aren't available in the Ruby libraries. For example, let's say a Rake script needs to kick off a Java process. The core Ruby libraries make this possible:

system("java", "-client -jar lib/foobar.jar")

But the system call is not specific to the Java executable so passing in JVM and Program options can be awkward. In Ant, we would use the java task:

<java classpath="lib/foobar.jar" classname="foo.bar.FooBar" fork="true">

    <jvmarg value="client"/>
    <arg value="argOne"/>
    <arg value="argTwo"/>
</java>

This is the equivalent using Antwrap:

@ant = AntProject.new(:ant_home => "/Users/caleb/tools/apache-ant-1.7.0")

@ant.java(:classpath => 'lib/foobar.jar', :classname => 'foo.bar.FooBar', 
          :fork => 'true'){
    jvmarg(:value => 'client')
    arg(:value => 'argOne')
    arg(:value => 'argTwo')
}

The Task attributes are passed to the method as a hash. What would normally be child elements in XML are passed to the method via a block. The AntProject instance doesn't actually have a java method defined. Rather, its method_missing method is invoked and ends up instantiating the appropriate Ant task for you. This is one example of Ruby's powerful meta-programming features; they make it easy to provide rich semantics (as opposed to invoking @ant.executeTask('java')).

Here is a more complicated example using the javac task. This simply illustrates that all of the normal Ant tasks are at your disposal, including Ant Properties and Refs:

@ant = AntProject.new(:ant_home => "/Users/caleb/tools/apache-ant-1.7.0")
@ant.property(:name => 'common.dir', :value => @current_dir)

@ant.path(:id => "other.class.path"){
    pathelement(:location => "classes")
    pathelement(:location => "config")
}

@ant.path(:id => "common.class.path"){
    fileset(:dir => "${common.dir}/lib"){
        include(:name => "**/*.jar")
    }
    pathelement(:location => "${common.classes}")
}

@ant.javac(:srcdir => "test", :destdir => "classes"){
    classpath(:refid => "common.class.path")
    classpath(:refid => "other.class.path")
}

The transition from an existing Ant build file to a Rake file is relatively easy. There is a handy conversion script available. So this:

<target name="clean" depends="init">

    <delete dir="classes" failonerror="false"/>
    <delete dir="${distro.dir}"/>
    <delete file="${outputjar}"/>

    <delete file="${output.dir}"/>
</target>

Becomes the Rake equivalent:

task :clean => [:init] do
    @ant.delete(:dir => "classes", :failonerror => "false")
    @ant.delete(:dir => "${distro.dir}")
    @ant.delete(:file => "${outputjar}")
    @ant.delete(:file => "${output.dir}")
end

Antwrap runs on the native Ruby and the JRuby interpreter. If running on the native Ruby interpreter, Antwrap depends on the Ruby Java Bridge (RJB) Gem which invokes Java classes via the Java Native Interface (JNI). Antwrap is currently being used for Ant tasks in the Raven (a.k.a. don't call me Maven) project. Raven is a JRuby implementation of the Rake tool that provides all kinds of utilities for a Java project.

In the long-term, I see Ruby/Rake scripts complementing a tool like Maven on Java projects. Ruby for fine-grained, project specific tasks and Maven for coarse-grained, boiler-plate tasks (compilation, unit-tests, packaging, documentation) that you need on most Java projects.

Other resources:
Martin Fowler on JRake
Groovy Antbuilder

Macrodef

The <macrodef/> task (new in Ant 1.6.5) can be used to define your own custom tasks from within an existing build file. In some cases it can eliminate the need to define your own custom tasks in the traditional fashion.

When possible, you should try to use <macrodef/> instead of the <antcall/> task. By using <antcall/>, you're invoking your target as if it were a task. This practice can become harder to maintain and result in performance problems in larger build files.

For example, here is a simple project file using the <antcall/> task to invoke the 'foo' target :

<project name="antcall">

<property name="A" value="A"/>
<property name="B" value="B"/>

<property name="C" value="C"/>

<target name="foo">
<echo>A: ${A}</echo>

<echo>B: ${B}</echo>
<echo>C: ${C}</echo>
</target>

<target name="foo.call.1">

<antcall target="foo">
</antcall>
</target>

<target name="foo.call.2">
<antcall target="foo">

 <param name="A" value="eh?" />
 <param name="B" value="bee" />

 <param name="C" value="sea" />
</antcall>
</target>

</project>

And here is the same project file rewritten so that 'foo' is a task:

<project name="macrodef">


<macrodef name="foo">
<attribute name="A" default="A" />
<attribute name="B" default="B" />

<attribute name="C" default="C" />
<sequential>
 <echo>A: @{A}</echo>

 <echo>B: @{B}</echo>
 <echo>C: @{C}</echo>
</sequential>
</macrodef>

<target name="foo.call.1">
<foo/>
</target>

<target name="foo.call.2">
<foo a="EH?" b="BEE" c="SEA"/>

</target>

</project>

The difference between the approaches is subtle but important;

• The 'macrodef' project is going to be more maintainable because the 'foo' task attributes are encapsulated (rather than using <property/> elements).
• Eclipse automatically detects the new task and provides the attributes via auto-completion. This saves a person having to scroll down to the target definition to determine what parameters for an <antcall> are required.
• The performance of the 'macrodef' project will be better than the 'antcall' project because the <antcall/> will actually create an entirely new project instance in memory. This is mentioned on the ant wiki