Paul René Jørgensen

Did you ever want to be the Big Brother of the Java Virtual Machine? Being able to do non intrusive monitoring of method calls without access to the source code, performing this in a production environment without a noticeable performance hit?

Do you want to know who is doing what to whom, that is, who invokes what method in context of which principal? Making certain that memory consumption is dynamically adjusted to avoid out of memory situations? Would you also like to use it in your legacy applications running pre Java 5 as well as utilize the new Java agent interface of the newer JVMs?

You know you have to collect vast amount of data and process it effectively and in real time. Finally you figure out that the amount of data exceeds the capabilities of a traditional data model and you decide that the only solution is a dimensional data model, using MDX for multi dimensional analysis.

There are numerous tools available for profiling and measuring your Java programs. However these tools are either: a) aimed at profiling or measuring your code in a development environment; b) expensive; c) not open source; d) requires a Ph.D. in debugging and profiling e) any combination of the above

Usemon attempts to alleviate this by providing an open source tool which is easy to use, collects data from environments based upon Java 1.4 through 6, has a flexible dimensional data model stored in a SQL database and attempts to be non-intrusive.

Usemon hooks into the Java virtual machine through a modified class loader on older JVMs or via the Java Agent Interface from version 5 and up. It automatically monitors Java EE components (EJBs, Servlets, MDBs, Queues, SQL Statements, etc), but can be configured to inspect any class in your JVM. Data is multicasted from agents to one or more collectors and stored in the DBMS of your choice. Usemon comes with a complete configuration for MySQL.

The overhead of running Usemon is negligible and thus you may collect run-time statistics around the clock all year, which is rather fascinating, given that you have enough disk space.

Usemon was developed for large Java EE environments at Telenor, which is one of the worlds largest telco to fulfill these requirements. You may utilize it in any environment in which you have control over the Java Virtual Machine start up parameters. For example Java EE applications, Spring based Web applications or simply your own home grown web application packaged in a .war file.

You may download or check out the source code at http://usemon.org

Prologue

Recently we experienced a problem regarding the RMI-compability between IBM WebSphere (WAS) version 5.1 and BEA WebLogic (WLS) version 8.3 both running a version 1.4.2 JVM. The WLS is a client of the WAS server and this worked fine since both the IBM and BEA RMI-implementation is based upon Sun’s RMI-ORB. Then an application came along that needed a custom login module and we had to enable security attribute propagation in WAS. All of a sudden hell broke loose, and the WLS was not able to speak with the WAS anymore. The problem was soon identified to be the wire format of the RMI-implementation. When you enable security attribute propagation, the wire format changes and WLS didn’t understand the WAS RMI response anymore.

The solution

In one way or the other I needed the RMI-implementation from WAS to work inside WLS. This is not an easy thing to do because a RMI-implementation register itself inside the JVM in several ways. I also wanted to hide the IBM classes from WLS to make sure I didn’t get any unexpected class ambiguities.

First I needed to just be able to call a EJB on a WAS server from a remote JVM. This was pretty easy and well documented from IBM. Since I didn’t need the support for DataSources or JMS resources I opted for the WebSphere pluggable client that only support EJB invocations. The following list includes the relevant WAS libraries needed to run the pluggable client.

"ibmext.jar", "ibmorb.jar", "iwsorbutil.jar", "naming.jar", "namingclient.jar", "properties.jar", "txClientPrivate.jar", "ras.jar", "wsexception.jar", "bootstrap.jar", "ecutils.jar", "iwsorb.jar", "ffdc.jar", "idl.jar", "txClient.jar"

The solution involved emulating the environment to let the IBM RMI-implementation believe it was running in a standard client JVM without knowing about BEA RMI and vica versa. I therefore created an IsolatingClassLoader that knew how to load the IBM client jars from the filesystem and not from the classpath. The same ClassLoader also knew what packages and classes it was supposed to isolate and to do the loading itself instead of delegating it to the parent ClassLoader.

  public class IsolatingClassLoader extends URLClassLoader {
    ...
    ...
    protected synchronized Class loadClass(String name, boolean resolve) throws ClassNotFoundException {
      boolean isolated = isolatedClassNames.contains(name);
      if (!isolated) {
        for (int i = 0; i < isolatedPackagePrefixes.length; i++) {
          if (name.startsWith(isolatedPackagePrefixes[i])) {
            isolated = true;
            break;
          }
        }
      }
      if (isolated) {
        Class c = findLoadedClass(name);
        if (c == null) c = findClass(name);
        if (resolve) resolveClass(c);
        return c;
      }
    return super.loadClass(name, resolve);
  }
}

By doing this I isolated all com.ibm. packages and classes. Before doing the lookups I had to substitute the context ClassLoader of the current thread to ensure that classes was loaded with the IsolatingClassLoader and register the thread to ensure the correct UtilDelegate is used (More on that below). This was restored and unregistered in a finally clause right after doing the lookup and narrowing.

ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
Thread.currentThread().setContextClassLoader(isolatingClassLoader);
ThreadRegistry.getInstance().register();
try {
  InitialContext ctx = new InitialContext(jndiConfig);
  Object o = ctx.lookup(”iiop://wasserverhostname:2809/cell/clusters/mycluster/ejb/com/path/to/ejb/SomeEJBHome”);
  SomeEJBHome home = (SomeEJBHome) PortableRemoteObject.narrow(o, SomeEJBHome.class);
  return (SomeEJBRemoteHome) Proxy.newProxyInstance(getClass().getClassLoader(), new Class[] \\
     { SomeEJBRemoteHome.class }, new IBMIvocationHandler(home.create(), isolatingClassLoader));
} finally {
  ThreadRegistry.getInstance().unregister();
  Thread.currentThread().setContextClassLoader(oldClassLoader);
}

The java.naming configuration also needed modification to use IBM’s implementation and using the constructor of the InitialContext class I supplied my own naming configuration.

As you can see from the lookup code I was forced to not return the actual RemoteHome object but actually a proxied version. Inside the IBMInvocationHandler I had to do the ContextClassLoader manipulation and thread registering again to ensure that IBM classes was available during the remote EJB invocation.

public class IBMIvocationHandler implements InvocationHandler {
  private ClassLoader isolatingClassLoader;
  private Object remoteStub;

  public MetroIvocationHandler(Object remoteStub, ClassLoader isolatingClassLoader) {
    this.remoteStub = remoteStub;
    this.isolatingClassLoader= isolatingClassLoader;
  }

  public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
    ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
    Thread.currentThread().setContextClassLoader(isolatingClassLoader);
    ThreadRegistry.getInstance().register();
    try {
      return method.invoke(remoteStub, args);
    } finally {
      ThreadRegistry.getInstance().unregister();
      Thread.currentThread().setContextClassLoader(oldClassLoader);
    }
  }
}

We have soon reached our goal, but one nasty obstacle is standing between ourselves and a solution. That is the javax.rmi.CORBA.Util class that holds a private reference to a singleton implementing the javax.rmi.CORBA.UtilDelegate inteface providing utility methods that can be used by stubs and ties to perform common operations. The implementation is defined by a system property (javax.rmi.CORBA.UtilClass) and is initialized during VM startup. We could simply have replaced the private field containting the UtilDelegate with IBM’s implementation (com.ibm.ws.orb.WSUtilDelegateImpl), but then BEA’s RMI-implementation would get into trouble and we would see some really unbelievable exceptions ;-). Instead I created a proxy version of the UtilDelegate that chooses between the original and IBM’s implementation depending on which RMI-implementation that was using it. The ThreadRegistry uses a WeakHashMap to store the current Thread to identify which Thread is coming through IBM code or BEA RMI code.

public class UtilDelegateInvocationHandler implements InvocationHandler {
  private UtilDelegate original;
  private UtilDelegate ibm;

  public UtilDelegateInvocationHandler(UtilDelegate original, UtilDelegate ibm) {
    this.original = original;
    this.ibm = ibm;
  }

  public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
    iif(ThreadRegistry.getInstance().isRegistered()) {
        return method.invoke(ibm, args);
      else
       return method.invoke(original, args);
  }

}

The proxied UtilDelegate was installed using the reflection API and required that Java2 Security allowed my class to modify the accessible flags.

UtilDelegate ibmUD = (UtilDelegate) \\
      isolatingClassLoader.loadClass("com.ibm.ws.orb.WSUtilDelegateImpl").newInstance();
Class c = Class.forName("javax.rmi.CORBA.Util");
Field ff = c.getDeclaredField("utilDelegate");
ff.setAccessible(true);
UtilDelegate originalUD = (UtilDelegate) ff.get(null);
UtilDelegate proxyUtilDelegate = (UtilDelegate) Proxy.newProxyInstance(getClass().getClassLoader(), \\
      new Class[] { UtilDelegate.class }, new UtilDelegateInvocationHandler(originalUD, ibmUD));
ff.set(null, proxyUtilDelegate);

Conclusion

After doing this we could successfully use both RMI-implementations interchangably inside the WLS server and finally invoke EJBs running inside a WAS cluster with the security attribute propagation enabled.

Source Code

You may download the complete source code: was-client-inside-wls-updated-src

Update: 2. May, 2008

After beeing challange by Asgeir Nilsen I came up with a much improved way of determining which UtilDelegate implementation to use. In the old version I parsed the stack trace, but in this new and improved version I register the Thread in a singleton instead. The source archive has been updated to reflect this change.