Determining the master node in a JBoss Cluster

Thanks to the original author and the source

How to determine which node in the cluster is the master node?

Have you ever dealt with clustered singleton service? How to determine which cluster node is the master? Well, if I am the current node, I can simply ask whether I am the master or not. But what if I already know that the current node is not the master, and I want to determine which node among other nodes in the cluster is the master?

First I would like to give brief summary about HASingleton service (HA stands for High Availability).

Summary:
HASingleton service, is a service that is deployed on every node in a cluster, but runs only on one node, while the other nodes remain passive. The node that the service runs on, is the master node.

How does JBoss selects the master node?
Well the first node in a cluster will become master node. If existing master node will leave the cluster as a result of a shutdown for example, another node is selected as master from the remaining nodes.

Master node can control which tasks will get executed, and how many times. HASingletons also have the ability to share a memory state across clustered partition. Something like caching ...

Solution:
Lets assume that I have a service bean that extends from HASingletonSupport class. HASingletonSupport in its turn extends from HAServiceMBeanSupport
and implements two interfaces: HASingletonMBean and HASingleton. All of them give me those wonderful APIs that can tell me whether the current node is the master or not, what the status of my cluster, how many nodes etc. etc.

1. public class MyHAService extends HASingletonSupport implements
2. MyHAServiceMBean {
3. 
   
4. private static Logger logger =
5. Logger.getLogger(MyHAService.class);
6. 
   
7. public void startService() throws Exception {
8. logger.info(" *** STARTED MY SINGLETON SERVICE *** ");
9. super.startService();
10. }
11. 
    
12. public void stopService() throws Exception {
13. logger.info(" *** STOPPED MY SINGLETON SERVICE *** ");
14. super.stopService();
15. }
16. 
    
17. public boolean isMasterNode() {
18. return super.isMasterNode();
19. }
20. 
    
21. 
    
22. public void startSingleton() {
23. logger.info(" *** CURRENT NODE IP:"
24. + this.getPartition().getClusterNode()
25. .getIpAddress().getHostAddress() +
26. " ELECTED AS A MASTER NODE *** ");
27. }
28. 
    
29. 
    
30. public void stopSingleton() {
31. logger.info(" *** CURRENT NODE IP:"
32. + this.getPartition().getClusterNode()
33. .getIpAddress().getHostAddress()
34. + " STOPPED ACTING AS A MASTER NODE *** ");
35. }
36. 
    
37. 
    
38. public void partitionTopologyChanged(List newReplicants, int newViewID) {
39. logger.info(" *** TOPOLOGY CHANGE STARTING *** ");
40. super.partitionTopologyChanged(newReplicants, newViewID);
41. 
    
42. }
43. }

public class MyHAService extends HASingletonSupport implements
      MyHAServiceMBean {

private static Logger logger =
  Logger.getLogger(MyHAService.class);

public void startService() throws Exception {
      logger.info(" *** STARTED MY SINGLETON SERVICE *** ");
      super.startService();
}

public void stopService() throws Exception {
      logger.info(" *** STOPPED MY SINGLETON SERVICE *** ");
      super.stopService();
}

public boolean isMasterNode() {
      return super.isMasterNode();
}


public void startSingleton() {
      logger.info(" *** CURRENT NODE IP:"
              + this.getPartition().getClusterNode()
   .getIpAddress().getHostAddress() +
   " ELECTED AS A MASTER NODE *** ");
}


public void stopSingleton() {
      logger.info(" *** CURRENT NODE IP:"
              + this.getPartition().getClusterNode()
   .getIpAddress().getHostAddress()
              + " STOPPED ACTING AS A MASTER NODE *** ");
}


public void partitionTopologyChanged(List newReplicants, int newViewID) {
      logger.info(" *** TOPOLOGY CHANGE STARTING *** ");
      super.partitionTopologyChanged(newReplicants, newViewID);

 }
}

startSingleton() - invoked when the current node elected as a master.
stopSingleton() - invoked when the current node stops acting as a master.
partitionTopologyChanged() - invoked when new node joins or leaves the cluster.

As i mentioned before, I have the ability to know whether the current node is the master node, by calling isMasterNode(). The method will return true if the node is master and false if its not.

In case I already know that the current node is not the master, I can ask the clustered partition (the cluster) which node is the master. For example I can request the current view of my cluster.

The implementation can be similar to the method below which you can have inside your service bean:

1. private String getMasterSocket() {
2. 
   
3. HAPartition partition = this.getPartition();
4. 
   
5. if (partition != null) {
6. 
   
7. if (partition.getCurrentView() != null) {
8. return partition.getCurrentView().get(0).toString();
9. 
   
10. } else {
11. return null;
12. }
13. } else {
14. return null;
15. }
16. }

private String getMasterSocket() {

      HAPartition partition = this.getPartition();

      if (partition != null) {

          if (partition.getCurrentView() != null) {
              return partition.getCurrentView().get(0).toString();

          } else {
              return null;
          }
      } else {
          return null;
      }
  }

The method above will return me a string contains port and ip of the master node, for example:

192.168.62.12:1099

The HAPartition service maintains across cluster a registry of nodes in a view order. Now, keep in mind that an order of the nodes in the view, does not necessary reflect the order nodes have joined the cluster.

So the first node in the view as you can see beloew, will be the master node.
Simple as that.

return partition.getCurrentView().get(0).toString();

Please note:
Method getPartition() may return null, if super.startService() hasnt been called. Have a look at implementation of HAServiceMBeanSupport and my other post JBoss Clustering - How many nodes in the cluster?.

Finding the number of cluster nodes in JBoss

Thanks to the Original Author and source

How to find out how many nodes in the cluster

If you want to know how many nodes there are in the current cluster partition, all you have to do is to ask HAPartition for the node list. HAPartition represents your cluster partition, and it contains all the information you need to know about your cluster and the nodes: their host names, IPs, position in the cluster view.

Lets assume you have a service bean that extends from HASingletonSupport. HASingletonSupport in its turn extends from HAServiceMBeanSupport.

HAServiceMBeanSupport is the one who gives you access to HAPartition object.

The code to request for HAPartition object and node list that you see below , you can put somewhere in your service bean:

HAPartition partition = getPartition();
ClusterNode[] nodes = partition.getClusterNodes();
System.out.println(nodes.length);

ClusterNode object represents your node in the cluster. It contains information about node's host name, its internet address and a few more things. getClusterNodes(), returns to you an array contains as many ClusterNode objects as you have currently in your cluster. So by getting the value of array length, you will know how many nodes your cluster has.

Another way, is to do practically the same, but to request from a HAPartition a current view of your cluster:

HAPartition partition = getPartition();
Vector v = partition.getCurrentView();

System.out.println(partition.getCurrentView().size());

for (Object o : v) {
  System.out.println(o.toString());
}

The view, which is a Vector contains information about node sockets. When printed, it will return to you a String representation of node ip + port: xxx.xxx.xxx.xxx:port. Also by printing size of the Vector, you will get number of nodes in the cluster.

Important note:
I noticed there is some delay happening from the time when node leaves the cluster to the time when HAPartition returns an updated view. In another words - after node has left the cluster and topology change has occurred, the HAPartition may return to you an old view still containing the dead node. So be careful.

Also, getPartition() may return null, if super.startService() hasnt been called. Have a look at implementation of HAServiceMBeanSupport and my other post JBoss Clustering - HASingleton service.

JBoss Clustering

Thanks to the original author and Source

Singleton Service

A clustered singleton service is deployed on multiple nodes in a cluster but runs on only one of the nodes. The node running the singleton service is typically called the master node. When the master fails, another master is selected from the remaining nodes and the service is restarted on the new master.

Figure 1. Clustered singleton service

Many times, an application needs to ensure that a certain task is run exactly once. Thus, only one of the nodes in a cluster should execute the task. The other nodes should knowingly remain passive. Examples of singleton tasks include:

• Sending email to the system administrator when the system is brought up or taken down. This notification does not take into account how many nodes are in the cluster. As long as at least one node is active, the system is up. When all nodes are down, the system is down.
• Database schema validation upon startup. When a database-driven application is brought up, it is a good practice for the middle tier servers to verify whether the version of the business logic they implement matches the database schema.
• Sending recurring notifications to system users. For example, a calendar application might send out email prior to each instance of a scheduled recurring meeting.
• Load balancing of queued tasks. It's popular to use a single coordinator that distributes tasks among nodes in a cluster.
• Fault tolerance. If an application uses a distributed cache, it is common to designate a single master node responsible for maintaining a current copy of distributed states. The other nodes make requests of the current master node. When the master fails, another node takes over its responsibilities.

While it is fairly easy to implement such singleton tasks in a single VM, the solution will usually not work immediately in a clustered environment. Even in the simple case of a task activated upon startup on one of the nodes in a two-node cluster, several problems must be addressed:

• When the application is started simultaneously on both nodes, which VM should run the singleton task?
• When the application is started on one node and then started later on another, how does the second node know not to run the singleton task again?
• When the node that started the task fails, how does another node know to resume the task?
• When the node that started the task fails, but later recovers, how do we ensure that the task remains running on only one of the nodes?

The logic to solve these problems is unlikely to be included in the design of a single-VM solution. However, a solution can be found to address the case at hand and it can be patched on the startup task. This is an acceptable approach for a few startup tasks and two-node clusters.

As the application grows and becomes more successful, more startup tasks may be necessary. The application may also need to scale to more than two nodes. The clustered singleton problem can quickly become mind-boggling for larger clusters, where the different node startup scenarios are far more difficult to enumerate than in the two-node case. Another complicating factor is communication efficiency. While two nodes can directly connect to each other and negotiate, 10 nodes will have to establish 45 total connections to use the same technique.

This is where JBoss comes in handy. It eliminates most of the complexity and allows application developers to focus on building singleton services regardless of the cluster topology.

We will illustrate how the JBoss clustered singleton facility works with an example. First, we will need a service archive descriptor. Let's use the one that ships with JBoss under server/all/farm/cluster-examples-service.xml. The following is an excerpt:

<!--    | This MBean is an example of a cluster Singleton    -->
<mbean code="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
         name="jboss.examples:service=HASingletonMBeanExample">
</mbean>
<!- - -->
  <!--    | This is a singleton controller which works similarly to the
          | SchedulerProvider (when a MBean target is used)    -->
  <mbean code="org.jboss.ha.singleton.HASingletonController"
         name="jboss.examples:service=HASingletonMBeanExample-HASingletonController">
     <depends>jboss:service=DefaultPartition</depends>
     <depends>jboss.examples:service=HASingletonMBeanExample</depends>
     <attribute name="TargetName">jboss:service=HASingletonMBeanExample</attribute>
     <attribute name="TargetStartMethod">startSingleton</attribute>
     <attribute name="TargetStopMethod">stopSingleton</attribute>
  </mbean>
<!- - -->

This file declares two MBeans, HASingletonMBeanExample and HASingletonController. The first one is a singleton service that contains the custom code. It is a simple JavaBean with the following source code:

public class HASingletonMBeanExample
 implements HASingletonMBeanExampleMBean {

   private boolean isMasterNode = false;

   public void startSingleton() {
       isMasterNode = true;
   }

   public boolean isMasterNode() {
       return isMasterNode; 
   }

   public void stopSingleton() {
       isMasterNode = false; 
   }
}

All of the custom logic for this particular singleton service is contained within this class. Our example is not too useful; it simply indicates, via the isMasterNode member variable, whether the master node is running the singleton. This value will be true only on the one node in the cluster where it is deployed.

HASingletonMBeanExampleMBean exposes this variable as an MBean attribute. It also exposes startSingleton() and stopSingleton() as managed MBean operations. These methods control the lifecycle of the singleton service. JBoss invokes them automatically when a new master node is elected.

How does JBoss control the singleton lifecycle throughout the cluster? The answer to this question is in the MBean declarations. Notice that the HASingletonMBeanExample-HASingletonController MBean also takes the name of the sample singleton MBean and its start and stop methods.

On each node in the cluster where these MBeans are deployed, the controller will work with all of the other controllers with the same MBean name deployed in the same cluster partition to oversee the lifecycle of the singleton. The controllers are responsible for tracking the cluster topology. Their job is to elect the master node of the singleton upon startup, as well as to elect a new master should the current one fail or shut down. In the latter case, when the master node shuts down gracefully, the controllers will wait for the singleton to stop before starting another instance on the new master node.

A singleton service is scoped in a certain cluster partition via its controller. Notice that, in the declaration above, the controller MBean depends on the MBean service DefaultPartition. If the partition where the singleton should run is different than the default, its name can be provided to the controller via the MBean attribute PartitionName.

Clustered singletons are usually deployed via the JBoss farming service. To test this example, just drop the service file above in the server/all/farm directory. You should be able to see the following in the JBoss JMX web console:

Figure 2. Controller MBean view. The MasterNode attribute will have value True on only one of the nodes.

Figure 3. Sample singleton MBean view. The MasterNode attribute will have the same value as the MasterNode attribute on the controller MBean.

Brief on JBoss EAR Deployment

EAR Deployment Process:

1) The EAR starts getting deployed and EARDeployer is the main class that does that.

2) All the dependent jars are scanned and deployed if not deployed. MainDeployer is the main class that does this.

3) The Queue Sevice and Topic Service will be started if there are any topics or queues defined in the Application. TopicService and QueueService are the main class that does this.

4) EJB3Deployment begins, which deploys Enterprise beans and Persistence Units in the application or module. EJB3Deployer iterates through the ear to find all the jars which has persistence units and enterprise beans for deployment.

a) If a persistence unit is detected, first an corresponding MDB will be created and registered into JMXServer. Similarly if an EJB is detected, a corresponding MDB is created and registered into JMX. JmxKernelAbstraction is the main class that does this.

b) Once PersistenceUnitMDB is registered, the service is started to create all the relevant tables n EntityManager relationships etc into db and PersistenceUnitDeployment is the class that does this.

c) Similarly once EJBMDB is registered, the service is started using EJBContainer class.

5) Once EJB Deployment is done, TomcatDeployer will deploy the web application if any in the particular module being scanned.

6) Cluster Session Management is started for this particular application, JBossCacheManager is the one responsible for this.

7) Once all the above steps are done, the application can have application specific database initializers, inventory trackers, etc started.


Before the EAR Deployment begins, all the services starting from Transaction Service, Timer Service, UDDI, WebService, etc would have started.
All the JBoss related Webapps like jmx-console, web-console will start. Once all these JBoss goes ahead with the EAR Deployment.

JBoss Lookup

Thanks to the original author.
You can find the original source here


Many a times when you are doing a lookup in the JNDI tree, you see javax.naming.NameNotFoundException. A simple code that does the lookup will look something like:

1. Context ctx = new InitialContext();
2. Object obj = ctx.lookup("somepath/somename");

Context ctx = new InitialContext(); Object obj = ctx.lookup("somepath/somename"); 

This code just looks up the JNDI tree to get an object bound by the name "somepath/somename". Looks simple. However, chances are that you might even see this exception:

1. javax.naming.NameNotFoundException: somepath not bound
2. at org.jnp.server.NamingServer.getBinding(NamingServer.java:529)
3. at org.jnp.server.NamingServer.getBinding(NamingServer.java:537)
4. at org.jnp.server.NamingServer.getObject(NamingServer.java:543)
5. at org.jnp.server.NamingServer.lookup(NamingServer.java:267)
6. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
7. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
8. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
9. at java.lang.reflect.Method.invoke(Method.java:585)
10. at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
11. at sun.rmi.transport.Transport$1.run(Transport.java:153)
12. at java.security.AccessController.doPrivileged(Native Method)
13. at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
14. at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
15. at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
16. at java.lang.Thread.run(Thread.java:595)
17. at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)
18. at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223)
19. at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126)
20. at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
21. at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:625)
22. at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:587)
23. at javax.naming.InitialContext.lookup(InitialContext.java:351)

javax.naming.NameNotFoundException: somepath not bound  at org.jnp.server.NamingServer.getBinding(NamingServer.java:529)  at org.jnp.server.NamingServer.getBinding(NamingServer.java:537)  at org.jnp.server.NamingServer.getObject(NamingServer.java:543)  at org.jnp.server.NamingServer.lookup(NamingServer.java:267)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  at java.lang.reflect.Method.invoke(Method.java:585)  at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)  at sun.rmi.transport.Transport$1.run(Transport.java:153)  at java.security.AccessController.doPrivileged(Native Method)  at sun.rmi.transport.Transport.serviceCall(Transport.java:149)  at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)  at java.lang.Thread.run(Thread.java:595)  at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)  at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223)  at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126)  at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:625)  at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:587)  at javax.naming.InitialContext.lookup(InitialContext.java:351) 

Look closely at the stacktrace. It shows that while looking up the JNDI tree it could not find the jndi name "somepath" (this name may vary). The reason is simple, the JNDI tree does not have any object bound by this name.

To quote the javadocs of this exception "This exception is thrown when a component of the name cannot be resolved because it is not bound."

So how do i know, what's the name to which my object is bound? Each application server, usually provides a JNDI view which can be used to see the contents of the JNDI tree. If you know what object you are looking for (ex: the name of the bean), then you can traverse this JNDI tree to see what name it is bound to. The JNDI view is specific to every application server.

To give an example, JBoss provides its JDNI tree view, through the JMX console. Here are the steps, one has to follow to check the JNDI tree contents on JBoss:

- Go to http://<>:<>/jmx-console (Ex: http://localhost:8080/jmx-console)
- Search for service=JNDIView on the jmx-console page
- Click on that link
- On the page that comes up click on the Invoke button beside the list() method
- The page that comes up will show the contents of the JNDI tree.

Here's an sample of how the output looks like(just a small part of the entire output):

1. java: Namespace
2. 
   
3. +- XAConnectionFactory (class: org.jboss.mq.SpyXAConnectionFactory)
4. +- DefaultDS (class: org.jboss.resource.adapter.jdbc.WrapperDataSource)
5. +- SecurityProxyFactory (class: org.jboss.security.SubjectSecurityProxyFactory)
6. +- DefaultJMSProvider (class: org.jboss.jms.jndi.JNDIProviderAdapter)
7. +- comp (class: javax.naming.Context)
8. +- JmsXA (class: org.jboss.resource.adapter.jms.JmsConnectionFactoryImpl)
9. +- ConnectionFactory (class: org.jboss.mq.SpyConnectionFactory)
10. +- jaas (class: javax.naming.Context)
11. | +- dukesbank (class: org.jboss.security.plugins.SecurityDomainContext)
12. | +- HsqlDbRealm (class: org.jboss.security.plugins.SecurityDomainContext)
13. | +- jbossmq (class: org.jboss.security.plugins.SecurityDomainContext)
14. | +- JmsXARealm (class: org.jboss.security.plugins.SecurityDomainContext)
15. 
    
16. 
    
17. Global JNDI Namespace
18. 
    
19. +- ebankTxController (proxy: $Proxy79 implements interface com.sun.ebank.ejb.tx.TxControllerHome,interface javax.ejb.Handle)
20. +- ebankAccountController (proxy: $Proxy75 implements interface com.sun.ebank.ejb.account.AccountControllerHome,interface javax.ejb.Handle)
21. +- TopicConnectionFactory (class: org.jboss.naming.LinkRefPair)
22. +- jmx (class: org.jnp.interfaces.NamingContext)
23. | +- invoker (class: org.jnp.interfaces.NamingContext)
24. | | +- RMIAdaptor (proxy: $Proxy48 implements interface org.jboss.jmx.adaptor.rmi.RMIAdaptor,interface org.jboss.jmx.adaptor.rmi.RMIAdaptorExt)
25. | +- rmi (class: org.jnp.interfaces.NamingContext)
26. | | +- RMIAdaptor[link -> jmx/invoker/RMIAdaptor] (class: javax.naming.LinkRef)
27. +- HTTPXAConnectionFactory (class: org.jboss.mq.SpyXAConnectionFactory)
28. +- ConnectionFactory (class: org.jboss.mq.SpyConnectionFactory)
29. +- ebankCustomer (proxy: $Proxy67 implements interface com.sun.ebank.ejb.customer.LocalCustomerHome)
30. +- UserTransactionSessionFactory (proxy: $Proxy14 implements interface org.jboss.tm.usertx.interfaces.UserTransactionSessionFactory)
31. +- ebankCustomerController (proxy: $Proxy77 implements interface com.sun.ebank.ejb.customer.CustomerControllerHome,interface javax.ejb.Handle)
32. +- HTTPConnectionFactory (class: org.jboss.mq.SpyConnectionFactory)
33. +- XAConnectionFactory (class: org.jboss.mq.SpyXAConnectionFactory)
34. +- TransactionSynchronizationRegistry (class: com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionSynchronizationRegistryImple)
35. +- ebankAccount (proxy: $Proxy68 implements interface com.sun.ebank.ejb.account.LocalAccountHome)
36. +- UserTransaction (class: org.jboss.tm.usertx.client.ClientUserTransaction)
37. +- UILXAConnectionFactory[link -> XAConnectionFactory] (class: javax.naming.LinkRef)
38. +- UIL2XAConnectionFactory[link -> XAConnectionFactory] (class: javax.naming.LinkRef)
39. +- queue (class: org.jnp.interfaces.NamingContext)
40. | +- A (class: org.jboss.mq.SpyQueue)
41. | +- testQueue (class: org.jboss.mq.SpyQueue)
42. | +- ex (class: org.jboss.mq.SpyQueue)
43. | +- DLQ (class: org.jboss.mq.SpyQueue)
44. | +- D (class: org.jboss.mq.SpyQueue)
45. | +- C (class: org.jboss.mq.SpyQueue)
46. | +- B (class: org.jboss.mq.SpyQueue)

java: Namespace    +- XAConnectionFactory (class: org.jboss.mq.SpyXAConnectionFactory)   +- DefaultDS (class: org.jboss.resource.adapter.jdbc.WrapperDataSource)   +- SecurityProxyFactory (class: org.jboss.security.SubjectSecurityProxyFactory)   +- DefaultJMSProvider (class: org.jboss.jms.jndi.JNDIProviderAdapter)   +- comp (class: javax.naming.Context)   +- JmsXA (class: org.jboss.resource.adapter.jms.JmsConnectionFactoryImpl)   +- ConnectionFactory (class: org.jboss.mq.SpyConnectionFactory)   +- jaas (class: javax.naming.Context)   |   +- dukesbank (class: org.jboss.security.plugins.SecurityDomainContext)   |   +- HsqlDbRealm (class: org.jboss.security.plugins.SecurityDomainContext)   |   +- jbossmq (class: org.jboss.security.plugins.SecurityDomainContext)   |   +- JmsXARealm (class: org.jboss.security.plugins.SecurityDomainContext)    Global JNDI Namespace    +- ebankTxController (proxy: $Proxy79 implements interface com.sun.ebank.ejb.tx.TxControllerHome,interface javax.ejb.Handle)   +- ebankAccountController (proxy: $Proxy75 implements interface com.sun.ebank.ejb.account.AccountControllerHome,interface javax.ejb.Handle)   +- TopicConnectionFactory (class: org.jboss.naming.LinkRefPair)   +- jmx (class: org.jnp.interfaces.NamingContext)   |   +- invoker (class: org.jnp.interfaces.NamingContext)   |   |   +- RMIAdaptor (proxy: $Proxy48 implements interface org.jboss.jmx.adaptor.rmi.RMIAdaptor,interface org.jboss.jmx.adaptor.rmi.RMIAdaptorExt)   |   +- rmi (class: org.jnp.interfaces.NamingContext)   |   |   +- RMIAdaptor[link -> jmx/invoker/RMIAdaptor] (class: javax.naming.LinkRef)   +- HTTPXAConnectionFactory (class: org.jboss.mq.SpyXAConnectionFactory)   +- ConnectionFactory (class: org.jboss.mq.SpyConnectionFactory)   +- ebankCustomer (proxy: $Proxy67 implements interface com.sun.ebank.ejb.customer.LocalCustomerHome)   +- UserTransactionSessionFactory (proxy: $Proxy14 implements interface org.jboss.tm.usertx.interfaces.UserTransactionSessionFactory)   +- ebankCustomerController (proxy: $Proxy77 implements interface com.sun.ebank.ejb.customer.CustomerControllerHome,interface javax.ejb.Handle)   +- HTTPConnectionFactory (class: org.jboss.mq.SpyConnectionFactory)   +- XAConnectionFactory (class: org.jboss.mq.SpyXAConnectionFactory)   +- TransactionSynchronizationRegistry (class: com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionSynchronizationRegistryImple)   +- ebankAccount (proxy: $Proxy68 implements interface com.sun.ebank.ejb.account.LocalAccountHome)   +- UserTransaction (class: org.jboss.tm.usertx.client.ClientUserTransaction)   +- UILXAConnectionFactory[link -> XAConnectionFactory] (class: javax.naming.LinkRef)   +- UIL2XAConnectionFactory[link -> XAConnectionFactory] (class: javax.naming.LinkRef)   +- queue (class: org.jnp.interfaces.NamingContext)   |   +- A (class: org.jboss.mq.SpyQueue)   |   +- testQueue (class: org.jboss.mq.SpyQueue)   |   +- ex (class: org.jboss.mq.SpyQueue)   |   +- DLQ (class: org.jboss.mq.SpyQueue)   |   +- D (class: org.jboss.mq.SpyQueue)   |   +- C (class: org.jboss.mq.SpyQueue)   |   +- B (class: org.jboss.mq.SpyQueue) 

Let's see what this tells us. Let's consider the Global JNDI Namespace first. It contains (among other things) the following:

1. +- ebankTxController (proxy: $Proxy79 implements interface com.sun.ebank.ejb.tx.TxControllerHome,interface javax.ejb.Handle)

+- ebankTxController (proxy: $Proxy79 implements interface com.sun.ebank.ejb.tx.TxControllerHome,interface javax.ejb.Handle) 

This tells me that an object which implements com.sun.ebank.ejb.tx.TxControllerHome and javax.ejb.Handle interfaces is bound to the JNDI tree by the jndi-name "ebankTxController". So if at all i have to lookup this object, my lookup code would be something like:

view plaincopy to clipboardprint?

1. Context ctx = new InitialContext();
2. ctx.lookup("ebankTxController");

Context ctx = new InitialContext(); ctx.lookup("ebankTxController"); 

Similarly in the same Global JDNI Namespace, we see :

1. +- queue (class: org.jnp.interfaces.NamingContext)
2. | +- A (class: org.jboss.mq.SpyQueue)
3. 
   

+- queue (class: org.jnp.interfaces.NamingContext)   |   +- A (class: org.jboss.mq.SpyQueue)    

Make note of the nesting of the names here. This tells me that an object of type org.jboss.mq.SpyQueue is bound by the name "A under the path queue". So your lookup for this object should look like:

1. Context ctx = new InitialContext();
2. ctx.lookup("queue/A");

Context ctx = new InitialContext(); ctx.lookup("queue/A"); 

Now let's move on to the java: namespace in the JNDI tree view above. The difference between a Global JNDI namespace and the java: namespace is that, the object bound in the java: namespace can be looked-up ONLY by clients within the SAME JVM. Whereas, in case of Global JNDI namespace, the objects bound in this namespace can be looked-up by clients, even if they are not in the same JVM as the server. One would ask, how does this matter? Consider a standalone java program(client) which tries to lookup some object on the server (running in its own JVM). Whenever a standalone client is started (using the java command), a new JVM is instantiated. As a result, the server (which is started in its own JVM) and the client are running on different JVMs. Effectively, the client will NOT be able to lookup objects bound in the java: namespace of the server. However, the client can lookup the objects present in the Global JNDI namespace of the server.

So, why are we discussing these details, in a topic which was meant to explain the NameNotFoundException? Let's consider the java: namespace output above. There's a

1. +- DefaultDS (class: org.jboss.resource.adapter.jdbc.WrapperDataSource)

+- DefaultDS (class: org.jboss.resource.adapter.jdbc.WrapperDataSource) 

This tells me that there's an object bound to the name DefaultDS in the java: namespace. So my lookup code would be:

1. Context ctx = new InitialContext();
2. ctx.lookup("java:/DefaultDS");

Context ctx = new InitialContext(); ctx.lookup("java:/DefaultDS"); 

As explained above, this code is going to return you the object, if this piece of code runs in the same JVM as the server. However, if this piece of code is run from a client in different JVM (maybe a standalone client), then it's going to run into NameNotFoundException. The reason i explained the java: and the Global JNDI namespace is that, sometimes people are surprised that even though the JNDI view shows that the object is bound in the java: namespace(with the same name as the one they pass to the lookup method), they still run into NameNotFoundException. The probable reason might be, the client is in a different JVM.

Advanced JBoss Class Loading

Advanced JBoss Class Loading

Introduction

One of the main concerns of a developer writing hot re-deployable JBoss applications is to understand how JBoss class loading works. Within the internals of the class loading mechanism lies the answer to questions like

• What happens if I pack a newer version of an utility library with my application, while an older version of the same library lingers somewhere in the server's lib directory?

• How can I use two different versions of the same utility library, simultaneously, within the same instance of the application server?

• What version of an utility class I am currently using?

or even the most fundamental of them all

• Why do I need to mess with all this class loading stuff anyway?

This article tries to provide the reader with the knowledge required to answer these questions. It will start by trying to answer the last one, and then it will present several often encountered use cases and explain the behavior of the JBoss class loading mechanism when faced with those situations.

The Need for Class Loaders and Class Loading Management

Class Namespace Isolation

An application server should ideally give its deployed applications the freedom to use whatever utility library and whatever version of the library they see fit, regardless of the presence of concurrent applications that want to use the same library. This is mandated by the J2EE specifications, which calls it class namespace isolation (Java EE 5 Specifications, Section EE.8.4). The fact that different applications load their classes in different class name spaces or class loading domains allow them to do just that: run whatever class version they like, oblivious to the fact that their neighbors use the same class.

Java doesn't provide the formal notion of class version. So how is it possible to implement a class loading domain? The runtime identity of a class in Java 2 is defined by the fully qualified class name and its defining class loader. This means that the same class, loaded by two different class loaders, is seen by the Virtual Machine as two completely different types.

If you like history, you probably know that this wasn't always the case. In Java 1.1 the runtime identity of a class was defined only by its fully qualified class name. That made Vijay Saraswat declare in 1997 that "[Java is not type-safe|http://matrix.research.att.com/vj/bug.html]" and Sheng Liang and Gilad Bracha fixed it by strengthening the type system to include a class's defining class loader in addition to the name of the class to fully define the type. This is good and ... not so good. It is good because now its not possible anymore that a rogue class loader would re-define your "java.lang.String" class. The VM will detect that and throw a ClassCastException. It is also good because now it is possible to have class loading domains within the same VM. Not so good however, is the fact that passing an object instance by reference between two class loading domains is not possible. Doing so results in the dreaded ClassCastException. If you would like to know more details about how this happens, please follow this link. Not being able to pass an Object by reference means you have to fall back on serialization and serialization means performance degradation.

Hot Redeployment

Returning to application servers, the need for class loaders becomes probably obvious: this is how an application server implements class namespace isolation: each application gets its own class loader at deployment, and hence, its own "version" of classes.

To extend this even more, it would be nice if we could re-deploy an application (i.e. instantiate a newer version of a class), at run-time, without necessarily bringing down the VM and re-starting it to reload the class. That would mean 24x7 uptime. Java 4 doesn't intrinsically support the concept of hot re-deployment. Once a dependent class reference was added to the runtime constant pool of a class, it is not possible to drop that reference anymore.

However, it is possible to trick the server (or the VM) into doing this. If application A interacts with application B, but doesn't have any direct references to the B classes, and B changes, let's say a newer and better version becomes available, it is possible to create a new class loader, load the new B classes and have the invocation bus (the application server) route all invocations from A to the new B classes. This way, A deals with a new version of B without even knowing it. If the application server is careful to drop all explicit references to the old B classes, they will eventually be garbage collected and the old B will eventually disappear from the system.

Sharing Classes

Isolating class loading domains is nice. Our applications will run happily and safe. But very slowly, when it comes to interacting with each other. This is because each interaction involves passing arguments by value, which means serialization, which means overhead.

We're sometimes (actually quite often) faced with the situation where in we would like to allow applications to share classes. We know precisely, for example, that in our environment, application A and B, otherwise independent, will always use the same version of the utility library and doing so, they could pass references among themselves without any problem. The added benefit in this case is that the invocations will be faster, given the fact serialization is cut out.

One word of caution though. In this situation, if we hot redeploy the utility library, we also must re-deploy the application A and B: the current A and B classes used direct references to the utility classes, so they're tainted forever, they won't ever be able to use the new utility classes.

JBoss makes possible for applications to share classes. JBoss 3.x does that by default. JBoss 4.0 does this for the "standard" configuration, but maintains class namespace isolation between applications for its "default" configuration. JBoss 4.0.1 reverts to the 3.x convention.

Class Repositories or How JBoss Class Loading Works

JBoss makes sharing classes possible by introducing the concept of class loader repository. The central piece of the class loading mechanism is the org.jboss.mx.loading.UnifiedClassLoader3 (UCL). UnifiedClassLoader3 extends URLClassLoader. Each UCL is associated with a shared repository of classes and resources, usually an instance of org.jboss.mx.loading.UnifiedLoaderRepository3. Every UCL is associated with a single instance of UnifiedLoaderRepository3, but a repository can have multiple UCLs. A UCL may have multiple URLs associated with it for class and resource loading. By default, there is a single UnifiedLoaderRepository3 shared across all UCL instances. UCLs form a single flat class namespace.

The class loader parent for each UCL is a NoAnnotationClassLoader instance. The NoAnnotationClassLoader extends URLClassLoader. A singleton NoAnnotationClassLoader instance is created during the server's boot process and its job is to define classes available in the $JBOSS_HOME/lib libraries (ex: commons-logging.jar, concurrent.jar, dom4j.jar, jboss-common.jar, jboss-jmx.jar, jboss-system.jar, log4j-boot.jar, xercesImpl.jar, etc.). The NoAnnotationClassLoader's parent is the system class loader (sun.misc.Launcher$AppClassLoader).

When a new UCL is created and associated with the repository, it contributes to the repository a map of packages it can potentially serve classes from. It doesn't add any class to the repository's class cache yet, because nobody has requested any class at this stage. The repository just walks through the class loader's URL to see what packages that UCL is capable of handling. So, the UCL just declares that it can potentially serve classes from the packages that are present in its classpath.

When requested to load a class, a UCL overrides the standard Java2 class loading model by first trying to load a class from its associated repository's cache. If it doesn't find it there, it delegates the task of loading the class to the first UCL associated with the repository that declared it can load that class. The order in which the UCLs have been added to the repository becomes important, because this is what defines "first" in this context. If no "available" UCL is found, the initiating UCL falls back to the standard Java2 parent delegation. This explains why you are still able to use "java.lang.String", for example.

At the end of this process, if no class definition is found in the bootstrap libraries, in the $JBOSS_HOME/lib libraries nor among the libraries associated with the repository's UCL, the UCL throws a ClassNotFoundException. However, if one of the pair UCL is able to load the class, the class will be added to the repository's class cache and from this moment on, it will be returned to any UCL requesting it.

Even if the Java bootstrap packages or $JAVA_HOME/lib packages are not added to the repository's package map, the classes belonging to those packages can be loaded through the process described above and they are added to the repository too. This explains why you'll find "java.lang.String" in the repository.

Note: Package Map is nothing but ...

/** A HashMap<String, Set<UCL>> of package names to the set of
    * ClassLoaders which have classes in the package.
    * Access synchronized via this.packagesMap monitor.
    */

Class sharing can be turned off. J2EE-style class namespace isolation is available. You get an "isolated" application by scoping the application's deployment. At the JBoss class loading management system level, scoping translates into creating a child repository. A scoped application still can load the classes present in the classpaths of the UCLs or the root repository. Depending on whether repository's "Java2ParentDelegation" flag is turned on or off, a scoped application even has access to the class instances available in the root repository's cache. However, sibling child repositories can never share classes.

Note: Even if an HierarchicalLoaderRepository3$NoParentClassLoader instance has its parent set to be an instance of NoAnnotationURLClassLoader, as represented above, the NoParentClassLoader implementation of loadClass() always throws a ClassNotFoundException to force the UCL to only load from its URLs. We will look closer at how NoParentClassLoader works and how a scoped application loads a class available in the system's bootstrap libraries when we present the Cases 3 and 4, below.

Real World Scenarios

We will explore the complex interactions presented above based on concrete use cases.

We start by assuming that we want to deploy our own application (be it a JBoss service, a complex enterprise archive or a simple stateless session bean), and this application relies on an external library. For simplicity, we could assume that the utility library contains only a single class, org.useful.Utility. The Utility class can be packed together with the application classes inside the application archive, or it could be packed in its own archive, utility.jar. We could also assume that we always use the JBoss' default configuration.

Our hypothetical application consists of a single class, org.pkg1.A. We will consider several common situations:

Case 1. The Utility.class is present in the application's archive, but nowhere else on the server.

The short story: The current UCL will become the defining class loader of the class, and the class will be added to the repository's class cache. The details of the process are presented below.

First time the application needs to use a strong-typed Utility reference, the VM asks the current UCL to load the class. The UCL tries to get the class from the repository's cache (1). If it is found, the class is returned and the process stops right here. If the class is not found, the UCL queries the repository for UCLs capable to load classes from the package the unknown class is part of (3). Being the single UCL able to define the class, the control returns to it and load manager calls loadClassLocally() on it (4). loadClassLocally() first tries to call super.loadClass() (5), which ends by involving the NoAnnotationClassLoader in the loading process. If the class is present in the bootstrap libraries or $JBOSS_HOME/lib (the URLs associated with the NoAnnotationClassLoader instance), it is loaded. Otherwise, the class is loaded from the URLs associated with the current UCL. Finally, the class is added to the repository's class cache (6).

This is the configuration of the UnifiedLoaderRepository after the class loading takes place.

Case 2. The Utility.class is present both in the application's archive AND server/default/lib. The deployment is non-scoped.

The short story: The version of the class available in server/default/lib/utility.jar will be used by the new deployment. The version of the class packed with the deployment will be ignored.

The key element here is that when getPackageClassLoaders() is invoked on the repository, the method calls returns two potential classloaders that can load org.useful.Utility: UCL0 and UCL1. The UCL0 is chosen, because it was added to the repository before UCL1 and it will be used to load org.useful.Utility.

This is the configuration of the UnifiedLoaderRepository after the class loading takes place.

Case 3. The Utility.class is present both in the application's archive AND server/default/lib. The deployment is scoped and Java2ParentDelegation is turned off (default).

The short story: The utility class is loaded from the application's archive.

Because Java2ParentDelegation is turned off by default, the Step (1.1) is never executed, parentRepository.getCachedClass() never gets called, so the UCL doesn't have access to the repository's cached classes.

Within the scope of the call to getPackageClassLoaders() at Step 3, the child repository also calls getPackageClassLoaders() on its parent, and also includes into the returned class loader set a UCL (constructed on the spot and associated to the child repository) that has among its ancestors an instance of NoAnnotationURLClassLoader, which ultimately can reach the system class loader. Why is that? Remember that the UCL's parent, HierarchicalLoaderRepository3$NoParentClassLoader, overrides loadClass() to always throw a ClassNotFoundException, thus forcing the UCL to only load from its URLs. If the UCL relies only on its class loader parent to load bootstrap classes, it will throw ClassNotFoundException and fail when your application wants to load "java.lang.String", for example. The NoAnnotationURLClassLoader-delegating UCL instance included in the return set provides a way load bootstrap library classes.

Always the HierarchialLoaderRepository's class loaders take precedence over the parent's (their "order" is lower). For the case depicted above, UCL1 is the preferred class loader.

This is the configuration of the UnifiedLoaderRepository after the class loading takes place.

Case 4. The Utility.class is present both in the application's archive AND server/default/lib. The deployment is scoped, but java2ParentDelegation is turned on.

When Java2ParentDelegation is turned on, the Step (1.1) is executed, and if a cached class is found in the parent repository, it is returned and the process stops here.

Within the scope of the call to getPackageClassLoaders() at Step (3), the child repository also calls getPackageClassLoaders() on its parent, but does not include into the returned class loader set a UCL with a parent to the system class loader. If there are no class loaders in the repository capable of handling the request ask the class loader itself in the event that its parent(s) can load the class (repository.loadClassFromClassLoader())

The HierarchialLoaderRepository's class loaders take precedence over the parent's (their "order" is lower). For the case depicted above, UCL1 is the preferred class loader.

This is the configuration of the UnifiedLoaderRepository after the class loading takes place.

Question: What happens if the parent delegation is true and a classloader already loaded the class in the parent repository's class cache?

Answer: My scoped application will use the already loaded class from the parent repository's class cache.

-

---

TO_DO

• Create similar use cases for WARs. Deploying a WAR involves an extra class loader (WebappClassLoader) created by the servlet container, that can be independently configured using the WAR's jboss-web.xml.

• Explain how UCL3 are created, by whom and why. Explain the relationship between UCL created during an EAR deployment. Multiple UCLs are created (the one corresponding to the EAR and then for the embedded JARs, WARs, etc).

• More details on Java2ParentDelegation flag. Make it clear that it only affects the relationship between repositories not classloaders. If a class is not found in any repository, then the standard delegation model is invoked (i.e. the parent classloader is asked to find the class).