JBoss Clustering

Thanks to the original author and Source

Singleton Service

A clustered singleton service is deployed on multiple nodes in a cluster but runs on only one of the nodes. The node running the singleton service is typically called the master node. When the master fails, another master is selected from the remaining nodes and the service is restarted on the new master.

Figure 1. Clustered singleton service

Many times, an application needs to ensure that a certain task is run exactly once. Thus, only one of the nodes in a cluster should execute the task. The other nodes should knowingly remain passive. Examples of singleton tasks include:

• Sending email to the system administrator when the system is brought up or taken down. This notification does not take into account how many nodes are in the cluster. As long as at least one node is active, the system is up. When all nodes are down, the system is down.
• Database schema validation upon startup. When a database-driven application is brought up, it is a good practice for the middle tier servers to verify whether the version of the business logic they implement matches the database schema.
• Sending recurring notifications to system users. For example, a calendar application might send out email prior to each instance of a scheduled recurring meeting.
• Load balancing of queued tasks. It's popular to use a single coordinator that distributes tasks among nodes in a cluster.
• Fault tolerance. If an application uses a distributed cache, it is common to designate a single master node responsible for maintaining a current copy of distributed states. The other nodes make requests of the current master node. When the master fails, another node takes over its responsibilities.

While it is fairly easy to implement such singleton tasks in a single VM, the solution will usually not work immediately in a clustered environment. Even in the simple case of a task activated upon startup on one of the nodes in a two-node cluster, several problems must be addressed:

• When the application is started simultaneously on both nodes, which VM should run the singleton task?
• When the application is started on one node and then started later on another, how does the second node know not to run the singleton task again?
• When the node that started the task fails, how does another node know to resume the task?
• When the node that started the task fails, but later recovers, how do we ensure that the task remains running on only one of the nodes?

The logic to solve these problems is unlikely to be included in the design of a single-VM solution. However, a solution can be found to address the case at hand and it can be patched on the startup task. This is an acceptable approach for a few startup tasks and two-node clusters.

As the application grows and becomes more successful, more startup tasks may be necessary. The application may also need to scale to more than two nodes. The clustered singleton problem can quickly become mind-boggling for larger clusters, where the different node startup scenarios are far more difficult to enumerate than in the two-node case. Another complicating factor is communication efficiency. While two nodes can directly connect to each other and negotiate, 10 nodes will have to establish 45 total connections to use the same technique.

This is where JBoss comes in handy. It eliminates most of the complexity and allows application developers to focus on building singleton services regardless of the cluster topology.

We will illustrate how the JBoss clustered singleton facility works with an example. First, we will need a service archive descriptor. Let's use the one that ships with JBoss under server/all/farm/cluster-examples-service.xml. The following is an excerpt:

<!--    | This MBean is an example of a cluster Singleton    -->
<mbean code="org.jboss.ha.singleton.examples.HASingletonMBeanExample"
         name="jboss.examples:service=HASingletonMBeanExample">
</mbean>
<!- - -->
  <!--    | This is a singleton controller which works similarly to the
          | SchedulerProvider (when a MBean target is used)    -->
  <mbean code="org.jboss.ha.singleton.HASingletonController"
         name="jboss.examples:service=HASingletonMBeanExample-HASingletonController">
     <depends>jboss:service=DefaultPartition</depends>
     <depends>jboss.examples:service=HASingletonMBeanExample</depends>
     <attribute name="TargetName">jboss:service=HASingletonMBeanExample</attribute>
     <attribute name="TargetStartMethod">startSingleton</attribute>
     <attribute name="TargetStopMethod">stopSingleton</attribute>
  </mbean>
<!- - -->

This file declares two MBeans, HASingletonMBeanExample and HASingletonController. The first one is a singleton service that contains the custom code. It is a simple JavaBean with the following source code:

public class HASingletonMBeanExample
 implements HASingletonMBeanExampleMBean {

   private boolean isMasterNode = false;

   public void startSingleton() {
       isMasterNode = true;
   }

   public boolean isMasterNode() {
       return isMasterNode; 
   }

   public void stopSingleton() {
       isMasterNode = false; 
   }
}

All of the custom logic for this particular singleton service is contained within this class. Our example is not too useful; it simply indicates, via the isMasterNode member variable, whether the master node is running the singleton. This value will be true only on the one node in the cluster where it is deployed.

HASingletonMBeanExampleMBean exposes this variable as an MBean attribute. It also exposes startSingleton() and stopSingleton() as managed MBean operations. These methods control the lifecycle of the singleton service. JBoss invokes them automatically when a new master node is elected.

How does JBoss control the singleton lifecycle throughout the cluster? The answer to this question is in the MBean declarations. Notice that the HASingletonMBeanExample-HASingletonController MBean also takes the name of the sample singleton MBean and its start and stop methods.

On each node in the cluster where these MBeans are deployed, the controller will work with all of the other controllers with the same MBean name deployed in the same cluster partition to oversee the lifecycle of the singleton. The controllers are responsible for tracking the cluster topology. Their job is to elect the master node of the singleton upon startup, as well as to elect a new master should the current one fail or shut down. In the latter case, when the master node shuts down gracefully, the controllers will wait for the singleton to stop before starting another instance on the new master node.

A singleton service is scoped in a certain cluster partition via its controller. Notice that, in the declaration above, the controller MBean depends on the MBean service DefaultPartition. If the partition where the singleton should run is different than the default, its name can be provided to the controller via the MBean attribute PartitionName.

Clustered singletons are usually deployed via the JBoss farming service. To test this example, just drop the service file above in the server/all/farm directory. You should be able to see the following in the JBoss JMX web console:

Figure 2. Controller MBean view. The MasterNode attribute will have value True on only one of the nodes.

Figure 3. Sample singleton MBean view. The MasterNode attribute will have the same value as the MasterNode attribute on the controller MBean.