|Configuring Client/Server Event Messaging / Configuring Highly Available Servers|
With server redundancy, each pool has a primary server and some number of secondaries. The primaries and secondaries are assigned on a per-pool basis and are generally spread out for load balancing, so a single client with multiple pools may have primary queues in more than one server.
The primary server pushes events to clients and the secondaries maintain queue backups. If the primary server fails, one of the secondaries becomes primary to provide uninterrupted event messaging.
For example, if there are six servers running and subscription-redundancy is set to two, one server is the primary, two servers are secondary, and the remaining three do not actively participate in HA for the client. If the primary server fails, the system assigns one of the secondaries as the new primary and attempts to add another server to the secondary pool to retain the initial redundancy level. If no new secondary server is found, then the redundancy level is not satisfied but the failover procedure completes successfully. As soon as another secondary is available, it is added.
By default, the primary server sends queue synchronization messages to the secondaries every second. You can change this interval with the server's cache message-sync-interval.
<!-- Set sync interval to 2 seconds --> <cache ... message-sync-interval="2" />
cache = CacheFactory.create(system); cache.setMessageSyncInterval(2);
Usually, all event messages are removed from secondary subscription queues based on the primary's synchronization messages. Occasionally, however, some messages are orphaned in the secondary queues. For example, if a primary fails in the middle of sending a synchronization message to its secondaries, some secondaries might receive the message and some might not. If the failover goes to a secondary that did receive the message, the system will have secondary queues holding messages that are no longer in the primary queue. The new primary will never synchronize on these messages, leaving them orphaned in the secondary queues.
To make sure these messages are eventually removed, the secondaries expire all messages that have been enqueued longer than the time indicated by the servers' message-time-to-live.
<!-- Set message ttl to 5 minutes --> <cache-server port="41414" message-time-to-live="300" />
Cache cache = ...; CacheServer cacheServer = cache.addCacheServer(); cacheServer.setPort(41414); cacheServer.setMessageTimeToLive(200); cacheServer.start();