activemq-artemis/docs/user-manual/en/message-grouping.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- ============================================================================= -->
<!-- Copyright © 2009 Red Hat, Inc. and others.                                    -->
<!--                                                                               -->
<!-- The text of and illustrations in this document are licensed by Red Hat under  -->
<!-- a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). -->
<!--                                                                               -->
<!-- An explanation of CC-BY-SA is available at                                    -->
<!--                                                                               -->
<!--            http://creativecommons.org/licenses/by-sa/3.0/.                    -->
<!--                                                                               -->
<!-- In accordance with CC-BY-SA, if you distribute this document or an adaptation -->
<!-- of it, you must provide the URL for the original version.                     -->
<!--                                                                               -->
<!-- Red Hat, as the licensor of this document, waives the right to enforce,       -->
<!-- and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent        -->
<!-- permitted by applicable law.                                                  -->
<!-- ============================================================================= -->

<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
<!ENTITY % BOOK_ENTITIES SYSTEM "ActiveMQ_User_Manual.ent">
%BOOK_ENTITIES;
]>
<chapter id="message-grouping">
   <title>Message Grouping</title>
   <para>Message groups are sets of messages that have the following characteristics:</para>
   <itemizedlist>
      <listitem>
         <para>Messages in a message group share the same group id, i.e. they have same group
            identifier property (<literal>JMSXGroupID</literal> for JMS, <literal
               >_HQ_GROUP_ID</literal> for ActiveMQ Core API).</para>
      </listitem>
      <listitem>
         <para>Messages in a message group are always consumed by the same consumer, even if there
            are many consumers on a queue. They pin all messages with the same group id to the same
            consumer. If that consumer closes another consumer is chosen and will receive all
            messages with the same group id.</para>
      </listitem>
   </itemizedlist>
   <para>Message groups are useful when you want all messages for a certain value of the property to
      be processed serially by the same consumer.</para>
   <para>An example might be orders for a certain stock. You may want orders for any particular
      stock to be processed serially by the same consumer. To do this you can create a pool of
      consumers (perhaps one for each stock, but less will work too), then set the stock name as the
      value of the _HQ_GROUP_ID property.</para>
   <para>This will ensure that all messages for a particular stock will always be processed by the
      same consumer.</para>
   <note>
      <para>Grouped messages can impact the concurrent processing of non-grouped messages due to the
         underlying FIFO semantics of a queue. For example, if there is a chunk of 100 grouped messages at
         the head of a queue followed by 1,000 non-grouped messages then all the grouped messages will need
         to be sent to the appropriate client (which is consuming those grouped messages serially) before
         any of the non-grouped messages can be consumed. The functional impact in this scenario is a
         temporary suspension of concurrent message processing while all the grouped messages are processed.
         This can be a performance bottleneck so keep it in mind when determining the size of your message
         groups, and consider whether or not you should isolate your grouped messages from your non-grouped
         messages.</para>
   </note>
   <section>
      <title>Using Core API</title>
      <para>The property name used to identify the message group is <literal
            >"_HQ_GROUP_ID"</literal> (or the constant <literal
         >MessageImpl.HDR_GROUP_ID</literal>). Alternatively, you can set <literal
            >autogroup</literal> to true on the <literal>SessionFactory</literal> which will pick a
         random unique id. </para>
   </section>
   <section id="message-grouping.jmsconfigure">
      <title>Using JMS</title>
      <para>The property name used to identify the message group is <literal
         >JMSXGroupID</literal>.</para>
      <programlisting>
 // send 2 messages in the same group to ensure the same
 // consumer will receive both
 Message message = ...
 message.setStringProperty("JMSXGroupID", "Group-0");
 producer.send(message);

 message = ...
 message.setStringProperty("JMSXGroupID", "Group-0");
 producer.send(message);</programlisting>
      <para>Alternatively, you can set <literal>autogroup</literal> to true on the <literal
            >ActiveMQConnectonFactory</literal> which will pick a random unique id. This can also be
         set in the <literal>activemq-jms.xml</literal> file like this:</para>
      <programlisting>
&lt;connection-factory name="ConnectionFactory">
   &lt;connectors>
      &lt;connector-ref connector-name="netty-connector"/>
   &lt;/connectors>
   &lt;entries>
      &lt;entry name="ConnectionFactory"/>
   &lt;/entries>
   &lt;autogroup>true&lt;/autogroup>
&lt;/connection-factory></programlisting>
      <para>Alternatively you can set the group id via the connection factory. All messages sent
         with producers created via this connection factory will set the <literal
            >JMSXGroupID</literal> to the specified value on all messages sent. To configure the
         group id set it on the connection factory in the <literal>activemq-jms.xml</literal> config
         file as follows
         <programlisting>
&lt;connection-factory name="ConnectionFactory">
   &lt;connectors>
      &lt;connector-ref connector-name="netty-connector"/>
   &lt;/connectors>
   &lt;entries>
      &lt;entry name="ConnectionFactory"/>
   &lt;/entries>
   &lt;group-id>Group-0&lt;/group-id>
&lt;/connection-factory></programlisting></para>
   </section>
   <section>
      <title>Example</title>
      <para>See <xref linkend="examples.message-group"/> for an example which shows how message
         groups are configured and used with JMS.</para>
   </section>
   <section>
      <title>Example</title>
      <para>See <xref linkend="examples.message-group2"/> for an example which shows how message
         groups are configured via a connection factory.</para>
   </section>
   <section>
      <title> Clustered Grouping</title>
      <para>Using message groups in a cluster is a bit more complex. This is because messages with a
         particular group id can arrive on any node so each node needs to know about which group
         id's are bound to which consumer on which node. The consumer handling messages for a
         particular group id may be on a different node of the cluster, so each node needs to know
         this information so it can route the message correctly to the node which has that consumer. </para>
      <para>To solve this there is the notion of a grouping handler. Each node will have its own
         grouping handler and when a messages is sent with a group id assigned, the handlers will
         decide between them which route the message should take.</para>
      <para id="message-grouping.type">There are 2 types of handlers; Local and Remote. Each cluster should choose 1 node to
         have a local grouping handler and all the other nodes should have remote handlers- it's the
         local handler that actually makes the decision as to what route should be used, all the
         other remote handlers converse with this. Here is a sample config for both types of
         handler, this should be configured in the <emphasis role="italic"
            >activemq-configuration.xml</emphasis>
         file.<programlisting>
&lt;grouping-handler name="my-grouping-handler">
   &lt;type>LOCAL&lt;/type>
   &lt;address>jms&lt;/address>
   &lt;timeout>5000&lt;/timeout>
&lt;/grouping-handler>

&lt;grouping-handler name="my-grouping-handler">
   &lt;type>REMOTE&lt;/type>
   &lt;address>jms&lt;/address>
   &lt;timeout>5000&lt;/timeout>
&lt;/grouping-handler></programlisting></para>
      <para id="message-grouping.address">The <emphasis role="italic">address</emphasis> attribute refers to a
      <link linkend="clusters.address">cluster connection and the address it uses</link>,
      refer to the clustering section on how to configure clusters. The
            <emphasis role="italic">timeout</emphasis> attribute referees to how long to wait for a
         decision to be made, an exception will be thrown during the send if this timeout is
         reached, this ensures that strict ordering is kept.</para>
      <para>The decision as to where a message should be routed to is initially proposed by the node
         that receives the message. The node will pick a suitable route as per the normal clustered
         routing conditions, i.e. round robin available queues, use a local queue first and choose a
         queue that has a consumer. If the proposal is accepted by the grouping handlers the node
         will route messages to this queue from that point on, if rejected an alternative route will
         be offered and the node will again route to that queue indefinitely. All other nodes will
         also route to the queue chosen at proposal time. Once the message arrives at the queue then
         normal single server message group semantics take over and the message is pinned to a
         consumer on that queue.</para>
      <para>You may have noticed that there is a single point of failure with the single local
         handler. If this node crashes then no decisions will be able to be made. Any messages sent
         will be not be delivered and an exception thrown. To avoid this happening Local Handlers
         can be replicated on another backup node. Simple create your back up node and configure it
         with the same Local handler.</para>
      <para/>
      <section>
         <title>Clustered Grouping Best Practices</title>
         <para>Some best practices should be followed when using clustered grouping:<orderedlist>
               <listitem>
                  <para>Make sure your consumers are distributed evenly across the different nodes
                     if possible. This is only an issue if you are creating and closing consumers
                     regularly. Since messages are always routed to the same queue once pinned,
                     removing a consumer from this queue may leave it with no consumers meaning the
                     queue will just keep receiving the messages. Avoid closing consumers or make
                     sure that you always have plenty of consumers, i.e., if you have 3 nodes have 3
                     consumers.</para>
               </listitem>
               <listitem>
                  <para>Use durable queues if possible. If queues are removed once a group is bound
                     to it, then it is possible that other nodes may still try to route messages to
                     it. This can be avoided by making sure that the queue is deleted by the session
                     that is sending the messages. This means that when the next message is sent it
                     is sent to the node where the queue was deleted meaning a new proposal can
                     successfully take place. Alternatively you could just start using a different
                     group id.</para>
               </listitem>
               <listitem>
                  <para>Always make sure that the node that has the Local Grouping Handler is
                     replicated. These means that on failover grouping will still occur.</para>
               </listitem>
            <listitem>
               <para>In case you are using group-timeouts, the remote node should have a smaller group-timeout with at least half
                     of the value on the main coordinator. This is because this will determine how often the last-time-use
                     value should be updated with a round trip for a request to the group between the nodes.</para>
            </listitem>
            </orderedlist></para>
      </section>
      <section>
         <title>Clustered Grouping Example</title>
         <para>See <xref linkend="examples.clustered.grouping"/> for an example of how to configure
            message groups with a ActiveMQ cluster</para>
      </section>
   </section>
</chapter>