mirror of
https://github.com/apache/activemq-artemis.git
synced 2025-02-08 19:15:08 +00:00
9a587c5633
https://issues.apache.org/jira/browse/ACTIVEMQ6-3 We are renaming packages from activemq6 to activemq as that's more generic and version independent The previous commit renamed the directories. On this commit now I'm changing the code. If we changed the code and the directories on the same commit git would remove and add a lot of files without recognizing the renames.
298 lines
16 KiB
XML
298 lines
16 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
||
<!-- ============================================================================= -->
|
||
<!-- Copyright © 2009 Red Hat, Inc. and others. -->
|
||
<!-- -->
|
||
<!-- The text of and illustrations in this document are licensed by Red Hat under -->
|
||
<!-- a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). -->
|
||
<!-- -->
|
||
<!-- An explanation of CC-BY-SA is available at -->
|
||
<!-- -->
|
||
<!-- http://creativecommons.org/licenses/by-sa/3.0/. -->
|
||
<!-- -->
|
||
<!-- In accordance with CC-BY-SA, if you distribute this document or an adaptation -->
|
||
<!-- of it, you must provide the URL for the original version. -->
|
||
<!-- -->
|
||
<!-- Red Hat, as the licensor of this document, waives the right to enforce, -->
|
||
<!-- and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent -->
|
||
<!-- permitted by applicable law. -->
|
||
<!-- ============================================================================= -->
|
||
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
|
||
<!ENTITY % BOOK_ENTITIES SYSTEM "HornetQ_User_Manual.ent">
|
||
%BOOK_ENTITIES;
|
||
]>
|
||
<chapter id="large-messages">
|
||
<title>Large Messages</title>
|
||
<para>HornetQ supports sending and receiving of huge messages, even when the client and server
|
||
are running with limited memory. The only realistic limit to the size of a message that can
|
||
be sent or consumed is the amount of disk space you have available. We have tested sending
|
||
and consuming messages up to 8 GiB in size with a client and server running in just 50MiB of
|
||
RAM!</para>
|
||
<para>To send a large message, the user can set an <literal>InputStream</literal> on a message
|
||
body, and when that message is sent, HornetQ will read the <literal>InputStream</literal>. A
|
||
<literal>FileInputStream</literal> could be used for example to send a huge message from
|
||
a huge file on disk.</para>
|
||
<para>As the <literal>InputStream</literal> is read the data is sent to the server as a stream
|
||
of fragments. The server persists these fragments to disk as it receives them and when the
|
||
time comes to deliver them to a consumer they are read back of the disk, also in fragments
|
||
and sent down the wire. When the consumer receives a large message it initially receives
|
||
just the message with an empty body, it can then set an <literal>OutputStream</literal> on
|
||
the message to stream the huge message body to a file on disk or elsewhere. At no time is
|
||
the entire message body stored fully in memory, either on the client or the server.</para>
|
||
<section id="large.message.configuring">
|
||
<title>Configuring the server</title>
|
||
<para>Large messages are stored on a disk directory on the server side, as configured on the
|
||
main configuration file.</para>
|
||
<para>The configuration property <literal>large-messages-directory</literal> specifies where
|
||
large messages are stored.</para>
|
||
<programlisting>
|
||
<configuration xmlns="urn:hornetq"
|
||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||
xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
|
||
...
|
||
<large-messages-directory>/data/large-messages</large-messages-directory>
|
||
...
|
||
</configuration</programlisting>
|
||
<para>By default the large message directory is <literal>data/largemessages</literal></para>
|
||
<para>For the best performance we recommend large messages directory is stored on a
|
||
different physical volume to the message journal or paging directory.</para>
|
||
</section>
|
||
<section>
|
||
<title>Configuring Parameters</title>
|
||
<para>Any message larger than a certain size is considered a large message. Large messages
|
||
will be split up and sent in fragments. This is determined by the parameter <literal
|
||
>min-large-message-size</literal></para>
|
||
<note>
|
||
<para>HornetQ messages are encoded using 2 bytes per character so if the message data is filled
|
||
with ASCII characters (which are 1 byte) the size of the resulting HornetQ message would roughly
|
||
double. This is important when calculating the size of a "large" message as it may appear to be
|
||
less than the <literal>min-large-message-size</literal> before it is sent, but it then turns into
|
||
a "large" message once it is encoded.</para>
|
||
</note>
|
||
<para>The default value is 100KiB.</para>
|
||
<section id="large-messages.core.config">
|
||
<title>Using Core API</title>
|
||
<para>If the HornetQ Core API is used, the minimal large message size is specified by
|
||
<literal>ServerLocator.setMinLargeMessageSize</literal>.</para>
|
||
<programlisting>
|
||
ServerLocator locator = HornetQClient.createServerLocatorWithoutHA(new TransportConfiguration(NettyConnectorFactory.class.getName()))
|
||
|
||
locator.setMinLargeMessageSize(25 * 1024);
|
||
|
||
ClientSessionFactory factory = HornetQClient.createClientSessionFactory();</programlisting>
|
||
<para><xref linkend="configuring-transports.client.side"/> will provide more information
|
||
on how to instantiate the session factory.</para>
|
||
</section>
|
||
<section>
|
||
<title>Using JMS</title>
|
||
<para>If JNDI is used to look up the connection factory, the minimum large message size
|
||
is specified in <literal>hornetq-jms.xml</literal></para>
|
||
<programlisting>...
|
||
<connection-factory name="ConnectionFactory">
|
||
<connectors>
|
||
<connector-ref connector-name="netty"/>
|
||
</connectors>
|
||
<entries>
|
||
<entry name="ConnectionFactory"/>
|
||
<entry name="XAConnectionFactory"/>
|
||
</entries>
|
||
|
||
<min-large-message-size>250000</min-large-message-size>
|
||
</connection-factory>
|
||
...</programlisting>
|
||
<para>If the connection factory is being instantiated directly, the minimum large
|
||
message size is specified by <literal
|
||
>HornetQConnectionFactory.setMinLargeMessageSize</literal>.</para>
|
||
</section>
|
||
<section>
|
||
<title>Compressed Large Messages</title>
|
||
<para>
|
||
You can choose to send large messages in compressed form using <literal>
|
||
compress-large-messages</literal> attributes.
|
||
</para>
|
||
<section>
|
||
<title><literal>compress-large-messages</literal></title>
|
||
<para>If you specify the boolean property <literal>compress-large-messages</literal> on
|
||
the <literal>server locator</literal> or <literal>ConnectionFactory</literal> as true, The
|
||
system will use the ZIP algorithm to compress the message body as the message is
|
||
transferred to the server's side. Notice that there's no special treatment at the
|
||
server's side, all the compressing and uncompressing is done at the client.</para>
|
||
<para>If the compressed size of a large message is below <literal>
|
||
min-large-message-size</literal>, it is sent to server as regular messages. This means
|
||
that the message won't be written into the server's large-message
|
||
data directory, thus reducing the disk I/O.</para>
|
||
</section>
|
||
<section>
|
||
<para>If you use JMS, you can achieve large messages compression by configuring your
|
||
connection factories. For example,</para>
|
||
<programlisting>...
|
||
<connection-factory name="ConnectionFactory">
|
||
<connectors>
|
||
<connector-ref connector-name="netty"/>
|
||
</connectors>
|
||
...
|
||
<compress-large-messages>true</compress-large-messages>
|
||
</connection-factory>
|
||
...</programlisting>
|
||
</section>
|
||
</section>
|
||
</section>
|
||
<section>
|
||
<title>Streaming large messages</title>
|
||
<para>HornetQ supports setting the body of messages using input and output streams (<literal
|
||
>java.lang.io</literal>)</para>
|
||
<para>These streams are then used directly for sending (input streams) and receiving (output
|
||
streams) messages.</para>
|
||
<para>When receiving messages there are 2 ways to deal with the output stream; you may
|
||
choose to block while the output stream is recovered using the method <literal
|
||
>ClientMessage.saveOutputStream</literal> or alternatively using the method <literal
|
||
>ClientMessage.setOutputstream</literal> which will asynchronously write the message
|
||
to the stream. If you choose the latter the consumer must be kept alive until the
|
||
message has been fully received.</para>
|
||
<para>You can use any kind of stream you like. The most common use case is to send files
|
||
stored in your disk, but you could also send things like JDBC Blobs, <literal
|
||
>SocketInputStream</literal>, things you recovered from <literal
|
||
>HTTPRequests</literal> etc. Anything as long as it implements <literal
|
||
>java.io.InputStream</literal> for sending messages or <literal
|
||
>java.io.OutputStream</literal> for receiving them.</para>
|
||
<section>
|
||
<title>Streaming over Core API</title>
|
||
<para>The following table shows a list of methods available at <literal
|
||
>ClientMessage</literal> which are also available through JMS by the use of
|
||
object properties.</para>
|
||
<table frame="topbot" id="large-messages.ClientMessageAPI">
|
||
<title>org.apache.activemq.api.core.client.ClientMessage API</title>
|
||
<tgroup cols="3">
|
||
<colspec colname="Name" colnum="1"/>
|
||
<colspec colname="Descr" colnum="2"/>
|
||
<colspec colname="JMS" colnum="3"/>
|
||
<thead>
|
||
<row>
|
||
<entry>Name</entry>
|
||
<entry>Description</entry>
|
||
<entry>JMS Equivalent Property</entry>
|
||
</row>
|
||
</thead>
|
||
<tbody>
|
||
<row>
|
||
<entry>setBodyInputStream(InputStream)</entry>
|
||
<entry>Set the InputStream used to read a message body when sending
|
||
it.</entry>
|
||
<entry>JMS_HQ_InputStream</entry>
|
||
</row>
|
||
<row>
|
||
<entry>setOutputStream(OutputStream)</entry>
|
||
<entry>Set the OutputStream that will receive the body of a message.
|
||
This method does not block.</entry>
|
||
<entry>JMS_HQ_OutputStream</entry>
|
||
</row>
|
||
<row>
|
||
<entry>saveOutputStream(OutputStream)</entry>
|
||
<entry>Save the body of the message to the <literal
|
||
>OutputStream</literal>. It will block until the entire content
|
||
is transferred to the <literal>OutputStream</literal>.</entry>
|
||
<entry>JMS_HQ_SaveStream</entry>
|
||
</row>
|
||
</tbody>
|
||
</tgroup>
|
||
</table>
|
||
<para>To set the output stream when receiving a core message: </para>
|
||
<programlisting>
|
||
...
|
||
ClientMessage msg = consumer.receive(...);
|
||
|
||
|
||
// This will block here until the stream was transferred
|
||
msg.saveOutputStream(someOutputStream);
|
||
|
||
ClientMessage msg2 = consumer.receive(...);
|
||
|
||
// This will not wait the transfer to finish
|
||
msg.setOutputStream(someOtherOutputStream);
|
||
...</programlisting>
|
||
<para> Set the input stream when sending a core message: </para>
|
||
<programlisting>
|
||
...
|
||
ClientMessage msg = session.createMessage();
|
||
msg.setInputStream(dataInputStream);
|
||
...</programlisting>
|
||
<para>Notice also that for messages with more than 2GiB the getBodySize() will return
|
||
invalid values since this is an integer (which is also exposed to the JMS API). On
|
||
those cases you can use the message property _HQ_LARGE_SIZE.</para>
|
||
</section>
|
||
<section id="large-messages.streaming.over.jms">
|
||
<title>Streaming over JMS</title>
|
||
<para>When using JMS, HornetQ maps the streaming methods on the core API (see <xref
|
||
linkend="large-messages.ClientMessageAPI"/>) by setting object properties . You
|
||
can use the method <literal>Message.setObjectProperty</literal> to set the input and
|
||
output streams.</para>
|
||
<para>The <literal>InputStream</literal> can be defined through the JMS Object Property
|
||
JMS_HQ_InputStream on messages being sent:</para>
|
||
<programlisting>
|
||
BytesMessage message = session.createBytesMessage();
|
||
|
||
FileInputStream fileInputStream = new FileInputStream(fileInput);
|
||
|
||
BufferedInputStream bufferedInput = new BufferedInputStream(fileInputStream);
|
||
|
||
message.setObjectProperty("JMS_HQ_InputStream", bufferedInput);
|
||
|
||
someProducer.send(message);</programlisting>
|
||
<para>The <literal>OutputStream</literal> can be set through the JMS Object Property
|
||
JMS_HQ_SaveStream on messages being received in a blocking way.</para>
|
||
<programlisting>
|
||
BytesMessage messageReceived = (BytesMessage)messageConsumer.receive(120000);
|
||
|
||
File outputFile = new File("huge_message_received.dat");
|
||
|
||
FileOutputStream fileOutputStream = new FileOutputStream(outputFile);
|
||
|
||
BufferedOutputStream bufferedOutput = new BufferedOutputStream(fileOutputStream);
|
||
|
||
// This will block until the entire content is saved on disk
|
||
messageReceived.setObjectProperty("JMS_HQ_SaveStream", bufferedOutput);</programlisting>
|
||
<para>Setting the <literal>OutputStream</literal> could also be done in a non blocking
|
||
way using the property JMS_HQ_OutputStream.</para>
|
||
<programlisting>
|
||
// This won't wait the stream to finish. You need to keep the consumer active.
|
||
messageReceived.setObjectProperty("JMS_HQ_OutputStream", bufferedOutput);</programlisting>
|
||
<note>
|
||
<para>When using JMS, Streaming large messages are only supported on <literal
|
||
>StreamMessage</literal> and <literal>BytesMessage</literal>.</para>
|
||
</note>
|
||
</section>
|
||
</section>
|
||
<section>
|
||
<title>Streaming Alternative</title>
|
||
<para>If you choose not to use the <literal>InputStream</literal> or <literal
|
||
>OutputStream</literal> capability of HornetQ You could still access the data
|
||
directly in an alternative fashion.</para>
|
||
<para>On the Core API just get the bytes of the body as you normally would.</para>
|
||
<programlisting>
|
||
ClientMessage msg = consumer.receive();
|
||
|
||
byte[] bytes = new byte[1024];
|
||
for (int i = 0 ; i < msg.getBodySize(); i += bytes.length)
|
||
{
|
||
msg.getBody().readBytes(bytes);
|
||
// Whatever you want to do with the bytes
|
||
}</programlisting>
|
||
<para>If using JMS API, <literal>BytesMessage</literal> and <literal>StreamMessage</literal>
|
||
also supports it transparently.</para>
|
||
<programlisting>
|
||
BytesMessage rm = (BytesMessage)cons.receive(10000);
|
||
|
||
byte data[] = new byte[1024];
|
||
|
||
for (int i = 0; i < rm.getBodyLength(); i += 1024)
|
||
{
|
||
int numberOfBytes = rm.readBytes(data);
|
||
// Do whatever you want with the data
|
||
} </programlisting>
|
||
</section>
|
||
<section id="large-messages.example">
|
||
<title>Large message example</title>
|
||
<para>Please see <xref linkend="examples.large-message"/> for an example which shows how
|
||
large message is configured and used with JMS.</para>
|
||
</section>
|
||
</chapter>
|