297 lines
16 KiB
XML
297 lines
16 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!-- ============================================================================= -->
|
|
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
|
|
<!-- contributor license agreements. See the NOTICE file distributed with -->
|
|
<!-- this work for additional information regarding copyright ownership. -->
|
|
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
|
|
<!-- (the "License"); you may not use this file except in compliance with -->
|
|
<!-- the License. You may obtain a copy of the License at -->
|
|
<!-- -->
|
|
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
|
|
<!-- -->
|
|
<!-- Unless required by applicable law or agreed to in writing, software -->
|
|
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
|
|
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
|
|
<!-- See the License for the specific language governing permissions and -->
|
|
<!-- limitations under the License. -->
|
|
<!-- ============================================================================= -->
|
|
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
|
|
<!ENTITY % BOOK_ENTITIES SYSTEM "ActiveMQ_User_Manual.ent">
|
|
%BOOK_ENTITIES;
|
|
]>
|
|
<chapter id="large-messages">
|
|
<title>Large Messages</title>
|
|
<para>ActiveMQ supports sending and receiving of huge messages, even when the client and server
|
|
are running with limited memory. The only realistic limit to the size of a message that can
|
|
be sent or consumed is the amount of disk space you have available. We have tested sending
|
|
and consuming messages up to 8 GiB in size with a client and server running in just 50MiB of
|
|
RAM!</para>
|
|
<para>To send a large message, the user can set an <literal>InputStream</literal> on a message
|
|
body, and when that message is sent, ActiveMQ will read the <literal>InputStream</literal>. A
|
|
<literal>FileInputStream</literal> could be used for example to send a huge message from
|
|
a huge file on disk.</para>
|
|
<para>As the <literal>InputStream</literal> is read the data is sent to the server as a stream
|
|
of fragments. The server persists these fragments to disk as it receives them and when the
|
|
time comes to deliver them to a consumer they are read back of the disk, also in fragments
|
|
and sent down the wire. When the consumer receives a large message it initially receives
|
|
just the message with an empty body, it can then set an <literal>OutputStream</literal> on
|
|
the message to stream the huge message body to a file on disk or elsewhere. At no time is
|
|
the entire message body stored fully in memory, either on the client or the server.</para>
|
|
<section id="large.message.configuring">
|
|
<title>Configuring the server</title>
|
|
<para>Large messages are stored on a disk directory on the server side, as configured on the
|
|
main configuration file.</para>
|
|
<para>The configuration property <literal>large-messages-directory</literal> specifies where
|
|
large messages are stored.</para>
|
|
<programlisting>
|
|
<configuration xmlns="urn:activemq"
|
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
|
xsi:schemaLocation="urn:activemq /schema/activemq-configuration.xsd">
|
|
...
|
|
<large-messages-directory>/data/large-messages</large-messages-directory>
|
|
...
|
|
</configuration</programlisting>
|
|
<para>By default the large message directory is <literal>data/largemessages</literal></para>
|
|
<para>For the best performance we recommend large messages directory is stored on a
|
|
different physical volume to the message journal or paging directory.</para>
|
|
</section>
|
|
<section>
|
|
<title>Configuring Parameters</title>
|
|
<para>Any message larger than a certain size is considered a large message. Large messages
|
|
will be split up and sent in fragments. This is determined by the parameter <literal
|
|
>min-large-message-size</literal></para>
|
|
<note>
|
|
<para>ActiveMQ messages are encoded using 2 bytes per character so if the message data is filled
|
|
with ASCII characters (which are 1 byte) the size of the resulting ActiveMQ message would roughly
|
|
double. This is important when calculating the size of a "large" message as it may appear to be
|
|
less than the <literal>min-large-message-size</literal> before it is sent, but it then turns into
|
|
a "large" message once it is encoded.</para>
|
|
</note>
|
|
<para>The default value is 100KiB.</para>
|
|
<section id="large-messages.core.config">
|
|
<title>Using Core API</title>
|
|
<para>If the ActiveMQ Core API is used, the minimal large message size is specified by
|
|
<literal>ServerLocator.setMinLargeMessageSize</literal>.</para>
|
|
<programlisting>
|
|
ServerLocator locator = ActiveMQClient.createServerLocatorWithoutHA(new TransportConfiguration(NettyConnectorFactory.class.getName()))
|
|
|
|
locator.setMinLargeMessageSize(25 * 1024);
|
|
|
|
ClientSessionFactory factory = ActiveMQClient.createClientSessionFactory();</programlisting>
|
|
<para><xref linkend="configuring-transports.client.side"/> will provide more information
|
|
on how to instantiate the session factory.</para>
|
|
</section>
|
|
<section>
|
|
<title>Using JMS</title>
|
|
<para>If JNDI is used to look up the connection factory, the minimum large message size
|
|
is specified in <literal>activemq-jms.xml</literal></para>
|
|
<programlisting>...
|
|
<connection-factory name="ConnectionFactory">
|
|
<connectors>
|
|
<connector-ref connector-name="netty"/>
|
|
</connectors>
|
|
<entries>
|
|
<entry name="ConnectionFactory"/>
|
|
<entry name="XAConnectionFactory"/>
|
|
</entries>
|
|
|
|
<min-large-message-size>250000</min-large-message-size>
|
|
</connection-factory>
|
|
...</programlisting>
|
|
<para>If the connection factory is being instantiated directly, the minimum large
|
|
message size is specified by <literal
|
|
>ActiveMQConnectionFactory.setMinLargeMessageSize</literal>.</para>
|
|
</section>
|
|
<section>
|
|
<title>Compressed Large Messages</title>
|
|
<para>
|
|
You can choose to send large messages in compressed form using <literal>
|
|
compress-large-messages</literal> attributes.
|
|
</para>
|
|
<section>
|
|
<title><literal>compress-large-messages</literal></title>
|
|
<para>If you specify the boolean property <literal>compress-large-messages</literal> on
|
|
the <literal>server locator</literal> or <literal>ConnectionFactory</literal> as true, The
|
|
system will use the ZIP algorithm to compress the message body as the message is
|
|
transferred to the server's side. Notice that there's no special treatment at the
|
|
server's side, all the compressing and uncompressing is done at the client.</para>
|
|
<para>If the compressed size of a large message is below <literal>
|
|
min-large-message-size</literal>, it is sent to server as regular messages. This means
|
|
that the message won't be written into the server's large-message
|
|
data directory, thus reducing the disk I/O.</para>
|
|
</section>
|
|
<section>
|
|
<para>If you use JMS, you can achieve large messages compression by configuring your
|
|
connection factories. For example,</para>
|
|
<programlisting>...
|
|
<connection-factory name="ConnectionFactory">
|
|
<connectors>
|
|
<connector-ref connector-name="netty"/>
|
|
</connectors>
|
|
...
|
|
<compress-large-messages>true</compress-large-messages>
|
|
</connection-factory>
|
|
...</programlisting>
|
|
</section>
|
|
</section>
|
|
</section>
|
|
<section>
|
|
<title>Streaming large messages</title>
|
|
<para>ActiveMQ supports setting the body of messages using input and output streams (<literal
|
|
>java.lang.io</literal>)</para>
|
|
<para>These streams are then used directly for sending (input streams) and receiving (output
|
|
streams) messages.</para>
|
|
<para>When receiving messages there are 2 ways to deal with the output stream; you may
|
|
choose to block while the output stream is recovered using the method <literal
|
|
>ClientMessage.saveOutputStream</literal> or alternatively using the method <literal
|
|
>ClientMessage.setOutputstream</literal> which will asynchronously write the message
|
|
to the stream. If you choose the latter the consumer must be kept alive until the
|
|
message has been fully received.</para>
|
|
<para>You can use any kind of stream you like. The most common use case is to send files
|
|
stored in your disk, but you could also send things like JDBC Blobs, <literal
|
|
>SocketInputStream</literal>, things you recovered from <literal
|
|
>HTTPRequests</literal> etc. Anything as long as it implements <literal
|
|
>java.io.InputStream</literal> for sending messages or <literal
|
|
>java.io.OutputStream</literal> for receiving them.</para>
|
|
<section>
|
|
<title>Streaming over Core API</title>
|
|
<para>The following table shows a list of methods available at <literal
|
|
>ClientMessage</literal> which are also available through JMS by the use of
|
|
object properties.</para>
|
|
<table frame="topbot" id="large-messages.ClientMessageAPI">
|
|
<title>org.apache.activemq.api.core.client.ClientMessage API</title>
|
|
<tgroup cols="3">
|
|
<colspec colname="Name" colnum="1"/>
|
|
<colspec colname="Descr" colnum="2"/>
|
|
<colspec colname="JMS" colnum="3"/>
|
|
<thead>
|
|
<row>
|
|
<entry>Name</entry>
|
|
<entry>Description</entry>
|
|
<entry>JMS Equivalent Property</entry>
|
|
</row>
|
|
</thead>
|
|
<tbody>
|
|
<row>
|
|
<entry>setBodyInputStream(InputStream)</entry>
|
|
<entry>Set the InputStream used to read a message body when sending
|
|
it.</entry>
|
|
<entry>JMS_HQ_InputStream</entry>
|
|
</row>
|
|
<row>
|
|
<entry>setOutputStream(OutputStream)</entry>
|
|
<entry>Set the OutputStream that will receive the body of a message.
|
|
This method does not block.</entry>
|
|
<entry>JMS_HQ_OutputStream</entry>
|
|
</row>
|
|
<row>
|
|
<entry>saveOutputStream(OutputStream)</entry>
|
|
<entry>Save the body of the message to the <literal
|
|
>OutputStream</literal>. It will block until the entire content
|
|
is transferred to the <literal>OutputStream</literal>.</entry>
|
|
<entry>JMS_HQ_SaveStream</entry>
|
|
</row>
|
|
</tbody>
|
|
</tgroup>
|
|
</table>
|
|
<para>To set the output stream when receiving a core message: </para>
|
|
<programlisting>
|
|
...
|
|
ClientMessage msg = consumer.receive(...);
|
|
|
|
|
|
// This will block here until the stream was transferred
|
|
msg.saveOutputStream(someOutputStream);
|
|
|
|
ClientMessage msg2 = consumer.receive(...);
|
|
|
|
// This will not wait the transfer to finish
|
|
msg.setOutputStream(someOtherOutputStream);
|
|
...</programlisting>
|
|
<para> Set the input stream when sending a core message: </para>
|
|
<programlisting>
|
|
...
|
|
ClientMessage msg = session.createMessage();
|
|
msg.setInputStream(dataInputStream);
|
|
...</programlisting>
|
|
<para>Notice also that for messages with more than 2GiB the getBodySize() will return
|
|
invalid values since this is an integer (which is also exposed to the JMS API). On
|
|
those cases you can use the message property _HQ_LARGE_SIZE.</para>
|
|
</section>
|
|
<section id="large-messages.streaming.over.jms">
|
|
<title>Streaming over JMS</title>
|
|
<para>When using JMS, ActiveMQ maps the streaming methods on the core API (see <xref
|
|
linkend="large-messages.ClientMessageAPI"/>) by setting object properties . You
|
|
can use the method <literal>Message.setObjectProperty</literal> to set the input and
|
|
output streams.</para>
|
|
<para>The <literal>InputStream</literal> can be defined through the JMS Object Property
|
|
JMS_HQ_InputStream on messages being sent:</para>
|
|
<programlisting>
|
|
BytesMessage message = session.createBytesMessage();
|
|
|
|
FileInputStream fileInputStream = new FileInputStream(fileInput);
|
|
|
|
BufferedInputStream bufferedInput = new BufferedInputStream(fileInputStream);
|
|
|
|
message.setObjectProperty("JMS_HQ_InputStream", bufferedInput);
|
|
|
|
someProducer.send(message);</programlisting>
|
|
<para>The <literal>OutputStream</literal> can be set through the JMS Object Property
|
|
JMS_HQ_SaveStream on messages being received in a blocking way.</para>
|
|
<programlisting>
|
|
BytesMessage messageReceived = (BytesMessage)messageConsumer.receive(120000);
|
|
|
|
File outputFile = new File("huge_message_received.dat");
|
|
|
|
FileOutputStream fileOutputStream = new FileOutputStream(outputFile);
|
|
|
|
BufferedOutputStream bufferedOutput = new BufferedOutputStream(fileOutputStream);
|
|
|
|
// This will block until the entire content is saved on disk
|
|
messageReceived.setObjectProperty("JMS_HQ_SaveStream", bufferedOutput);</programlisting>
|
|
<para>Setting the <literal>OutputStream</literal> could also be done in a non blocking
|
|
way using the property JMS_HQ_OutputStream.</para>
|
|
<programlisting>
|
|
// This won't wait the stream to finish. You need to keep the consumer active.
|
|
messageReceived.setObjectProperty("JMS_HQ_OutputStream", bufferedOutput);</programlisting>
|
|
<note>
|
|
<para>When using JMS, Streaming large messages are only supported on <literal
|
|
>StreamMessage</literal> and <literal>BytesMessage</literal>.</para>
|
|
</note>
|
|
</section>
|
|
</section>
|
|
<section>
|
|
<title>Streaming Alternative</title>
|
|
<para>If you choose not to use the <literal>InputStream</literal> or <literal
|
|
>OutputStream</literal> capability of ActiveMQ You could still access the data
|
|
directly in an alternative fashion.</para>
|
|
<para>On the Core API just get the bytes of the body as you normally would.</para>
|
|
<programlisting>
|
|
ClientMessage msg = consumer.receive();
|
|
|
|
byte[] bytes = new byte[1024];
|
|
for (int i = 0 ; i < msg.getBodySize(); i += bytes.length)
|
|
{
|
|
msg.getBody().readBytes(bytes);
|
|
// Whatever you want to do with the bytes
|
|
}</programlisting>
|
|
<para>If using JMS API, <literal>BytesMessage</literal> and <literal>StreamMessage</literal>
|
|
also supports it transparently.</para>
|
|
<programlisting>
|
|
BytesMessage rm = (BytesMessage)cons.receive(10000);
|
|
|
|
byte data[] = new byte[1024];
|
|
|
|
for (int i = 0; i < rm.getBodyLength(); i += 1024)
|
|
{
|
|
int numberOfBytes = rm.readBytes(data);
|
|
// Do whatever you want with the data
|
|
} </programlisting>
|
|
</section>
|
|
<section id="large-messages.example">
|
|
<title>Large message example</title>
|
|
<para>Please see <xref linkend="examples.large-message"/> for an example which shows how
|
|
large message is configured and used with JMS.</para>
|
|
</section>
|
|
</chapter>
|