Openwire clients create consumers to advisory topics to receive
notifications. As a result there are internal queues created
on advisory topics. Those consumer shouldn't be exposed via
management APIs which are used by the Console
To fix that the broker doesn't register any queues from
advisory addresses.
Also refactors a code to remove Openwire specific contants
from AddressInfo class.
I'm doing an overal improvement on large message support for AMQP
However this commit is just about a Bug on the converter.
It will be moot after all the changes I'm making, but I would rather keep this separate
as a way to cherry-pick on previous versions eventually.
This test starts 2 servers and send messages to
a queue until it enters into paging state. Then
it changes the address max-size to -1, restarts
the 2 servers again and consumes all the messages.
It verifies that even if the max-size has changed
all the paged messages will be depaged and consumed.
No stuck messages after restarting.
The tests is there to guard a case where messages
won't be depaged on server restart after the max-size
is changed to -1. This issue has been fixed into
master along with the fix for ARTEMIS-581, particularly
the changes to the method PagingStoreImpl.getMaxSize().
Server.stop is currently waiting completions on Sessions just because of test cases.
With the recent changes made into the Executors this is not needed any longer
Instead of flushing we just need to make sure there are no more calls into
page executors as we stop the PageManager.
This will avoid any possible starvations or deadlocks here.
The MappedSequentialFile relies on the assumption that any writers
won't exceed the maximum capacity of the file, leaving the JVM to crash otherwise.
This commit adds proper bounds checking on write operations (and position changes too)
in order to provide recoverable effects if such scenario should occour.
In addition are provided minor fixes on Mapped and Nio SequentialFile::fill behaviour
to match the original contract.
- Added Wait.assert methods, what would make it easier to assert on future conditions
- Moved Wait to artemis-junit, we are now using that module on the testsuite
The Critical analyzer is supposed to catch Deadlocks, and such
can only be resolved by killing the VM.
This test is using shutdown as it would be a bit more complex to
handle Byteman to then halt the VM. This will validate the
CriticalAnalyzer is capturing the event, and in such it would halt the VM.
Added a new trustAll flag which will support trusting any client
keystore when doing testing against a broker. This setting should not
be used in production and is strictly for testing.
Extend test cases in MessageTypesTest to cover Core to AMQP combinations for all JMSTypeTests
Fix ServerJMSBytesMessage to correctly return bodyLength
Remove unused/dead code
Openwire consumer is listed twice below "consumers" tab.
First it shows correctly the requested queue consume.
Second it shows consumer from multicast queue ActiveMQ.Advisory.
The second one is internal and should be hidden.
In a cluster if a node is shut down (or crashed) when a
message is being routed to a remote binding, a internal
property may be added to the message and persisted. The
name of the property is like _AMQ_ROUTE_TOsf.my-cluster*.
if the node starts back, it will load and reroute this message
and if it goes to a local consumer, this property won't
get removed and goes to the client.
The fix is to remove this internal property before it
is sent to any client.
Update Tranformer to be able to handle initiation via propertiers (map<string, string>)
Update Configuration to have more specific transfromer configuration type, and to take properties.
Support back compatibility.
Add AddHeadersTransformer which is a main use case, and can act as example also.
Update Control's to expose new property configuration
Add test cases
Update examples for new transformer config style
- it is now possible to disable the TimedBuffer
- this is increasing the default on libaio maxAIO to 4k
- The Auto Tuning on the journal will use asynchronous writes to simulate what would happen on faster disks
- If you set datasync=false on the CLI, the system will suggest mapped and disable the buffer timeout
This closes#1436
This commit superseeds #1436 since it's now disabling the timed buffer through the CLI
Allows for JMS selectors on JMSCorrelationID as well as JMSXGroupID
and JMSXUserID along with some fixes to avoid an NPE case and fixes
to the conversion of AMQP MessageID and CorrelationID values when
doing cross protocol mappings. Adds new tests to cover more cases
of using the JMS selector with Qpid JMS and the AMQP test client.
If message senders and receivers uses different
wireformat.tightEncodingEnabled options, broker will get marshalling
problem. This is because when openwire messages are converted to
core messages, and later these core messages converted to openwire
messages, the broker uses a mashaller that comes with the connection
used to carry the messages.
For example, if a producer sents a message using option "wireformat
.tightEncodingEnabled=false" and a receiver tries to receive it
using 'true' for the same option, it'll never get it because the
broker will fail to use a "tight encoding" marshaller to
decode a 'loose encoded' message.
To fix the problem, we always use 'tight encoding' for internal
message converters.
Currently this test can take over 30seconds just itself. It seems we can safely reduce the number of messages to validate perf improvement greatly reducing test time.
By default, every openwire connection will create a queue
under the multicast address ActiveMQ.Advisory.TempQueue.
If a openwire client is create temporary queues these queues
will fill up with messages for as long as the associated
openwire connection is alive. It appears these messages
do not get consumed from the queues.
The reason behind is that advisory messages don't require
acknowledgement so the messages stay at the queue.
Added integration test, to prove issue, and assert fix.
Fix PersistentQueueBindingEncoding to return value, not false.
Fix some method arg name to align with class interface arg name
Added test case for cross protocol on JMSDeliveryMode proving issue, and asserting fix
Added fix to AmqpCoreConverter to ensure durability (JMSDeliveryMode) is retained.
Similar issue spotted with JMSPriority as with JMSDeliveyMode, fixing at the same time.
Added extra test case for jmspriority
Added fix for jmspriority
Add support to update Queue config via reload using existing updateQueue method at runtime.
Add/extend unit test cases to include testing reload of queue config.
Instead of wait to flush an executor,
I have added a method isFlushed() which will just translate to the
state on the OrderedExecutor.
In the case another executor is provided (for tests) there's a delegate
into normal executors.
delegate to the jdk saslServer. Allow acceptor configuration of supported mechanismis; saslMechanisms=<a,b>
and allow login config scope for krb5 to be configured via saslLoginConfigScope=x
When calling a consumer to receive message with a timeout
(receive(long timeout), if the consumer's buffer is empty, it sends
a 'forced delivery' the waiting forever, expecting the server to
send back a 'forced delivery" message if the queue is empty.
If the connection is disconnected as the arrived 'forced
delivery' message is corrupted, this 'forced delivery' message
never gets to consumer. After the session is reconnected,
the consumer never knows that and stays waiting.
To fix that we can send a 'forced delivery' to server right
after the session is reconnected.
The test is broken. It can be fixed by doing
```java
FakeTransportConnector(URI uri) {
setServer(new TransportServer {
@Override
public URI getConnectURI() {
return uri;
}
but then the test would fail because message
eviction is not supported by Artemis.
This change results in many more tests being run. Previously, 662 tests
were executed. Now, 1166 tests is executed. The running time has increased
from ~20 minutes to ~50 minutes.
This is replacing an executor on ServerSessionPacketHandler
by a this actor.
This is to avoid creating a new runnable per packet received.
Instead of creating new Runnable, this will use a single static runnable
and the packet will be send by a message, which will be treated by a listener.
Look at ServerSessionPacketHandler on this commit for more information on how it works.
Add krb5sslloginmodule that will populate userPrincipal that can be mapped to roles independently
Generalised callback handlers to take a connection and pull certs or peerprincipal based on
callback. This bubbled up into api change in securitystore and security manager
If replication blocked anything on the journal
the processing from clients would be blocked
and nothing would work.
As part of this fix I am using an executor on ServerSessionPacketHandler
which will also scale better as the reader from Netty would be feed immediately.
Core client with netty connector and acceptor doing kerberos
jaas.doAs around sslengine init such that the SSL handshake can do kerberos ticket
generaton and validation.
The kerberos authenticated user is then validated with the security manager before
being populated into the message userId.
The feature is enabled with the kerb5Config property. When lowercase it is the
principal. With a leading uppercase char it is the login.config entry to use.
The MAPPED journal refactoring include:
- simplified lifecycle and logic (eg fixed file size with single mmap memory region)
- supports for the TimedBuffer to coalesce msyncs (via Decorator pattern)
- TLAB pooling of direct ByteBuffer like the NIO journal
- remove of old benchmarks and benchmark dependencies
When a large message is replicated to backup, a pendingID is generated
when the large message is finished. This pendingID is generated by a
BatchingIDGenerator at backup.
It is possible that a pendingID generated at backup may be a duplicate
to an ID generated at live server.
This can cause a problem when a large message with a messageID that is
the same as another largemessage's pendingID is replicated and stored
in the backup's journal, and then a deleteRecord for the pendingID
is appended. If backup becomes live and loads the journal, it will
drop the large message add record because there is a deleteRecord of
the same ID (even though it is a pendingID of another message).
As a result the expecting client will never get this large message.
So in summary, the root cause is that the pendingIDs for large
messages are generated at backup while backup is not alive.
The solution to this is that instead of the backup generating
the pendingID, we make them all be generated in advance
at live server and let them replicated to backup whereever needed.
The ID generater at backup only works when backup becomes live
(when it is properly initialized from journal).
This method name would clash with ServiceComponent
As the real meaning here on this method is just to failover
So I've renamed the method to avoid the clash with my next commit
(I've done this on a separate commit as you may need to redo this
commit from scratch again in other branches instead of lots of clashes on cherry-pick)
When a large message is being diverted, a new copy of the original
message is created and replicated (if there is a backup) to the backup.
In LargeServerMessageImpl.copy(long) it reuse a byte array to copy
message body. It is possible that one block of date is read into
the byte array before the previous read has been replicated,
causing the replicated bytes to corrupt.
If we make a copy of the byte array before replication, the corruption
of data will be avoided.
Before sending of messages to server 0 begins, the test
should wait until consumer is registered at RemoteQueueBindingImpl
on server 0. Otherwise some messages may not be rebalanced
to server 1.
it doesn't really matter the number of files.. as long as the data is valid.
This type of assertion limits the implementation. it's mocking test with too much intrusion
over the implementation. Hence I'm removing these clauses that will fail eventually.
Add extra configuration to address-settings to be able to
control / enable address/queue deletion by pattern,
rather than a global toggle.
Add support in the reload logic to remove address
and/or queues if the address matches an address setting,
where it is enabled.
When a broken packet arrives at client side it causes decoding error.
Currently artemis doesn't handle it properly. It should catch such
errors and disconnect the underlying connection, logging a proper
warning message
Use AcitveMQDestination for subscription naming, fixing and aligning queue naming in the process.
The change is behind a configuration toggle so to avoid causing any breaking changes for uses not expecting.
In a cluster if a node is shut down (or crashed) when a
message is being routed to a remote binding, a internal
property may be added to the message and persisted. The
name of the property is like _AMQ_ROUTE_TOsf.my-cluster*.
if the node starts back, it will load and reroute this message
and if it goes to a local consumer, this property won't
get removed and goes to the client.
The fix is to remove this internal property before it
is sent to any client.
Update ActiveMQConnection to change/alighn behaviour for addtional methods:
- getMetaData
- stop
Adding test to avoid regression.
This is aligning with qpid-jms and openwire clients
Adds headers AMQ_SCHEDULED_DELAY and AMQ_SCHEDULED_TIME to STOMP
protocol handling to allow for delayed and scheduled time of a
message. The AMQ_SCHEDULED_DELAY brings forward the same option
from the 5.x broker and the AMQ_SCHEDULED_TIME option adds a fixed
time of delivery alternative to match that of AMQP and others.
Add test case, to prove the issue, and then obviously ensure it works, post fix.
Apply changes in logic of createQueueName to handle global better and fix the behaviour.
Create queues so names are same as behaviour with core client.
Add topic and queue cache maps in Session.
Add configuration to use cache or not with defaulting to false, which keeps existing behaviour as the default.
Need to configure the WS Handshaker in the test client's netty transport
with the same value given to the proton connection via setMaxFrameSize
so that incoming frames larger than the default 65535 over WS don't
trigger netty to fail the connection.
Building on ARTEMIS-905 JCtools ConcurrentMap replacement first proposed but currently parked by @franz1981, replace the collections with primitive key concurrent collections to avoid auto boxing.
The goal of this is to reduce/remove autoboxing on the hot path.
We are just adding jctools to the broker (should not be in client dependencies)
Like wise targeting specific use case with specific implementation rather than a blanket replace all.
Using collections from Bookkeeper, reduces outside tlab allocation, on resizing compared to JCTools, which occurs frequently on testing.
Port fixes to the AMQP test client recently made in the 5.x version.
Fixes some thread safety issues in the Transport. Ensures more
timely shutdown of the Connection executor. Uses a dynamic Proxy
to generate Read-Only Proton wrappers instead of the hand crafted
versions. Adds additional logging for test data
- NIO/ASYNCIO new TimedBuffer with adapting batch window heuristic
- NIO/ASYNCIO improved TimedBuffer write monitoring with
lightweight concurrent performance counters
- NIO/ASYNCIO journal/paging operations benefit from less buffer copy
- NIO/ASYNCIO any buffer copy is always performed with raw batch copy
using SIMD instrinsics (System::arrayCopy) or memcpy under the hood
- NIO improved clear buffers using SIMD instrinsics (Arrays::fill) and/or memset
- NIO journal operation perform by default TLABs allocation pooling (off heap)
retaining only the last max sized buffer
- NIO improved file copy operations using zero-copy FileChannel::transfertTo
- NIO improved zeroing using pooled single OS page buffer to clean the file
+ pwrite (on Linux)
- NIO deterministic release of unpooled direct buffers to avoid OOM errors
due to slow GC
- Exposed OS PAGE SIZE value using Env class
move classes and methods to their correct location to avoid cyclic dependencies between packages and classes.
ARTEMIS-904 Remove cyclic dependencies from artemis-cli
move classes and methods to their correct location to avoid cyclic dependencies between packages and classes.
ARTEMIS-904 Remove cyclic dependencies from artemis-cli
move classes and methods to their correct location to avoid cyclic dependencies between packages and classes.
Added a wait-for-activation option to shared-store master HA policies.
This option is enabled by default to ensure unchanged server startup behavior.
If this option is enabled, ActiveMQServer.start() with a shared-store master server will not return
before the server has been activated.
If this options is disabled, start() will return after a background activation thread has been started.
The caller can use waitForActivation() to wait until server is activated, or just check the current activation status.
Adds a test that validates that messages that are either lacking a
header or are set to be non-durable are not persisted and are not
recovered on broker restart.
Adding a new ActievMQServerPlugin interface to support adding custom
behavior to the broker at certain events such as connection or session
creation.
https://issues.apache.org/jira/browse/ARTEMIS-898
When creating some AMQP resources (senders, receivers, etc) the broker
can return an error of 'failed' instead of the security error that is
expected in these cases. In the case of a receiver being created and
a security error happening the broker fails to send back a response
causing the client to hang waiting for an attach response.
Refactor the AMQP test suite grouping tests into more logical unit
tests and adding additional coverage in many areas. Adds some negative
validation tests to cover features that were only partially tested.
Brings in tests from ActiveMQ 5.x that were not yet ported to Artemis
to increase coverage amd test scenarios previously seen to have issues
in the 5.x broker.
Improve tests that were failing sporadically due to not waiting for
broker stats to be updated after async calls were made.
Instead of going directly into backup mode within the shared-store
live activation, we just change the HA-policy to slave and return
to the caller - ActiveMQServerImpl.internalStart().
The caller will then handle the backup activation as usual
in a separate thread, such that EmbeddedJMS.start() can return.
Also added a related integration test.
sendMessage() may throw ActiveMQException that causes CNFE
at the management client. Also it should check if headers
in the message is null (to prevent NPE).
Broker should support full qualified queue names (FQQN)
as well as bare queue names. This means when clients access
to a queue they have two equivalent ways to do so. One way
is by queue names and the other is by FQQN (i.e. address::qname)
names. Currently only receiving is supported.
On link attach we currently default out SenderSettleMode to MIXED which
while legal doesn't truly reflect what the client asked for. We instead
now update the link to reflect the mode requested by the client
Also add some tests to ensure that we always return the
ReceiverSettleMode as FIRST since we don't support SECOND.
Adds some new AMQP protocol handling tests brought forward from
ActiveMQ 5.x as well as cleaning up some of th existing tests
code to make adding some other tests easier.
This is fixing an issue introduced on 4b47461f03 (ARTEMIS-822)
The Transactions were being looked up without the readLock and some of the controls for Read and Write lock
were broken after this.
removing JmsTopicWildcardSendReceiveTest::testReceiveWildcardTopicMatchDoubleWildcard
Accordingly to bisect, this test was broken at 21b64b3e4f
and it contradicts the commit done at 21b64b3e4f
So, this is being removed
AIOFileLockManager doesn't work on NFS-mounted share store directories.
Since the GFS2 bug https://bugzilla.redhat.com/show_bug.cgi?id=678585
has been fixed end of 2011, the class AIOFileLockManager is no longer needed and I have removed it.
Broker should support full qualified queue names (FQQN)
as well as bare queue names. This means when clients access
to a queue they have two equivalent ways to do so. One way
is by queue names and the other is by FQQN (i.e. address::qname)
names. Currently only receiving is supported.
Add tests for this management operation with both core and AMQP encoded
messages. Also fix a few problems with the implementation like not
checking the passed-in headers for null and not counting messages
properly.
Fix the getUserID and getTimestamp methods in AMQPMessage to read and
return the correct values. Adds some tests to cover these cases and
cleans up some others.
Ensure that the header value for priority is read and returned in a form
that is scaled such that it won't cause an IndexOutOfBoundsException
from the QueueImpl priority array. Adds some additional testing for
message priority support.
This is moving the smoke tests creates as part of the relication tests.
They are also now based on junit tests.
And to support starting servers I am exposing basedir to unit tests in general.
When populate-validated-user = true AMQP messages can cause exceptions.
This feature isn't particularly applicable to AMQP so this commit
eliminates the exception and leaves the AMQP messages untouched
even if populate-validated-user = true. In other words,
populate-validated-user + AMQP is not supported.
If broker fails to decode any packets from buffer, it should
treat it as a critical bug and disconnect immediately.
Currently broker only logs an error message.
When I added flow control, some tests that were using reflection started to fail.
Also as a precaution I'm using <= on the flow control low credit check
Update the AMQP test client to allow for better inspection of the
delivery updates that happen during normal use. Use those modification
to check that when the broker's sender accepts and settles a non-settled
disposition it adds a proper TransactionState disposition with the
correct outcome and txn-id in that state.
When a message is sent to the broker with a TransactionState indicating
that the message should be included in a transaction the disposition from
the broker indicating acceptance of the message should be done using a
TransactionState value that contained the TX ID and the Accepted
disposition.
The coordinator needs to refill credit on the receiver once it has been
exhausted, otherwise the remote cannot send additional declare or
discharge commands to the broker.
Adjust slow-consumer detection logic to use the number of messages in
the queue and not just the number of messages added since the last
check. This means the getRate() method now returns the rate of messages
which it *could* have dispatched since the last check rather than the
rate at which it received messages. This is a more reliable metric to
ensure the slow-consumer detection logic doesn't flag a consumer as
slow unfairly. Although the reliability will come at a performance cost
since getMessageCount() must lock the queue.
As part of my refactoring on AMQP, the broker shouldn't rely on Application properties
for any broker semantic changes on delivery.
I am removing any access to those now, so we can properly deal with this post 2.0.0.
Artemis expose createQueue() method to management console like Jon.
If the queue to be created already exists it throws an ActiveMQException
back to the console, which will get a ClassNotFoundException when
deserializing the exception.
The same issue goes also with AddressControl.sendMessage() method.
To fix that it should throw a common java exception like IllegalStateException.
with this we could send and receive message in their raw format,
without requiring conversions to Core.
- MessageImpl and ServerMessage are removed as part of this
- AMQPMessage and CoreMessage will have the specialized message format for each protocol
- The protocol manager is now responsible to send the message
- The message will provide an encoder for journal and paging
When openwire sends back an exception response, it doesn't set
the correct correlation id. This causes the client to miss the
response and the exception won't get caught.
To fix it we need to add the correlation id before sending.
When sending an empty ObjectMessage, broker doesn't
write a 'length' field to the message buffer. In delivery
the broker tries to read the length from the buffer, which
causes "IndexOutOfBoundsException".
To fix it, we need to check if the buffer is empty or not,
and only read it if the buffer is not empty.
When a producer sends a messages to a temp destination created from
another connection, it fails. The reason behind it is that the
producer's connection didn't receive the advisory message (notification)
from broker about this temp destination, and it will throw an exception
if it doesn't know this temp destination.
The fix is send the advisory to the client so that it knows this destination.
When creating a 'no-local' openwire consumer, it doesn't work,
meaning it can still receive messages from the same connection.
The fix is similar to what Artemis client does, which is adding
a 'filter' to the consumer/subscription.
The difference is that with OpenWire we have to do it on the
broker side.
Adds tests for handling of Rejected, Released and Modified outcomes for
a delivery sent to a receiver. Tests show that for the Modified outcome
the broker is redelivering the message to the same receiver when the
undeliverable here value is set which violates the AMQP 1.0 specified
handling of that field. Also for Rejected outcome the broker should
be sending the rejected message to the DLQ as Rejected is supposed to
be a terminal outcome.
Small fix included to not adjust the delivery count if the Modified
outcome does not indicate that the delivery failed.
If the SASL plain mechanism arrives with the authzid value set the
mechanism needs to account for its presence and use the correct fields
of the exchange to get the username and password values. Adds some
tests to validate this fix.
Two issues encountered here:
i - ClosingConnectionTest was intermittently breaking other tests, in particular it was breaking PagingLeakTest for no apparent reason
. apparently it was a dead lock from removeAddress on the ServerController
ii - it still showing issues after removing the not needed synchronziation. Since the test is not really needed I am just removing the offending test.
Tests for the management of temporary destinations using the dynamic
node feature. Failing case, the broker return a source or target that
indicates it will honor the lifetime policy of delete on close but the
temporary destination remains in existence after the link it closed.
Some new tests that cover some AMQP credit handling scenarios. Test
case 'testCloseBusyReceiver' currently fails due to being dispatched a
duplicate message.
The broker needs to return only the filters that are supported on a
receiver attach otherwise the remote is not aware that the broker is not
able to honor the requested configuration of the receiver.
Two new tests that acquire messages inside a TX and manage their
settlement and retirement in differing ways, one test works the other
'testReceiversCommitAndRollbackWithMultipleSessionsInSingleTXNoSettlement'
fails due to messages that were committed being redelivered to the
client unexpectedly.
Refactor the AMQP Message transformers both for better performance and
also to fix a number of issues with the transformers creating inbound
and outbound messages with incorrectly mapped values or extra data
appended where it should not be.
Since we don't need client implementations any longer, given the maturity level of
qpid jms, these classes can go, as a result a lot of the interfaces can be removed.
As part of this I am removing proton-plug, and reorganizing the packages in a way I think it
makes more sense and easier to other developers to understand and maintain it.
https://issues.apache.org/jira/browse/ARTEMIS-751
* add DataSource property to DatabaseStorageConfiguration to be able to
communicate with the data store using this DataSource instance instead
of relying on the creation the SQL connnection using the JDBC connection
URL/driver class name tuple.
* add SQLProvider.Factory property to DatabaseStorageConfiguration to
externalize the choice of the SQLProvider instead of relying on
hard-coded choices. If the property is null, the current behaviour will
be used (determing the SQLProvider based on the driver class name)
* bindingsJournal and messageJournal are already started in the start()
method. Remove redundant calls that were creating unused JDBC
connections that are never closed.
JIRA: https://issues.apache.org/jira/browse/ARTEMIS-714
max-disk-usage = how much of a disk we can use before the system blocks
global-max-size = how much bytes we can take from memory for messages before we start enter into the configured page mode
This will also change the default created configuration into page-mode as that's more reliable for systems.
Clebert Suconic
Tomas Kratky
saurabhrai
Stanislav Knot
Pat Fox
Andy Taylor
Justin Bertram
Martyn Taylor
Howard Gao
Francesco Nigro
Chris Burlinchon
Christopher L. Shannon (cshannon)
Michael André Pearce
Robert Godfrey
Timothy Bish
Jeff Mesnil
Jens Reimann
Jiri Danek
Robbie Gemmell
gtully
Erich Duda
xstefank
Odyldzhon Toshbekov
Guillaume Nodet
Michal Tóth
Ingo Weiss
Ville Skyttä
Ulf Lilleengen
Bennet Schulz
Bernd Gutjahr
Zoran Regvart
Dejan Bosanac
Erich Duda
Will Reichert
Pat Fox
jbertram
barreiro
Tomas Hofman
Paul Gallagher
Francesco Nigro
Gordon Sim