- LargeServerMessageImpl.finalize is eventually causing deadlocks
- CoreMessage needs to check properties before decoding
- PagingTest tweaks
- ServerLocatorImpl can deadlock eventually, avoiding a lock and using actors
- ActiveMQServerImpl.finalize is also evil and can cause deadlocks on the testsuite
- MqttClusterRemoteSubscribeTest needs to setup the Address now on the setup
This is good when you are a customer and an artemis engineer (e.g. me) asks your journal print-data but you can't do it because that would expose your user's data. If you do artemis data print --safe, that will only expose the journal structure without exposing user's data and eliminate any liability between the engineer and users.
Adding new metrics for tracking message counts and sizes on a Queue.
This includes tracking metrics for pending, delivering and scheduled
messages. The paging store also tracks message size now.
Support exlusive consumer
Allow default address level settings for exclusive consumer
Allow queue level setting in broker.xml
Add the ability to set queue settings via Core JMS using address. Similar to ActiveMQ 5.X
Allow for Core JMS client to define exclusive consumer using address parameters
Add tests
HornetQClientProtocolManager is used to connect HornteQ servers.
During reconnect, it sends a CheckFailoverMessage packet to the
server as part of reconnection. This packet is not supported by
HornetQ server (existing release), so it will break the backward
compatibility.
Also fixed a failover issue where a hornetq NettyConnector's
ConnectorFactory is serialized to the clients who cannot
instantiate it because class not found exception.
We provide a feature to mask passwords in the configuration files.
However, passwords in the bootstrap.xml (when the console is
secured with HTTPS) cannot be masked. This enhancement has
been opened to allow passwords in the bootstrap.xml to be masked
using the built-in masking feature provided by the broker.
Also the LDAPLoginModule configuration (in login.config) has a
connection password attribute that also needs this mask support.
In addition the ENC() syntax is supported for password masking
to replace the old 'mask-password' flag.
Change all use from Set<RoutingType> to EnumSet<RoutingType>
Deprecating any old exposed interfaces but keeping for back compatibility.
Address info to avoid iterator on getRoutingType hotpath, like wise can be avoided where single RoutingType is passed in.
* Move byte util code into ByteUtil
* Re-use the new equals method in SimpleString
* Apply same pools/interners to client decode
* Create String to SimpleString pools/interners for property access via String keys (producer and consumer benefits)
* Lazy init the pools on withing the get methods of CoreMessageObjectPools to get the specific pool, to avoid having this scattered every where.
* reduce SimpleString creation in conversion to/from core message methods with JMS wrapper.
* reduce SimpleString creation in conversion to/from Core in OpenWire, AMQP, MQTT.
Expose User associated with creating Queue on JMX QueueControl (as attribute)
Allow setting of the user to associate with creating the queue when configured in broker.xml (before only if created over wire is it possible to set the user)
Revert "ARTEMIS-1545 Adding HornetQ 2.4.7 on the mesh to validate send-acks"
I'm reverting this as the testsuite is broken..
We will send it back once worked out.
This reverts commit 8f5b7a1e73.
This reverts commit 9b982b3e30.
add connection ID to the trace logging output
dealing with reading/writing packets. This helps correlate
the packets read/written to an individual tcp connection
Openwire clients create consumers to advisory topics to receive
notifications. As a result there are internal queues created
on advisory topics. Those consumer shouldn't be exposed via
management APIs which are used by the Console
To fix that the broker doesn't register any queues from
advisory addresses.
Also refactors a code to remove Openwire specific contants
from AddressInfo class.
The temporary deadlock is avoided by removing 'synchronized' from
ClientSessionImpl::getCredits method. As the method uses only
a producerCreditManger, only this object is guarded against
the parallel access.
When the execution of code jumps out of the while cycle,
it should check whether the blocking-call-timeout expired or not.
If no, ActiveMQUnBlockedException should be thrown
instead of ActiveMQConnectionTimedOutException.
The changes to remove netty-all removes the classifiers that add the
dependency to the netty transport that includes the compiled native
library wrapper. Add those classifiers back in.
Added a new trustAll flag which will support trusting any client
keystore when doing testing against a broker. This setting should not
be used in production and is strictly for testing.
In a cluster if a node is shut down (or crashed) when a
message is being routed to a remote binding, a internal
property may be added to the message and persisted. The
name of the property is like _AMQ_ROUTE_TOsf.my-cluster*.
if the node starts back, it will load and reroute this message
and if it goes to a local consumer, this property won't
get removed and goes to the client.
The fix is to remove this internal property before it
is sent to any client.
Update Tranformer to be able to handle initiation via propertiers (map<string, string>)
Update Configuration to have more specific transfromer configuration type, and to take properties.
Support back compatibility.
Add AddHeadersTransformer which is a main use case, and can act as example also.
Update Control's to expose new property configuration
Add test cases
Update examples for new transformer config style
- it is now possible to disable the TimedBuffer
- this is increasing the default on libaio maxAIO to 4k
- The Auto Tuning on the journal will use asynchronous writes to simulate what would happen on faster disks
- If you set datasync=false on the CLI, the system will suggest mapped and disable the buffer timeout
This closes#1436
This commit superseeds #1436 since it's now disabling the timed buffer through the CLI
priority is stored as a byte in the ICoreMessage's map.
It is stored then in a int when it is converted to JMS (as JMSPriority
header is an Integer).
JIRA: https://issues.apache.org/jira/browse/ARTEMIS-1448
add the adaptTransportConfiguration() method to the
ClientProtocolManagerFactory so that transport configurations used by
the ClientProtocolManager have an opportunity to adapt their transport
configuration.
This allows the HornetQClientProtocolManagerFactory to adapt the
transport configuration received by remote HornetQ broker to replace the
HornetQ-based NettyConnectorFactory by the Artemis-based one.
JIRA: https://issues.apache.org/jira/browse/ARTEMIS-1431
delegate to the jdk saslServer. Allow acceptor configuration of supported mechanismis; saslMechanisms=<a,b>
and allow login config scope for krb5 to be configured via saslLoginConfigScope=x
When calling a consumer to receive message with a timeout
(receive(long timeout), if the consumer's buffer is empty, it sends
a 'forced delivery' the waiting forever, expecting the server to
send back a 'forced delivery" message if the queue is empty.
If the connection is disconnected as the arrived 'forced
delivery' message is corrupted, this 'forced delivery' message
never gets to consumer. After the session is reconnected,
the consumer never knows that and stays waiting.
To fix that we can send a 'forced delivery' to server right
after the session is reconnected.
This is replacing an executor on ServerSessionPacketHandler
by a this actor.
This is to avoid creating a new runnable per packet received.
Instead of creating new Runnable, this will use a single static runnable
and the packet will be send by a message, which will be treated by a listener.
Look at ServerSessionPacketHandler on this commit for more information on how it works.
Add krb5sslloginmodule that will populate userPrincipal that can be mapped to roles independently
Generalised callback handlers to take a connection and pull certs or peerprincipal based on
callback. This bubbled up into api change in securitystore and security manager
Core client with netty connector and acceptor doing kerberos
jaas.doAs around sslengine init such that the SSL handshake can do kerberos ticket
generaton and validation.
The kerberos authenticated user is then validated with the security manager before
being populated into the message userId.
The feature is enabled with the kerb5Config property. When lowercase it is the
principal. With a leading uppercase char it is the login.config entry to use.
The default id-cache-size is 20000 and the default
confirmation-window-size is 1MB. It turns out the 1MB
size is too small for id-cache-size.
To fix it we adjust the confirmation-window-size to 10MB. Also
a test is added to guarantee it won't break this rule when this
default value is to be changed to any new value.
It fixes compatibility issues with JMS Core clients using the old address model, allowing the client to query JMS temporary queues too.
you would eventually see this issue when using older clients:
AMQ119019: Queue already exists
When a broken packet arrives at client side it causes decoding error.
Currently artemis doesn't handle it properly. It should catch such
errors and disconnect the underlying connection, logging a proper
warning message
Use AcitveMQDestination for subscription naming, fixing and aligning queue naming in the process.
The change is behind a configuration toggle so to avoid causing any breaking changes for uses not expecting.
Added a wait-for-activation option to shared-store master HA policies.
This option is enabled by default to ensure unchanged server startup behavior.
If this option is enabled, ActiveMQServer.start() with a shared-store master server will not return
before the server has been activated.
If this options is disabled, start() will return after a background activation thread has been started.
The caller can use waitForActivation() to wait until server is activated, or just check the current activation status.
Wrap the host added to the HTTP request headers with IPV6Util.encloseHost
to ensure that load balancers that reads the header will have a valid
IPv6 address.
JIRA: https://issues.apache.org/jira/browse/ARTEMIS-1043
Use `long` array for hourly counters instead of `int` array.
Prevents overflow when the number of new messages (a `long`) is added.
Fixes one of the "Implicit narrowing conversion in compound assignment"
alerts on https://lgtm.com/projects/g/apache/activemq-artemis/alerts.
When populate-validated-user = true AMQP messages can cause exceptions.
This feature isn't particularly applicable to AMQP so this commit
eliminates the exception and leaves the AMQP messages untouched
even if populate-validated-user = true. In other words,
populate-validated-user + AMQP is not supported.
If broker fails to decode any packets from buffer, it should
treat it as a critical bug and disconnect immediately.
Currently broker only logs an error message.
The following changes are made to support Epoll.
Refactored SharedNioEventLoopGroup into renamed SharedEventLoopGroup to be generic (as so we can re-use for both Nio and Epoll)
Add support and toggles for Epoll in NettyAcceptor and NettyConnector (with fall back to NIO if cannot load Epoll)
Removal from code of PartialPooledByteBufAllocator, caused bad address when doing native, and no longer needed - see jira discussion
New Connector Properties:
useEpoll - toggles to use epoll or not, default true (but we failback to nio gracefully)
remotingThreads = same behaviour as nioRemotingThreads. Previous property is depreated.
useGlobalWorkerPool = same behaviour as useNioGlobalWorkerPool. Old property is deprecated.
New Acceptor Properties:
useEpoll - toggles to use epoll or not, default true (but we failback to nio gracefully)
useGlobalWorkerPool = same behaviour as useNioGlobalWorkerPool but for Epoll.
This closes#1093
As part of my refactoring on AMQP, the broker shouldn't rely on Application properties
for any broker semantic changes on delivery.
I am removing any access to those now, so we can properly deal with this post 2.0.0.
with this we could send and receive message in their raw format,
without requiring conversions to Core.
- MessageImpl and ServerMessage are removed as part of this
- AMQPMessage and CoreMessage will have the specialized message format for each protocol
- The protocol manager is now responsible to send the message
- The message will provide an encoder for journal and paging
There are some operations in ActiveMQServerControl that don't have
@Parameter annotations. That will make clients like JBoss Jon failed
to invoke those operations.
Also in AddressControl there is a typo in sendMessage. The second
parameter's name should be 'type' not 'headers'.
When HTTP Upgrade is enabled, update Netty's pipeline only after the
HTTP Upgrade handshake is successful *and* the trailing
EMPTY_LAST_CONTENT is received.
Otherwise, this EMPTY_LAST_CONTENT is handled by
ActiveMQChannelHandler which is only expected to handle ByteBuf
JIRA: https://issues.apache.org/jira/browse/ARTEMIS-963
* Fix isEquivalent() method to take into account the activemqServerName
property when httpUpgradeEnabled is true. Two ActiveMQ server hosted on
the same app server may have the same host and port (corresponding to
the Web server HTTP port). The activemqServerName property is used to
distinguish them.
* Iron out HTTP upgrade handler so that the latch is always count down
and the channel context is closed unless the handshake was completed
successfully
JIRA: https://issues.apache.org/jira/browse/ARTEMIS-931
* Do not offset ports for Netty connector/acceptor with http-upgrade
enabled.
* Pass the name of the ActiveMQ server to the HTTP request to initiate
the Upgrade so that the HTTP endpoint on the app server can find the
correct ActiveMQ broker that must handle the upgrade.
JIRA: https://issues.apache.org/jira/browse/ARTEMIS-803
max-disk-usage = how much of a disk we can use before the system blocks
global-max-size = how much bytes we can take from memory for messages before we start enter into the configured page mode
This will also change the default created configuration into page-mode as that's more reliable for systems.
This avoids an issue where a broker would discover itself, causing an unexpected behavior when using
core bridges to forward messages:
* Make channel manager a singleton ensuring that only one channel with a given name exists
* Ensure that messages are marked with NON_LOOPBACK to avoid receiving messages originating from
itself
Javax.json is a newer JSR, but has an ASF compliant version, is pretty close to the original JSON.org API and will support a standard annotation based JSON-B solution at some point soon.
Updated integration tests and removed JSON.org from license.
Using array() is a bit dangerous as it's an optional part of any
ByteBuffer implementation. This new method will deal with various
ByteBuffer implementations appropriately.
Implements a new feature for the broker whereby it may automatically create and
delete JMS topics which are not explicitly defined through the management API
or file-based configuration. A JMS topic is created in response to a sent
message or connected subscriber. The topic may subsequently be deleted when it
no longer has any subscribers. Auto-creation and auto-deletion can both be
turned on/off via address-setting.
The previous fix was breaking compatibility with older servers.
We need to check the default address if an exception happened during the send (due to flow control or blocker)
Obfuscate truststore password in TransportConfiguration.toString()
in the same way as keystore. The password will not be logged in
plain text when bridge is connected.
https://issues.apache.org/jira/browse/ARTEMIS-524
I am keeping all the debug ad tracing I added during the debug of this issue,
for that reason this commit may look longer than expected
The fix will be highlited by the tests added on org.apache.activemq.artemis.tests.integration.client.PagingTest
- Added a thread pool executor, that combines cached and fixed size thread pooling.
It behaves like a cached thread pool in that it reuses exising threads and removes
idle threads after a timeout, limits the maximum number of threads in the pool, but
queue additional request instead of rejecting them.
- changed existing code to use the new thread pool instead of a fixed-size thread pool in
all places that are configured with a client thread pool size.
When NettyConnection.classSSLAndChannel is called from the EventLoop,
waiting for the SSL handler to close will always take 10 seconds, because
the sslCloseFuture is from a task that is scheduled with the same
EventLoop. But since the EventLoop is a single threaded executor, it
will only be executed after the current task is completed.
Due to the single threaded nature of the EventLoop, all blocking calls
should be avoided. Therefore, I removed both awaitUninterruptibly calls
if the closing happens within an event loop tasks. As a side effect,
the annoying server log timeout warnings will go away.
Changed the ActiveMQClient interface to expose global thread pools as
ExecutorService and ScheduledExecutorService interface. This is necessary
to allow injecting thread pool implementations that are not based on
ThreadPoolExecutor or ScheduledThreadPoolExecutor.
1. Changed public fields in ActiveMQClient to private and added getters.
Exposing fields for thread pool sized allow to modify them in undesired ways.
I made these fields private and added corresponding getter methods.
In addition, I renamed the field 'globalThreadMaxPoolSize'
to 'globalThreadPoolSize' to be more consistent with the
'globalScheduledThreadPoolSize' field name.
I also adapted some tests to always call clearThreadPools after
the thread pool size configuration has been changed.
2. Protect against injecting null as thread pools
ActiveMQClient.injectPools allowed null as injected thread pools.
The effect was that internal threads pools were created,
but not shutdown correctly.
Adapted code to handle -1 correctly to configure an unbounded thread pool.
In addition, I removed the capability to reconfigure the max pool size
of existing thread pools, because the global thread pool can either be
an unbounded cached pool, or a bounded fixed size pool.
These 2 kinds of pool also differ in the used blocking queue,
therefore cannot be converted into each other.
Communication between nodes will fail under certain topologies
JGroups has something called JForkChannel that could be used on container systems.
And be injected into Artemis.
For some reason that channel cannot be reused for more than one channel per VM.
And it cannot ever be closed.
I am keeping the trace logs I used to debug this issue in case anything similar to this happens again.
https://issues.apache.org/jira/browse/ARTEMIS-484
The File copy after the initial synchronization on large messages was broken.
On this commit we fix how the buffer is cleaned up before each read since
a previously unfinished body read would make the buffer dirty.
I'm keeping also lots of Traces I have added to debug this issue, so they will
be useful if anything like this happens again.
DelegatingSession class wraps ClientSessionImpl and attempts to close
session should it not be closed by the user. It does this by
implementing finalize. However, the order in which finalize runs can be
difficult to predict as compilers, and JIT compilers are able to
optimize early.
The current DelegatingSession was causing problems of finalize getting
called early (before consumers, producers were finished with the
session). This was causing tests to fail on the IBM JDK (which
optimizes early). The same happens on OpenJDK if the GC is forced.
Its now possible to also add the broker name to jmx tree avoiding clashes when multiple brokers are in a single vm. This is now the default but the old way can be used with some configuration
https://issues.apache.org/jira/browse/ARTEMIS-311
Ive renamed the current isSameHost method to isSameparams as thats what it checked and added a new method for isSameHost that checks the appropriate params for the connector. Ive changed ClientSessionFactoryImp to use this to correct the behaviour.
https://issues.apache.org/jira/browse/ARTEMIS-292
When server sends disconnect to the client, the ClientSession schedules
a close task on it's ordered executor. Once the close method starts
it's waits to check to see if all jobs in it's executor has completed.
To do this it adds a job to it's ordered executor, once it is run it
knows there is nothing more to do and thus is ready to close. However,
this causes a deadlock as both jobs are running in the ordered executor
and thus are both waiting on each other. The close eventually timesout
which is why we see the logs as reported in the JIRA.
This commit runs the close method in it's own ordered executor, thus
preventing the two jobs blocking each other.
The failback process needs to be deterministic rather than relying on various
incarnations of Thread.sleep() at crucial points. Important aspects of this
change include:
1) Make the initial replication synchronization process block at the very
last step and wait for a response from the replica to ensure the replica has
as the necessary data. This is a critical piece of knowledge during the
failback process because it allows the soon-to-become-backup server to know
for sure when it can shut itself down and allow the soon-to-become-live
server to take over. Also, introduce a new configuration element called
"initial-replication-sync-timeout" to conrol how long this blocking will occur.
2) Set the state of the server as 'LIVE' only after the server is fully
started. This is necessary because once the soon-to-be-backup server shuts
down it needs to know that the soon-to-be-live server has started fully before
it restarts itself as the new backup. If the soon-to-be-backup server restarts
before the soon-to-be-live is fully started then it won't actually become a
backup server but instead will become a live server which will break the
failback process.
3) Wait to receive the announcement of a backup server before failing-back.
Netty 4.x uses pooled buffers. These buffers can run out of memory when
transferring large amounts of data over connection. This was causing an
OutOfMemory exception to be thrown on the CoreBridge when tranferring
large messages. Netty provides a callback handler to notify listeners
when a Connection is writable. This patch adds the ability to register
connection writable listeners to the Netty connection and registers the
relevant callback from the Bridge to avoid writing when the buffers are
full.
https://issues.apache.org/jira/browse/ARTEMIS-217 fixing dead lock
This is using a separate lock for notifications, this way we won't hold a lock while communicating on netty which was the issue here.
Inbound sessions are always created from the same ActiveMQConnectionFactory
which means the load-balancing policy is applied to them in the expected
manner. However, outbound sessions are created from independent, unique
ActiveMQConnectionFactory instances which means that the load-balancing
doesn't follow the expected pattern.
This commit changes this behavior by caching each unique
ActiveMQConnectionFactory instance and using it for both inbound and outbound
sessions potentially. This ensures the sessions are load-balanced as
expected.
this is just calling Idea format on all the files using the new style
I am separating manual changes from automatic changes in case I have to repeat the manual changes again
https://issues.apache.org/jira/browse/ARTEMIS-142
I have previously exposed this exception as some errors were being hidden
Now that I have cleared those, It's best to remove this as some errors are now happening
https://issues.apache.org/jira/browse/ARTEMIS-136
From what I researched from implementers of XA TM if you throw ERR over communication errors the transaction manager will create
an heuristic transaction to be manually dealt with.
Other XA Implementations (such as Oracle JDBC) are return FAIL over communication failures during any XA operation.
"mvn install" now works without the lint, but a "mvn install javadoc:jar" still fails. Since that is what the release plugin uses, need to keep the lint there for now. Still lots of failures.
To reproduce this commit, apply a replace regex rule using:
search regex: /\*\*\n \* Licensed
replace: /\*\n \* Licensed
These files had to be changed manually:
artemis-selector/src/main/javacc/HyphenatedParser.jj
artemis-selector/src/main/javacc/StrictParser.jj
artemis-website/src/main/resources/styles/impact/css/pygmentize.css
artemis-website/src/main/resources/styles/impact/css/site.css
We had a few reported small issues on the codebase from the recent introduced google error prone.
This should eliminate any issues, and I am making sure these won't happen again
Lots of work on the test-suite in this commit including:
- Rename ServiceTestBase to ActiveMQTestBase
- Make AddressSettings fluent
- Remove unnecessary tearDown() implementations
- Use ActiveMQTestBase.create*Locator() instead of
ActiveMQClient.createServerLocator*(..)
- Use fluent ServerLocator methods
- Make sure all ActiveMQServers.newActiveMQServer invocations
are surrounded with addServer() where appropriate
- Create a few example tests to be references from hacking-guide
- Update hacking-guide with more info on writing tests
- Refactor config creation methods in ActiveMQTestBase
This has bothered me for awhile, but writing the hacking guide has
given me an opportunity to refactor some of our test-suite to be
simpler, more consistent, and easier to understand. This is
important if we want users to provide well-written tests. Our
test-suite is an important part of the code-base and it should be
easy to write good tests.
Basically I just consolidated CoreUnitTestCase, UnitTestCase, and
ServiceTestBase into a single class named ServiceTestBase. I also
simplified some of the configuration creation methods to reduce
duplicated code.
Stanislav Knot
Clebert Suconic
Justin Bertram
Ingo Weiss
Christopher L. Shannon (cshannon)
Michael André Pearce
yang wei
Dmitrii Tikhomirov
rico.pahlisch
Martyn Taylor
Howard Gao
Francesco Nigro
Ulf Lilleengen
Jiri Danek
art-licis
raul.valdoleiros
Andy Taylor
gtully
saurabhrai
Pat Fox
Erich Duda
Timothy Bish
ederign1
Jeff Mesnil
Daniel Kulp
Tomas Hofman
Bernd Gutjahr
Aditya Sharad
Zoran Regvart
Armand Roelens
Will Reichert
Dejan Bosanac
jbertram
Shaun Appleton
barreiro
Paul Gallagher
Ville Skyttä
Francesco Nigro
bayern39
John D. Ament
Martin Styk
Martin Šmérek
Erich Duda
bayern39
kurobako
Stuart Douglas
Ivo Studensky
Petter Nordlander
Julian Scheid
Otavio Rodolfo Piske
Thiago Kronig
Thiago Kronig