When performing concurrent user admin actions (e.g. resetUser, addUser,
removeUser on ActiveMQServerControl) when using the
PropertiesLoginModule with reload=true the underlying user and role
properties files can get corrupted.
This commit fixes the issue via the following changes:
- Add synchronization to the management commands
- Add concurrency controls to underlying file access
- Change CLI user commands to use remote methods instead of modifying
the files directly. This avoids potential concurrent changes. This
change forced me to modify the names of some of the commands'
parameters to disambiguate them from connection-related parameters.
Both authentication and authorization will hit the underlying security
repository (e.g. files, LDAP, etc.). For example, creating a JMS
connection and a consumer will result in 2 hits with the *same*
authentication request. This can cause unwanted (and unnecessary)
resource utilization, especially in the case of networked configuration
like LDAP.
There is already a rudimentary cache for authorization, but it is
cleared *totally* every 10 seconds by default (controlled via the
security-invalidation-interval setting), and it must be populated
initially which still results in duplicate auth requests.
This commit optimizes authentication and authorization via the following
changes:
- Replace our home-grown cache with Google Guava's cache. This provides
simple caching with both time-based and size-based LRU eviction. See more
at https://github.com/google/guava/wiki/CachesExplained. I also thought
about using Caffeine, but we already have a dependency on Guava and the
cache implementions look to be negligibly different for this use-case.
- Add caching for authentication. Both successful and unsuccessful
authentication attempts will be cached to spare the underlying security
repository as much as possible. Authenticated Subjects will be cached
and re-used whenever possible.
- Authorization will used Subjects cached during authentication. If the
required Subject is not in the cache it will be fetched from the
underlying security repo.
- Caching can be disabled by setting the security-invalidation-interval
to 0.
- Cache sizes are configurable.
- Management operations exist to inspect cache sizes at runtime.
In certain cases with shared-store HA a broker's activation can fail but
the broker will still be holding the journal lock. This results in a
"zombie" broker which can't actually service clients and prevents the
backup from activating.
This commit adds an ActivationFailureListener to catch activation
failures and stop the broker completely.
The calculation used by
ActiveMQServerControlImpl.getDiskStoreUsagePercentage() is incorrect. It
uses disk space info with global-max-size which is for address memory.
Also, the existing getDiskStoreUsage() method *already* returns a
percentage of total disk store usage so this method seems redundant.
Add the command `check` to the Command Line utility. This command exposes some
checks for nodes and queues using the management API for most of them.
The checks have been implemented to be modular. Each user can compose his own
health check, ie to produce and consume from a queue the command is
`artemis check queue --name TEST --produce 1 --consume 1`.
it is intentional to compare brokerURL == DEFAULT_BROKER_URL here
so, I added a @SuppressWarnings to clear the false positivie.
And also added some comment on why this is intentional.
This commit does the following:
- Deprecates existing overloaded createQueue, createSharedQueue,
createTemporaryQueue, & updateQueue methods for ClientSession,
ServerSession, ActiveMQServer, & ActiveMQServerControl where
applicable.
- Deprecates QueueAttributes, QueueConfig, & CoreQueueConfiguration.
- Deprecates existing overloaded constructors for QueueImpl.
- Implements QueueConfiguration with JavaDoc to be the single,
centralized configuration object for both client-side and broker-side
queue creation including methods to convert to & from JSON for use in
the management API.
- Implements new createQueue, createSharedQueue & updateQueue methods
with JavaDoc for ClientSession, ServerSession, ActiveMQServer, &
ActiveMQServerControl as well as a new constructor for QueueImpl all
using the new QueueConfiguration object.
- Changes all internal broker code to use the new methods.
FQQN support for the CLI was implemented via ARTEMIS-1840 before general
FQQN support was added for producers via ARTEMIS-1867. The CLI's FQQN
functionality is slightly different from what is now generally available
and it can be confusing for users. By refactoring the CLI to use the
general FQQN support the code can be much simpler and consistent with
the expected behavior. Refactoring includes:
- Deprecating the use of "fqqn://". The CLI commands use JMS so using
"fqqn://" (instead of "queue://" or "topic://") makes the destination
type ambiguous which can yield unexpected message routing behavior.
Now "queue://" and "topic://" can be used with the normal FQQN syntax
(e.g. address::queue).
- Eliminating the use of the _AMQ_ROUTE_TO header when sending messags
to an FQQN. The _AMQ_ROUTE_TO header is an internal header used when
routing messages over a cluster bridge. Using it in the CLI for FQQN
support was a clever hack, but using the general FQQN support
eliminates complexity and makes behavior consistent between
standalone JMS clients using FQQN and the CLI.
- De-duplicating MessageSerializer initialization boilerplate.
- Removing limitation where using an FQQN with an anycast address
required the same name for the address and queue.
This is a Large commit where I am refactoring largeMessage Body out of CoreMessage
which is now reused with AMQP.
I had also to fix Reference Counting to fix how Large Messages are Acked
And I also had to make sure Large Messages are transversing correctly when in cluster.
There is an optimization in AMQP, that properties are only parsed over demand.
It happens that after ARTEMIS-2294 (commit 2dd0671698),
every send would request for the property on the message, resulting the properties to always be parsed upon send.
Even when there's no use of application properties.
Add the config parameter `page-sync-timeout` to set a customized value,
because if the broker is configured to use ASYNCIO journal, the timeout
has the same value of NIO default journal buffer timeout ie 3333333.
* Upgrading versions
* Adding wildfly-common dependency as jboss-logmanager now depends on it
for simple common operations such as getting hostname or process id
* Updating bootclasspath with wildfly-common
The Audit log allows user to log some important actions,
such as ones performed via management APIs or clients,
like queue management, sending messages, etc.
The log tries to record who (the user if any) doing what
(like deleting a queue) with arguments (if any) and timestamps.
By default the audit log is disabled. Through configuration can
be easily turned on.
This is the second commit. It improves the windows service
configuration file so that the service can be shutdown
gracefully (clean shutdown) and allow for a chance to
clean up the web tmp dir.
These improvements were also part of this task:
- Routing is now cached as much as possible.
- A new Runnable is avoided for each individual message,
since we use the Netty executor to perform delivery
https://issues.apache.org/jira/browse/ARTEMIS-2205
Optionally specify a queue routing type when creating a broker. Example:
"create --queues myqueue,mytopic:multicast". Defaults to anycast if
unspecified.
Added new cases:
- export from anycast queue, then import to multicast topic where number of queues > 1
- export from anycast queue, then import to multicast fqqn where number of queues > 1
- export from multicast topic, then import to anycast queue (both address)
- export from anycast queue, then import to multicast topic (both address)
Previously the port was always random. This caused problems with
remote JMX connections that needed to overcome firewalls. As of
this patch it's possible to make the RMI port static and whitelist
it in the firewall settings.
JMX configuration is now done via management.xml. Configuring JMX via
the start scripts could result in unexpected behavior since the
authorisation configuration from management.xml would be ignored.
Fix warning seen in logs during test case runs, caused by CF not being closed.
Oct 10, 2018 8:53:18 PM org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl finalize
WARN: AMQ212008: I am closing a core ClientSessionFactory you left open. Please make sure you close all ClientSessionFactories explicitly before letting them go out of scope! 639,542,871
java.lang.Exception
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.<init>(ClientSessionFactoryImpl.java:171)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:779)
at org.apache.activemq.artemis.cli.commands.messages.DestAbstract.getManagementSession(DestAbstract.java:103)
at org.apache.activemq.artemis.cli.commands.messages.DestAbstract.getQueueAttribute(DestAbstract.java:127)
at org.apache.activemq.artemis.cli.commands.messages.DestAbstract.getQueueIdFromName(DestAbstract.java:116)
at org.apache.activemq.artemis.cli.commands.messages.Producer.execute(Producer.java:75)
at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:150)
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:98)
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:125)
at org.apache.activemq.artemis.cli.Artemis.main(Artemis.java:81)
Replace guava Preconditions with artemis Preconditions
Replace guava Predicate with java Predicate
Replace guava Ordering with java Comparator
Replace guava Immutable, with ArrayList/Set and then wrap with unmodifiable
cannot find login.config
The Windows variable ARTEMIS_ETC_INSTANCE was not set correctly,
as it was used as ARTEMIS_INSTANCE_ETC. So the fix is to make sure
that the two match, and additionally they it was renamed to more
closely match the other variables naming convention. This was done
in two files; so when 'artemis create mybroker' is run on a Windows
machine, the broker is correctly configured with login.config.
Minor fix of changing use of single quotes to correct usage of
double quotes when declaring variable
This option was removed in JRE 10 and its presence causes error on startup.
Artemis still does not compile on JDK 9 and up, so what I tried was
to compile on JDK 1.8 and then start Artemis on 1.8 and 10.
That now works for me.
It avoid using the system clock to perform the locks logic
by using the DBMS time.
It contains several improvements on the JDBC error handling
and an improved observability thanks to debug logs.
It allows a user to customize the max allowed distance between system and DB time,
improving HA reliability by shutting down the broker when the misalignment
exceeds configured limit.
The JDBC Lock Acquisition Timeout is no longer exposed to any user configuration and defaulted to infinite to match the behaviour of the journal (file-based) one.
Logging for the "fast-tests" profile used for PR builds could be reduced
significantly. This would save time as well as prevent log truncation
(Travis CI only supports logs up to 4MB).
This is good when you are a customer and an artemis engineer (e.g. me) asks your journal print-data but you can't do it because that would expose your user's data. If you do artemis data print --safe, that will only expose the journal structure without exposing user's data and eliminate any liability between the engineer and users.
Before this the test would use a string and a temporary file on the user's folder.
After this the test will use a temporary file with the proper File.createTemporaryFile method.
When using the tool to import more than one large messages
from xml exported file, this utility class will create some
tmp files, each for one large message. However it only delete
one of the tmp files. All the rest of tmp files won't get
cleaned up.
We provide a feature to mask passwords in the configuration files.
However, passwords in the bootstrap.xml (when the console is
secured with HTTPS) cannot be masked. This enhancement has
been opened to allow passwords in the bootstrap.xml to be masked
using the built-in masking feature provided by the broker.
Also the LDAPLoginModule configuration (in login.config) has a
connection password attribute that also needs this mask support.
In addition the ENC() syntax is supported for password masking
to replace the old 'mask-password' flag.
Change all use from Set<RoutingType> to EnumSet<RoutingType>
Deprecating any old exposed interfaces but keeping for back compatibility.
Address info to avoid iterator on getRoutingType hotpath, like wise can be avoided where single RoutingType is passed in.
- it is now possible to disable the TimedBuffer
- this is increasing the default on libaio maxAIO to 4k
- The Auto Tuning on the journal will use asynchronous writes to simulate what would happen on faster disks
- If you set datasync=false on the CLI, the system will suggest mapped and disable the buffer timeout
This closes#1436
This commit superseeds #1436 since it's now disabling the timed buffer through the CLI
Instead of wait to flush an executor,
I have added a method isFlushed() which will just translate to the
state on the OrderedExecutor.
In the case another executor is provided (for tests) there's a delegate
into normal executors.
This is replacing an executor on ServerSessionPacketHandler
by a this actor.
This is to avoid creating a new runnable per packet received.
Instead of creating new Runnable, this will use a single static runnable
and the packet will be send by a message, which will be treated by a listener.
Look at ServerSessionPacketHandler on this commit for more information on how it works.
The MAPPED journal refactoring include:
- simplified lifecycle and logic (eg fixed file size with single mmap memory region)
- supports for the TimedBuffer to coalesce msyncs (via Decorator pattern)
- TLAB pooling of direct ByteBuffer like the NIO journal
- remove of old benchmarks and benchmark dependencies
remove custom repo
update groupid to match artifact in maven central.
bump version also to that now deployed to maven central.
bump checkstyle version to 7.7 to make compatible.
updated checkstyle.xml to ignore existing issues which are prolific
which are now flagged in latest version as some bugs in previous meant they we'ren't detected e.g. https://github.com/checkstyle/checkstyle/issues/3320
fixing some violations which are not too prolific.
- NIO/ASYNCIO new TimedBuffer with adapting batch window heuristic
- NIO/ASYNCIO improved TimedBuffer write monitoring with
lightweight concurrent performance counters
- NIO/ASYNCIO journal/paging operations benefit from less buffer copy
- NIO/ASYNCIO any buffer copy is always performed with raw batch copy
using SIMD instrinsics (System::arrayCopy) or memcpy under the hood
- NIO improved clear buffers using SIMD instrinsics (Arrays::fill) and/or memset
- NIO journal operation perform by default TLABs allocation pooling (off heap)
retaining only the last max sized buffer
- NIO improved file copy operations using zero-copy FileChannel::transfertTo
- NIO improved zeroing using pooled single OS page buffer to clean the file
+ pwrite (on Linux)
- NIO deterministic release of unpooled direct buffers to avoid OOM errors
due to slow GC
- Exposed OS PAGE SIZE value using Env class
move classes and methods to their correct location to avoid cyclic dependencies between packages and classes.
ARTEMIS-904 Remove cyclic dependencies from artemis-cli
move classes and methods to their correct location to avoid cyclic dependencies between packages and classes.
ARTEMIS-904 Remove cyclic dependencies from artemis-cli
move classes and methods to their correct location to avoid cyclic dependencies between packages and classes.
with this we could send and receive message in their raw format,
without requiring conversions to Core.
- MessageImpl and ServerMessage are removed as part of this
- AMQPMessage and CoreMessage will have the specialized message format for each protocol
- The protocol manager is now responsible to send the message
- The message will provide an encoder for journal and paging
Due to recent changes, the web component is shutdown by the
server, but the shutdown flag is lost so the web component's
cleanup check method is not get called and the web's tmp
dir is left there after user stopped the broker (control-c).
The fix is add a suitable API to allow passing of the
flag so the web component can make sure its tmp dir gets
cleaned up properly before exiting the VM.
https://issues.apache.org/jira/browse/ARTEMIS-943
queueType in the artemis-import-export schema needs to be marked with
min and max occurs attributes to specify that the element can occur more than one
time. This is important so JAXB generates the correct bindings.
- user passwords for PropertiesLoginModule stored using PBKDF2 algothrim
by default
- implements cli user command to help create and manage user/roles
- adds a mask cli command to mask passwords
Clebert Suconic
Domenico Francesco Bruscino
franz1981
Justin Bertram
Andy Taylor
brusdev
gtully
Emmanuel Hugonnet
Wei Yang
Shailendra Kumar Singh
Emmanuel Hugonnet
Jiri Danek
Howard Gao
King Ramos
feuillemorte
Šmucr Jan
feuillemorte
Michael André Pearce
Zachary Simon
vkamble60
Robbie Gemmell
acemanace
Martyn Taylor
Ingemar Allqvist
Benjamin Graf
Dejan Bosanac
Derek Wickern
Pat Fox
Stanislav Knot
Jens Reimann
老袁
Bennet Schulz
Christopher L. Shannon (cshannon)
jbertram
Paul Gallagher