The index audit trail is currently using a BulkProcessor directly, which under
certain conditions can result in a deadlock. This occurs when the BulkProcessor
is executing a bulk request that triggers another request on the same node and
a flush of the BulkProcessor is also triggered at the same time. The flush
operation holds the lock on the bulk processor but block on acquiring a permit
from the semaphore. The request that was triggered by the bulk request blocks
the release of the semaphore permit since it needs to add a new audit message
to the BulkProcessor.
This commit works around this issue by making use of a bounded queue between the
index audit trail and the BulkProcessor with a consumer thread that handles the
add calls to the BulkProcessor.
Additionally, a new state, INITIALIZED, was added for the lifecycle of the index
audit trail. This is needed for tests since the audit trail can stop, a new
cluster state update is received, and the ShieldLifecycleService will restart the
index audit trail. At the end of the tests, the test infrastructure interrupts all
the threads and this was causing tests to fail with a InterruptedException.
Finally, the test infrastructure was also deleting the template for the index audit
logs, so this commit adds the necessary logic to prevent the deletion of this
template.
Closeselastic/elasticsearch#920
Original commit: elastic/x-pack-elasticsearch@f1b0b47b99
The change fixes two bugs in the index audit trail implementation. The first is that
we did not always store the origin type with rest requests. The second is that a
conditional statement controlled the storage of the rest requests content, but the
conditional was based on a log level that had nothing to do with the index based
audit implementation.
Closeselastic/elasticsearch#932
Original commit: elastic/x-pack-elasticsearch@b309e261c3
we're not using the _timestamp field and the path option is no longer supported
in elasticsearch 2.0 so this commit removes the field from the mapping.
Original commit: elastic/x-pack-elasticsearch@399d835d1f
We need this as the `XContentSource` supports all xcontent constructs as the root construct, while xcontent in core only supports objects. For this reason, we can't rely on xcontent auto-detection of the xcontent type. We need to be explicit about it.
Original commit: elastic/x-pack-elasticsearch@a2ed944a21
This commit is a backwards compatibilty break for all watcher indices
that had a `throttel_period` set on their watches. `throttle_period` used
to be a numeric value but now is stored as a string AND requires a unit
like seconds or minutes etc. to prevent errors. All other time valiues like
http timeouts also require units now.
Closeselastic/elasticsearch#598
Original commit: elastic/x-pack-elasticsearch@e3b2c2a4af
- lowercase `beta` and `rc`
- replaced `.betaXXX` and `.rcXXXX` suffix with `-betaXXX` and `-rcXXX`
Original commit: elastic/x-pack-elasticsearch@843d01c647
This change renames the shield audit indices to keep naming consistent with other plugins.
The name of the index uses '_' to separate words, a '-' to separate the prefix from the time
portion, and '.'s to separate the different portions of the date.
Closeselastic/elasticsearch#925
Original commit: elastic/x-pack-elasticsearch@8ca6856e4a
Previously, we were just using the current time in milliseconds from the system
for dates and the indices were not being created for UTC dates. This change
uses UTC dates for timestamps and indices resolution for index auditing.
This also ensures that custom shield forbidden apis for tests are enforced.
Closeselastic/elasticsearch#916
Original commit: elastic/x-pack-elasticsearch@724d12cb7a
We currently store the names of indices as a comma separated string instead
of an array. An array is the proper format for this information so this commit
changes the index audit trail to store the indices as an array.
Closeselastic/elasticsearch#917
Original commit: elastic/x-pack-elasticsearch@025393d91c
- moved to 2.0.0-beta1
- moved the min license version to 2.0.0
- moved to min shield version to 2.0.0
- lowercased the "beta" and "rc" part of the version
Original commit: elastic/x-pack-elasticsearch@fab1983bbb
If nodes drop and .watches / .triggered_watches shards are available after those shards were started a new cluster state update will come along that triggers the start watcher logic.
Original commit: elastic/x-pack-elasticsearch@af36f8b078
Before if timing permitted it a start and stop could be executed very close to each other time wise. This could make the manual stopped check seem to be valid, because the stop hadn't yet been performed.
Original commit: elastic/x-pack-elasticsearch@c1865c1069
This enables different constructs (primarily scripts) to set variables that can be access by subsequent constructs throughout the wathc execution. These variables are scoped to a single execution, that is, they are not persisted across multiple executions of the same watch.
Closeselastic/elasticsearch#589
Original commit: elastic/x-pack-elasticsearch@34223d1991
There are a lot of settigns which we don't need at all or are already
defined in the parent. For instance dependencies are not needed if
they are included in elasticsearch-parent.
This commit also removes the shading which we don't do anymore in
core and we include guava already.
Original commit: elastic/x-pack-elasticsearch@c4f951a751
Upgrade to Elasticsearch 2.0.0-SNAPSHOT
This moves the master branch to follow Elasticsearch 2.0.0-SNAPSHOT and fixes most problems that occurred during the upgrade. The remaining issues not yet fixed are:
* `HttpClient` and the `Account` used for Email support need to install security manager which is not supported by the elasticsearch security policy. This is not yet resolved an requires fundamental changes and/or a rule in the core policy file. See elastic/elasticsearch#597
* Due to changes to the way Time/Byte settings are parsed settings without a unit must be upgraded. See elastic/elasticsearch#598
* REST tests are currently disabled due to some limitations from Elasticsearch core that don't allow to run 3rd party REST tests. See elastic/elasticsearch#599
Watcher now also inherits the elaticsearch-parent pom file and all it's properties.
Original commit: elastic/x-pack-elasticsearch@1e03234e3e
The execution service will throw an exection if we acquire a watch lock or add a watch to the current executions if Watcher has stopped or is stopping. We should deal with this sitation by properly catching those failures. FYI if a watch managed to put itself to the current executions a shutdown will wait until a watch execution is finished.
Original commit: elastic/x-pack-elasticsearch@5692316072
The beforeIndexDeletion method expects that nothing is still indexing when it runs
but this is not the case as the index audit trail will continue indexing events that
occur as checks are being performed in the cluster.
Original commit: elastic/x-pack-elasticsearch@02001a5222
Previously we relied on the default dynamic mapping for the audit index output, which did
not create an ideal mapping. This change adds a mapping file with default settings and
dynamic mapping disabled for the audit indexes.
Additionally, the ability to override settings for the audit indexes has been provided so that
users can customize the number of shards and replicas to meet their needs.
In order to implement these changes, the index audit service had to be moved from a lifecycle
component to an abstract component that had its own custom lifecycle on top of the cluster
state. A ShieldLifecycleService class was added to accomplish this. In the future, this class
can be used for other services that need to perform index based operations.
Closeselastic/elasticsearch#913
Original commit: elastic/x-pack-elasticsearch@231740c1cc
uboness
Martijn van Groningen
jaymode
Simon Willnauer
Boaz Leskes
Simon Willnauer
Robert Muir
Alexander Reelsen