Commit Graph

1535 Commits

Author SHA1 Message Date
jaymode d65bfd1721 use doc_values for most fields in audit indices
This changes the mappings for the audit indices to use doc_values for all fields
other than the request_body, which will have a lot of variance. Additionally, the
request_body field is no longer indexed.

Closes elastic/elasticsearch#918

Original commit: elastic/x-pack-elasticsearch@4917529ffa
2015-06-23 09:40:27 -04:00
Martijn van Groningen d2d1c44d07 test: increased test logging
Original commit: elastic/x-pack-elasticsearch@455892c628
2015-06-23 15:31:06 +02:00
uboness 5daddf3fc1 Fixed version scheme
- Lowercase `beta` and `rc`
- use `-beta` and `-rc` suffixes instead of `.beta`and `.rc`

Original commit: elastic/x-pack-elasticsearch@74860d8252
2015-06-23 11:40:34 +02:00
Martijn van Groningen 075e6caba0 logging: if starting watcher fails this should be more visible
Original commit: elastic/x-pack-elasticsearch@26da9d7661
2015-06-23 08:42:56 +02:00
jaymode f0d5c2da61 make addition of message to the bulk processor asynchronous
The index audit trail is currently using a BulkProcessor directly, which under
certain conditions can result in a deadlock. This occurs when the BulkProcessor
is executing a bulk request that triggers another request on the same node and
a flush of the BulkProcessor is also triggered at the same time. The flush
operation holds the lock on the bulk processor but block on acquiring a permit
from the semaphore. The request that was triggered by the bulk request blocks
the release of the semaphore permit since it needs to add a new audit message
to the BulkProcessor.

This commit works around this issue by making use of a bounded queue between the
index audit trail and the BulkProcessor with a consumer thread that handles the
add calls to the BulkProcessor.

Additionally, a new state, INITIALIZED, was added for the lifecycle of the index
audit trail. This is needed for tests since the audit trail can stop, a new
cluster state update is received, and the ShieldLifecycleService will restart the
index audit trail. At the end of the tests, the test infrastructure interrupts all
the threads and this was causing tests to fail with a InterruptedException.

Finally, the test infrastructure was also deleting the template for the index audit
logs, so this commit adds the necessary logic to prevent the deletion of this
template.

Closes elastic/elasticsearch#920

Original commit: elastic/x-pack-elasticsearch@f1b0b47b99
2015-06-22 16:34:23 -04:00
jaymode 9fcd68c8f4 always store origin type and request content for rest requests
The change fixes two bugs in the index audit trail implementation. The first is that
we did not always store the origin type with rest requests. The second is that a
conditional statement controlled the storage of the rest requests content, but the
conditional was based on a log level that had nothing to do with the index based
audit implementation.

Closes elastic/elasticsearch#932

Original commit: elastic/x-pack-elasticsearch@b309e261c3
2015-06-22 15:05:34 -04:00
jaymode 4e11cbebad remove _timestamp from audit index mapping
we're not using the _timestamp field and the path option is no longer supported
in elasticsearch 2.0 so this commit removes the field from the mapping.

Original commit: elastic/x-pack-elasticsearch@399d835d1f
2015-06-22 14:54:37 -04:00
Simon Willnauer 58bd0cc509 [TEST] Remove awaits-fix, the issue is resovled in es-core master
Original commit: elastic/x-pack-elasticsearch@eb813e83dd
2015-06-22 12:54:03 +02:00
uboness 3cf6b32f6a [cleanup] XContentSource now requires XContentType
We need this as the `XContentSource` supports all xcontent constructs as the root construct, while xcontent in core only supports objects. For this reason, we can't rely on xcontent auto-detection of the xcontent type. We need to be explicit about it.

Original commit: elastic/x-pack-elasticsearch@a2ed944a21
2015-06-22 12:24:56 +02:00
Simon Willnauer 797945b586 [TEST] Beef up test to sometimes pass a plain number as a string
Original commit: elastic/x-pack-elasticsearch@f42662c719
2015-06-22 12:11:17 +02:00
Simon Willnauer 5ea2d0528c Require units for all time values
This commit is a backwards compatibilty break for all watcher indices
that had a `throttel_period` set on their watches. `throttle_period` used
to be a numeric value but now is stored as a string AND requires a unit
like seconds or minutes etc. to prevent errors. All other time valiues like
http timeouts also require units now.

Closes elastic/elasticsearch#598

Original commit: elastic/x-pack-elasticsearch@e3b2c2a4af
2015-06-22 11:47:37 +02:00
uboness 5766c64f94 added support for 1.0.0-rc1 in WatcherVersion
Original commit: elastic/x-pack-elasticsearch@02c018c0b9
2015-06-19 15:51:22 +02:00
uboness fa57b2c78e removed javax.mail from test dependencies
Original commit: elastic/x-pack-elasticsearch@294f461456
2015-06-19 11:45:57 +02:00
uboness b4342d6bd4 changed version number to the new scheme
- lowercase `beta` and `rc`
- replaced `.betaXXX` and `.rcXXXX` suffix with `-betaXXX` and `-rcXXX`

Original commit: elastic/x-pack-elasticsearch@843d01c647
2015-06-19 11:26:12 +02:00
uboness fad95315fb remove @Repeat annotation from test
Original commit: elastic/x-pack-elasticsearch@d5414a32e8
2015-06-19 01:46:32 +02:00
jaymode d302b04256 rename shield audit indices to keep naming consistent
This change renames the shield audit indices to keep naming consistent with other plugins.
The name of the index uses '_' to separate words, a '-' to separate the prefix from the time
portion, and '.'s to separate the different portions of the date.

Closes elastic/elasticsearch#925

Original commit: elastic/x-pack-elasticsearch@8ca6856e4a
2015-06-18 15:53:51 -04:00
uboness d41815ca18 fixed version scheme in WatcherVersion
Original commit: elastic/x-pack-elasticsearch@1007cd71a3
2015-06-18 21:50:39 +02:00
jaymode bf9a8024ca Test: remove @Repeat instead of suppressing
Original commit: elastic/x-pack-elasticsearch@cc61a9d14a
2015-06-18 15:06:25 -04:00
Simon Willnauer bc4ce6f153 Remove @Repeat - don't commit hardcoded repeats
Original commit: elastic/x-pack-elasticsearch@e9de83bb3a
2015-06-18 20:59:46 +02:00
jaymode 0533b55dbc Test: add SuppressForbidden for tests with @Repeat
Original commit: elastic/x-pack-elasticsearch@41247fa507
2015-06-18 14:55:33 -04:00
jaymode a5eefb6259 use UTC dates for audit indexing
Previously, we were just using the current time in milliseconds from the system
for dates and the indices were not being created for UTC dates. This change
uses UTC dates for timestamps and indices resolution for index auditing.

This also ensures that custom shield forbidden apis for tests are enforced.

Closes elastic/elasticsearch#916

Original commit: elastic/x-pack-elasticsearch@724d12cb7a
2015-06-18 14:25:38 -04:00
Simon Willnauer 11e928bbe4 Reenable Security Manager
Closes elastic/elasticsearch#597

Original commit: elastic/x-pack-elasticsearch@d7737e6f2f
2015-06-18 20:23:10 +02:00
jaymode 2b3c157c97 store names of indices as an array instead of a string
We currently store the names of indices as a comma separated string instead
of an array. An array is the proper format for this information so this commit
changes the index audit trail to store the indices as an array.

Closes elastic/elasticsearch#917

Original commit: elastic/x-pack-elasticsearch@025393d91c
2015-06-18 13:26:51 -04:00
uboness 38f3c8b607 Fixed the version
- moved to 2.0.0-beta1
- moved the min license version to 2.0.0
- moved to min shield version to 2.0.0
- lowercased the "beta" and "rc" part of the version

Original commit: elastic/x-pack-elasticsearch@fab1983bbb
2015-06-18 17:06:44 +02:00
Martijn van Groningen 7a55a957d3 removed retry start watcher mechanism
If nodes drop and .watches / .triggered_watches shards are available after those shards were started a new cluster state update will come along that triggers the start watcher logic.

Original commit: elastic/x-pack-elasticsearch@af36f8b078
2015-06-18 15:48:16 +02:00
Martijn van Groningen ca58e93150 applied feedback
Original commit: elastic/x-pack-elasticsearch@4767935457
2015-06-18 15:26:14 +02:00
jaymode 6202914ffc Test: add '.' to allowed name characters
Original commit: elastic/x-pack-elasticsearch@26a254a4c2
2015-06-18 09:20:30 -04:00
Martijn van Groningen c4cb85104e lifecycle: Verify whether Watcher was stopped manually under a lock
Before if timing permitted it a start and stop could be executed very close to each other time wise. This could make the manual stopped check seem to be valid, because the stop hadn't yet been performed.

Original commit: elastic/x-pack-elasticsearch@c1865c1069
2015-06-18 13:45:55 +02:00
uboness 8af461daaf removed all timezone'less DateTime constructions
Now as this rule was added to the forbidden APIs in core, this is required for the build to run.

Original commit: elastic/x-pack-elasticsearch@a3b570e7f1
2015-06-18 13:28:25 +02:00
uboness aef6ad5c12 Added dynamic variables registry to the execution context
This enables different constructs (primarily scripts) to set variables that can be access by subsequent constructs throughout the wathc execution. These variables are scoped to a single execution, that is, they are not persisted across multiple executions of the same watch.

Closes elastic/elasticsearch#589

Original commit: elastic/x-pack-elasticsearch@34223d1991
2015-06-18 12:17:07 +02:00
Martijn van Groningen bcb7428868 Renamed `email.email` to `email.message` in the email action result.
Closes elastic/elasticsearch#592

Original commit: elastic/x-pack-elasticsearch@043a4084e6
2015-06-18 11:20:55 +02:00
Simon Willnauer 4c6d709bca Add back lost tests.timewarp settings
This setting was lost during the upgrade to Elasticsearch 2.0

Original commit: elastic/x-pack-elasticsearch@d0e3b0e080
2015-06-18 10:24:18 +02:00
Simon Willnauer 64fce2b360 Cleanup pom.xml after using elasticsearch-parent pom
There are a lot of settigns which we don't need at all or are already
defined in the parent. For instance dependencies are not needed if
they are included in elasticsearch-parent.

This commit also removes the shading which we don't do anymore in
core and we include guava already.

Original commit: elastic/x-pack-elasticsearch@c4f951a751
2015-06-18 10:19:06 +02:00
Simon Willnauer e96eab8ba0 Enable REST tests
Enable REST tests again after core support for 3rd party REST tests
was added back.

Closes elastic/elasticsearch#599

Original commit: elastic/x-pack-elasticsearch@a5fffbca6e
2015-06-18 09:25:13 +02:00
Boaz Leskes 3394c2e883 Add a debug log to IndexAuditTrail
Original commit: elastic/x-pack-elasticsearch@ccff079b12
2015-06-18 08:55:34 +02:00
Simon Willnauer 483fc360f4 [BUILD] Remove dead files from ancient build system
Original commit: elastic/x-pack-elasticsearch@98aa655945
2015-06-17 21:14:19 +02:00
Simon Willnauer 6d255c0c49 [TEST] Use temp directory to fix script loading on windows
Original commit: elastic/x-pack-elasticsearch@6fc6c29718
2015-06-17 21:06:33 +02:00
uboness 110a9ed6a0 Added missing mail jar in the distribution
Original commit: elastic/x-pack-elasticsearch@d05af66792
2015-06-17 20:23:15 +02:00
Simon Willnauer d730917b53 Merge pull request elastic/elasticsearch#594 from elastic/upgrade_to_es_20
Upgrade to Elasticsearch 2.0.0-SNAPSHOT

This moves the master branch to follow Elasticsearch 2.0.0-SNAPSHOT and fixes most problems that occurred during the upgrade. The remaining issues  not yet fixed are:

 * `HttpClient` and the `Account` used for Email support need to install security manager which is not supported by the elasticsearch security policy. This is not yet resolved an requires fundamental changes and/or a rule in the core policy file. See elastic/elasticsearch#597
 * Due to changes to the way Time/Byte settings are parsed settings without a unit must be upgraded.  See elastic/elasticsearch#598
 * REST tests are currently disabled due to some limitations from Elasticsearch core that don't allow to run 3rd party REST tests. See elastic/elasticsearch#599

Watcher now also inherits the elaticsearch-parent pom file and all it's properties.

Original commit: elastic/x-pack-elasticsearch@1e03234e3e
2015-06-17 17:30:40 +02:00
Simon Willnauer de39879558 Merge branch 'master' into upgrade_to_es_20
Conflicts:
	src/main/java/org/elasticsearch/watcher/execution/CurrentExecutions.java
	src/main/java/org/elasticsearch/watcher/watch/WatchLockService.java
	src/test/java/org/elasticsearch/watcher/actions/throttler/ActionThrottleTests.java

Original commit: elastic/x-pack-elasticsearch@ab51f0104f
2015-06-17 15:56:45 +02:00
Simon Willnauer d5c94148a5 remove nocommits
Original commit: elastic/x-pack-elasticsearch@8e0f99ea0e
2015-06-17 15:53:19 +02:00
Simon Willnauer b0986aec1c disable REST
Original commit: elastic/x-pack-elasticsearch@c15b03e388
2015-06-17 15:47:40 +02:00
Simon Willnauer ef7d216b4e make the rest pass
Original commit: elastic/x-pack-elasticsearch@3d60f13994
2015-06-17 15:47:39 +02:00
jaymode 853f8d50ef Test: assert that the current thread is not already interrupted
Original commit: elastic/x-pack-elasticsearch@acc517ca53
2015-06-17 09:38:50 -04:00
Robert Muir 9886ecab5c fix maven to work at all
Original commit: elastic/x-pack-elasticsearch@28f61700e6
2015-06-17 09:15:17 -04:00
Martijn van Groningen b82fca3379 We should cope with the situation when a watch is going to be executed whilst shutting down.
The execution service will throw an exection if we acquire a watch lock or add a watch to the current executions if Watcher has stopped or is stopping. We should deal with this sitation by properly catching those failures. FYI if a watch managed to put itself to the current executions a shutdown will wait until a watch execution is finished.

Original commit: elastic/x-pack-elasticsearch@5692316072
2015-06-17 13:38:38 +02:00
jaymode d036fc505d Test: override beforeIndexDeletion for audit trail tests
The beforeIndexDeletion method expects that nothing is still indexing when it runs
but this is not the case as the index audit trail will continue indexing events that
occur as checks are being performed in the cluster.

Original commit: elastic/x-pack-elasticsearch@02001a5222
2015-06-17 06:26:12 -04:00
Alexander Reelsen 22c77805bd Test: Removed river handler due to removal in master
Original commit: elastic/x-pack-elasticsearch@c6a3372ce0
2015-06-17 12:07:02 +02:00
Martijn van Groningen 2f14240915 test: increased time throttle timeout to deal with timing issues
Original commit: elastic/x-pack-elasticsearch@90fd3c9cd9
2015-06-17 09:17:13 +02:00
jaymode 38837b310c [Test] disable random dynamic templates for audit index
Original commit: elastic/x-pack-elasticsearch@914e0990ef
2015-06-16 15:43:58 -04:00