4837 Commits

Author SHA1 Message Date
Daniel Mitterdorfer
fa1219fd13 Merge remote-tracking branch 'origin/master' into strict-booleans
Original commit: elastic/x-pack-elasticsearch@6fbbaf19dd
2016-12-21 08:47:45 +01:00
Martijn van Groningen
0b5b26284b Validate if job index already exists after job validation.
Closes elastic/elasticsearch#594

Original commit: elastic/x-pack-elasticsearch@b3b7b086a7
2016-12-21 08:34:16 +01:00
Jason Tedor
1178112c4a Skip pre-built transport client test on JDK 9
This test will currently not pass because Netty tries to do unsafe
operations that JDK 9 shoots down.

Relates elastic/elasticsearch#4430

Original commit: elastic/x-pack-elasticsearch@42aa250d4c
2016-12-20 20:25:41 -05:00
Tim B
f45c24faf2 Fix x-pack readme to reflect project name and dev-setup changes (elastic/elasticsearch#4402)
* Update readme to reflect new dev setup directory structure.

* Fix typo in elasticsearch-extra path in readme

* Update gradle exception for x-pack directory structure.

* Make directory path where x-pack must be checked out explicit in the gradle exception

Original commit: elastic/x-pack-elasticsearch@91f1d04542
2016-12-20 12:43:32 -06:00
Nik Everett
b2dc1bdca3 Introduce NamedXContentRegistry (elastic/elasticsearch#4399)
This "super registry" will eventually replace things like
`IndiciesQueriesRegistry` but for now it is just another thing
to plumb across requests.

Original commit: elastic/x-pack-elasticsearch@da26a42b36
2016-12-20 11:05:42 -05:00
Alexander Reelsen
6f7a065605 Watcher: Parse index action results to prevent exceptions (elastic/elasticsearch#4424)
When the index action is used to do some bulk indexing, the single
items of the response were not checked to have been indexed successful.
This could lead to NPEs due to an index response being null when the index
operation had failed. The action was still logged as a success though.

This commit only returns SUCCESS for the action, if all items were indexed
successfully. If all items failed, the result will be FAILED as well. Lastly
the result status PARTIAL_FAILURE is used if there were successful and unsuccessful
index operations.

Additionally some minor cleanups happened, like changing equals/hashcode.

Closes elastic/elasticsearch#4416

Original commit: elastic/x-pack-elasticsearch@692687e1af
2016-12-20 16:53:12 +01:00
Dimitrios Liappis
17c3164979 [ci] Fix bug with -Dtests.jvm.argline
The special `-Dtests.jvm.argline` params needed for jdk-9 builds do not
get passed correctly if enclosed within single quotes.

Fix jdk9 target for `dev-tools/ci` script to correctly pass the
-Dtests.jvm.argline parameters.

Relates:  elastic/elasticsearch#4428 

Original commit: elastic/x-pack-elasticsearch@6cd329b8da
2016-12-20 16:35:18 +02:00
Alexander Reelsen
0f8e6354f6 Watcher: Allow deletion of watches regardless of license (elastic/elasticsearch#4414)
The user might want to delete watches on expiry to prevent
further execution, without any actions being executed.

Closes elastic/elasticsearch#4413

Original commit: elastic/x-pack-elasticsearch@244a1b9cd8
2016-12-20 15:26:06 +01:00
Yannick Welsch
68bfc90cea Let ClusterStateObserver only hold onto state that's needed for change detection
Companion commit for elastic/elasticsearchelastic/elasticsearch#21631

Original commit: elastic/x-pack-elasticsearch@3333480707
2016-12-20 15:23:00 +01:00
Martijn van Groningen
999a1eb234 fix JobTests#testEquals_noId by fixing regex expression and including -
Original commit: elastic/x-pack-elasticsearch@b295d5f621
2016-12-20 15:15:47 +01:00
David Roberts
8b135226f6 Fix long line
Original commit: elastic/x-pack-elasticsearch@8d030d8cf8
2016-12-20 14:09:39 +00:00
David Kyle
537fea313f Add job config option to set the index name (elastic/elasticsearch#560)
* Add job config option to set the index name

* Check index does not already exist if ’index_name’ is set

* Don’t create alias if ‘index_name’ is the same as ‘job_id’

* Default index_name value

Set it to job_id if null and only create the index alias if job_id != index_name

* Fix compile errors after rebasing

* Address review comments

* Test if the index exists by checking the cluster state

* Update comment

Original commit: elastic/x-pack-elasticsearch@a3e7f1a5bb
2016-12-20 14:03:24 +00:00
David Roberts
3383c67cf2 Ensure automatically generated job IDs meet the validation criteria (elastic/elasticsearch#586)
Original commit: elastic/x-pack-elasticsearch@34b67cb509
2016-12-20 13:53:29 +00:00
Alexander Reelsen
b994f16f4a Watcher: Ensure non duplicate Wid ids (elastic/elasticsearch#4423)
Removing the WatchLockService could result in duplication of wids, because of a wrong
call to replace underscores with dashes. As UUIDs.createBase64UUID() can contain underscores
but they are kind of reserved in the Wid class due to handling of watch ids, this just uses
the toString() representation of a random UUID.

Closes elastic/elasticsearch#4422

Original commit: elastic/x-pack-elasticsearch@dceb01ae5e
2016-12-20 14:02:00 +01:00
David Roberts
5b029ee595 Correct test condition and add a message in case of future failures
Original commit: elastic/x-pack-elasticsearch@55729b39fa
2016-12-20 11:22:03 +00:00
David Roberts
1615be4ff3 Moved the C++ code under the cpp sub-directory (elastic/elasticsearch#584)
The $PRELERT_SRC_HOME environment variable is replaced with $CPP_SRC_HOME,
which points to the cpp sub-directory off the repository root

Original commit: elastic/x-pack-elasticsearch@02ef6d6be6
2016-12-20 11:03:00 +00:00
David Kyle
78fdca65d5 Add job Id to state document IDs (elastic/elasticsearch#579)
* Persist quantile documents with the jobId in the document Id

* Add job Id to snapshot Id

* Add job Id to categoriser state document Id

* Rename quantiles doc to start with job id as the other state docs do

* Fix restoring categoriser state

Original commit: elastic/x-pack-elasticsearch@3e5d3368b5
2016-12-20 10:47:37 +00:00
Dimitrios Athanasiou
316f6bf6cb Remove unused methods and fix test
Original commit: elastic/x-pack-elasticsearch@12290517d0
2016-12-20 10:15:53 +00:00
Dimitris Athanasiou
bee58f8b81 Add get API for schedulers (elastic/elasticsearch#582)
Original commit: elastic/x-pack-elasticsearch@59c555d60a
2016-12-20 10:09:45 +00:00
Daniel Mitterdorfer
b9a38d9b97 Enable trace logging for LocalExporterTests#testSimpleExport()
This test fails due to a changing cluster state(?).  The
test checks that a local exporter is ready and then continues.
However, during the test, we see output similar to:

skipping exporter [_local] as it isn't ready yet

Which indicates that the cluster state has changed and the
exporter does not return a bulk anymore. Hence, the test is
failing although at one point in time it returned a bulk.

By enabling trace logging we should be able to find out
what's going on.

Original commit: elastic/x-pack-elasticsearch@d7e2200dd9
2016-12-20 08:54:26 +01:00
Ryan Ernst
789df7d2fa Internal: Update signatures for EnvironmentAwareCommand (elastic/elasticsearch#4367)
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#22175

Original commit: elastic/x-pack-elasticsearch@4359cb1947
2016-12-19 15:23:53 -08:00
javanna
adf22c3d24 Adapt to strict parsing mode removal
Original commit: elastic/x-pack-elasticsearch@2d6276a60f
2016-12-19 19:39:08 +01:00
Dimitrios Liappis
8a3181ceed [ci] Add jdk9 target to the dev-tools/ci script
To add support for jdk9 tests for x-pack we need a number of parameters
for the jvm, passed in the gradle command. To resolve issues with arg
passing through bash variables and improve readability, switch to an
array construct for storing the arguments.

Add new jdk9 target to allow execution of jdk-9-ea based tests in the CI
infra.

Relates: elastic/elasticsearch#4415 
Relates: https://github.com/elastic/infra/issues/1573#issuecomment-267832307

Original commit: elastic/x-pack-elasticsearch@b41e5f4b98
2016-12-19 19:53:00 +02:00
David Roberts
9b3764b7fa Max jobs per node cannot be dynamic due to threadpool (elastic/elasticsearch#578)
The threadpool that supplies the threads used for job IO cannot be
resized, so the number of jobs cannot be dynamic either

Original commit: elastic/x-pack-elasticsearch@c584bf7147
2016-12-19 17:48:22 +00:00
Alexander Reelsen
79a8f27569 Watcher: Remove WatchLockService (elastic/elasticsearch#4365)
The watch lock service is not really needed, as there is already
a data structure that has information about the currently executing
watches, that can be consulted before executed.

This change will now check, if there is already a watch running with
the current id. If there is not, execution will happen as usual. If
there is however, than a watch record will be created, stating that
the watch is currently being executed - which means that it is either
being executed or in the list of planned executions.

This way users can check in the watch history, if a watch has been executed
more often than it should.

In order to easily search for this, a new execution state called
`NOT_EXECUTED_ALREADY_QUEUED` has been added.

Original commit: elastic/x-pack-elasticsearch@867acec3c3
2016-12-19 17:33:48 +01:00
David Roberts
f65125df55 No point rethrowing exception in thread pool task
It will just get logged twice, but with a less useful message the
second time

Original commit: elastic/x-pack-elasticsearch@7d32f65cc1
2016-12-19 16:24:45 +00:00
Dimitrios Athanasiou
081bebbbcd Remove unused constants
Original commit: elastic/x-pack-elasticsearch@0f72f1fc6e
2016-12-19 14:22:05 +00:00
Dimitris Athanasiou
8750e5f926 Redesign the get anomaly_detectors APIs (elastic/elasticsearch#570)
* Redesign the get anomaly_detectors APIs

This commit redesigns the APIs to get anomaly_detectors.

The new design has 2 GET APIs:

- An API to get the configurations: /anomaly_detectors/{job_id}
- An API to get the stats: /anomaly_detectors/{job_id}/_stats

For both APIs entering "_all" as the job_id returns results for
all jobs.

Note that page params have been removed. They were useful
when the configs were stored in an index. Now that they are part
of cluster state there is no need. Additionally, future support
for wildcard job_id expressions will give users a tool to narrow
down the GET actions to a certain subset of jobs which will be
more useful than the from/size approach.

Follow up:

- Implement similar GET APIs for schedulers
- Remove scheduler_stats from the anomaly_detectors _stats API
as it will be part of the schedulers _stats API

Closes elastic/elasticsearch#548

Original commit: elastic/x-pack-elasticsearch@046a0db8f5
2016-12-19 13:09:54 +00:00
Daniel Mitterdorfer
2cb41e690d Address initial review comments
Original commit: elastic/x-pack-elasticsearch@bcd6433afb
2016-12-19 13:27:40 +01:00
Daniel Mitterdorfer
24a4290379 Merge remote-tracking branch 'origin/master' into strict-booleans
Original commit: elastic/x-pack-elasticsearch@32a533f38f
2016-12-19 12:58:30 +01:00
David Kyle
4eda09d24a Don’t write job description if not set
Original commit: elastic/x-pack-elasticsearch@6e7ff939d3
2016-12-19 11:27:32 +00:00
David Roberts
854dc88cbd Corrections to previous commit
Original commit: elastic/x-pack-elasticsearch@a9e94119e1
2016-12-19 10:28:23 +00:00
David Roberts
028a505fe5 Update copyright headers (elastic/elasticsearch#572)
* Added the Elastic copyright header to C++ files

* Added the name of the copyright holder to Java files

Original commit: elastic/x-pack-elasticsearch@aea1b5a656
2016-12-19 10:19:21 +00:00
Dimitris Athanasiou
6a14f52d2e Remove aggs and retrieve_whole_source from SchedulerConfig (elastic/elasticsearch#569)
Aggs does not need to be a separate member field. There can simply
be an aggs parse field which also then stored onto the aggregations
parse field.

Finally, retrieve_whole_source is unnecessary as we move towards
a node client based data extraction.

Original commit: elastic/x-pack-elasticsearch@14024c2ee5
2016-12-19 09:54:59 +00:00
David Roberts
89e506f050 Merge branch 'analytics-dev' into master
Original commit: elastic/x-pack-elasticsearch@dc9d10a502
2016-12-19 09:07:12 +00:00
Daniel Mitterdorfer
7a652fa090 Merge pull request elastic/elasticsearch#4398 from danielmitterdorfer/xcontent_strict_duplicate_checks
Enable strict duplicate checks for all XContent types

Original commit: elastic/x-pack-elasticsearch@ad249c2e65
2016-12-19 09:30:19 +01:00
Tim Vernum
df7c528c66 Change classification of event_type in index audit trail (elastic/elasticsearch#4375)
In LoggingAuditTrail and the audit-event filtering, we distinguish between single-realm authentication failures ("realm_authentication_failure") and global failures ("authentication_failure").

Update the message output of IndexAuditTrail to reflect this distinction.

Original commit: elastic/x-pack-elasticsearch@56802ae2df
2016-12-19 14:39:21 +11:00
Simon Willnauer
c9cfedc3cb [TEST] check if .watches index / alias exists before creating it
Original commit: elastic/x-pack-elasticsearch@74d85121eb
2016-12-18 10:17:43 +01:00
Areek Zillur
278496d5e5 x-pack changes for removing deprecated _suggest endpoint (elasticsearchelastic/elasticsearch#22203)
Original commit: elastic/x-pack-elasticsearch@b9bba20fae
2016-12-16 13:06:35 -05:00
David Kyle
aacb6f3949 Move JobResultsPersister.getJobIndexName to new class AnomalyDetectorsIndex (elastic/elasticsearch#567)
Original commit: elastic/x-pack-elasticsearch@bf54dd56a9
2016-12-16 17:47:52 +00:00
Martijn van Groningen
e3ec908828 run with native process by default
Original commit: elastic/x-pack-elasticsearch@c73beec522
2016-12-16 15:21:11 +01:00
Martijn van Groningen
eaca8fb06a Stop local allocated jobs before closing node.
Original commit: elastic/x-pack-elasticsearch@ad3ed7c86a
2016-12-16 14:59:35 +01:00
David Roberts
c43a9ba1dd Fix intermittent failure of renormalization (elastic/elasticsearch#558)
The synchronization was flawed

Original commit: elastic/x-pack-elasticsearch@a968c68c7d
2016-12-16 13:45:47 +00:00
Daniel Mitterdorfer
d3c1fbe002 Enable strict duplicate checks for all XContent types
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default for all XContent types (not only JSON).

We have also changed the name of the system property to disable this feature
from `es.json.strict_duplicate_detection` to the now more appropriate name
`es.xcontent.strict_duplicate_detection`.

Relates elastic/elasticsearchelastic/elasticsearch#19614
Relates elastic/elasticsearchelastic/elasticsearch#22073

Original commit: elastic/x-pack-elasticsearch@c92e2a6c55
2016-12-16 13:57:55 +01:00
Colin Goodheart-Smithe
5e6c63bc51 Removes warnings
Original commit: elastic/x-pack-elasticsearch@59626a3a3a
2016-12-16 12:35:45 +00:00
Alexander Reelsen
0d9cb5f7bd Docs: Updating execution phases of watch execution
Closes elastic/elasticsearch#4079

Original commit: elastic/x-pack-elasticsearch@658ba0a7f7
2016-12-16 11:30:27 +01:00
Dimitris Athanasiou
c932588c68 Repackage scheduler classes (elastic/elasticsearch#559)
* Repackage scheduler classes

* Rename and move scheduler status tests

Original commit: elastic/x-pack-elasticsearch@a9d7cc4e48
2016-12-16 09:50:06 +00:00
Daniel Mitterdorfer
e3302da589 Make boolean conversion strict
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.

Original commit: elastic/x-pack-elasticsearch@6400f18911
2016-12-15 19:06:59 +01:00
Boaz Leskes
b2972a142c Remove Cluster State Status (elastic/elasticsearch#4313)
A companion PR for https://github.com/elastic/elasticsearch/pull/21817

Original commit: elastic/x-pack-elasticsearch@392235877f
2016-12-15 17:07:02 +01:00
David Kyle
8d5c8b24d6 Make ModelDebugOutput fields snake case (elastic/elasticsearch#555)
* snake_case model debug output fields

* Rename CategorizerState and BucketInfluencer types to snake_case

Original commit: elastic/x-pack-elasticsearch@da94dc7ec1
2016-12-15 15:42:38 +00:00