Commit Graph

6291 Commits

Author SHA1 Message Date
Lisa Cawley 2abf36dd2c [DOCS] Add custom URL instructions (elastic/x-pack-elasticsearch#2215)
* [DOCS] Add custom URL instructions

* [DOCS] Update custom URL examples

* [DOCS] Add example for custom URL

* [DOCS] Add ML category terms custom URL example

* [DOCS] Add tip for pop-up ML custom URLs

* [DOCS] Address feedback about ML custom URLs

* [DOCS] Clarify impact of interval on custom URL

Original commit: elastic/x-pack-elasticsearch@eee192226f
2017-08-15 10:32:11 -07:00
David Roberts db8885a46e [ML] Do not download the ml-cpp zip when building it locally (elastic/x-pack-elasticsearch#2262)
When the machine-learning-cpp repo is built locally, the zip file it
creates is preferred over that downloaded from s3 when creating the
overall x-pack-elasticsearch zip.  However, prior to this change the
build would ALSO download an ml-cpp zip from s3, and just not use it.

Original commit: elastic/x-pack-elasticsearch@bd71637edd
2017-08-15 16:31:23 +01:00
Jason Tedor f3a7d46698 Rename CONF_DIR to ES_PATH_CONF
This commit is following upstream Elasticsearch which has renamed the
environment variable used to specify a custom configuration directory
from CONF_DIR to ES_PATH_CONF.

Relates elastic/x-pack-elasticsearch#2261

Original commit: elastic/x-pack-elasticsearch@9ae29941e5
2017-08-15 06:19:39 +09:00
Tim Vernum a27dc257c9 Gracefully handle no content(-type) in Put License (elastic/x-pack-elasticsearch#2258)
PUT /_xpack/license with no content or content-type should fail with an appropriate error message rather than throwing NPE.

Original commit: elastic/x-pack-elasticsearch@f8c744d2a2
2017-08-14 20:39:39 +10:00
Alexander Reelsen 69b3ffa40a Tests: Remove useless wait time in watcher REST test
The rest test waited for the watch to run in the background, but there
were no guarantees that this really happened. Also it waited for five
seconds, instead of just executing the watch manually.

relates elastic/x-pack-elasticsearch#2255

Original commit: elastic/x-pack-elasticsearch@56765a649e
2017-08-14 11:52:26 +02:00
Alexander Reelsen 5416a6afd4 Tests: Fix timeout in watcher history template test
Due to an invalid timeout the test failed earlier than it should have.

relates elastic/x-pack-elasticsearch#2222

Original commit: elastic/x-pack-elasticsearch@2265c419e3
2017-08-14 09:53:52 +02:00
Lisa Cawley ea05ddd513 [DOCS] Fix principal access_granted attribute (elastic/x-pack-elasticsearch#2257)
Original commit: elastic/x-pack-elasticsearch@9c33afce9f
2017-08-11 16:53:21 -07:00
lcawley e2f7081693 [DOCS] Add abbreviated titles
Original commit: elastic/x-pack-elasticsearch@a4cf8a363f
2017-08-11 10:00:35 -07:00
lcawley 8d9dc6ff65 [DOCS] Clarify title for X-Pack install
Original commit: elastic/x-pack-elasticsearch@9712bbd1d5
2017-08-11 09:15:20 -07:00
David Roberts b5d159bc1c [ML] Handle simultaneous force delete datafeed and stop datafeed (elastic/x-pack-elasticsearch#2243)
This is an important case as the UI force stops datafeeds now.

Fixes elastic/x-pack-kibana#2083

Original commit: elastic/x-pack-elasticsearch@4d0f62ad2d
2017-08-11 14:43:24 +01:00
Alexander Reelsen 3ad2f5e9f5 Tests: Increase logging for watcher smoke tests
These tests have repeating but not reproducible failures,
where the stash is filled with a second PUT operation and the
watcher stats response does not match. Setting the log to trace
should shed some light on this.

As the smoke tests are only four tests this will not lead to a
log explosion.

Relates elastic/x-pack-elasticsearch#1513, elastic/x-pack-elasticsearch#1874

Original commit: elastic/x-pack-elasticsearch@5832dc7990
2017-08-11 15:29:52 +02:00
Alexander Reelsen 26c5766a0d Tests: Do not delete index templates on bwc tests
In order to run the bwc tests there is no need to delete the
index template at the end of a test run which results
in recreation of those due to all the cluster state listener.

Relates elastic/x-pack-elasticsearch#2228

Original commit: elastic/x-pack-elasticsearch@702d1c61ed
2017-08-11 14:11:50 +02:00
David Roberts cb3f3d2d04 [ML] Switch from max_running_jobs to xpack.ml.max_open_jobs (elastic/x-pack-elasticsearch#2232)
This change makes 2 improvements to the max_running_jobs setting:

1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
   is not used elsewhere

The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified.  max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.

Relates elastic/x-pack-elasticsearch#2185

Original commit: elastic/x-pack-elasticsearch@18c539f9bb
2017-08-11 09:00:33 +01:00
Alexander Reelsen 5f30508efd Tests: Remove AwaitsFix annotation and increase logging
This test does not reproduce locally but fails regularly in CI.
Added more logging and proper comments.

Relates elastic/x-pack-elasticsearch#2222

Original commit: elastic/x-pack-elasticsearch@bf6b590629
2017-08-11 09:44:09 +02:00
Alexander Reelsen 11334b2df3 Tests: Fix TimeThrottleIntegrationTests to not rely on shard actions (elastic/x-pack-elasticsearch#2234)
These tests used to fail rarely, because during a watch execution
one of the watcher shards was relocated resulting in a second execution
of watch.

In order to prevent this, the tests do not need to actually create any
shards, which causes watcher potentially to be rebalanced.

This simplifies and speeds up the test as well.

relates elastic/x-pack-elasticsearch#1608

Original commit: elastic/x-pack-elasticsearch@1cfac1145d
2017-08-11 09:19:25 +02:00
Lisa Cawley cc7c9aeddb [DOCS] Remove redundant Logstash security page (elastic/x-pack-elasticsearch#2239)
Original commit: elastic/x-pack-elasticsearch@8f66e85fb0
2017-08-10 15:31:41 -07:00
Lisa Cawley e500fba354 [DOCS] Update links to Kibana security (elastic/x-pack-elasticsearch#2235)
Original commit: elastic/x-pack-elasticsearch@88f29b3321
2017-08-10 12:56:03 -07:00
Lisa Cawley 1066a6f877 [DOCS] Add loopback to X-Pack install info (elastic/x-pack-elasticsearch#2237)
Original commit: elastic/x-pack-elasticsearch@8c105818e2
2017-08-10 12:37:14 -07:00
lcawley 8c7828413d [DOCS] Fix typo
Original commit: elastic/x-pack-elasticsearch@03c2f708be
2017-08-10 09:20:57 -07:00
Lisa Cawley a03fa9dd45 [DOCS] Change indexes to indices (elastic/x-pack-elasticsearch#2231)
Original commit: elastic/x-pack-elasticsearch@1c2d24848e
2017-08-10 09:13:53 -07:00
David Kyle c4910098b0 [Docs] Change indexes -> indices in datafeed config example (elastic/x-pack-elasticsearch#2227)
Original commit: elastic/x-pack-elasticsearch@7ddd36360e
2017-08-10 16:22:59 +01:00
Alexander Reelsen 35ee552923 Tests: Fix logger initialization to include hostname in timewarped watcher
This cleans up logging, when starting several elasticsearch instances,
as otherwise you cannot see, which node emits this log message.

Original commit: elastic/x-pack-elasticsearch@c8c2819d86
2017-08-10 14:09:14 +02:00
Alexander Reelsen ceb13988e3 Watcher: Do not update watch status state during execution (elastic/x-pack-elasticsearch#2204)
When a watch is executed, it sends an update request to the watch to
udpate its status.

This update request also updates the status.state field, which contains
information, if the watch is active. If the watch gets executed, and
during execution a watch gets disabled, then the current execution will
set the watch back to active.

This commit fixes the current behaviour and never changes the state of
a watch when updating the status after executing, allowing
activate/deactivate calls to work as expected, regardless if a watch
is being executed.

This will fix not only the current behaviour but also some flaky tests.

Original commit: elastic/x-pack-elasticsearch@ca69109ecb
2017-08-10 13:16:39 +02:00
Alexander Reelsen 4012da662d Tests: Add @AwaitsFix to failing test
Original commit: elastic/x-pack-elasticsearch@545155ec13
2017-08-10 10:17:22 +02:00
lcawley 1840e952af [DOCS] Remove outdated SSL info
Original commit: elastic/x-pack-elasticsearch@aeddc6dd5b
2017-08-09 10:11:57 -07:00
Lisa Cawley 0d79c80a8f [DOCS] Add user setup to X-Pack install info (elastic/x-pack-elasticsearch#2189)
* [DOCS] Add user setup to X-Pack install info

* [DOCS] Add TSL steps to X-Pack install

* [DOCS] Clarify SSL settings in Xpack install

Original commit: elastic/x-pack-elasticsearch@eee37729ff
2017-08-09 10:05:43 -07:00
Lisa Cawley 1bb3ad38c7 [DOCS] Update APIs for multiple jobs or datafeeds (elastic/x-pack-elasticsearch#2105)
* [DOCS] Update APIs for multiple jobs or datafeeds

* [DOCS] Fix syntax diagrams for ML stop/close APIs

* [DOCS] Removed TBD authorization for ML APIs

Original commit: elastic/x-pack-elasticsearch@1a9137a5a7
2017-08-09 08:30:21 -07:00
Colin Goodheart-Smithe a4dd177978 Migrates ToXContent classes (elastic/x-pack-elasticsearch#2024)
* Migrates ToXContent classes

* review comments

Original commit: elastic/x-pack-elasticsearch@74ce3755ab
2017-08-09 15:55:04 +01:00
Lisa Cawley ccf0b6f2ed [DOCS] Fix typo (elastic/x-pack-elasticsearch#2211)
Original commit: elastic/x-pack-elasticsearch@6efb78c1b2
2017-08-08 12:52:52 -07:00
Alexander Reelsen 0b5909fc65 Watcher: Stop swallowing exceptions, always return them instead of message (elastic/x-pack-elasticsearch#1933)
It is really hard to debug some issues with watcher, when only the
e.getMessage() is returned as failure reasons instead of the whole
stack trace.

This commit gets rid of ExceptionsHelper.detailedMessage(e) and always
returns the whole exception.

This commit also extends the watch history to have all fields named
error be treated like an object to be sure they do not get
indexed. No matter where it's placed in the hierarchy

In addition a few Field interface classes were removed, that only contained parse fields.

relates elastic/x-pack-elasticsearch#1816

Original commit: elastic/x-pack-elasticsearch@b2ce680139
2017-08-08 18:36:22 +02:00
David Roberts 22da5cf89e [ML] Add max open jobs per node as a node attribute (elastic/x-pack-elasticsearch#2203)
This commit adds the max_running_jobs setting from elasticsearch.yml
into a node attribute called ml.max_open_jobs.  Previously there was
an assumption that max_running_jobs would be the same for all nodes in
the cluster.  However, during a rolling cluster restart where the value
of the setting is being changed this clearly cannot be the case, and
would cause unexpected/unpredictable limits to be used during the period
when different nodes had different settings.

For backwards compatibility, if another node in the cluster has not added
its setting for max_running_jobs to the cluster state then the old
(flawed but better than nothing) approach is applied, i.e. assume the
remote node's setting for max_running_jobs is equal to that of the node
deciding the job allocation.

Relates elastic/x-pack-elasticsearch#2185

Original commit: elastic/x-pack-elasticsearch@1e62b89183
2017-08-08 16:16:27 +01:00
Dimitris Athanasiou 2864078da2 [ML] Move job group validation after parsing (elastic/x-pack-elasticsearch#2207)
Validating job groups during parsing results into
the validation error being wrapped into a parse
exception. The UI then does not display the cause of the
error. Finally, it is conceptually not a parse error, so
it belongs outside the parsing phase.

Original commit: elastic/x-pack-elasticsearch@a03f002bdc
2017-08-08 15:59:04 +01:00
Alexander Reelsen cff3418e96 Tests: Only restart watcher on watcher tests
In order to not restart watcher on every test, this checks
if this is a watcher test and only restarts watcher in that case.

In addition this also checks if watcher is not marked as started.
as otherwise restarting does not make sense. Lastly, this waits until
watcher is marked as started before proceeding.

Original commit: elastic/x-pack-elasticsearch@a8d72f3ebb
2017-08-08 14:28:44 +02:00
David Roberts 6a159d2127 [ML] Fix fallout from bulk action requiring newlines (elastic/x-pack-elasticsearch#2205)
Only unit tests were broken.  Production ML code was always terminating
bulk requests with newlines.

Original commit: elastic/x-pack-elasticsearch@96ed06fed3
2017-08-08 11:07:13 +01:00
Alexander Reelsen 55e88d6857 Watcher: Ignore if template is missing when upgrade is running (elastic/x-pack-elasticsearch#2199)
If one of the old watcher templates does not exist when we try 
to delete it, the upgrade should just continue.

Original commit: elastic/x-pack-elasticsearch@6a52bad329
2017-08-08 10:17:58 +02:00
Chris Earle 1d6f82dbe3 [Monitoring] Remove Dedicated IndicesStatsCollector (elastic/x-pack-elasticsearch#2192)
This removes the `IndicesStatsCollector` and, instead, it reuses the superset version of the call from the `IndexStatsCollector`.

On clusters with a large number of indices, this should actually help a good amount in reducing wasted calls and memory allocation without any difference in the output.

Original commit: elastic/x-pack-elasticsearch@93b09878e4
2017-08-07 13:00:41 -04:00
Tim Sullivan 7d86d75aa4 [Monitoring/Cluster Alerts] default admin email is snake_case (elastic/x-pack-elasticsearch#2177)
Original commit: elastic/x-pack-elasticsearch@d6129a0065
2017-08-07 09:15:14 -07:00
Alexander Reelsen 2d08477093 Tests: Fix upgrade tests by not running stats against arbitrary hosts
Original commit: elastic/x-pack-elasticsearch@1b858aad52
2017-08-07 13:27:36 +02:00
Alexander Reelsen 8f6874abf9 Tests: Increase logging in watcher upgrade tests to debug test failures
Original commit: elastic/x-pack-elasticsearch@380a8541dc
2017-08-07 11:31:16 +02:00
Luca Cavanna b530da945d Adapt to yaml test client upstream changes (elastic/x-pack-elasticsearch#2172)
Original commit: elastic/x-pack-elasticsearch@d984f71801
2017-08-07 11:02:25 +02:00
David Roberts 05cbe8dc0c [ML] Disallow creating a job against a closed results or state index (elastic/x-pack-elasticsearch#2186)
Previously if this was attempted you'd get an NPE (5.x) or hang (6.x).
Following this change you get an error message telling you what the
problem is.

relates elastic/x-pack-elasticsearch#2170

Original commit: elastic/x-pack-elasticsearch@ea12a9ff46
2017-08-07 08:53:12 +01:00
Chris Earle 87bc215b91 [ML] Use try-with-resources for InputStream (elastic/x-pack-elasticsearch#2182)
Original commit: elastic/x-pack-elasticsearch@88d5e73fec
2017-08-04 13:26:35 -04:00
Alexander Reelsen 36bad2079d Tests: Fix watcher test when aliases were used
Original commit: elastic/x-pack-elasticsearch@d1c38cb85a
2017-08-04 18:01:26 +02:00
Lisa Cawley a6a76f8d2a [DOCS] Clarify xpack.security.enabled info (elastic/x-pack-elasticsearch#2166)
Original commit: elastic/x-pack-elasticsearch@140e330973
2017-08-04 08:59:50 -07:00
Jay Modi 8b0fb5eae8 Re-enable OpenLDAP tests and run against vagrant instance (elastic/x-pack-elasticsearch#2121)
This commit re-enables the OpenLDAP tests that were previously running against a one-off instance
in AWS but now run against a vagrant fixture. There were some IntegTests that would run against the
OpenLDAP instance randomly but with this change they no longer run against OpenLDAP. This is ok as
the functionality that is tested by these has coverage elsewhere.

relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@ac9bc82297
2017-08-04 09:44:08 -06:00
David Roberts 39bc5886f2 [ML] Remove record_count from ML mappings (elastic/x-pack-elasticsearch#2183)
record_count is no longer written to new results, but is still tolerated
for backwards compatibility.  However, in the backwards compatibility case
the results index must already contain the required mapping.  There's no
need to add this mapping to newly created results indices.

Original commit: elastic/x-pack-elasticsearch@e586f3ba96
2017-08-04 10:52:35 +01:00
Jay Modi a7d6138f83 Fix the building of the default URL for the setup password tool (elastic/x-pack-elasticsearch#2176)
This commit fixes the building of the default URL for the setup password tool so that a default
elasticsearch.yml file will still result in a succesful run of the tool.

relates elastic/x-pack-elasticsearch#2174

Original commit: elastic/x-pack-elasticsearch@2291b14875
2017-08-03 15:14:24 -06:00
Dimitris Athanasiou 2d9d4c41d8 [ML] Add ability to assign groups to jobs (elastic/x-pack-elasticsearch#2155)
In particular:

  - adds a `groups` field in a job
  - group names can be used in multi-job APIs

relates elastic/x-pack-elasticsearch#2097

Original commit: elastic/x-pack-elasticsearch@c8517221ae
2017-08-03 17:32:05 +01:00
David Kyle 4b531c4dbb [ML] Check histogram interval is a divisor of bucketspan (elastic/x-pack-elasticsearch#2153)
Original commit: elastic/x-pack-elasticsearch@356dfa719c
2017-08-03 11:58:58 +01:00
Alexander Reelsen 65ea1b3bc4 Tests: Remove unneeded classes, fix AwaitsFix watcher tests (elastic/x-pack-elasticsearch#2139)
Fix TemplateTransformMappingTests to work, even if date rolls over
during execution.

Reenable test in BootStrapTests, was forgotten.

Remove the SecurityF/MonitoringF/WatcherF classes, as there is a gradle
command to easily start elasticsearch with xpack

Remove HasherBenchmark, as it is not a test and relies on RandomContext
that is not available anymore (also I think a JMH benchmark would be
needed here).

Remove ManualPublicSmtpServersTester, was not usable anymore.

Remove OldWatcherIndicesBackwardsCompatibilityTests, now in dedicated
rolling upgrade tests.

Remove unused EvalCron class.

Original commit: elastic/x-pack-elasticsearch@100fa9e9b0
2017-08-03 12:51:35 +02:00