Commit Graph

7994 Commits

Author SHA1 Message Date
Chris Earle 065f9aa834 [Monitoring] Remove support for setting interval -1 (elastic/x-pack-elasticsearch#4035)
This removes the ability to set `xpack.monitoring.collection.interval` to
`-1`.

Original commit: elastic/x-pack-elasticsearch@60f2db4bd1
2018-02-26 17:53:09 -08:00
David Roberts 2dd0fbca7a [TEST] Mute tests that will need updating following an ML C++ change
Relates elastic/x-pack-elasticsearch#4033
Relates elastic/ml-cpp#7

Original commit: elastic/x-pack-elasticsearch@de297097f3
2018-02-26 09:09:40 +00:00
David Kyle 9786b38747 [ml] Don’t open failed jobs after restart (elastic/x-pack-elasticsearch#3997)
Original commit: elastic/x-pack-elasticsearch@f4b2ff084b
2018-02-25 13:47:15 +00:00
Costin Leau b8c9c5325c SQL: add catalog filter to SYS COLUMNS command (elastic/x-pack-elasticsearch#3978)
Add basic support for catalog parameters in SYS COLUMN
Pass an empty string instead of a null inside the prepared statement
Don't use pattern for catalog in getColumns

Original commit: elastic/x-pack-elasticsearch@17e9e851a0
2018-02-25 05:08:48 +02:00
Costin Leau b2631f9ac8 SQL: Extend field resolution to deal with unquoted qualified columns (elastic/x-pack-elasticsearch#3894)
* foo.bar can mean either: no table field foo.bar or table foo, field bar
The resolution rule has been extended to include the latter case as
fallback.
* Always check field ambiguity
* Since field with dots can create confusion (when not qualified), the
analyzer always now always checks for both qualified and not-qualified
fields and throws an ambiguity message with the potential candidates.
This forces the use of qualifiers or quotes to better indicate the
desired field.
* Add example of aliasing the table to remove ambiguity

Original commit: elastic/x-pack-elasticsearch@8b70b9c4ee
2018-02-25 04:39:48 +02:00
Zachary Tong 3b474d8868 [Test] Fix slow rollup job task test
Incorrect latch caused this test to run slowly (until the await
finished), and could probably cause failure due to incorrect ordering

Original commit: elastic/x-pack-elasticsearch@ebeb8655da
2018-02-24 20:04:33 +00:00
Zachary Tong eb82e3cf61 [Test] Fix bad latches in rollup state tests
The latches were not placed correctly, allowing the aborts
to be set before we checked the state for Indexing the first time.

This was due to using the DelayingIndexer's built in latch, which
isn't placed quite where we needed it.

Original commit: elastic/x-pack-elasticsearch@590cfa07b0
2018-02-24 18:44:51 +00:00
Zachary Tong ea531c0135 [Rollup] Silence delete job yaml tests
See issue elastic/x-pack-elasticsearch#4041

Original commit: elastic/x-pack-elasticsearch@e5e589593b
2018-02-24 18:15:57 +00:00
Zachary Tong 390e64aabd Add empty policy file to Rollups
Packaging tests seem to require a policy file for the time being

Original commit: elastic/x-pack-elasticsearch@ce34b023db
2018-02-24 03:28:28 +00:00
Ryan Ernst d7c097e2b4 Build: Remove specifying zip distribution in qa tests (elastic/x-pack-elasticsearch#4037)
This is the xpack side of https://github.com/elastic/elasticsearch/pull/28805

Original commit: elastic/x-pack-elasticsearch@48cbe6623f
2018-02-23 18:17:59 -08:00
Zachary Tong e3543b06ba [Docs] Remove bad cross-book link
Temporary to keep the build green, will figure this out during
the next round of rollup docs work.

Original commit: elastic/x-pack-elasticsearch@7657938ffb
2018-02-23 23:23:51 +00:00
Lee Hinman c763a14132 Pass InputStream when creating XContent parser (elastic/x-pack-elasticsearch#4001)
* Pass InputStream when creating XContent parser

Rather than passing the raw `BytesReference` in when creating the xcontent
parser, this passes the StreamInput (which is an InputStream), this allows us to
decouple XContent from BytesReference.

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28754

* Use the streamInput variant, not sourceAsString

Original commit: elastic/x-pack-elasticsearch@dd5d8b1654
2018-02-23 15:22:28 -07:00
Zachary Tong bf1550a0b2 Rollups for Elasticsearch (elastic/x-pack-elasticsearch#4002)
This adds a new Rollup module to XPack, which allows users to configure periodic "rollup jobs" to pre-aggregate data.  That data is then available later for search through a special RollupSearch API, which mimics the DSL and functionality of regular search.

Rollups are used to drastically reduce the on-disk footprint of metric-based data (e.g. timestamped document with numeric and keyword fields).  It can also be used to speed up aggregations over large datasets, since the rolled data will be considerably smaller and fewer documents to search.

The PR adds seven new endpoints to interact with Rollups; create/get/delete job, start/stop job, a capabilities API similar to field-caps, and a Rollup-enabled search.

Original commit: elastic/x-pack-elasticsearch@dcde91aacf
2018-02-23 17:10:37 -05:00
Chris Earle 6e5343192b [TEST] Wait for index to be green before searching (elastic/x-pack-elasticsearch#4034)
Since adding, I have not been able to reproduce shard failure.

Original commit: elastic/x-pack-elasticsearch@9f9fc6fb8f
2018-02-23 13:44:04 -05:00
Chris Earle 71372752dc [TEST] Remove AwaitsFix after elastic/x-pack-elasticsearch#4031
Original commit: elastic/x-pack-elasticsearch@240168baae
2018-02-23 13:13:18 -05:00
Chris Earle 0402ef4853 [TEST] Wait for Exporters independent of results (elastic/x-pack-elasticsearch#4031)
This should force waiting on the exporters so that the data can be awaited
separately.

Original commit: elastic/x-pack-elasticsearch@1e675f2786
2018-02-23 13:09:10 -05:00
Lisa Cawley 67b6009944 [DOCS] Add introductory monitoring information (elastic/x-pack-elasticsearch#3775)
Original commit: elastic/x-pack-elasticsearch@fbd164351c
2018-02-23 08:23:16 -08:00
Chris Earle cf0fe1bbff [Monitoring] Deprecate setting interval to -1 (elastic/x-pack-elasticsearch#4023)
This deprecates setting `xpack.monitoring.collection.interval` to `-1`.

Original commit: elastic/x-pack-elasticsearch@7c5fa35aad
2018-02-23 10:33:34 -05:00
Alexander Reelsen 7f166e1927 Cleanup: Move tests calling external services into QA projects (elastic/x-pack-elasticsearch#3869)
This also moves the hipchat/pagerduty/slack tests to REST tests which put real watches and also check the history.

Relates elastic/x-pack-elasticsearch#3800


Original commit: elastic/x-pack-elasticsearch@fce08d9cb7
2018-02-23 13:31:28 +01:00
Alexander Reelsen 9624788c17 Monitoring: Mute MonitoringIT tests
Relates elastic/x-pack-elasticsearch#4027, elastic/x-pack-elasticsearch#3962

Original commit: elastic/x-pack-elasticsearch@9ef5cac4ba
2018-02-23 12:12:40 +01:00
Luca Cavanna df8dcb22b8 [TEST] share code between streamable/writeable/xcontent base test classes (elastic/x-pack-elasticsearch#4021)
This is the xpack side of https://github.com/elastic/elasticsearch/pull/28785

Original commit: elastic/x-pack-elasticsearch@f1202e23af
2018-02-23 10:49:04 +01:00
Alexander Reelsen 2ebb21f108 Watcher: Reduce threads on non-data nodes (elastic/x-pack-elasticsearch#3932)
The watcher thread pool is scaled by the number of CPUs and has by
default up to 5x the number of cores. This is needed because we assumme
I/O based waiting workloads for watcher. However if the node is not a
data node, there will not be any execution of watches with the exception
of a user calling the execute watch API on that node.

This means, we can get away with just one thread, so that there is no
need for the JVM to manage more threads on master/client or ingest only
nodes.

Original commit: elastic/x-pack-elasticsearch@b5899401d3
2018-02-23 09:16:32 +01:00
Alexander Reelsen 45fb3db062 Tests: Fix JiraIssueTests to not hide secrets
Original commit: elastic/x-pack-elasticsearch@f4c83c2da3
2018-02-23 09:10:56 +01:00
Lee Hinman 13e47ab315 Decouple XContentGenerator and JsonXContentGenerator from BytesReference (elastic/x-pack-elasticsearch#4009)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28772

Original commit: elastic/x-pack-elasticsearch@8a37e38f62
2018-02-22 14:28:44 -07:00
Chris Earle b009dec503 [Monitoring] Clean .watcher-history-* indices by default in 6.3+ (elastic/x-pack-elasticsearch#2720)
This flips the setting default to true for 6.3+.

Original commit: elastic/x-pack-elasticsearch@47a09d8449
2018-02-22 13:20:53 -05:00
jaymode 020f5496be Test: update IndexAuditUpgradeIT after backport
This commit updates the IndexAuditUpgradeIT test after backporting. The
backport fixes a bug that caused the node name field to be null, which
affected the expected number of nodes in the tests.

Original commit: elastic/x-pack-elasticsearch@9df99d8800
2018-02-22 09:55:01 -07:00
Tim Brooks 33ae455e6c Selectors operate on channel contexts (elastic/x-pack-elasticsearch#3803)
This is related to elastic/elasticsearch#28468. It is a compatibility
commit to ensure that x-pack is compatible with those changes.

Original commit: elastic/x-pack-elasticsearch@3ddf719adc
2018-02-22 09:45:07 -07:00
javanna 531d44f446 Adjust serialization versions after backport
Now that elastic/x-pack-elasticsearch#3985 has been backported to 6.x, we can update the version conditions from 7.0.0-alpha1 to 6.3.0

Original commit: elastic/x-pack-elasticsearch@b068ca92a7
2018-02-22 16:00:47 +01:00
David Kyle 26516c507e [ML][DOCS] Fix substitution in ML docs code snipppet (elastic/x-pack-elasticsearch#4006)
* Revert "Mute failing ML datafeed docs code snippet"

* Fix substitution

Original commit: elastic/x-pack-elasticsearch@8442863480
2018-02-22 09:32:52 +00:00
Luca Cavanna 79bc6d9a53 Remove AcknowledgedRestListener in favour of RestToXContentListener (elastic/x-pack-elasticsearch#3985)
Adapt to AcknowledgedRestListener removal 

Original commit: elastic/x-pack-elasticsearch@74c08fcf02
2018-02-22 09:13:58 +01:00
Aaron Bull Schaefer 28bbb8b3cd Add license.key option to CI releaseTest build script (elastic/x-pack-elasticsearch#3973)
See https://groups.google.com/a/elastic.co/d/msgid/build-elasticsearch/CAN1nV7FC6Fij-oSOmWz9WU47WJsJw_8EmiRx22GiLEtDDRj7Pw%40mail.gmail.com?utm_medium=email&utm_source=footer

Related to https://github.com/elastic/infra/pull/3916

Original commit: elastic/x-pack-elasticsearch@15b435274f
2018-02-22 08:27:23 +01:00
Chris Earle 309adaf38e [Monitoring] Disable Monitoring Collection by Default (elastic/x-pack-elasticsearch#3962)
This adds a new setting, `xpack.monitoring.collection.enabled`, and
disables it by default (`false`).

Original commit: elastic/x-pack-elasticsearch@4b3a5a1161
2018-02-21 23:10:20 -05:00
Tim Vernum 828387a25a Add more info logging when loading roles.yml (elastic/x-pack-elasticsearch#3992)
Although not frequently used in production, we make heavy use of the
FileRolesStore within integration tests. This change adds a little bit
more logging at INFO level when the roles.yml file is (re)loaded.

Original commit: elastic/x-pack-elasticsearch@bbacd46e28
2018-02-22 11:10:27 +11:00
Lisa Cawley 1eca36bda9 [DOCS] Clarified model snapshot retention (elastic/x-pack-elasticsearch#4000)
Original commit: elastic/x-pack-elasticsearch@f1bdf5454d
2018-02-21 08:58:17 -08:00
David Kyle 92a9fc8b48 Mute failing ML datafeed docs code snippet
Original commit: elastic/x-pack-elasticsearch@9cfea037bc
2018-02-21 10:03:20 +00:00
Alexander Reelsen 90c360d9d0 Watcher: Replace YAML tests needing to wait for watch execution (elastic/x-pack-elasticsearch#3995)
In order to check for the REST tests if triggering of watches with
security enabled works as expected, we have to add a watch and wait for
its background execution. In the REST tests the only wait is to wait for
this with a timeout. If the timeout is reached but the watch has not
been executed yet, the test will fail.

This commit replaces the YAML with a java based REST test, so that
helper methods like assertBusy() can be used and waiting for a watch to
be executed now works as expected.

relates elastic/x-pack-elasticsearch#3753

Original commit: elastic/x-pack-elasticsearch@fc39636ef7
2018-02-21 09:58:11 +01:00
Alexander Reelsen 0bf354eb38 Watcher: Fix hipchat message failure serialization (elastic/x-pack-elasticsearch#3939)
The current toXContent serialization of a failed hipchat message writes
the same field called status twice and thus cannot be stored in the
watch history.

This commit ensures the field gets only written once.

relates elastic/x-pack-elasticsearch#3919

Original commit: elastic/x-pack-elasticsearch@fb499e8055
2018-02-21 09:38:53 +01:00
lcawley 0e2a39603e [DOCS] Fixed ml.machine_memory example testing
Original commit: elastic/x-pack-elasticsearch@d0fa44ab20
2018-02-20 18:36:56 -08:00
Lisa Cawley e9b4a2d063 [DOCS] Enabled code snippet testing for more ML APIs (elastic/x-pack-elasticsearch#3990)
Original commit: elastic/x-pack-elasticsearch@1b631adff6
2018-02-20 11:08:37 -08:00
Jay Modi e3e82ccb2d Security: IndexAuditTrail updates mappings when indexing locally (elastic/x-pack-elasticsearch#3944)
This change makes the IndexAuditTrail update mappings for the current
index even when indexing locally. This is needed in a couple of
scenarios; the first being upgrades from 5x to 6x where the document
type was changed and the second is when upgrading minor versions and
a new field has been introduced. A new field was introduced in 6.2.

Additionally, writing tests for the rolling upgrade revealed a bug
where the actual node name was never being set as it was using the name
key instead of the node.name key.

relates elastic/x-pack-elasticsearch#3901
relates elastic/x-pack-elasticsearch#3718

Original commit: elastic/x-pack-elasticsearch@7374a62ff1
2018-02-20 11:09:56 -07:00
Jay Modi 0fc0034509 Test: remove outdated logic for file writing in security tests (elastic/x-pack-elasticsearch#3947)
This commit removes some outdated logic in the SecurityTestUtils class
for writing files and creating directories. A long, long time ago there
was a global cluster for tests that was used across suites and because
of this there were calls to delete files if they already existed. The
global cluster has been removed, so we no longer need the code that
deletes the content of a directory if it already exists.

Additionally, the file writing used in SecurityTestUtils did not use
atomic moves when possible and this commit changes the code such that a
temp file is written and we try to atomically move it to the correct
path; if atomic moves are not supported a regular move is performed.

relates elastic/x-pack-elasticsearch#3912

Original commit: elastic/x-pack-elasticsearch@973fcfe2e1
2018-02-20 09:34:17 -07:00
Lee Hinman 6728912c87 Handle removing log4j dependency from elasticsearch-core jar (elastic/x-pack-elasticsearch#3976)
This handles the removal of the log4j dependency from our elasticsearch-core
project. It was originally necessary only for our jar classpath checking. It is
now replaced by a `Consumer<String>`.

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28705

Original commit: elastic/x-pack-elasticsearch@1186a7eaf6
2018-02-20 09:15:56 -07:00
David Kyle e365a255c7 [ML] Filter non-ml attributes from data feed stats (elastic/x-pack-elasticsearch#3994)
Original commit: elastic/x-pack-elasticsearch@9cdc78c000
2018-02-20 12:22:28 +00:00
Alexander Reelsen c9d77d20fd Watcher: Never return credentials after watch creation... (elastic/x-pack-elasticsearch#3581)
... yet support updates. This commit introduces a few changes of how
watches are put.

The GET Watch API will never return credentials like basic auth
passwords, but a placeholder instead now. If the watcher is enabled to
encrypt sensitive settings, then the original encrypted value is
returned otherwise a "::es_redacted::" place holder.

There have been several Put Watch API changes.

The API now internally uses the Update API and versioning. This has
several implications. First if no version is supplied, we assume an
initial creation. This will work as before, however if a credential is
marked as redacted we will reject storing the watch, so users do not
accidentally store the wrong watch.

The watch xcontent parser now has an additional methods to tell the
caller if redacted passwords have been found. Based on this information
an error can be thrown.

If the user now wants to store a watch that contains a password marked
as redacted, this password will not be part of the toXContent
representation of the watch and in combinatination with update request
the existing password will be merged in. If the encrypted password is
supplied this one will be stored.

The serialization for GetWatchResponse/PutWatchRequest has changed.
The version checks for this will be put into the 6.x branch.

The Watcher UI now needs specify the version, when it wants to store a
watch. This also prevents last-write-wins scenarios and is the reason
why the put/get watch response now contains the internal version.

relates elastic/x-pack-elasticsearch#3089

Original commit: elastic/x-pack-elasticsearch@bb63be9f79
2018-02-20 10:09:27 +01:00
Tim Vernum 56c761f241 Log roles.yml changes in TokenAuthIntegTests
This test recently failed and it appears that the cause was that
roles.yml was deleted while the cluster was still running.
This turns on logging for the FileRolesStore so that changes to file
based roles can traced more easily.

Original commit: elastic/x-pack-elasticsearch@fbce5a8027
2018-02-20 17:19:39 +11:00
Tim Vernum c4582cdcd0 Additional settings for SAML NameID policy (elastic/x-pack-elasticsearch#3969)
* Additional settings for SAML NameID policy

We should not be populating SPNameQualifier by default as it is
intended to be used to specify an alternate SP EntityID rather than
our own. Some IdPs (ADFS) fail when presented with this value.

This commit
- makes the SPNameQualifier a setting that defaults to blank
- adds a setting for "AllowCreate"
- documents the above

Original commit: elastic/x-pack-elasticsearch@093557e88f
2018-02-20 13:51:42 +11:00
Lisa Cawley 64653e525a [DOCS] Identify informational ML properties (elastic/x-pack-elasticsearch#3773)
Original commit: elastic/x-pack-elasticsearch@cb310b360d
2018-02-19 11:48:09 -08:00
Lisa Cawley 530b709948 [DOCS] Add skip_time to Flush Jobs API (elastic/x-pack-elasticsearch#1955)
Original commit: elastic/x-pack-elasticsearch@352bd336d8
2018-02-19 11:04:12 -08:00
Lisa Cawley ecfd8892b1 [DOCS] Expanded conceptual information about jobs (elastic/x-pack-elasticsearch#3332)
Original commit: elastic/x-pack-elasticsearch@18b32bd7b0
2018-02-19 10:38:38 -08:00
Albert Zaharovits 5833ed296e Fix embarrassing NPE in audit trail.
relates elastic/x-pack-elasticsearch#3988

Original commit: elastic/x-pack-elasticsearch@72c16ea50f
2018-02-19 20:14:37 +02:00