Commit Graph

7986 Commits

Author SHA1 Message Date
Zachary Tong cf965dca27 [Docs] Mark Rollup as Experimental instead of Beta
Original commit: elastic/x-pack-elasticsearch@bce120daae
2018-04-10 15:51:09 +00:00
Lee Hinman 9e3b03531c Remove custom PeriodType for watcher PeriodThrottler (elastic/x-pack-elasticsearch#4327)
This constructor was actually never used, other than in tests, and even then,
there is no need for a custom period type as the human-readable toString value
will suffice.

Original commit: elastic/x-pack-elasticsearch@fc666a04b9
2018-04-10 08:01:32 -06:00
Dimitris Athanasiou 4c7bd71bdf [ML] Improve comment on why task is marked completed on kill process
Original commit: elastic/x-pack-elasticsearch@5c16cf8e26
2018-04-10 13:36:13 +01:00
David Kyle 7e4e1dabcf [ML] Add categorical exclude condition (elastic/x-pack-elasticsearch#4326)
Original commit: elastic/x-pack-elasticsearch@6c80988e08
2018-04-10 13:19:00 +01:00
Tim Vernum 411f683521 Document ssl.verification_mode (elastic/x-pack-elasticsearch#4253)
It is common for users to wish to adjust the verification_mode in SSL
settings, usually with the intention of skipping hostname
verification. This has been supported for a long time, but the
relevant configuration setting was not clearly documented, which would
sometimes lead users to set `verification_mode` to `none`, and disable
more checks than they intended.

This commit adds clearer documentation regarding the options available
for `verification_mode` and actively discourages the use of `none`.

Original commit: elastic/x-pack-elasticsearch@2fdf53b42f
2018-04-10 20:27:23 +10:00
Igor Motov 2ba7258603 SQL: Extract CSV spec parser into Utils class (elastic/x-pack-elasticsearch#4317)
For Geo SQL we need to have a separate set of CSV tests. This commit
extracts CSV spec parsing logic into a separate file for a more
straight forward reuse.

Relates elastic/x-pack-elasticsearch#4080

Original commit: elastic/x-pack-elasticsearch@29034ef051
2018-04-09 17:01:24 -04:00
Lisa Cawley 17543dbc00 [DOCS] Augmented token API description (elastic/x-pack-elasticsearch#4321)
Original commit: elastic/x-pack-elasticsearch@c24713ee43
2018-04-09 11:39:21 -07:00
Lee Hinman 218e9a57bb Correct documentation for number of salt generation rounds (elastic/x-pack-elasticsearch#4322)
For the user cache, the crypt option rounds are actually the log2 of the number
of rounds. This commits updates the documentation to reflect this.

Original commit: elastic/x-pack-elasticsearch@d3cc2b7f29
2018-04-09 10:59:06 -06:00
David Kyle 11a6cd18ac [ML] Create calendars with job groups (elastic/x-pack-elasticsearch#4308)
Original commit: elastic/x-pack-elasticsearch@dc42dccb1f
2018-04-09 16:18:01 +01:00
Alexander Reelsen 6bd5e9ef91 Watcher: Reenable email property settings (elastic/x-pack-elasticsearch#4319)
With the change of requiring to configure account settings properly by
using affix settings, we forgot another special snowflake, namely the
configuration of mail properties, which can be arbitrary in the
configuration. Those properties are used when an email is sent.

This commit adds a few (but not all of those) options back and removes
the link in the documentation to refer to all of those settings.

Some settings are useless, as they only change the execution
expectations when a mail is sent, which the watch has control over.

The following settings are supported

* smtp.{host,port,user,password}
* smtp.auth
* smtp.starttls.{enable,required}
* smtp.{timeout,connection_timeout,write_timeout}
* smtp.{local_address,local_port}
* smtp.send_partial
* smtp.wait_on_quit

relates elastic/x-pack-elasticsearch#4048

Original commit: elastic/x-pack-elasticsearch@39d5624710
2018-04-09 15:42:16 +02:00
Lisa Cawley b6d901f530 [DOCS] Augmented security configuration steps (elastic/x-pack-elasticsearch#4316)
Original commit: elastic/x-pack-elasticsearch@48b47b19ae
2018-04-06 11:48:03 -07:00
Zachary Tong 7810dc6146 [Rollup] Add `value_count` metric (elastic/x-pack-elasticsearch#4315)
Adds `value_count` as one of the accepted metrics.  The caveat is that
it only accepts numeric values for two reasons:

- Job validation at creation makes sure all metrics are numeric fields.
Changing this would require new syntax (or disallowing anything but
value_count on mixed fields)
- when `toBuilders()` is called, we have to supply a ValueSource to
the ValueCountBuilder, and we don't know what the field type is at that
time.

These are both fixable, but relatively more involved.  I think numeric-only
is a reasonable limitation to start with

Original commit: elastic/x-pack-elasticsearch@270f24c8bf
2018-04-06 10:47:33 -07:00
Lee Hinman 5e81e91df8 Adjust to Streams.copy moving into elasticsearch-core (elastic/x-pack-elasticsearch#4263)
In https://github.com/elastic/elasticsearch/pull/29322 Streams.copy was moved
into the elasticsearch-core project, this is the x-pack side of it.

Original commit: elastic/x-pack-elasticsearch@5803b8a042
2018-04-06 11:07:25 -06:00
Lee Hinman 752ef086f0 Adjust to ObjectParser moving into the x-content lib (elastic/x-pack-elasticsearch#4297)
* Adjust to ObjectParser moving into the x-content lib

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29373

Original commit: elastic/x-pack-elasticsearch@93741602c7
2018-04-06 09:41:20 -06:00
Christoph Büscher 5a59c5394f Add migration check for deprecated `delimited_payload_filter` (elastic/x-pack-elasticsearch#4310)
Add a check to the migration assistant to warn about the renaming of
`delimited_payload_filter` to `delimited_payload`. This should still
word for old indices from 7.0 on but will throw an error for newly
created indices and the user should be warned about it when running the
migration checks.

Original commit: elastic/x-pack-elasticsearch@5d55e4e499
2018-04-06 13:41:00 +02:00
Dimitris Athanasiou d340cd5a00 [ML] Add more trace logging around job task cancelling
Original commit: elastic/x-pack-elasticsearch@3143c39b6e
2018-04-06 10:58:48 +01:00
Yogesh Gaikwad ed6a6af64c SAML: Make alias for signing key optional (elastic/x-pack-elasticsearch#4248)
We specify an alias for signing key, but when we just have
a single key in key store this is an additional setting which
is annoying. This PR addresses this issue by making it optional.

- Changes in SamlRealmSettings to make signing/encryption
  key alias optional
- Checks if none of the keys are useful for given operation
  signing or encryption throws an error.
- Checks for no of aliases in key-store, if more than one and alias
  is not specified throws error.
- If an alias is not specified and there is just one alias in
  keystore then use it as the credential.
- Unit Tests

Note: A side effect of this change the above-mentioned behavior is
it's also applicable for encryption keys currently, but it is going
to change when fixing elastic/x-pack-elasticsearch#3980 for supporting multiple encryption keys.

relates elastic/x-pack-elasticsearch#3981

Original commit: elastic/x-pack-elasticsearch@2b5af1d8a8
2018-04-06 10:43:35 +10:00
Lisa Cawley 3b876262e2 [DOCS] Clarify certgen deprecation (elastic/x-pack-elasticsearch#4099)
Original commit: elastic/x-pack-elasticsearch@dfbe38325d
2018-04-05 12:55:38 -07:00
Lisa Cawley bbcb33b519 [DOCS] Security disabled by default (elastic/x-pack-elasticsearch#4288)
Original commit: elastic/x-pack-elasticsearch@110df8a58e
2018-04-05 12:06:43 -07:00
Zachary Tong f682ecc576 [Rollup] Remove `computed` field from rollup docs
The computed field contained a list of all aggs that were computed
for this particular rollup doc.  It was used to help filter to the
correct rollup job/set of jobs.

But this functionality was never perfect, and has been obsoleted by
validating the rollup caps while searching.  So we can remove the
computed field and save a bunch of space (since they were quite bulky)

Original commit: elastic/x-pack-elasticsearch@455644488f
2018-04-05 15:25:20 +00:00
Jay Modi b4bf9ed87e Security: minimize automatons as they are combined (elastic/x-pack-elasticsearch#4300)
This commit changes the combination of multiple automatons representing
a pattern so that the result of each step is minimal. Previously, the
code unioned the automata and performed the minimization operation
after all of the automata had been combined. This resulted in patterns
with lots of overlap causing a TooComplexToDeterminizeException even
though the end result could be a automaton that is total. Minimizing
the automata as we go, allows us to build an automata that could not
previously be built at the cost of additional operations. Automata are
typically cached in the security code, so the net performance impact
should be minimal.

Original commit: elastic/x-pack-elasticsearch@b59fe8d690
2018-04-05 08:50:23 -06:00
Alan Woodward be92ee1fb1 Upgrade to lucene-7.3.0 (elastic/x-pack-elasticsearch#4305)
Original commit: elastic/x-pack-elasticsearch@31eba0f383
2018-04-05 12:49:52 +01:00
Albert Zaharovits f3a7c58413 Optimize volatile flags for logging audit (elastic/x-pack-elasticsearch#4265)
Remove unnecessary volatile qualifier.

Original commit: elastic/x-pack-elasticsearch@2889bc40e7
2018-04-05 13:35:32 +03:00
Alexander Reelsen f709fcd083 Watcher: Refactor/Fix TransportWatcherServiceAction (elastic/x-pack-elasticsearch#4240)
This commit fixes several issues with the current implementation of
starting & stopping watcher

1. The WatcherServiceResponse was always returning a message, that the
   request was acknowledged, completely independent from the fact if it was
   or not.
2. A new cluster state instance was always returned, regardless if the
   state had changed or not (which is explicitely mentioned in the
   javadocs to check for this)
3. The AckedClusterStateUpdateTask now returns a proper WatcherServiceResponse
4. A failure now gets logged

Relates elastic/x-pack-elasticsearch#4225 (this is just a hunch for now)

Original commit: elastic/x-pack-elasticsearch@f4c1749f95
2018-04-05 09:33:28 +02:00
Zachary Tong 3852b41330 [Rollup] Validate field mapping before creating Rollup Job (elastic/x-pack-elasticsearch#4274)
This PR adds logic to ensure that the fields (and field types) configured
in the Rollup Job are present in the index/indices specified by the job's
index pattern.  If a field is missing, or is not aggregatable, it
will throw an exception before the job is created.

This is important for user-friendliness, because otherwise the user
only discovers an issue with mapping when the job is started and
fails to rollup correctly (and only really noticeable by looking at logs,
since it's a runtime failure).

Original commit: elastic/x-pack-elasticsearch@686cd03072
2018-04-04 15:32:26 -07:00
Lisa Cawley 3ae8c71f16 [DOCS] Added exporter details (elastic/x-pack-elasticsearch#4171)
Original commit: elastic/x-pack-elasticsearch@791d9d6a45
2018-04-04 15:25:14 -07:00
Lisa Cawley 3e4ceec0c2 [DOCS] Removed X-Pack installation info (elastic/x-pack-elasticsearch#4299)
Original commit: elastic/x-pack-elasticsearch@5af83b7531
2018-04-04 11:59:31 -07:00
Chris Earle aa5ee1ade9 [Monitoring] HttpExportBulk shouldn't bother setting payload to null (elastic/x-pack-elasticsearch#4287)
This avoid setting the payload to `null` when sending it to the remote
monitoring cluster. The payload will be GCed when this overall object is
GCed, which should be very soon anyway.

Original commit: elastic/x-pack-elasticsearch@04f63c4150
2018-04-04 13:42:00 -04:00
lcawley 510f667ea8 [DOCS] Fixed link text
Original commit: elastic/x-pack-elasticsearch@4f42744ee1
2018-04-04 10:39:13 -07:00
Dimitris Athanasiou aee8416649 [ML][TEST] Disable other plugins for internal cluster integ tests (elastic/x-pack-elasticsearch#4285)
ML internal cluster integration tests should not depend on any of
the other plugins. In addition, monitoring can interfere with
some of the tests, making them slower and more fragile. This
reverts a change that got in elastic/x-pack-elasticsearch#3643.

Relates elastic/x-pack-elasticsearch#4279

Original commit: elastic/x-pack-elasticsearch@340455c848
2018-04-04 11:56:18 +01:00
Alexander Reelsen 2da6d14859 Watcher: Prevent watch acknowledgement while watch is executing (elastic/x-pack-elasticsearch#4224)
When a watch is acknowledged, while it is also being executed, the
acknowledgment information can get lost. The reason for this is the
fact, that the execution writes the watch status inside of the watch
regardless, if other writes happened inbetween to make sure the
execution state is caught.

This commit checks the current executions in the execution service and
aborts the API call, if the specified watch ID can be found in those.

Note, this does not prevent this issue fully, as a watch could be
triggered, while the acknowledgement update is running, but it does
reduce the surface area of this problem. In order to properly solve
this, indexing the watch status as part of a watch would need to be
changed.

relates elastic/x-pack-elasticsearch#4003

Original commit: elastic/x-pack-elasticsearch@d7e218b2ac
2018-04-04 10:15:20 +02:00
Alexander Reelsen 681a5ba624 Watcher: Execute Watch API returns better error message (elastic/x-pack-elasticsearch#4228)
Many users take the JSON from an PUT watch API and put it into the
execute watch API and then start to wonder why there is an error, as
they forget to wrap the watch inside a 'watch' field.

This commit adds a better error message in this case hinting at the user
to add a 'watch' field.

Original commit: elastic/x-pack-elasticsearch@5b56b4abad
2018-04-04 09:37:13 +02:00
Chris Earle 70e0012ec4 [TEST][Monitoring] Remove Unneeded Assert (elastic/x-pack-elasticsearch#4284)
This removes the `assert` that has been causing some very unexpected test
failures due to unexpected threading issues.

Some of the code changed and became async, so it is no longer guaranteed
that the same thread calls `doFlush` and `doClose`. We could similarly
make the field `volatile`, but since this `assert` is not really helping
anything it's easier to remove it.

Original commit: elastic/x-pack-elasticsearch@ba39de605f
2018-04-03 15:28:48 -04:00
Jason Tedor ca48a22ca7 Remove license key leniency (elastic/x-pack-elasticsearch#4277)
If the license key specified by the system property license.key does not
exist, Gradle does not care. Gradle should care, so this commit makes it
care.

Original commit: elastic/x-pack-elasticsearch@afc0a1443c
2018-04-03 13:45:07 -04:00
Dimitris Athanasiou 3d8c438403 [ML][DOCS] Correct path for get-stats for all jobs/datafeeds (elastic/x-pack-elasticsearch#4280)
Original commit: elastic/x-pack-elasticsearch@dfb7166f95
2018-04-03 18:00:34 +01:00
Lee Hinman 637301beca Adjust to Nullable moving into core (elastic/x-pack-elasticsearch#4275)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29341

Original commit: elastic/x-pack-elasticsearch@08b6220009
2018-04-03 07:56:21 -06:00
Adrien Grand b7996d70a6 Update shas after Lucene upgrade.
Original commit: elastic/x-pack-elasticsearch@ddc6095933
2018-04-03 09:32:56 +02:00
Tim Brooks 383cadaee7 Fix SSL renegoation test (elastic/x-pack-elasticsearch#4273)
This commit fixes an issue in the security nio transport tests where
renegotiation was not actually being tested. We were not waiting on the
handshake complete latches. This commit fixes this issue.

Original commit: elastic/x-pack-elasticsearch@47bebc5c13
2018-04-02 21:35:35 -04:00
Lee Hinman 0013de7aed Adjust to xcontent being a separate lib (elastic/x-pack-elasticsearch#4259)
* Adjust to xcontent being a separate lib

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29300
where xcontent is now starting to be a separate jar/dependency

Original commit: elastic/x-pack-elasticsearch@1eb249565a
2018-04-02 15:58:48 -06:00
David Leatherman b3cc7fc2c5 Java versions for ci (elastic/x-pack-elasticsearch#4268)
* Add test matrix axis files for periodic java testing

* Add properties file defining java versions to use

Original commit: elastic/x-pack-elasticsearch@d679b9ab3e
2018-04-02 16:50:43 -04:00
Alexander Reelsen ca6879b83e Watcher: Fix JIRA tests (elastic/x-pack-elasticsearch#4271)
The response JSON has changed.

Original commit: elastic/x-pack-elasticsearch@86f9973a68
2018-04-02 13:59:31 -06:00
Aaron Bull Schaefer 1f48d6d2da Add packagingTestSample option to dev-tools/ci script (elastic/x-pack-elasticsearch#4266)
This is going to be run on every pull request.

Related to https://github.com/elastic/infra/issues/4258

Original commit: elastic/x-pack-elasticsearch@5752d779b0
2018-04-02 16:04:44 +02:00
Igor Motov 51602ddb97 SQL: Extract H2 initialization logic (elastic/x-pack-elasticsearch#4260)
Makes it possible to specify custom initialization logic to the test H2
instance. This is will be used to separate H2 setup between geo and
non-geo use cases.

Relates elastic/x-pack-elasticsearch#4080

Original commit: elastic/x-pack-elasticsearch@fa064e35de
2018-04-02 09:45:53 -04:00
Andy Bristol 0144d20213 [test] remove Streamable serde assertions (elastic/x-pack-elasticsearch#4261)
Serialization assertions in ElasticsearchAssertions, a transport
interceptor that used them, and a plugin that added that interceptor
were removed from the test framework. This test case no longer needs to
exclude them from its plugins.

Original commit: elastic/x-pack-elasticsearch@07e5c58983
2018-03-30 14:09:36 -07:00
Zachary Tong b91640877a [Docs] Fix bad doc test
Typo, needed to use a test setup instead of a continuation

Original commit: elastic/x-pack-elasticsearch@cfbc1202c6
2018-03-30 21:07:19 +00:00
David Kyle a4fca07c01 Replace ThrottlerField -> Field in comments and string constants (elastic/x-pack-elasticsearch#4238)
Original commit: elastic/x-pack-elasticsearch@97b449d29d
2018-03-30 21:52:28 +01:00
Zachary Tong 574ce84885 [Docs] Add quickstart and limitation documentation for Rollups
Original commit: elastic/x-pack-elasticsearch@cb4aaa0992
2018-03-30 20:43:33 +00:00
Zachary Tong e8a6c9f5d1 [Rollup] Delegate GetJobs to master (elastic/x-pack-elasticsearch#4247)
If a job is deleted and then GetJobs API is immediately called,
it is possible for a job to be returned in the response.  This is likely
due to the GetJobs API being executed on a node with a slightly
stale cluster state which shows the job as still existing.

So we delegate to the master node so the list of jobs/tasks is current.
After routing to the master, we need to check if the rollup job
is in the PersistentTask's CS.  A job can be acknowledged canceled,
removed from the CS, but the allocated task is still alive.  So we
first check the CS to make sure it's really there before going to the
allocated task to get the status.

As extra precaution, when running local to the task, we also make
sure the task isn't canceled before including it in the response.

relates elastic/x-pack-elasticsearch#4041 


Original commit: elastic/x-pack-elasticsearch@3b6fb65e12
2018-03-30 06:24:29 -07:00
Zachary Tong 54539a1eb0 [Rollup] Make Rollup a Basic license feature (elastic/x-pack-elasticsearch#4246)
* Make Rollup a Basic license feature

Original commit: elastic/x-pack-elasticsearch@ef1ee98855
2018-03-30 06:23:08 -07:00
Dimitris Athanasiou 2aeff7713c [ML] Parsing objects from internal indices should be lenient (elastic/x-pack-elasticsearch#4256)
All ML objects stored in internal indices are currently parsed
strictly. This means unknown fields lead to parsing failures.
In turn, this means we cannot add new fields in any of those
objects (e.g. bucket, record, calendar, etc.) as it is not
backwards compatible.

This commit changes this by introducing lenient parsing when
it comes to reading those objects from the internal indices.
Note we still use strict parsing for the objects we read from
the c++ process, which is nice as it guarantees we would detect
if any of the fields were renamed on one side but not the other.

Also note that even though this is going in from 6.3, we cannot
introduce new fields until 7.0.

relates elastic/x-pack-elasticsearch#4232

Original commit: elastic/x-pack-elasticsearch@3f95d3c7b9
2018-03-29 17:32:57 +01:00