41040 Commits

Author SHA1 Message Date
Jay Modi
ea52277a1e
HLRest: add put user API ()
This commit adds a security client to the high level rest client, which
includes an implementation for the put user api. As part of these
changes, a new request and response class have been added that are
specific to the high level rest client. One change here is that the response
was previously wrapped inside a user object. The plan is to remove this
wrapping and this PR adds an unwrapped response outside of the user
object so we can remove the user object later on.

See 
2018-09-05 10:56:30 -06:00
Adrien Grand
7eef7f441b Add info about the running threads in case of failure.
Relates 
2018-09-05 18:41:26 +02:00
Christoph Büscher
eafc2a5470
Don't count metadata fields towards index.mapping.total_fields.limit ()
The maximum number of fields per index is limited to 1000 by default by the
`index.mapping.total_fields.limit` setting to prevent accidental mapping
explosions due to too many fields. Currently all metadata fields also count
towards this limit, which can lead to some confusion when using lower limits.
It is not obvious for users that they cannot actually add as many fields as
are specified by the limit in this case.

This change takes the number of metadata fields out of the field count that we
check against the field limit. It also adds tests that check that we can add
fields up to the specified limit, but throw an exception for any additional field added.

Closes 
2018-09-05 18:27:21 +02:00
Jason Tedor
23934e39d2
Fix deprecated setting specializations ()
Deprecating a some setting specializations (e.g., list settings) does
not cause deprecation warning headers and deprecation log messages to
appear. This is due to a missed check for deprecation. This commit fixes
this for all setting specializations, and ensures that this can not be
missed again.
2018-09-05 11:01:58 -04:00
Michael Basnight
7319bc7411
HLRC: split cluster request converters ()
In an effort to encapsulate the different clients, the request
converters are being shuffled around. This splits the ClusterClient
request converters.
2018-09-05 09:34:47 -05:00
Dimitris Athanasiou
24776b2b80
HLRC: Add ML get influencers API ()
Relates 
2018-09-05 15:06:55 +01:00
Alan Woodward
636442700c
Add conditional token filter to elasticsearch ()
This allows tokenfilters to be applied selectively, depending on the status of the current token in the tokenstream.  The filter takes a scripted predicate, and only applies its subfilter when the predicate returns true.
2018-09-05 14:52:43 +01:00
Nik Everett
74b87989d9
Build: Merge xpack checkstyle config into core ()
Solves all of the xpack line length suppressions and then merges the
remainder of the xpack checkstyle_suppressions.xml file into the core
checkstyle_suppressions.xml file. At this point that just means the
antlr generated files for sql.

It also adds an exclusion to the line length tests for javadocs that
are just a URL. We have one such javadoc and breaking up the line would
make the link difficult to use.
2018-09-05 09:17:02 -04:00
Adrien Grand
913d5fd820 Disable IndexRecoveryIT.testRerouteRecovery.
Relates .
2018-09-05 14:53:22 +02:00
Armin Braun
46774098d9
INGEST: Implement Drop Processor ()
* INGEST: Implement Drop Processor
* Adjust Processor API
* Implement Drop Processor
* Closes 
2018-09-05 14:25:29 +02:00
David Roberts
a296829205
[ML] Add field stats to log structure finder ()
The log structure endpoint will return these in addition to
pure structure information so that it can be used to drive
pre-import data visualizer functionality.

The statistics for every field are count, cardinality
(distinct count) and top hits (most common values).  Extra
statistics are calculated if the field is numeric: min, max,
mean and median.
2018-09-05 12:57:20 +01:00
Paul Sanwald
c303006e6b
Add interval response parameter to AutoDateInterval histogram ()
Adds the interval used to the aggregation response.
2018-09-05 07:35:59 -04:00
Armin Braun
4156cc3fae
MINOR+CORE: Remove Dead Methods ClusterService ()
* None of these methods are used anywhere
2018-09-05 12:08:28 +02:00
Costin Leau
d7965ba681
SQL: Align SYS TABLE for ODBC SQL_ALL_* args ()
Fix a bug in SYS TABLES command that did skipped SQL_ALL_* arguments for
catalog and table types

Fix 
2018-09-05 10:44:18 +03:00
Alpar Torok
9f96d2ce17
Don't build bwc on assemble ()
Gradle triggers the build of artifacts even if assemble is disabled.
Most users will not need bwc distributions after running `./gradlew
assemble` so instead of forcing them to add `-x buildBwcVersion`, we
detect this and skip the configuration of the artifacts.
2018-09-05 08:24:44 +03:00
Gordon Brown
cfd3fa72ed
Add user-defined cluster metadata ()
Adds a place for users to store cluster-wide data they wish to associate
with the cluster via the Cluster Settings API. This is strictly for
user-defined data, Elasticsearch makes no other other use of these
settings.
2018-09-04 16:14:18 -06:00
Jason Tedor
48790b0aad
Allow query caching by default again ()
With the introduction of the default distribution, it means that by
default the query cache is wrapped in the security implementation of the
query cache. This cache does not allow caching if the request does not
carry indices permissions. Yet, this will not happen if authorization is
not allowed, which it is not by default. This means that with the
introduction of the default distribution, query caching was disabled by
default! This commit addresses this by checking if authorization is
allowed and if not, delegating to the default indices query
cache. Otherwise, we proceed as before with security. Additionally, we
clear the cache on license state changes.
2018-09-04 17:50:31 -04:00
Lisa Cawley
f3f8d9b833
[DOCS] Moves monitoring pages to docs folder () 2018-09-04 10:02:13 -07:00
Dimitris Athanasiou
bd8d82d859 [ML] Add missing param description in GetOverallBucketsRequest 2018-09-04 17:31:49 +01:00
lcawl
c5109a54ee [DOCS] Revert fix for broken link 2018-09-04 09:26:28 -07:00
Costin Leau
43f80fa82b DOCS: Fix anchor and example typos 2018-09-04 19:06:44 +03:00
lcawl
303ae25a6a [DOCS] Fixes broken link 2018-09-04 09:05:30 -07:00
Nik Everett
ebd5eb6dc2 ML: Fix build after HLRC change
I recently merged a HLRC change that passed the PR builds but didn't
compile after merging. Sad time. This fixes the compilation.
2018-09-04 11:10:44 -04:00
Jim Ferenczi
dbc7102c86
Fix inner hits retrieval when stored fields are disabled (_none_) ()
Now that types are unique per mapping we can retrieve the document mapper
without referencing the type. This fixes an NPE when stored fields are disabled.
For 6x we'll need a different fix since mappings can still have multiple types.

Relates 
2018-09-04 16:25:52 +02:00
Costin Leau
17c7f99343
SQL: Show/desc commands now support table ids ()
Extend SHOW TABLES, DESCRIBE and SHOW COLUMNS to support table
identifiers not just SQL LIKE pattern.
This allows both Elasticsearch-style multi-index patterns and SQL LIKE.
To disambiguate between the two (as the " vs ' can be easy to miss),
the grammar now requires LIKE keyword as a prefix for all LIKE-like
patterns.

Also added some docs comparing the two types of patterns.

Fix 
2018-09-04 16:54:10 +03:00
Nhat Nguyen
16b53b5ab5 Mute testValidateFollowingIndexSettings
Tracked at 
2018-09-04 09:03:26 -04:00
Sohaib Iftikhar
761e8c461f HLRC: Add delete by query API ()
Adds the delete-by-query API to the High Level REST Client.
2018-09-04 08:56:26 -04:00
Dimitris Athanasiou
1457b07a06
[ML] The sort field on get records should default to the record_score ()
This is not changing the behaviour as when the sort field was set
to `influencer_score` the secondary sort would be used and that
was using the `record_score` at the highest priority.
2018-09-04 11:38:24 +01:00
David Roberts
84eaac79d7
[ML] Minor improvements to categorization Grok pattern creation ()
1. The TOMCAT_DATESTAMP format needs to be checked before
   TIMESTAMP_ISO8601, otherwise TIMESTAMP_ISO8601 will
   match the start of the Tomcat datestamp.
2. Exclude more characters before and after numbers.  For
   example, in 1.2.3 we don't want to match 1.2 as a float.
2018-09-04 09:43:49 +01:00
Nikolay Vasiliev
d9f394b099 [DOCS] fix a couple of typos () 2018-09-04 10:07:11 +02:00
Alpar Torok
7f7e8fd733
Disable assemble task instead of removing it () 2018-09-04 07:32:14 +03:00
Julie Tibshirani
78df00ff24
Simplify the return type of FieldMapper#parse. () 2018-09-04 01:15:19 +00:00
Benjamin Trent
767d8e0801
[ML] Delete forecast API () ()
* Delete forecast API ()
2018-09-03 19:06:18 -05:00
Jason Tedor
09bf4e5f00
Introduce private settings ()
This commit introduces the formal notion of a private setting. This
enables us to register some settings that we had previously not
registered as fully-fledged settings to avoid them being exposed via
APIs such as the create index API. For example, we had hacks in the
codebase to allow index.version.created to be passed around inside of
settings objects, but was not registered as a setting so that if a user
tried to use the setting on any API then they would get an
exception. This prevented users from setting index.version.created on
index creation, or updating it via the index settings API. By
introducing private settings, we can continue to reject these attempts,
yet now we can represent these settings as actual settings. In this
change, we register index.version.created as an actual setting. We do
not cutover all settings that we had been treating as private in this
pull request, it is already quite large due to moving some tests around
to account for the fact that some tests need to be able to set the
index.version.created. This can be done in a follow-up change.
2018-09-03 19:17:57 -04:00
Christoph Büscher
79db16f9bb
[Docs] Add search timeout caveats ()
Global search timeouts and timeouts specified in the search request body use the
same internal mechanism as search cancellation. Therefore the same caveats
apply, mostly around the responsiveness of the timeout which gets only checked
by a running search on segment boundaries by default.

Closes 
2018-09-03 20:56:05 +02:00
Armin Braun
1f046617bf
TESTS: Fix Race Condition in Temp Path Creation ()
* TESTS: Fix Race Condition in Temp Path Creation

* Calling `createTempDir` concurrently here in
the `Follower`s causes collisions at times
which lead to `createEngine` throwing because
of unexpected files in the newly created temp
dir
   * Fixed by creating all temp dirs in the main test thread
* closes 
2018-09-03 19:55:59 +02:00
Nhat Nguyen
24d60c7f4b
Fix from_range in search_after in changes snapshot ()
We can have multiple documents in Lucene with the same seq_no for
parent-child documents (or without rollback). In this case, the usage
"lastSeenSeqNo + 1" is an off-by-one error as it may miss some
documents. This error merely affects the `skippedOperations` contract.

See: https://github.com/elastic/elasticsearch/pull/33222#discussion_r213842257

Closes 
2018-09-03 11:58:49 -04:00
Armin Braun
42424aff21
TESTS+DISTR.: Fix testIndexCheckOnStartup Flake ()
* Ignore all `RuntimeException` since random
file corruption triggers other RTE in addition
to the randomly caught one
* closes 
2018-09-03 17:06:12 +02:00
tony-dillon
a9d2b1dde8 Null completion field should not throw IAE ()
Ignore null value on the completion field

Closes 
2018-09-03 16:49:53 +02:00
Colin Goodheart-Smithe
0bf36253a9
Adds code to help with IndicesRequestCacheIT failures ()
* Adds code to help with IndicesRequestCacheIT failures

Relates to 

* Adds comment

* Fixes test failure
2018-09-03 14:54:17 +01:00
David Kyle
ccb2ad25cc
Prevent NPE parsing the stop datafeed request. ()
The issue depends on the request parameters being passed in the request
body rather than as query parameters.
2018-09-03 13:35:04 +01:00
Dimitris Athanasiou
54fe7fb5a1
HLRC: Add ML get overall buckets API ()
Relates 
2018-09-03 13:34:36 +01:00
Alexander Reelsen
246a7df8c2
Core: Fix epoch millis java time formatter ()
The existing implemention could not deal with negative numbers as well
as +- 999 milliseconds around the epoch.

This commit uses Instant.ofEpochMilli() and parses the input to
a number instead of using a date formatter.
2018-09-03 13:13:19 +02:00
Christoph Büscher
978d1ed257
[Docs] Improve tuning for speed advice ()
This change merges two sections in the "Tune for search speed" documentation
that recommend mapping numeric identifiers as keywords. Both sections contain
mostly the same advice, so they can be merged.

Closes 
2018-09-03 11:09:30 +02:00
Zachary Tong
90ce3a6224 [Rollup] Fix Caps Comparator to handle calendar/fixed time ()
The comparator used TimeValue parsing, which meant it couldn't handle
calendar time.  This fixes the comparator to handle either (and potentially
mixed).  The mixing shouldn't be an issue since the validation code
upstream will prevent it, but was simplest to allow the comparator
to handle both.
2018-09-03 10:49:19 +02:00
Jim Ferenczi
9310d2eaf3 [CI] Mute IndexShardTests#testIndexCheckOnStartup fails 2018-09-03 10:27:42 +02:00
Jim Ferenczi
2fa75b4438 [CI] Mute LuceneChangesSnapshotTests#testUpdateAndReadChangesConcurrently 2018-09-03 10:14:00 +02:00
Jim Ferenczi
f0a61b6dec
Security for _field_names field should not override field statistics ()
In Lucene 8 the statistics for a field (doc_count, sum_doc_count, ...) are
checked and invalid values (v < 0) are rejected. Though for the _field_names
field we hide the statistics of the field if security is enabled since
some terms (field names) may be filtered. However this statistics are never
used, this field is not used for ranking and cannot be used to generate
term vectors. For these reasons this commit restores the original statistics
for the field in order to be compliant with Lucene 8.
2018-09-03 09:36:39 +02:00
Jim Ferenczi
713c07e14d
Add early termination support to BucketCollector ()
This commit adds the support to early terminate the collection of a leaf
in the aggregation framework. This change introduces a MultiBucketCollector which
handles CollectionTerminatedException exactly like the Lucene MultiCollector.
Any aggregator can now throw a CollectionTerminatedException without stopping
the collection of a sibling aggregator. This is useful for aggregators that
can infer their result without visiting all documents (e.g.: a min/max aggregation on a match_all query).
2018-09-03 09:34:35 +02:00
Alpar Torok
3c367a2c46
Fix extractjar task ci ()
Remove tasks to check license and notice and add build integration test instead.

Closes 
2018-09-03 10:18:34 +03:00