Commit Graph

41084 Commits

Author SHA1 Message Date
Alan Woodward 636442700c
Add conditional token filter to elasticsearch (#31958)
This allows tokenfilters to be applied selectively, depending on the status of the current token in the tokenstream.  The filter takes a scripted predicate, and only applies its subfilter when the predicate returns true.
2018-09-05 14:52:43 +01:00
Nik Everett 74b87989d9
Build: Merge xpack checkstyle config into core (#33399)
Solves all of the xpack line length suppressions and then merges the
remainder of the xpack checkstyle_suppressions.xml file into the core
checkstyle_suppressions.xml file. At this point that just means the
antlr generated files for sql.

It also adds an exclusion to the line length tests for javadocs that
are just a URL. We have one such javadoc and breaking up the line would
make the link difficult to use.
2018-09-05 09:17:02 -04:00
Adrien Grand 913d5fd820 Disable IndexRecoveryIT.testRerouteRecovery.
Relates #32686.
2018-09-05 14:53:22 +02:00
Armin Braun 46774098d9
INGEST: Implement Drop Processor (#32278)
* INGEST: Implement Drop Processor
* Adjust Processor API
* Implement Drop Processor
* Closes #23726
2018-09-05 14:25:29 +02:00
David Roberts a296829205
[ML] Add field stats to log structure finder (#33351)
The log structure endpoint will return these in addition to
pure structure information so that it can be used to drive
pre-import data visualizer functionality.

The statistics for every field are count, cardinality
(distinct count) and top hits (most common values).  Extra
statistics are calculated if the field is numeric: min, max,
mean and median.
2018-09-05 12:57:20 +01:00
Paul Sanwald c303006e6b
Add interval response parameter to AutoDateInterval histogram (#33254)
Adds the interval used to the aggregation response.
2018-09-05 07:35:59 -04:00
Armin Braun 4156cc3fae
MINOR+CORE: Remove Dead Methods ClusterService (#33346)
* None of these methods are used anywhere
2018-09-05 12:08:28 +02:00
Costin Leau d7965ba681
SQL: Align SYS TABLE for ODBC SQL_ALL_* args (#33364)
Fix a bug in SYS TABLES command that did skipped SQL_ALL_* arguments for
catalog and table types

Fix #33312
2018-09-05 10:44:18 +03:00
Alpar Torok 9f96d2ce17
Don't build bwc on assemble (#33372)
Gradle triggers the build of artifacts even if assemble is disabled.
Most users will not need bwc distributions after running `./gradlew
assemble` so instead of forcing them to add `-x buildBwcVersion`, we
detect this and skip the configuration of the artifacts.
2018-09-05 08:24:44 +03:00
Gordon Brown cfd3fa72ed
Add user-defined cluster metadata (#33325)
Adds a place for users to store cluster-wide data they wish to associate
with the cluster via the Cluster Settings API. This is strictly for
user-defined data, Elasticsearch makes no other other use of these
settings.
2018-09-04 16:14:18 -06:00
Jason Tedor 48790b0aad
Allow query caching by default again (#33328)
With the introduction of the default distribution, it means that by
default the query cache is wrapped in the security implementation of the
query cache. This cache does not allow caching if the request does not
carry indices permissions. Yet, this will not happen if authorization is
not allowed, which it is not by default. This means that with the
introduction of the default distribution, query caching was disabled by
default! This commit addresses this by checking if authorization is
allowed and if not, delegating to the default indices query
cache. Otherwise, we proceed as before with security. Additionally, we
clear the cache on license state changes.
2018-09-04 17:50:31 -04:00
Lisa Cawley f3f8d9b833
[DOCS] Moves monitoring pages to docs folder (#33324) 2018-09-04 10:02:13 -07:00
Dimitris Athanasiou bd8d82d859 [ML] Add missing param description in GetOverallBucketsRequest 2018-09-04 17:31:49 +01:00
lcawl c5109a54ee [DOCS] Revert fix for broken link 2018-09-04 09:26:28 -07:00
Costin Leau 43f80fa82b DOCS: Fix anchor and example typos 2018-09-04 19:06:44 +03:00
lcawl 303ae25a6a [DOCS] Fixes broken link 2018-09-04 09:05:30 -07:00
Nik Everett ebd5eb6dc2 ML: Fix build after HLRC change
I recently merged a HLRC change that passed the PR builds but didn't
compile after merging. Sad time. This fixes the compilation.
2018-09-04 11:10:44 -04:00
Jim Ferenczi dbc7102c86
Fix inner hits retrieval when stored fields are disabled (_none_) (#33018)
Now that types are unique per mapping we can retrieve the document mapper
without referencing the type. This fixes an NPE when stored fields are disabled.
For 6x we'll need a different fix since mappings can still have multiple types.

Relates #32941
2018-09-04 16:25:52 +02:00
Costin Leau 17c7f99343
SQL: Show/desc commands now support table ids (#33363)
Extend SHOW TABLES, DESCRIBE and SHOW COLUMNS to support table
identifiers not just SQL LIKE pattern.
This allows both Elasticsearch-style multi-index patterns and SQL LIKE.
To disambiguate between the two (as the " vs ' can be easy to miss),
the grammar now requires LIKE keyword as a prefix for all LIKE-like
patterns.

Also added some docs comparing the two types of patterns.

Fix #33294
2018-09-04 16:54:10 +03:00
Nhat Nguyen 16b53b5ab5 Mute testValidateFollowingIndexSettings
Tracked at #33379
2018-09-04 09:03:26 -04:00
Sohaib Iftikhar 761e8c461f HLRC: Add delete by query API (#32782)
Adds the delete-by-query API to the High Level REST Client.
2018-09-04 08:56:26 -04:00
Dimitris Athanasiou 1457b07a06
[ML] The sort field on get records should default to the record_score (#33358)
This is not changing the behaviour as when the sort field was set
to `influencer_score` the secondary sort would be used and that
was using the `record_score` at the highest priority.
2018-09-04 11:38:24 +01:00
David Roberts 84eaac79d7
[ML] Minor improvements to categorization Grok pattern creation (#33353)
1. The TOMCAT_DATESTAMP format needs to be checked before
   TIMESTAMP_ISO8601, otherwise TIMESTAMP_ISO8601 will
   match the start of the Tomcat datestamp.
2. Exclude more characters before and after numbers.  For
   example, in 1.2.3 we don't want to match 1.2 as a float.
2018-09-04 09:43:49 +01:00
Nikolay Vasiliev d9f394b099 [DOCS] fix a couple of typos (#33356) 2018-09-04 10:07:11 +02:00
Alpar Torok 7f7e8fd733
Disable assemble task instead of removing it (#33348) 2018-09-04 07:32:14 +03:00
Julie Tibshirani 78df00ff24
Simplify the return type of FieldMapper#parse. (#32654) 2018-09-04 01:15:19 +00:00
Benjamin Trent 767d8e0801
[ML] Delete forecast API (#31134) (#33218)
* Delete forecast API (#31134)
2018-09-03 19:06:18 -05:00
Jason Tedor 09bf4e5f00
Introduce private settings (#33327)
This commit introduces the formal notion of a private setting. This
enables us to register some settings that we had previously not
registered as fully-fledged settings to avoid them being exposed via
APIs such as the create index API. For example, we had hacks in the
codebase to allow index.version.created to be passed around inside of
settings objects, but was not registered as a setting so that if a user
tried to use the setting on any API then they would get an
exception. This prevented users from setting index.version.created on
index creation, or updating it via the index settings API. By
introducing private settings, we can continue to reject these attempts,
yet now we can represent these settings as actual settings. In this
change, we register index.version.created as an actual setting. We do
not cutover all settings that we had been treating as private in this
pull request, it is already quite large due to moving some tests around
to account for the fact that some tests need to be able to set the
index.version.created. This can be done in a follow-up change.
2018-09-03 19:17:57 -04:00
Christoph Büscher 79db16f9bb
[Docs] Add search timeout caveats (#33354)
Global search timeouts and timeouts specified in the search request body use the
same internal mechanism as search cancellation. Therefore the same caveats
apply, mostly around the responsiveness of the timeout which gets only checked
by a running search on segment boundaries by default.

Closes #31263
2018-09-03 20:56:05 +02:00
Armin Braun 1f046617bf
TESTS: Fix Race Condition in Temp Path Creation (#33352)
* TESTS: Fix Race Condition in Temp Path Creation

* Calling `createTempDir` concurrently here in
the `Follower`s causes collisions at times
which lead to `createEngine` throwing because
of unexpected files in the newly created temp
dir
   * Fixed by creating all temp dirs in the main test thread
* closes #33344
2018-09-03 19:55:59 +02:00
Nhat Nguyen 24d60c7f4b
Fix from_range in search_after in changes snapshot (#33335)
We can have multiple documents in Lucene with the same seq_no for
parent-child documents (or without rollback). In this case, the usage
"lastSeenSeqNo + 1" is an off-by-one error as it may miss some
documents. This error merely affects the `skippedOperations` contract.

See: https://github.com/elastic/elasticsearch/pull/33222#discussion_r213842257

Closes #33318
2018-09-03 11:58:49 -04:00
Armin Braun 42424aff21
TESTS+DISTR.: Fix testIndexCheckOnStartup Flake (#33349)
* Ignore all `RuntimeException` since random
file corruption triggers other RTE in addition
to the randomly caught one
* closes #33345
2018-09-03 17:06:12 +02:00
tony-dillon a9d2b1dde8 Null completion field should not throw IAE (#33268)
Ignore null value on the completion field

Closes #33200
2018-09-03 16:49:53 +02:00
Colin Goodheart-Smithe 0bf36253a9
Adds code to help with IndicesRequestCacheIT failures (#33313)
* Adds code to help with IndicesRequestCacheIT failures

Relates to #32827

* Adds comment

* Fixes test failure
2018-09-03 14:54:17 +01:00
David Kyle ccb2ad25cc
Prevent NPE parsing the stop datafeed request. (#33347)
The issue depends on the request parameters being passed in the request
body rather than as query parameters.
2018-09-03 13:35:04 +01:00
Dimitris Athanasiou 54fe7fb5a1
HLRC: Add ML get overall buckets API (#33297)
Relates #29827
2018-09-03 13:34:36 +01:00
Alexander Reelsen 246a7df8c2
Core: Fix epoch millis java time formatter (#33302)
The existing implemention could not deal with negative numbers as well
as +- 999 milliseconds around the epoch.

This commit uses Instant.ofEpochMilli() and parses the input to
a number instead of using a date formatter.
2018-09-03 13:13:19 +02:00
Christoph Büscher 978d1ed257
[Docs] Improve tuning for speed advice (#33315)
This change merges two sections in the "Tune for search speed" documentation
that recommend mapping numeric identifiers as keywords. Both sections contain
mostly the same advice, so they can be merged.

Closes #32733
2018-09-03 11:09:30 +02:00
Zachary Tong 90ce3a6224 [Rollup] Fix Caps Comparator to handle calendar/fixed time (#33336)
The comparator used TimeValue parsing, which meant it couldn't handle
calendar time.  This fixes the comparator to handle either (and potentially
mixed).  The mixing shouldn't be an issue since the validation code
upstream will prevent it, but was simplest to allow the comparator
to handle both.
2018-09-03 10:49:19 +02:00
Jim Ferenczi 9310d2eaf3 [CI] Mute IndexShardTests#testIndexCheckOnStartup fails #33345 2018-09-03 10:27:42 +02:00
Jim Ferenczi 2fa75b4438 [CI] Mute LuceneChangesSnapshotTests#testUpdateAndReadChangesConcurrently 2018-09-03 10:14:00 +02:00
Jim Ferenczi f0a61b6dec
Security for _field_names field should not override field statistics (#33261)
In Lucene 8 the statistics for a field (doc_count, sum_doc_count, ...) are
checked and invalid values (v < 0) are rejected. Though for the _field_names
field we hide the statistics of the field if security is enabled since
some terms (field names) may be filtered. However this statistics are never
used, this field is not used for ranking and cannot be used to generate
term vectors. For these reasons this commit restores the original statistics
for the field in order to be compliant with Lucene 8.
2018-09-03 09:36:39 +02:00
Jim Ferenczi 713c07e14d
Add early termination support to BucketCollector (#33279)
This commit adds the support to early terminate the collection of a leaf
in the aggregation framework. This change introduces a MultiBucketCollector which
handles CollectionTerminatedException exactly like the Lucene MultiCollector.
Any aggregator can now throw a CollectionTerminatedException without stopping
the collection of a sibling aggregator. This is useful for aggregators that
can infer their result without visiting all documents (e.g.: a min/max aggregation on a match_all query).
2018-09-03 09:34:35 +02:00
Alpar Torok 3c367a2c46
Fix extractjar task ci (#33272)
Remove tasks to check license and notice and add build integration test instead.

Closes #33201
2018-09-03 10:18:34 +03:00
Nhat Nguyen 3a1dad1050 Mute testFollowIndexAndCloseNode
Tracked at #33337
2018-09-02 19:17:51 -04:00
Nik Everett f8b7a4dbc8
Logging: Drop Settings from some logging ctors (#33332)
Drops `Settings` from some logging ctors now that they are no longer
needed. This should allow us to stop passing `Settings` around to quite
as many places.
2018-09-02 16:51:26 -04:00
Jason Tedor ea4eef8641
Merge branch 'master' into ccr
* master:
  HLREST: add update by query API (#32760)
2018-09-02 16:07:50 -04:00
Sohaib Iftikhar 389bf67275 HLREST: add update by query API (#32760)
Adds update by query to the high level rest client.
2018-09-02 15:15:00 -04:00
Nhat Nguyen 3197a6bbdd Merge branch 'master' into ccr
* master:
  HLRC: ML Flush job (#33187)
  HLRC: Adding ML Job stats (#33183)
  LLREST: Drop deprecated methods (#33223)
  Mute testSyncerOnClosingShard
  [DOCS] Moves machine learning APIs to docs folder (#31118)
2018-09-02 09:30:51 -04:00
Nhat Nguyen c6b011f8ea
TEST: Increase timeout testFollowIndexAndCloseNode (#33333)
This test fails several times due to timeout when asserting the number
of docs on the following and leading indices. This change reduces
the number of docs to index and increases the timeout.
2018-09-02 09:28:47 -04:00