Commit Graph

50541 Commits

Author SHA1 Message Date
Dimitris Athanasiou 94da4ca3fc
[7.x][ML] Extend classification to support multiple classes (#53539) (#53597)
Prepares classification analysis to support more than just
two classes. It introduces a new parameter to the process config
which dictates the `num_classes` to the process. It also
changes the max classes limit to `30` provisionally.

Backport of #53539
2020-03-16 15:00:54 +02:00
David Kyle a38e5ca8e7
Mute TimeSeriesLifecycleActionsIT.testHistoryIsWrittenWithFailure (#53595)
Failure tracked in #50353
2020-03-16 12:30:56 +00:00
Marios Trivyzas 1272ae411e SQL: Fix issue with LIKE/RLIKE as painless script (#53495)
Add missing asScript() implementation for LIKE/RLIKE expressions.

When LIKE/RLIKE are used for example in GROUP BY or are wrapped with
scalar functions in a WHERE clause, the translation must produce a
painless script which will be executed to implement the correct
behaviour and previously this was completely missing, and as a
consquence wrong results were silently (no error) returned.

Fixes: #53486
(cherry picked from commit eaa8ead6742a8e7dcf343bcbaff8de031550fd77)
2020-03-16 12:27:45 +01:00
bellengao e2effa9fab Fix inaccurate total hit count in _search template api (#53155)
When 'rest_track_total_hits_as_int' is set to true, the total hits count in the response should be accurate. So we should set trackTotalHits to true if need when parsing the inline script of a search template request.

Closes #52801
2020-03-16 11:48:43 +01:00
Martijn van Groningen 3b9545848f
Reenable watcher rest tests (#53532)
Also log a message instead of failing if there are active watches at a beginning of a test.

Relates to #53177
2020-03-16 10:24:14 +01:00
David Turner 7e82a4f78c Do not log no-op reconnections at DEBUG (#53469)
Today the NodeConnectionsService emits a DEBUG-level log message each time it
calls TransportService#connectToNode, which happens for every node in the
cluster every ten seconds, and also at every cluster state update. That's a lot
of log messages. Most of these calls are no-ops and can be ignored, but if the
call was not a no-op then it may be worth investigating further. Since the logs
do not distinguish the interesting and uninteresting cases, they are not
useful.

This commit distinguishes the two cases and pushes the noisy logging for the
common no-op case down to TRACE level, leaving only useful and actionable
information in the DEBUG-level logs.
2020-03-16 08:56:20 +00:00
Mark Vieira 2f0aca992b
Revert "Upgrade to Jackson 2.10.3 and GeoIP2 to 2.13.1 (#53576)"
This reverts commit b7dbadeea0.
2020-03-15 18:10:40 -07:00
Jason Tedor 66374b61ca
Remove extra code in allocation commands parsing (#53579)
This commit removes some code that is duplicated in the parsing of
allocation commands in the cluster reroute API.
2020-03-14 18:14:13 -04:00
Jason Tedor b7dbadeea0
Upgrade to Jackson 2.10.3 and GeoIP2 to 2.13.1 (#53576)
This commit upgrades our Jackson dependency to 2.10.3 and our GeoIP2
dependency to 2.13.1.

Relates #53523
2020-03-14 13:28:06 -04:00
Benjamin Trent 1262ab2762
[ML] [Inference] fix number inference models returned in x-pack info call (#53540) (#53560)
the ML portion of the x-pack info API was erroneously counting configuration documents and definition documents. The underlying implementation of our storage separates the two out.

This PR filters the query so that only trained model config documents are counted.
2020-03-13 16:53:34 -04:00
Benjamin Trent 4e43ede735
[ML] renaming inference processor field field_mappings to new name field_map (#53433) (#53502)
This renames the `inference` processor configuration field `field_mappings` to `field_map`.

`field_mappings` is now deprecated.
2020-03-13 15:40:57 -04:00
Marios Trivyzas b6c94fd73e
Fix Term Vectors with artificial docs and keyword fields (#53504) (#53550)
Previously, Term Vectors API was returning empty results for
artificial documents with keyword fields. Checking only for `string()`
on `IndexableField` is not enough, since for `KeywordFieldType`
`binaryValue()` must be used instead.

Fixes #53494

(cherry picked from commit 1fc3fe3d32f41eab2101c0536751b7c47e63cc48)
2020-03-13 19:26:14 +01:00
Tom Veasey 690099553c
[7.x][ML] Adds the class_assignment_objective parameter to classification (#53552)
Adds a new parameter for classification that enables choosing whether to assign labels to
maximise accuracy or to maximise the minimum class recall.

Fixes #52427.
2020-03-13 17:35:51 +00:00
Lisa Cawley b1d589f276 [DOCS] Adds operations_behind to transform stats (#53518) 2020-03-13 09:34:49 -07:00
Marios Trivyzas f8fe1d3344
Fix YAML test for search.allow_expensive_queries (#53541) (#53548)
Remove excessive testing and keep only the checks for when the queries
are disallowed. Fix also the check for the initial value of the setting
to be conmbatible with Go client tests.

(cherry picked from commit 314145294ea926e069c6f8629dfc622a7f31a0fb)
2020-03-13 16:07:38 +01:00
Dan Hermann fb29c2dccf
Fix ingest pipeline _simulate api with empty docs never returns a res… (#52937) (#53547) 2020-03-13 09:41:14 -05:00
Jim Ferenczi e7c393a518 Fix broken link in client documentation (#51834)
This change fixes the link to the RankFeatureQueryBuilder javadocs
in the documentation.

Closes #51800
2020-03-13 14:38:11 +01:00
William Brafford 5b718d2565
Use snake case for nodes stats/info metric names (#53446) (#53535)
* Use snake case for nodes stats/info metric names (#53446)

The REST API uses "thread_pool" as the name of the thread pool metric.
If we use this name internally when we serialize nodes stats and info
requests, we won't need to do any fancy logic to check for and switch
out "threadPool", which was the previous internal name.
2020-03-13 07:49:14 -04:00
James Rodewig 1e31031b73 [DOCS] Clarify `max_shingle_size` parm def (#53480)
Rewrites the `search_as_you_type` field datatype's `max_shingle_size`
mapping parameter to improve clarity and better communicate trade-offs
regarding index size.

Relates to [elastic/kibana#55161][0].

Closes #51774.

[0]: https://github.com/elastic/kibana/pull/55161#discussion_r368107177
2020-03-13 04:08:44 -04:00
Tim Vernum a8677499d7
[Backport] Add support for secondary authentication (#53530)
This change makes it possible to send secondary authentication
credentials to select endpoints that need to perform a single action
in the context of two users.

Typically this need arises when a server process needs to call an
endpoint that users should not (or might not) have direct access to,
but some part of that action must be performed using the logged-in
user's identity.

Backport of: #52093
2020-03-13 16:30:20 +11:00
Tim Vernum bac1740d44
Support authentication without anonymous user (#53528)
This change adds a new parameter to the authenticate methods in the
AuthenticationService to optionally exclude support for the anonymous
user (if an anonymous user exists).

Backport of: #52094
2020-03-13 14:27:29 +11:00
Jim Ferenczi 9dfcc07401 Fix pre-sorting of shards in the can_match phase (#53397)
This commit fixes a bug on sorted queries with a primary sort field
that uses different types in the requested indices. In this scenario
the returned min/max values to sort the shards are not comparable so
we should avoid the sorting rather than throwing an obscure exception.
2020-03-13 01:28:11 +01:00
Jason Tedor f696360517
Fix SHAs for :x-pack:snapshot-tool
This commit fixes the SHA for jackson-databind in :x-pack:snapshot-tool.
2020-03-12 20:24:29 -04:00
Jason Tedor 32dd852210
Update jackson-databind to 2.8.11.6 (#53522)
This commit upgrades the jackson-databind depdendency to
2.8.11.6. Additionally, we revert a previous change that put
ingest-geoip on the version of jackson-databind from the version
properties file. This is because upgrading ingest-geoip to a later
version of jackson-databind also requires an upgrade to the geoip2
dependency which is currently blocked. Therefore, if we can get to a
point where we otherwise upgrade our Jackson dependencies, we do not
want ingest-geoip to automatically come along with it.
2020-03-12 20:15:13 -04:00
Nhat Nguyen fe2f6b359e Fix concurrent requests race over scroll context limit (#53449)
Concurrent search scroll requests can lead to more scroll contexts than the limit.
2020-03-12 17:56:51 -04:00
Lee Hinman 2789fe4179
[7.x] Add ComponentTemplate to MetaData (#53290) (#53489)
* Add ComponentTemplate to MetaData (#53290)

* Add ComponentTemplate to MetaData

This adds a `ComponentTemplate` datastructure that will be used as part of #53101 (Index Templates
v2) to the `MetaData` class. Currently there are no APIs for interacting with this class, so it will
always be an empty map (other than in tests). This infrastructure will be built upon to add APIs in
a subsequent commit.

A `ComponentTemplate` is made up of a `Template`, a version, and a MetaData.Custom class. The
`Template` contains similar information to an `IndexTemplateMetaData` object— settings, mappings,
and alias configuration.

* Update minimal supported version constant

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-12 15:33:32 -06:00
Nik Everett 9dcd64c110
Preserve metric types in top_metrics (backport of #53288) (#53440)
This changes the `top_metrics` aggregation to return metrics in their
original type. Since it only supports numerics, that means that dates,
longs, and doubles will come back as stored, with their appropriate
formatter applied.
2020-03-12 17:17:09 -04:00
Jim Ferenczi 97621e7f65 Removes old Lucene's experimental flag from analyzer documentations (#53217)
This change removes the Lucene's experimental flag from the documentations of the following
tokenizer/filters:
  * Simple Pattern Split Tokenizer
  * Simple Pattern tokenizer
  * Flatten Graph Token Filter
  * Word Delimiter Graph Token Filter

The flag is still present in Lucene codebase but we're fully supporting these tokenizers/filters
in ES for a long time now so the docs flag is misleading.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-03-12 21:18:19 +01:00
Jay Modi 0353b804bf
Mute testKeepTranslogAfterGlobalCheckpoint (#53510)
This change mutes a test that fails reproducibly in
InternalEngineTests.

Relates #53505
2020-03-12 13:08:13 -06:00
Jason Tedor 5b08ea84c9
Add deprecation check for listener thread pool (#53438)
This commit adds a deprecation check for the listener thread pool
settings as these will be removed in 8.0.0.
2020-03-12 14:32:41 -04:00
Mark Vieira 3972cbeba9
Remove Java 13 from runtime testing matrix
With Java 14 soon to be GA 13 will be EOL so there's no need to continue
testing on it.

Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-03-12 10:11:10 -07:00
István Zoltán Szabó 969164cc47 [DOCS] Adds a warning about reindexing docs with the same ID to the PUT DFA docs. (#53490) 2020-03-12 18:01:43 +01:00
Lee Hinman 67fffe676e
[7.x] Add read/writeOptionalVLong to StreamInput/Output (#5314… (#53491)
The spirit of StreamInput/StreamOutput is that common I/O patterns should
be handled by these classes so that the persistence methods in
application classes can be kept short, which facilitates easy visual
comparison between read and write methods, and reduces risks of having
serialization issues due to mismatched implementations.

To this end, this change adds readOptionalVLong and writeOptionalVLong
methods to these classes as we have started to build up cases where
that conditional/null logic has been implemented directly in the read &
write methods.

Co-authored-by: Tim Vernum <tim.vernum@elastic.co>
2020-03-12 10:59:31 -06:00
Jay Modi af36665b08
Deprecate the logstash enabled setting (#53487)
The setting, `xpack.logstash.enabled`, exists to enable or disable the
logstash extensions found within x-pack. In practice, this setting had
no effect on the functionality of the extension. Given this, the
setting is now deprecated in preparation for removal.

Backport of #53367
2020-03-12 10:18:39 -06:00
Dan Hermann 34adfd9611
Validate SSL settings at parse time (#49196) (#53473) 2020-03-12 10:14:51 -05:00
Aleksandr Maus 31d45b3c95
EQL: Improve query folder test suite (#53187) (#53476)
Related to https://github.com/elastic/elasticsearch/issues/52775
2020-03-12 10:58:07 -04:00
Aleksandr Maus e3a1291adf
EQL: Add more rest client tests (#53422) (#53474) 2020-03-12 09:44:46 -04:00
David Kyle 9face1be38
[7.x] Add ObjectParser.declareNamedObject (singular) method (#53017) (#53395)
Add the convenience method AbstractObjectParser.declareNamedObject (singular) to 
complement the existing declareNamedObjects (plural).
2020-03-12 13:21:36 +00:00
Yannick Welsch 48124807d5 Fix SourceOnlySnapshotIT (#53462)
The tests in this class had been failing for a while, but went unnoticed as not tested by CI (see #53442).

The reason the tests fail is that the can-match phase is smarter now, and filters out access to a non-existing field.

Closes #53442
2020-03-12 14:15:03 +01:00
James Rodewig af987fb2d4
[DOCS] Reduce content reuse in enrich docs (#53460)
Restructures the 'Update an enrich policy' section to:

* Migrate the content to the section. It was previously stored in the
  Put Enrich Policy API docs.
* Remove the warning tag admonition from the section content.
* Replace a reused section earlier in the "Set up an enrich processor"
  page with a link.

No substantive changes were made to the content.
2020-03-12 05:57:23 -04:00
Przemyslaw Gomulka 2438b899eb
Support joda style date patterns in 7.x (#52555)
If an index was created in version 6 and contain a date field with a joda-style pattern it should still be allowed to search and insert document into it.
Those created in 6 but date pattern starts with 8, should be considered as java style.
2020-03-12 08:57:03 +01:00
Ryan Ernst 4301c35783 Use exclusiveContent gradle feature for jdk and distro downloads (#53358)
The jdk and distribution download plugins create fake ivy repositories,
and use group based repository filtering to ensure no other artifacts
try to resolve against the fake repos. Currently this works by adding a
blanket exclude to all repositories for the given group name. This
commit changes to using the new exclusiveContent feature in
Gradle to do the exclusion.
2020-03-11 16:35:15 -07:00
Jason Tedor d8e70d4688
Enable deprecation checks for removed settings (#53317)
Today we do not have any infrastructure for adding a deprecation check
for settings that are removed. This commit enables this by adding such
infrastructure. Note that this infrastructure is unused in this commit,
which is deliberate. However, the primary target for this commit is 7.x
where this infrastructue will be used, in a follow-up.
2020-03-11 16:49:16 -04:00
Benjamin Trent 89668c5ea0
[ML][Inference] adds new default_field_map field to trained models (#53294) (#53419)
Adds a new `default_field_map` field to trained model config objects.

This allows the model creator to supply field map if it knows that there should be some map for inference to work directly against the training data.

The use case internally is having analytics jobs supply a field mapping for multi-field fields. This allows us to use the model "out of the box" on data where we trained on `foo.keyword` but the `_source` only references `foo`.
2020-03-11 13:49:39 -04:00
Nik Everett 9ada508347
Fix date_nanos in composite aggs (backport of #53315) (#53347)
It looks like `date_nanos` fields weren't likely to work properly in
composite aggs because composites iterate field values using points and
we weren't converting the points into milliseconds. Because the doc
values were coming back in milliseconds we ended up geting very confused
and just never collecting sub-aggregations.

This fixes that by adding a method to `DateFieldMapper.Resolution` to
`parsePointAsMillis` which is similarly in name and function to
`NumberFieldMapper.NumberType`'s `parsePoint` except that it normalizes
to milliseconds which is what aggs need at the moment.

Closes #53168
2020-03-11 13:00:07 -04:00
Lisa Cawley c408a34a21 [DOCS] Fixes link to custom realm examples (#53205) 2020-03-11 09:15:48 -07:00
Jay Modi 9a21a8abf2
Opt-in logstash plugin to formatting (#53413)
This change opts-in the logstash plugin for enforced formatting.

Backport of #53370
2020-03-11 09:58:37 -06:00
Nhat Nguyen 1fd56698fa Adjust wire version for search context id
Relates #53143
2020-03-11 11:48:11 -04:00
Nhat Nguyen 6665ebe7ab Harden search context id (#53143)
Using a Long alone is not strong enough for the id of search contexts
because we reset the id generator whenever a data node is restarted.
This can lead to two issues:

1. Fetch phase can fetch documents from another index
2. A scroll search can return documents from another index

This commit avoids these issues by adding a UUID to SearchContexId.
2020-03-11 11:48:11 -04:00
Przemysław Witek 8c4c19d310
Perform evaluation in multiple steps when necessary (#53295) (#53409) 2020-03-11 15:36:38 +01:00