Commit Graph

50518 Commits

Author SHA1 Message Date
James Rodewig 8f4a3eb07f [DOCS] Add token graph concept docs (#53339)
Adds conceptual docs for token graphs.
These docs cover:

* How a token graph is constructed from a token stream
* How synonyms and multi-position tokens impact token graphs
* How token graphs are used during search
* Why some token filters produce invalid token graphs

Also makes the following supporting changes:
* Adds anchors to the 'Anatomy of an Analyzer' docs for cross-linking
* Adds several SVGs for token graph diagrams
2020-03-19 07:43:18 -04:00
Jim Ferenczi 4b0ae15a9d Disable distributed sort optimization on scroll requests (#53759)
This commit disables the sort optimization added in #51852 for scroll requests.
Scroll queries keep a state per shard so we cannot modify the request on
the first round (submit).
This bug was introduced in non-released versions which is why this pr
is marked as a non-issue.
2020-03-19 08:11:23 +01:00
Ioannis Kakavas 4a36894a48
Mute failing tests (#53781)
See #53738
2020-03-19 08:16:23 +02:00
Mark Vieira 9b3b08318d
Remove unused import 2020-03-18 21:07:17 -07:00
Jason Tedor bc5dae2713
Fix compilation in RoutingNode
This commit fixes compilation in RoutingNode.java after a backport
brought back usage of an API not available in JDK 8.
2020-03-18 22:21:54 -04:00
Leaf-Lin b649b0d273
Update jvm options doc
There's a typo with extra `
2020-03-19 12:00:01 +11:00
Jason Tedor 90ab949415
Improve performance of shards limits decider (#53577)
On clusters with a large number of shards, the shards limits allocation
decider can exhibit poor performance leading to timeouts applying
cluster state updates. This occurs because for every shard, we do a loop
to count the number of shards on the node, and the number of shards for
the index of the shard. This is roughly quadratic in the number of
shards. This loop is not necessary, since we already have a O(1) method
to count the number of non-relocating shards on a node, and with this
commit we add some infrastructure to RoutingNode to make counting the
number of shards per index O(1).
2020-03-18 20:58:22 -04:00
Benjamin Trent 415d73c27d
[Transform] renamed _cat/transform to _cat/transforms (#53743) (#53771)
renaming _cat/transform to  _cat/transforms for uniformity with the other _cat apis.
2020-03-18 19:54:03 -04:00
Lisa Cawley 268e512f0b [DOCS] Add transform nodes (#53698) 2020-03-18 15:26:37 -07:00
Stuart Tettemer cdbee32f55
Scripting: Per-context script cache, default off (#52855) (#53756)
* Adds per context settings:
  `script.context.${CONTEXT}.cache_max_size` ~
  `script.cache.max_size`

  `script.context.${CONTEXT}.cache_expire` ~
  `script.cache.expire`

  `script.context.${CONTEXT}.max_compilations_rate` ~
  `script.max_compilations_rate`

* Context cache is used if:
  `script.max_compilations_rate=use-context`.  This
  value is dynamically updatable, so users can
  switch back to the general cache if desired.

* Settings for context caches take the first value
  that applies:
  1) Context specific settings if set, eg
     `script.context.ingest.cache_max_size`
  2) Correlated general setting is set to the non-default
     value, eg `script.cache.max_size`
  3) Context default

The reason for 2's inclusion is to allow an easy
transition for users who've customized their general
cache settings.

Using the general cache settings for the context caches
results in higher effective settings, since they are
multiplied across the number of contexts.  So a general
cache max size of 200 will become 200 * # of contexts.
However, this behavior it will avoid users snapping to a
value that is too low for them.

Backport of: #52855
Refs: #50152
2020-03-18 14:44:04 -06:00
Jason Tedor 569323928b
Upgrade the bundled JDK to JDK 14 (#53748)
This commit upgrades the bundled JDK to JDK 14.
2020-03-18 16:18:12 -04:00
Christoph Büscher 9a328c2b83 Add unsupported parameters to HLRC search request (#53745)
Currently we don't send values for the `pre_filter_shard_size` and
`max_concurrent_shard_requests` SearchRequest parameters over http when using
the High Level Rest Client. This change adds these parameters to the
RequestConverters and tests.
2020-03-18 20:00:31 +01:00
James Rodewig 0e2e06bd7e [DOCS] Remove incorrect parms from put index template API docs (#53750)
Removes the `flat_settings` and `timeout` query parameters from the JSON
spec and asciidoc docs for the put index template API.

These parameters are not supported by the API.
2020-03-18 14:36:27 -04:00
Ioannis Kakavas af519cccff Revert "Mute TimeSeriesLifecycleActionsIT (#53741)"
This reverts commit df0ad7569b.
2020-03-18 18:51:06 +02:00
Jim Ferenczi 8e17322b3a
Shortcut query phase using the results of other shards (#51852) (#53659)
This commit, built on top of #51708, allows to modify shard search requests based on informations collected on other shards. It is intended to speed up sorted queries on time-based indices. For queries that are only interested in the top documents.

This change will rewrite the shard queries to match none if the bottom sort value computed in prior shards is better than all values in the shard.
For queries that mix top documents and aggregations this change will reset the size of the top documents to 0 instead of rewriting to match none.
This means that we don't need to keep a search context open for this shard since we know in advance that it doesn't contain any competitive hit.
2020-03-18 17:20:35 +01:00
markharwood ae19802e29
Fix highlighter support in PinnedQuery and added test (#53716) (#53729)
CappedScoreQuery was not delegating queryVisitor calls

Closes #53699
2020-03-18 15:39:17 +00:00
Ioannis Kakavas df0ad7569b
Mute TimeSeriesLifecycleActionsIT (#53741)
see #53738
2020-03-18 17:38:24 +02:00
markharwood 598d4c1bf9
Formatting fix
Bullet points were not rendered correctly
2020-03-18 15:36:52 +00:00
Lisa Cawley c6e37c7662 [DOCS] Adds stub for cat transform API (#53737) 2020-03-18 08:31:10 -07:00
Alan Woodward d325899c54 Use QueryVisitor when extracting PercolatorQuery list for highlighting (#53728)
The highlighting phase for percolator queries currently uses some custom query
traversal logic to find all instances of PercolatorQuery in the query tree for the
current search context. This commit converts things to instead use a QueryVisitor,
which future-proofs us against new wrapper queries or queries from custom
plugins that the percolator module doesn't know about.
2020-03-18 15:24:49 +00:00
Nhat Nguyen 1615c4b379
Fix testKeepTranslogAfterGlobalCheckpoint (#53704)
Read the global checkpoint after flushed as we might advance it while flushing.

Closes #53505
2020-03-18 11:24:19 -04:00
Luca Cavanna 75c367de13 [TEST] Replace agg key in async search yaml test (#53727)
Some clients have problems running this test as a numeric key is treated like an array index by default.
We can work around this by renaming the aggregation key to not be a numeric.
2020-03-18 16:16:15 +01:00
Benjamin Trent 2ccb963f1d
Create GET _cat/transforms API Issue (#53643) (#53726)
Adds new` _cat/transform` and `_cat/transform/{transform_id}` endpoints.
2020-03-18 10:45:28 -04:00
Alan Woodward 580bc40c0c Make it possible to deprecate all variants of a ParseField with no replacement (#53722)
Sometimes we want to deprecate and remove a ParseField entirely, without replacement;
for example, the various places where we specify a _type field in 7x. Currently we can
tell users only that a particular field name should not be used, and that another name should
be used in its place. This commit adds the ability to say that a field should not be used at
all.
2020-03-18 14:16:19 +00:00
Marios Trivyzas d56dee599a
Increase step between checks for cancellation (#53712)
The introduction of the ExitableDirectoryReader showed increase of
latencies for range queries using pointvalues.

Check for cancellation every 1024 docs instead of every 15 to lower
the impact of the check in query's performance.

Follows: #52822
Fixes: #53496
(cherry picked from commit 6b5fc35e4458e60a7ca5822584ec6a60562f2c01)
2020-03-18 14:52:40 +01:00
James Rodewig 0732475cb9 [DOCS] Streamline `analyzer` mapping parm def (#51874)
Simplifies the `analyzer` mapping parameter definition to remove
duplicated analysis content and examples.
2020-03-18 09:43:59 -04:00
Ioannis Kakavas e5aa0906f7
Mute testHistoryIsWrittenWithDeletion (#53721)
see #53718
2020-03-18 14:49:57 +02:00
James Rodewig e4b7af11ab [DOCS] Remove `light_bengali` stemmer (#53697)
Only the `bengali` stemmer is available in Lucene and surfaced through
Elasticsearch. This removes the incorrect `light_bengali` link in our
docs.
2020-03-18 08:34:27 -04:00
Tanguy Leroux 6cc564d677 Restore off-heap loading for term dictionary in ReadOnlyEngine (#53713)
This is a partial restore of #43158, following decision taken in #51247

Closes #51247
2020-03-18 13:24:34 +01:00
Dan Hermann 94ac979c66
Support array for all string ingest processors (#53694) 2020-03-18 07:07:49 -05:00
Christoph Büscher 2384c1359d Revert "Fix ReloadSynonymAnalyzerIT failure (#53663)"
This reverts commit 2c32173fce.
2020-03-18 12:44:23 +01:00
Christoph Büscher 2c32173fce Fix ReloadSynonymAnalyzerIT failure (#53663)
There is an assertion in ReloadAnalyzersResponse.merge that compares index names
of merged responses that was falsely using object equality instead of
String.equals(). In the past this didn't seem to matter but with changes in the
test setup we started to see failures. Correcting this and also simplifying test
a bit to be able to run it repeatedly if needed.

Closes #53443
2020-03-18 11:55:37 +01:00
Ralph Ursprung 474dfbf9c7 [Docs] Fix highlighting in match-query example (#52426) 2020-03-18 11:53:14 +01:00
Tianlun Li e7ae9ae596 Deprecate delaying state recovery for master nodes (#53646)
It is useful to be able to delay state recovery until enough data nodes have
joined the cluster, since this gives the shard allocator a decent opportunity
to re-use as much existing data as possible. However we also have the option to
delay state recovery until a certain number of master-eligible nodes have
joined, and this is unnecessary: we require a majority of master-eligible nodes
for state recovery, and there is no advantage in waiting for more.

This commit deprecates the unnecessary settings in preparation for their
removal.

Relates #51806
2020-03-18 10:04:22 +00:00
Przemysław Witek ec13c093df
Make ML index aliases hidden (#53160) (#53710) 2020-03-18 10:28:45 +01:00
Yash Joshi 85cc7a06c4 [Docs] Fix typo in range query (#53656) 2020-03-18 10:18:08 +01:00
Ioannis Kakavas 873d0ecd09
Fix potential bug in concurrent token refresh support (#53668) (#53705)
Ensure that we do not proceed execution after calling the
listerer's onFailure
2020-03-18 09:43:26 +02:00
Hendrik Muhs 7a12300ce6
[7.x][Transform] enhance the output of preview to return full… (#53695)
changes the output format of preview regarding deduced mappings and enhances
it to return all the details about auto-index creation. This allows the user
to customize the index creation. Using HLRC you can create a index request
from the output of the response.

backport #53572
2020-03-18 08:37:56 +01:00
Hendrik Muhs a6dca577e5 [Transform] data nanos/date histogram IT (#53654)
add an integration test for date nanos in combination with date_histogram
2020-03-17 20:58:57 +01:00
Lee Hinman 9c0e846db3
[7.x] Add REST API for ComponentTemplate CRUD (#53558) (#53681)
* Add REST API for ComponentTemplate CRUD

This adds the Put/Get/DeleteComponentTemplate APIs that allow inserting, retrieving, and removing
ComponentTemplateMetadata into the cluster state metadata.

These APIs are currently only available behind a feature flag system property -
`es.itv2_feature_flag_registered`.

Relates to #53101

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-17 13:23:28 -06:00
Karen Metts 3c5437894e
Remove link to old settings 7.x (#53639) 2020-03-17 14:38:50 -04:00
Ryan Ernst 169308656c Actually add licenses for jackson
Missed in 1d9f57b
2020-03-17 11:13:20 -07:00
Ryan Ernst 1d9f57bfc1 Fix databind version reference
This fixes fallout from a bad backport of #53642
2020-03-17 10:40:56 -07:00
Ryan Ernst 5c472fcb47 Upgrade jackson to 2.10.3 and GeoIP to 2.13.1 (#53642)
Re-applies the change from #53523 along with test fixes.

closes #53626
closes #53624
closes #53622
closes #53625

Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Jake Landis <jake.landis@elastic.co>
2020-03-17 10:28:51 -07:00
Mark Vieira 497250ba9c
Improve performance of docs snippet length checkstyle rule (#53636)
The multiline regex rule used to detect docs code snippets greater than
76 characters in length has considerable cost to. For the high level
rest client project alone, this was taking upwards of 3 minutes. This
commit updates the rule regex pattern to use non-greedy matching when
appropriate which results in a lot fewer backtracks and about a 70%
reduction in execution time on the high level rest client module.
2020-03-17 09:58:07 -07:00
David Turner 347b7594b7 Fix deprecation in history retention docs (#53655)
This commit adjusts a `deprecation[...]` message in the docs since such
messages must be on a single line. It also moves this message to the start of
the description of the deprecated setting as is the case with other such
messages.
2020-03-17 14:06:37 +00:00
David Kyle 2b635737e1
[ML] Parse single named object in config classes (#53472) (#53542) 2020-03-17 13:59:52 +00:00
Alan Woodward 71b703edd1 Rename AtomicFieldData to LeafFieldData (#53554)
This conforms with lucene's LeafReader naming convention, and
matches other per-segment structures in elasticsearch.
2020-03-17 12:30:12 +00:00
Andrei Stefan 79600eb38b
SQL: add support for index aliases for SYS COLUMNS command (#53525) (#53653)
(cherry picked from commit f65e4d6ff7b2e00eb6f9c985fbe7cb24de00f045)
2020-03-17 12:49:08 +02:00
Hendrik Muhs a0314ad015 [Transform] add transform discovery node role (#53616)
Enhancement of #52712: Add a discovery node role using the letter t for transform.

Fixes #53156
2020-03-17 11:39:20 +01:00