Commit Graph

50628 Commits

Author SHA1 Message Date
Dan Hermann 3ffd34617f
Switch to AtomicLong for ingestCurrent metric to prevent negative values (#52581) (#52834) 2020-02-26 13:26:26 -06:00
Jay Modi 07ef8ccff4
Allow dynamic updates for index.hidden setting (#52837)
This commit changes the `index.hidden` setting from being final to a
dynamic setting. While the setting being final allows for easier
reasoning about an index, making this setting update-able has more
benefits in that we can upgrade existing indices to be hidden and it
will enable future features that would dynamically make indices hidden.

Backport of #52772
2020-02-26 11:46:29 -07:00
Nik Everett bfaa487757
Switch pipeline agg parsing to ContextParser (#52776) (#52832)
We've pretty well settled on `ContextParser` for a generic interface to
`ObjectParser`-like-things. This switches the interface used for
building parsing pipeline aggregations to `ContextParser` which saves a
couple of little wrappers around `ObjectParser`.
2020-02-26 12:57:20 -05:00
James Rodewig f5253d20f7 [DOCS] Update term vectors snippet to prevent CI failure (#52819)
Adds the `?refresh=wait_for` query argument to an index API snippet in
the term vectors API docs.

This should ensure the document is indexed and available before a
subsequent term vectors API request executes.

Fixes #52814.
2020-02-26 12:41:40 -05:00
Armin Braun f7d71b5930
Fix GCS Mock Range Downloads (#52804) (#52830)
We were not correctly respecting the download range which lead
to the GCS SDK client closing the connection at times.
Also, fixes another instance of failing to drain the request fully before sending the response headers.

Closes #51446
2020-02-26 18:38:02 +01:00
Lisa Cawley b788ec7157 [DOCS] Adds cat datafeeds API (#52738) 2020-02-26 09:28:57 -08:00
Tim Brooks be8d704e2b
Remove seeds depedency for remote cluster settings (#52829)
Currently 3 remote cluster settings (ping interval, skip unavailable,
and compression) have a dependency on the seeds setting being
comfigured. With proxy mode, it is now possible that these settings the
seeds setting has not been configured. This commit removes this
dependency and adds new validation for these settings.
2020-02-26 10:17:25 -07:00
Ioannis Kakavas 2d01c005ba
Update commons-collections test dependency to 3.2.2 (#52808) (#52817)
This is only a test dependency but it trips scanners so upgrade to
3.2.2 which doesn't suffer from the issues mentioned in i.e.
https://snyk.io/vuln/SNYK-JAVA-COMMONSCOLLECTIONS-472711
2020-02-26 17:03:45 +02:00
Adrien Grand 1807f86751
Generalize how queries on `_index` are handled at rewrite time (#52815)
Generalize how queries on `_index` are handled at rewrite time (#52486)

Since this change refactors rewrites, I also took it as an opportunity to adrress #49254: instead of returning the same queries you would get on a keyword field when a field is unmapped, queries get rewritten to a MatchNoDocsQueryBuilder.

This change exposed a couple bugs, like the fact that the percolator doesn't rewrite queries at query time, or that the significant_terms aggregation doesn't rewrite its inner filter, which I fixed.

Closes #49254
2020-02-26 15:37:43 +01:00
David Kyle ad3a3b1af9 Mute RunDataFrameAnalyticsIT.testOutlierDetectionStopAndRestart 2020-02-26 14:31:00 +00:00
Jake Landis 8d311297ca
[7.x] Smarter copying of the rest specs and tests (#52114) (#52798)
* Smarter copying of the rest specs and tests (#52114)

This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default
the rest specs will get copied if the project applies
`elasticsearch.standalone-rest-test` or `esplugin` and the project
has rest tests or you configure the custom extension `restResources`.

This PR also removes the need for dozens of places where the x-pack
specs were copied by supporting copying of the x-pack rest specs too.

The plugin/task introduced here can also copy the rest tests to the
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of
which rest specs are copied. Per project can be configured to include
only a subset of the specs (or tests). Configuring a project to only
copy the specs when actually needed should help with build cache hit
rates since we can better define what is actually in use.
However, project level optimizations for build cache hit rates are
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on
integTest task.

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources`
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-26 08:13:41 -06:00
Ioannis Kakavas 2a6c3bea3f
Update oauth2-oidc-sdk to 7.0 (#52489) (#52806)
Resolves: #48409
Other changes:
https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect
-extensions/src/7.0.2/CHANGELOG.txt
2020-02-26 16:02:10 +02:00
Nhat Nguyen f0bc8abcd0 Fix translog stats on closed indices yaml test (#52800)
We need to wait for no initializing shards before closing; otherwise, 
we might fail to close some recovering replicas.

Closes #52701
2020-02-26 08:14:35 -05:00
Luca Cavanna 9e38125464
Clarify when shard iterators get sorted (#52810)
Currently we have two ways to create a GroupShardsIterator: one that will resort the iterators based on their natural ordering, and another one that will leave them in their original order. This is currently done through two constructors, one that accepts a single argument which does the sorting, and another which accepts a second boolean argument to control whether sorting should happen or not. This second constructor is only called externally to disable the sorting.

By introducing a specific method to create a sorted shard iterator we clarify and make it easier to track when we do sort and when we do not as the iterators are externally sorted.
2020-02-26 13:58:20 +01:00
Bogdan Pintea 304e1e69b8 remove references to the SQL API from ODBC config (#52765)
Remove reference to an "SQL API" which could suggest that one needs to
treat this in a special way when configuring the ODBC driver.

(cherry picked from commit 451c341e0193b542409e8891ec2a31e62529a5e7)
2020-02-26 13:39:54 +01:00
Alan Woodward a76ec765e5 Ensure that percolator sorting also works (#52758)
Commit #52748 fixed a bug where percolate queries wrapped in a constant score
could report incorrect matches. This commit adds a test to check that it also fixes
the case where a percolate query is sorted by something other than score.

Closes #52618
2020-02-26 10:52:04 +00:00
István Zoltán Szabó f57422bbfd [DOCS] Adds cat data frame analytics API (#52764)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-02-26 11:10:42 +01:00
Jim Ferenczi a73ad248e8
Fix backport of #46731 (#52744)
This change fixes the incomplete backport of #46731 in 7.x (as of 7.5).
We now check if `max_children` is set on the top level nested sort and fails with an
exception if it's not the case.

Relates #46731
Closes #52202
2020-02-26 10:46:51 +01:00
David Kyle 37be695d5c
[ML] Handle failed datafeed in MlDistributedFailureIT (#52631) (#52789) 2020-02-26 08:18:37 +00:00
Lisa Cawley 05f1cd74a6 [DOCS] Fixes monitoring links (#52790) 2020-02-25 18:08:23 -08:00
Florian Kelbert 85e5e56510 [DOCS] Fixes missing colon (#52797) 2020-02-25 16:51:18 -08:00
Sachin Frayne d3c0a2f013 Improve the error message when loading text fielddata. (#52753)
Emphasize keyword over fielddata as the preferred way to use String fields for aggregations or sorting.
2020-02-25 15:45:44 -08:00
Lee Hinman 662f21fcea Remove TODO in MaxAgeCondition serialization (#52794)
* Remove TODO in MaxAgeCondition serialization

This removes the TODO with a message for any future readers regarding the code in question.

Resolves #52505
2020-02-25 15:47:36 -07:00
Tim Brooks 6669e53f08
Do not lock on reads of XPackLicenseState (#52492)
XPackLicenseState reads to necessary to validate a number of cluster
operations. This reads occasionally occur on transport threads which
should not be blocked. Currently we sychronize when reading. However,
this is unecessary as only a single piece of state is updateable. This
commit makes this state volatile and removes the locking.
2020-02-25 15:38:35 -07:00
Tim Brooks c8ef9649e2
Force execution of finish shard bulk request (#51957) (#52484)
Currently the shard bulk request can be rejected by the write threadpool
after a mapping update. This introduces a scenario where the mapping
listener thread will attempt to finish the request and fsync. This
thread can potentially be a transport thread. This commit fixes this
issue by forcing the finish action to happen on the write threadpool.

Fixes #51904.
2020-02-25 14:37:11 -07:00
Nhat Nguyen 848d3bc153 Revert "Fix testKeepTranslogAfterGlobalCheckpoint"
This reverts commit a88d54eb2d.
2020-02-25 14:12:35 -05:00
Lisa Cawley 924f0bd243 [DOCS] Updates custom rules example (#52731) 2020-02-25 09:32:52 -08:00
Mark Vieira 528e77cd5e
Update Docker on linux exclusion list (#52736)
(cherry picked from commit 5cfbf4dac11225d9f48fc25919e29713dd53a279)
2020-02-25 09:24:38 -08:00
Nhat Nguyen a88d54eb2d Fix testKeepTranslogAfterGlobalCheckpoint
Read the last synced global checkpoint after flushing as
we might advance it during committing.

CI: https://gradle-enterprise.elastic.co/s/7o6qengg4gva2
2020-02-25 11:49:24 -05:00
Andrei Stefan 51c6aefa55
SQL: Use calendar_interval of 1d for HISTOGRAMs with 1 DAY intervals (#52749) (#52771)
(cherry picked from commit 556f5fa33be88570c4f8550cb8f784323d26a707)
2020-02-25 18:44:02 +02:00
Costin Leau a8911802d3 EQL: transform query AST into queryDSL (#52432)
(cherry picked from commit 94cef29df259319dfe2a3bf92d3f1a42d7e45781)
2020-02-25 17:53:59 +02:00
Alan Woodward 638f3e4183 Use ByteBuffersDirectory rather than RAMDirectory (#52768)
Lucene's RAMDirectory has been deprecated. This commit replaces all uses of
RAMDirectory in elasticsearch with the newer ByteBuffersDirectory. Most uses
are in tests, but the percolator and painless executor may get some small speedups.
2020-02-25 15:46:35 +00:00
Nik Everett 02b23c37d1 Another test fix
Another attempt to fix a test that fails rarely and randomly. This time
try locking the query to just a single index.
2020-02-25 10:22:12 -05:00
Aleksandr Maus a6f5b4bb78
Unmute EqlActionIT (#52757)
Related to https://github.com/elastic/elasticsearch/issues/52737
2020-02-25 10:22:07 -05:00
Pius 563f033511 Update ilm-settings.asciidoc (#51577) 2020-02-25 10:18:55 -05:00
Henning Andersen 3ad1783a41 Delete by query test on low free disk block (#52759)
The block setup by the test could be released by the nodes cluster info
thread before the disk threshold decider was disabled, now disable
decider first.
2020-02-25 15:54:30 +01:00
bellengao d2db16e046 [DOCS] Correct policy name in ILM docs example (#52354)
Updates an example snippet to use a consistent policy name.
2020-02-25 09:36:22 -05:00
David Pilato 6c6ab8fa47 [DOS] Fix typo in CSV processor docs (#52649)
Corrects an example array in a snippet of the CSV processor docs.
2020-02-25 08:48:50 -05:00
bellengao 49f37989c4 [DOCS] Fix typo in ingest node docs (#52671) 2020-02-25 07:57:52 -05:00
David Roberts cf122d13b8 [ML] Use event.timezone in file_structure_finder ingest pipeline (#52720)
This is because beat.timezone was renamed to event.timezone in
elastic/beats#9458
2020-02-25 12:33:53 +00:00
James Rodewig 9b05f6a668 [DOCS] Add admonition for app using cat APIs (#52727)
Adds an explicit "important" admonition discouraging apps from using
cat APIs.

cat APIs are intended for human consumption via the command line or
Kibana console only. They are not intended for consumption by
applications.
2020-02-25 07:20:33 -05:00
Aleksandr Maus b2cb38ccf5
EQL: Expand verification tests (#52664) (#52725)
* EQL: Expand verification tests (#52664)

Expand verification tests
Fix some error messaging consistency in EqlParser

Related to https://github.com/elastic/elasticsearch/issues/51873

* Adjust for 7.x compatibility
2020-02-25 07:19:33 -05:00
Alan Woodward 18663b0a85 Don't index ranges including NOW in percolator (#52748)
Currently, date ranges queries using NOW-based date math are rewritten to
MatchAllDocs queries when being preprocessed for the percolator. However,
since we added the verification step, this can result in incorrect matches when
percolator queries are run without scores. This commit changes things to instead
wrap date queries that use NOW with a new DateRangeIncludingNowQuery.
This is a simple wrapper query that returns its delegate at rewrite time, but it can
be detected by the percolator QueryAnalyzer and be dealt with accordingly.

This also allows us to remove a method on QueryRewriteContext, and push all
logic relating to NOW-based ranges into the DateFieldMapper.

Fixes #52617
2020-02-25 12:18:16 +00:00
James Rodewig 1a14ae4e1b [DOCS] Document `include_in_*` nested mapping parms (#52648)
Adds documentation for the `include_in_parent` and `include_in_root`
mapping parameters for the `nested` mapping datatype.
2020-02-25 07:13:49 -05:00
Ioannis Kakavas e2aa5bc174
Adjust file realm docs (#52471) (#52745)
The existing wording in the file realm docs proved confusing
for users as it seemed to indicate that it should _only_ be
used as a fallback/recovery realm and that it is not a
first class realm.

This change attempts to clarify this and point out that recovery
is _a_ use case for the file realm but not the only intended one.
2020-02-25 11:59:13 +02:00
Adrien Grand 5f81906fcf Discourage from opting in for the `niofs` store. (#52638)
Indices open with the `niofs` store type load much more data on-heap than
indices open with the `mmapfs` store type. This limitation is now documented
and examples have been updated to show how to update settings to use the
`mmapfs` store type rather than `niofs`.
2020-02-25 08:54:11 +01:00
Adrien Grand 9b0ddc1c03 Clarify the resiliency trade-off of disabling replicas to speed up indexing. (#52714)
We should be more explicit about the downsides of disabling replicas and
explain that users should be ready to re-do the entire load in case of
issues mid-way.
2020-02-25 08:54:10 +01:00
Adrien Grand 5ce66b8b3c Document how CCR may be used to speed up indexing. (#52717)
One architecture that we have recommended to several users to speed up
indexing involved using CCR to prevent searching from stealing resources
from indexing.
2020-02-25 08:54:10 +01:00
Bob Blank 28d4b71947
Clarified http.max_content_length description (#52329)
Adding "greater than" based on discussion with @jasontedor for clarity.
2020-02-24 21:01:14 -05:00
Mark Vieira ac654692d4
Upgrade to Gradle 6.2.1 (#52732) 2020-02-24 16:34:53 -08:00