Commit Graph

42383 Commits

Author SHA1 Message Date
Jim Ferenczi 999f8f5850 Adapt Lucene BWC version
Bump the Lucene version used by ES 6.6 now that the 6.x branch is upgraded to Lucene 7.6.
2018-11-06 12:15:33 +01:00
Costin Leau 75e9a639ee
SQL: Introduce Coalesce function (#35253)
Add Coalesce conditional for replacing null values

Fix #35060
2018-11-06 13:12:24 +02:00
Nick Knize a5e1f4d3a2 Upgrade to lucene-8.0.0-snapshot-31d7dfe6b1 (#35224) 2018-11-06 11:55:23 +01:00
Alan Woodward 9f4b93fd5e
Fix failing ICU tests (#35207)
Fixes #35173
2018-11-06 09:02:40 +00:00
Simon Willnauer 833e0f8ecf
Prevent throttled indices to be searched through wildcards by default (#34354)
Today if a wildcard, date-math expression or alias expands/resolves
to an index that is search-throttled we still search it. This is likely
not the desired behavior since it can unexpectedly slow down searches
significantly.

This change adds a new indices option that allows `search`, `count`
and `msearch` to ignore throttled indices by default. Users can
force expansion to throttled indices by using `ignore_throttled=true`
on the rest request to expand also to throttled indices.

Relates to #34352
2018-11-06 09:45:30 +01:00
Martijn van Groningen 46c238d792
[CCR] Improve error when operations are missing (#35179)
Improve error when operations are missing
2018-11-06 08:42:47 +01:00
Martijn van Groningen cac67f8bcc
[CCR] Add extra validation to unfollow api (#35245)
Validate whether the follow index actually exists and
whether the follow index actually has custom ccr metadata.
2018-11-06 08:00:34 +01:00
Jeff Soloshy 14c8a483d5 [Docs] Minor formatting and wording fixes (#35278) 2018-11-06 07:52:13 +01:00
Tim Vernum 7cd27d1a96 [DOCS] Fix missing float in breaking changes 2018-11-06 16:59:11 +11:00
Tim Vernum 3776de5f20
HLRC: Add InvalidateToken security API (#35114)
This change adds the Invalidate Token API
(DELETE /_xpack/security/oauth2/token) to the Elasticsearch
High Level Rest Client.

Relates: #29827
2018-11-06 15:26:12 +11:00
Tim Vernum 574ec6686e
Include realm type in Security Realm setting keys (#30241)
This moves all Realm settings to an Affix definition.
However, because different realm types define different settings
(potentially conflicting settings) this requires that the realm type
become part of the setting key.

Thus, we now need to define realm settings as:

    xpack.security.authc.realms:
      file.file1:
        order: 0

      native.native1:
        order: 1

- This is a breaking change to realm config
- This is also a breaking change to custom security realms (SecurityExtension)
2018-11-06 14:56:50 +11:00
Shaunak Kashyap 00e66bab36
[Elasticsearch Monitoring] Collect only display_name (for now) (#35265)
This is a forward port of some of the changes made in #8445, specifically the change mentioned in https://github.com/elastic/elasticsearch/pull/34023#issuecomment-433212636.

Currently, in master, the `cluster_stats` collector collects _all_ cluster metadata and indexes it into `.monitoring-es-*`. However, per the discussion linked to above, we decided to collect _only_ the `display_name` cluster metadata setting for now. This PR makes this change.
2018-11-05 16:27:05 -08:00
lcawl 8280596f16 [DOCS] Fixes formatting error in auditing settings 2018-11-05 15:30:35 -08:00
Jake Landis 9150a93b1c
ingest: dot_expander_processor prevent null add/append to source document (#35106)
* don't allow null values to be added or appended to the 
source document if the field does not exist.
2018-11-05 17:16:42 -06:00
Marios Trivyzas 9ac7af6b3e
SQL: Introduce NotEquals node to simplify expressions (#35234)
Add NotEquals node in parser to simplify expressions so that <value1> != <value2> is
no longer translated internally to NOT(<value1> = <value2>)

Closes: #35210
Fixes: #35233
2018-11-05 22:23:07 +01:00
Nik Everett f72ef9b5fd
Build: Pull "skip assemble on qa" to common build (#35214)
Pull all of the logic that we use to skip the `assemble` and
`dependenciesInfo` tasks on `qa` projects into one spot in our root
build file.
2018-11-05 16:16:00 -05:00
Armin Braun 216c761a5d
MINOR: Remove Dead Code in Routing (#35074)
* MINOR: Remove Dead Code in Routing
2018-11-05 20:40:27 +01:00
Lee Hinman 3ee004c7b9
Add dedicated step for checking shrink allocation status (#35161)
This adds a new step for checking whether an index is allocated correctly based
on the rules added prior to running the shrink step. It also fixes a bug where
for shrink we are not allowed to have the shards relocating for the shrink step.

This also allows us to simplify AllocationRoutedStep and provide better
feedback in the step info for why either the allocation or the shrink checks
have failed.

Resolves #34938
2018-11-05 12:02:59 -07:00
Alexander Zhukov 842809ef37 Doc: Drop an extra 'a' in snapshots.asciidoc (#35251) 2018-11-05 13:31:35 -05:00
Nik Everett 59a43180a5 Reindex: Clean up docs around multi-index
We have an example in `reindex`'s docs about copying from many indices
at once. It doesn't work at the moment because we only allow a single
type per index. We didn't notice it in the docs tests because those
tests didn't copy any documents. This change:
1. Adds documents to the docs tests to fully exercise the snippet.
2. Fixes the example by moving all copied documents to the same type.
3. Moves the note about id collisions and expands on it because it is
even more likely than before.

Closes #35150
2018-11-05 12:52:11 -05:00
Marios Trivyzas fd82813660
SQL: [tests] Fix select csv spec and enable it (#35239)
Also, replace `||` and `&&` with `OR` and `AND` as the former are not
SQL standard operators.
2018-11-05 18:45:58 +01:00
Jake Landis c2766b65cf
ingest: raise visibility of ingest plugin documentation (#35048)
* move the set security user processor to the main documentation
* link to plugin processors

part of #33188
2018-11-05 11:44:10 -06:00
Armin Braun a95f05780d
NETWORK: Remove Dead Code from Netty4CorsConfig (#34947)
* Same as #34324 for the Netty transport, the `isNullOriginAllowed` setting is always false
2018-11-05 17:53:02 +01:00
Gordon Brown 0fbb8a16bc
Skip Rollover step if next index already exists (#35168)
If the Rollover step would fail due to the next index in sequence
already existing, just skip to the next step instead of going to the
Error step.

This prevents spurious `ResourceAlreadyExistsException`s created by
simultaneous RolloverStep executions from causing ILM to error out
unnecessarily.
2018-11-05 09:20:43 -07:00
Yannick Welsch 4f35eea8fe [TEST] Fix testConcurrentTermIncreaseOnReplicaShard
This test has a bug that got introduced during the refactoring of #32442. With 2 concurrent term increments,
we can only assert under the operation permit that we are in the correct operation term, not that there is
not already another term bump pending.

Closes #34862
2018-11-05 16:18:20 +01:00
Lee Hinman 3473217563
Remove Joda usage from ILM (#35220)
This commit removes the Joda time usage from ILM and the HLRC components of ILM.
It also fixes an issue where using the `?human=true` flag could have caused the
parser not to work. These millisecond fields now follow the standard we use
elsewhere in the code, with additional fields added iff the `human` flag is
specified.

This is a breaking change for ILM, but since ILM has not yet been released, no
compatibility shim is needed.
2018-11-05 08:17:15 -07:00
Benjamin Trent a4442dacd7
HLRC: Add ML API PUT filter (#35175) 2018-11-05 08:56:53 -06:00
Vladimir Dolzhenko 9d28a104e1
Discovery EC2: Utilize Amazon SDK to resolve EC2 metadata server (#35246)
* Discovery EC2: Utilize Amazon SDK to resolve EC2 metadata server

Closes #35141
2018-11-05 15:02:09 +01:00
Christoph Büscher 02043a2260
[Tests] Fix rare edge case in SimpleQueryStringBuilderTests (#35201)
If the random query string is "now" by accident _and_ we are also not setting
some field names to use explicitely, then we can hit the "mapped_date" field
from default test setup. This correctly leads to the query being was marked as
not cacheable, but we assume and check so later. This change fixes this rare
edge case by making sure we don't hit the "date" field in this rare cases.

Closes #35183
2018-11-05 13:31:13 +01:00
Alexander Reelsen 409050e8de
Refactor: Remove settings from transport action CTOR (#35208)
As settings are not used in the transport action constructor, this
removes the passing of the settings in all the transport actions.
2018-11-05 13:08:18 +01:00
Martijn van Groningen ddda2d419c
[CCR] Change max_read_request_size default (#35247)
This changes the max_read_request_size default from unlimited to 32MB.
2018-11-05 12:51:42 +01:00
David Kyle 85f8458f06 [ML] Add comment describing test behaviour 2018-11-05 11:21:59 +00:00
Mayya Sharipova a2c22ad7ae
Small corrections to HLRC doc for _termvectors (#35221)
Relates to #33447
2018-11-04 18:29:07 -05:00
Boaz Leskes 28078642b3
Engine.newChangesSnapshot may cause unneeded refreshes if called concurrently (#35169)
When the engine is asked for historical operations, we check if some of the requested operations
are not yet refreshed and if so we refresh before returning the operations. The refresh check is
based on capturing the local checkpoint before each refresh and comparing that value to the one
requested when `newChangesSnapshot` was called. If the requested range is above the captured
local checkpoint we issue a refresh.

This can currently cause unneeded extra refreshes if the method is called concurrently which may cause unwanted degradation in indexing performance. This is especially relevant for CCR where we always ask for a range below the global checkpoint. That range is guaranteed to be below the local
checkpoint of the shard and one refresh is enough to serve multiple changes requests.

This commit fixes this by introducing a dedicated mutex to make sure the test for whether a refresh
is needed actually wait for concurrents for concurrent refreshes that were caused by another
change refresh. 

Note that this is not a big change in semantics as refreshes are serialized by lucene anyway. I also
opted not to keep the synchronization to the changes snapshot request only even if in theory we
can apply it to all refreshes, not matter where they come from.
2018-11-04 13:43:33 +01:00
Nhat Nguyen 54e1231ebd
CCR/TEST: Limit indexing docs in FollowerFailOverIT (#35228)
The suite FollowerFailOverIT is failing because some documents are not
replicated to the follower. Maybe the FollowTask is not working as
expected or the background indexers eat all resources while the follower
cluster is trying to reform after a failover; then CI is not fast enough
to replicate all the indexed docs within 60 seconds (sometimes I see 80k
docs on the leader).

This commit limits the number of documents to be indexed into the leader
index by the background threads so that we can eliminate the latter
case. This change also replaces a docCount assertion with a docIds
assertion so we can have more information if these tests fail again.

Relates #33337
2018-11-03 10:00:54 -04:00
Nhat Nguyen 855ab3fa1e
Add equals/hashCode to SeqNoStats (#35223)
This commit adds equals/hashCode to SeqNoStats so we can verify it wholly in tests.
2018-11-02 21:31:36 -04:00
Jack Conradson 44f08717ba
[Scripting] Make Max Script Length Setting Dynamic (#35184)
This changes the current script.max_size_in_bytes to be dynamic so it can be 
set through the cluster settings API. This setting is also applied to inline scripts 
in the compile method of ScriptService to prevent excessively long inline 
scripts from being compiled. The script length limit is removed from Painless as 
this is no longer necessary with the protection in compile.
2018-11-02 16:07:54 -07:00
Tim Brooks 0166388d74
Use single netty event loop group for transports (#35181)
Currently we create a new netty event loop group for client connections
and all server profiles. Each new group creates new threads for io
processing. This means 2 * num of processors new threads for each group.
A single group should be able to handle all io processing (for the
transports). This also brings the netty module inline with what we do
for nio.

Additionally, this PR renames the worker threads to be the same for
netty and nio.
2018-11-02 16:31:19 -06:00
Lisa Cawley 3c36ba1f5e
[DOCS] Move CCR overview pages (#34918) 2018-11-02 14:23:13 -07:00
Jay Modi ec1c2c4a76
Remove AbstractComponent usage in x-pack core (#35187)
This change removes the use of AbstractComponent in the security
module. The classes now declare their own loggers.

Relates #34488
2018-11-02 14:30:43 -06:00
Gordon Brown b3da3eae08
[ILM] Fix race condition in test (#35143)
Previously, testRunStateChangePolicyWithNextStep asserted that the
ClusterState before and after running the steps were equal. The test
only passed due to a race condition: The latch would be triggered by the
step execution, but the cluster state update thread would continue
running before committing the change to the cluster state. This allowed
the test to read the old cluster state and pass the equality check about
99.99% of the time.

The test now waits for the new cluster state to be committed before
checking that it is _not_ equal to the old cluster state.
2018-11-02 11:09:48 -06:00
Nhat Nguyen d6e44129b1
TEST: Only check max_seq_no_of_updates when rollback (#35170)
Currently, we assume that rollback always happens in the test
testRestoreLocalHistoryFromTranslogOnPromotion. However, if the global
checkpoint equals max_seq_no, we won't rollback. This causes the
max_seq_no_of_updates assertion failed because max_seq_no_of_updates
won't be advanced to the global checkpoint. With this commit, we assert
max_seq_no_of_updates in two different paths.
2018-11-02 12:27:48 -04:00
Gordon Brown 9b3d581339 [DOCS] Remove link to nonexistent ILM API 2018-11-02 09:38:40 -06:00
Jason Tedor 1e241190eb
Disable assemble task from ILM qa projects
This commit disables the assemble tasks from all ILM qa projects. These
projects do not have an assemble task to execute.
2018-11-02 11:16:34 -04:00
David Kyle 43cff56aec Mute FollowerFailOverIT.testFollowIndexAndCloseNode
Issue #33337
2018-11-02 15:13:09 +00:00
Alan Woodward e2af849f70
Move ObjectPath and XContentUtils to libs/x-content (#34803)
These are generally useful utility classes that do not need to live in the Watcher code
2018-11-02 15:12:09 +00:00
Jay Modi 6b08d5fc89
Remove use of AbstractComponent in security (#35186)
This change removes the use of AbstractComponent in the security
module. The classes now declare their own loggers.

Relates #34488
2018-11-02 08:33:26 -06:00
Milan Mrdjen 34677b9c83 HLRC: Add document _count API (#34267)
Add `count()` api method, `CountRequest` and `CountResponse` classes to HLRC. Code in server module is unchanged.

Relates to #27205
2018-11-02 08:21:19 -05:00
Vladimir Dolzhenko 12072b200e
Run code snippets in ReindexDocumentationIT (#35165)
Closes #32093
2018-11-02 13:52:15 +01:00
Nhat Nguyen e753e12f61
Do not alloc full buffer for small change requests (#35158)
Today we always allocate a full buffer (1024 elements) in a
LuceneChangesSnapshot even though the requesting size is smaller.
With this change, we will use the requesting size as the buffer size if
it's smaller than the default batch size; otherwise uses the default
batch size.
2018-11-02 08:49:55 -04:00