Commit Graph

47214 Commits

Author SHA1 Message Date
David Roberts f617585dbd [ML] Improve CSV header row detection in find_file_structure (#45099)
When doing a fieldwise Levenshtein distance comparison
between CSV rows, this change ignores all fields that
have long values, not just the longest field.

This approach works better for CSV formats that have
multiple freeform text fields rather than just a single
"message" field.

Fixes #45047
2019-08-02 09:08:21 +01:00
Armin Braun 9450505d5b
Stop Passing Around REST Request in Multiple Spots (#44949) (#45109)
* Stop Passing Around REST Request in Multiple Spots

* Motivated by #44564
  * We are currently passing the REST request object around to a large number of places. This works fine since we simply copy the full request content before we handle the rest itself which is needlessly hard on GC and heap.
  * This PR removes a number of spots where the request is passed around needlessly. There are many more spots to optimize in follow-ups to this, but this one would already enable bypassing the request copying for some error paths in a follow up.
2019-08-02 07:31:38 +02:00
Tim Vernum e21d58541a
Improve errors when TLS files cannot be read (#45122)
This change improves the exception messages that are thrown when the
system cannot read TLS resources such as keystores, truststores,
certificates, keys or certificate-chains (CAs).

This change specifically handles:

- Files that do not exist
- Files that cannot be read due to file-system permissions
- Files that cannot be read due to the ES security-manager

Backport of: #44787
2019-08-02 12:29:43 +10:00
Tim Vernum 590777150f
Explicitly fail if a realm only exists in keystore (#45091)
There are no realms that can be configured exclusively with secure
settings. Every realm that supports secure settings also requires one
or more non-secure settings.
However, sometimes a node will be configured with entries in the
keystore for which there is nothing in elasticsearch.yml - this may be
because the realm we removed from the yml, but not deleted from the
keystore, or it could be because there was a typo in the realm name
which has accidentially orphaned the keystore entry.

In these cases the realm building would fail, but the error would not
always be clear or point to the root cause (orphaned keystore
entries). RealmSettings would act as though the realm existed, but
then fail because an incorrect combination of settings was provided.

This change causes realm building to fail early, with an explicit
message about incorrect keystore entries.

Backport of: #44471
2019-08-02 12:28:59 +10:00
Yogesh Gaikwad ae5c01e2d2 Do not use scroll when finding duplicate API key (#45026)
When we create API key we check if the API key with the name
already exists. It searches with scroll enabled and this causes
the request to fail when creating large number of API keys in
parallel as it hits the number of open scroll limit (default 500).
We do not need the search context to be created so this commit
removes the scroll parameter from the search request for duplicate
API key.
2019-08-02 10:16:48 +10:00
Dimitris Athanasiou 8a6675b994
[7.x][ML] Check dest index is empty when starting DF analytics (#45094) (#45112)
If one tries to start a DF analytics job that has already run,
the result will be that the task will fail after reindexing the
dest index from the source index. The results of the prior run
will be gone and the task state is not properly set to failed
with the failure reason.

This commit improves the behavior in this scenario. First, we
set the task state to `failed` in a set of failures that were
missed. Second, a validation is added that if the destination
index exists, it must be empty.
2019-08-02 00:19:48 +03:00
Mark Vieira c13285a382
Remove unnecessary plugin application and project configuration (#45100) 2019-08-01 14:18:24 -07:00
Jim Ferenczi 3f94e2ea43 Sparse role queries can throw an NPE (#45053)
Sparse role queries are executed differently than other queries in order
to account for the fact that most of the documents are filtered from search.
However this special execution does not set the scorer for the query so any
collector that needs to access the score of a document fails with an NPE.
This change fixed this bug by setting the scorer before collecting any hits
when intersecting the main query and the sparse role.
2019-08-01 20:21:53 +02:00
William Brafford 5f50da947a
Fix bug in the Settings#processSetting method (#45095)
The Settings#processSetting method is intended to take a setting map and add a
setting to it, adjusting the keys as it goes in case of "conflicts" where the
new setting implies an object where there is currently a string, or vice
versa. processSetting was failing in two cases: adding a setting two levels
under a string, and adding a setting two levels under a string and four levels
under a map. This commit fixes the bug and adds test coverage for the
previously faulty edge cases.

* fix issue #43791 about settings
* add unit test in testProcessSetting()
2019-08-01 13:27:08 -04:00
James Rodewig 138865a58e [DOCS] Reformat script score query (#45087) 2019-08-01 12:23:35 -04:00
Lisa Cawley 7f74790db2
[DOCS] Adds 7.3.0 ml-cpp release notes (#45019) 2019-08-01 09:12:33 -07:00
lcawl 4e00e119ed [DOCS] Fixes missing discrete attribute 2019-08-01 09:11:14 -07:00
Yannick Welsch 917510d3e4 Always use primary term of operation in InternalEngine (#45083)
We keep adding the current primary term to operations for which we do not assign a sequence
number. This does not make sense anymore as all operations which we care about have
sequence numbers now. The goal of this commit is to clean things up in InternalEngine and
reduce the complexity.
2019-08-01 17:30:00 +02:00
Armin Braun 48dc53f8d2
Make PathTrieIterator a Little more Memory Efficient (#44951) (#45070)
* There's no need to have the trie iterator hold another reference to the request object (which could be huge, see #44564)
* Also removed unused boolean field from trie node
2019-08-01 17:26:08 +02:00
James Rodewig 187f1e776b [DOCS] Offset clone index API headings to correct nav (#45097) 2019-08-01 11:22:41 -04:00
Nhat Nguyen 3a487379c3 Tighten no pending scheduled refresh check (#45025)
Previously, we use ThreadPoolStats to ensure that the scheduledRefresh
triggered by the internal refresh setting update is executed before we
index a new document. With that change (#40387), this test did not fail for 
the last 3 months. However, using ThreadPoolStats is not entirely watertight
as both "active" and "queue" count can be 0 in a very small interval
when ThreadPoolExecutor pulls a task from the queue but before marking
the corresponding worker as active (i.e., lock it).

Closes #39565
2019-08-01 09:06:22 -04:00
David Turner c088bafbbc Wait for events in waitForRelocation (#45074)
Adds a `waitForEvents(Priority.LANGUID)` to the cluster health request in
`ESIntegTestCase#waitForRelocation()` to deal with the case that this health
request returns successfully despite the fact that there is a pending reroute task which
will relocate another shard.

Relates #44433
Fixes #45003
2019-08-01 13:47:39 +01:00
Przemysław Witek 6c87845fc1
Persist DatafeedTimingStats with RefreshPolicy.NONE by default (#44940) (#45079) 2019-08-01 14:36:59 +02:00
Benjamin Trent 2fc4b76ba8
[ML][Data Frame] adding breaking changes to doc (#45051)
* [ML][Data Frame] adding breaking changes to doc

* [DOCS] Edits data frame transform breaking change

* Update migrate_7_3.asciidoc
2019-08-01 07:30:41 -05:00
David Turner 532ade7816 More logging for slow cluster state application (#45007)
Today the lag detector may remove nodes from the cluster if they fail to apply
a cluster state within a reasonable timeframe, but it is rather unclear from
the default logging that this has occurred and there is very little extra
information beyond the fact that the removed node was lagging. Moreover the
only forewarning that the lag detector might be invoked is a message indicating
that cluster state publication took unreasonably long, which does not contain
enough information to investigate the problem further.

This commit adds a good deal more detail to make the issues of slow nodes more
prominent:

- after 10 seconds (by default) we log an INFO message indicating that a
  publication is still waiting for responses from some nodes, including the
  identities of the problematic nodes.

- when the publication times out after 30 seconds (by default) we log a WARN
  message identifying the nodes that are still pending.

- the lag detector logs a more detailed warning when a fatally-lagging node is
  detected.

- if applying a cluster state takes too long then the cluster applier service
  logs a breakdown of all the tasks it ran as part of that process.
2019-08-01 13:20:46 +01:00
Hendrik Muhs b3be8f75f0 Fix version logic after 7.3 release (BWC) (#45077)
removes unreleased version 7.2.2 after release of 7.3.0 as it breaks the version verifier, add documentation that explains the logic
2019-08-01 12:43:23 +02:00
Christoph Büscher a669efd2a4
Remove left-over AwaitsFix in RateClusterStateIT (#45043)
Issues are closed and fixes in #42580 and #42430 seem to be merged to 7.x at
least.
2019-08-01 12:03:29 +02:00
Lisa Cawley 9e901c5f94
[DOCS] Edits 7.3 release highlights (#45066) 2019-07-31 15:49:27 -07:00
Tim Brooks fdc6c9853f
Do not write if connect incomplete (#44466)
Currently, we do not handle READ or WRITE events until the channel
connection process is complete. However, the external write queue path
allows a write to be attempted when the conneciton is not complete. This
commit closes the loophole and only queues write operations when the
connection process is not complete.
2019-07-31 14:30:14 -06:00
Tim Brooks aff66e3ac5
Add Cors integration tests (#44361)
This commit adds integration tests to ensure that the basic cors
functionality works for the netty and nio transports.
2019-07-31 14:24:23 -06:00
Lisa Cawley 78c254a9c3
[DOCS] Adds notable 7.3 breaking changes (#45057) 2019-07-31 11:47:10 -07:00
James Rodewig 3c4150cf72 [DOCS] Reformat rank feature query. Add relevance score section. (#44975) 2019-07-31 14:31:50 -04:00
James Rodewig 728b0cf9ff [DOCS] Update parameter format (#44703) 2019-07-31 14:18:55 -04:00
Flavio Pompermaier f1bab2fa89 [DOCS] Correct sum_other_doc_count value in terms agg example (#45028)
Closes issue #41902
2019-07-31 14:10:36 -04:00
Armin Braun 8d63bd1d1e
Cleanup Various Action- Listener and Runnable Usages (#42273) (#45052)
* Dry up code for creating simple `ActionRunnable` a little
* Shorten some other code around `ActionListener` usage, in particular
when wrapping it in a `TransportResponseListener`
2019-07-31 18:55:31 +02:00
Armin Braun ee663dc9ac
Reenable Parallel Restore Test on Windows (#45037) (#45050)
* As a result of #44096 this test shouldn't fail anymore on `master` and `7.4`+ so we should reenable it there
  * For older versions we won't backport that change so the tests should stay disabled there
* Closes #44671
2019-07-31 18:35:34 +02:00
Mayya Sharipova 0c68765088
Adds usage stats for vectors (#45023)
Example of usage:

_xpack/usage

"vectors": {
    "available": true,
    "enabled": true,
    "dense_vector_fields_count" : 1,
    "sparse_vector_fields_count" : 1,
    "dense_vector_dims_avg_count" : 100
}
Backport for #44512
2019-07-31 12:32:41 -04:00
Jack Conradson 5202d2624e Add several context examples for Painless date documentation (#44985) 2019-07-31 08:23:17 -07:00
Christoph Büscher 35291ae175
Remove muted AckIT and AckClusterUpdateSettingsIT (#45044)
Reading up on #33673 it looks like parts of these tests have been reworked and
there is no intention to fix the remains on 7.x, so I think we can remove the
entire test.
2019-07-31 17:17:21 +02:00
Luca Cavanna 8cc3c0dd93 Remove task null check in TransportAction (#45014)
The task that TaskManager#register returns cannot be null. The method
enforces that it is not null after calling request#createTask. It is
then needless to check for null in the listener later. Also, added the
call to the delegate listener in a finally block, just to make sure.
2019-07-31 17:16:41 +02:00
Christoph Büscher e85b53a955
Remove left-over AwaitsFix in DedicatedClusterSnapshotRestoreIT (#45042)
The issue mentioned (#38845) seems to have been closed with #38891 so the test
can be re-activated.
2019-07-31 17:15:41 +02:00
Zachary Tong 6ed2a6e9dc [DOCS] Add breaking change re: direct alias modification on followers
Relates: #43017
2019-07-31 11:11:56 -04:00
Armin Braun c7d7230524
Stop Recreating Wrapped Handlers in RestController (#44964) (#45040)
* We shouldn't be recreating wrapped REST handlers over and over for every request. We only use this hook in x-pack and the wrapper there does not have any per request state.
  This is inefficient and could lead to some very unexpected memory behavior
   => I made the logic create the wrapper on handler registration and adjusted the x-pack wrapper implementation to correctly forward the circuit breaker and content stream flags
2019-07-31 17:11:34 +02:00
Ioannis Kakavas 56da35b706 Indicate that some user APIs handle built-in users (#44857)
The Get Users API also returns users form the restricted realm or built-in users,
 as we call them in our docs. One can also change the passwords of built-in 
users with the Change Password API
2019-07-31 17:55:28 +03:00
Zachary Tong c25f3dd5d0
Introduce 7.3.1 version (#45046) 2019-07-31 10:53:55 -04:00
Christoph Büscher f6922bca2d
Unmute test that seems to be fixed (#44432)
Since #42509 is closed and the fix seems to have been backported to 7.x (#43539)
the test can be enabled again.
2019-07-31 16:33:21 +02:00
Andrey Ershov c27ac3d24c Unmute testClusterJoinDespiteOfPublishingIssues and testElectMasterWithLatestVersion (#38555)
See my comments for #37539 and #37685

(cherry picked from commit 038d4ab2940340eca942e32b54044f183b7804d9)
2019-07-31 14:55:02 +02:00
Daniel Mitterdorfer 5dd0e74e79 Clarify which circuit breaker settings are static (#44992)
Most of the circuit breaker settings are dynamically configurable.
However, `indices.breaker.total.use_real_memory` is not. With this
commit we add a clarifying note that this specific setting is static.

Closes #44974
2019-07-31 13:15:33 +02:00
Jesse Wright f19f2adbe6 [Docs] Fix typo in rank-eval.asciidoc (#44978) 2019-07-31 12:37:49 +02:00
Tim Vernum 3c17d4379d
Expand logging when SAML Audience condition fails (#45027)
A mismatched configuration between the IdP and SP will often result in
SAML authentication attempts failing because the audience condition is
not met (because the IdP and SP disagree about the correct form of the
SP's Entity ID).

Previously the error message in this case did not provide sufficient
information to resolve the issue because the IdP's expected audience
would be truncated if it exceeeded 32 characters. Since the error did
not provide both IDs in full, it was not possible to determine the
correct fix (in detail) based on the error alone.

This change expands the message that is included in the thrown
exception, and also adds additional logging of every failed audience
condition, with diagnostics of the match failure.

Backport of: #44334
2019-07-31 19:40:17 +10:00
David Roberts 5e3010a606 Use system context for looking up connected nodes (#43991)
When finding nodes in a connected cluster for cross cluster
search the requests to get cluster state on the connected
cluster should be made in the system context because
logically they are equivalent to checking a single detail
in the local cluster state and should not require that the
user who made the request that is using this method in its
implementation is authorized to view the entire cluster
state.

Fixes #43974
2019-07-31 09:09:56 +01:00
Lisa Cawley 899c62ad02 [DOCS] Remove xpack terminology from installation pages (#44973) 2019-07-30 15:29:59 -07:00
Igor Motov 1a1bb4707d Geo: move indexShape to AbstractGeometryFieldMapper.Indexer (#44979)
Move indexShape functionality into AbstractGeometryFieldMapper to make
it more unit testable.

Relates to #43644
2019-07-30 14:50:23 -04:00
Lisa Cawley e4b7ae211b [DOCS] Updates terms in machine learning get APIs (#44986) 2019-07-30 11:29:25 -07:00
Mayya Sharipova a154b73d99 Assure index ops are successful for SimpleNestedIT (#44815)
relates to #44486
2019-07-30 14:24:28 -04:00