Commit Graph

51649 Commits

Author SHA1 Message Date
Jake Landis 25ea6a74f0
[7.x] Validate REST specs against schema (#55117) (#55563)
A JSON schema was recently introduced for the REST API specification. #54252
This PR introduces a 3rd party validation tool to ensure that the
REST specification conforms to the schema.

The task is applied to the 3 projects that contain REST API specifications.
The plugin wires this task into the precommit commit task, and should be
considered as part of the public API for the build tools for any plugin
developer to contribute their plugin's specification.

An ignore parameter has been introduced for the task to allow specific
file to be ignored from the validation. The ignored files in this PR
will soon get issues logged and a link so they can be fixed.

Closes #54314
2020-04-22 14:14:03 -05:00
Albert Zaharovits 82ed0ab420
Update the audit logfile list of system users (#55578)
Out of the box "access granted" audit events are not logged
for system users. The list of system users was stale and included
only the _system and _xpack users. This commit expands this list
with _xpack_security and _async_search, effectively reducing the
auditing noise by not logging the audit events of these system
users out of the box.

Closes #37924
2020-04-22 21:59:31 +03:00
Tal Levy c370b83bd7
Fix locale lowercase test issue in GenerateSnapshotNameStepTests (#55597) (#55605)
The testPerformAction test has been failing periodically due to
how Hamcrest's containsStringIgnoringCase does not lowercase using
the same Locale set in the test infrastructure.

This commit falls back to explicitly lowercasing using the root
locale
2020-04-22 11:29:57 -07:00
Tal Levy f27ce69f0c
[backport] Add geo_bounds aggregation support for geo_shape (#55328) (#55600)
This commit adds a new GeoShapeBoundsAggregator to the spatial plugin and registers it with the GeoShapeValuesSourceType. This enables geo_bounds aggregations on geo_shape fields
2020-04-22 11:29:35 -07:00
Lisa Cawley 314ca78e31
[7.x][DOCS] Update example and nesting in get data frame analytics job stats API (#55612) 2020-04-22 10:58:26 -07:00
Przemko Robakowski 6e1b958069
Fix updating Index Templates V2 (#55556) (#55610)
This change fixes problem with updating Index Templates V2.
Validatation added in #54933 didn't filter list of conflicting templates correctly so
new template was always clashing with itself unless patterns were not changed completely.
2020-04-22 19:42:32 +02:00
Stuart Tettemer 41748f02a5
Test: don't modify defaultConfig on upgrade (#55560) (#55599)
Backport: 58ec9c3
2020-04-22 11:07:27 -06:00
Igor Motov 3504755f44
Add InstantiatingObjectParser (#55483) (#55604)
Introduces InstantiatingObjectParser which is similar to the
ConstructingObjectParser, but instantiates the object using its constructor
instead of a builder function.

Closes #52499
2020-04-22 12:28:52 -04:00
Tal Levy 0844455505
Add geo_shape mapper supporting doc-values in Spatial Plugin (#55037) (#55500)
After #53562, the `geo_shape` field mapper is registered within
a module. This opens the door for introducing a new `geo_shape`
field mapper into the Spatial Plugin that has doc-values support.

This is very much an extension of server's GeoShapeFieldMapper,
but with the addition of the doc values implementation.
2020-04-22 08:12:54 -07:00
James Rodewig 8d05d7dace
[DOCS] Add collapsible sections to 7.x breaking changes (#55334)
Adds collapsible sections and new format to the 7.x breaking changes.

Relates to #53229.
2020-04-22 10:56:38 -04:00
James Rodewig 6f9513915d
[DOCS] Add 'how to' doc about avoiding oversharding (#55480)
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
2020-04-22 10:44:16 -04:00
James Rodewig 414f9c98f3
[DOCS] Document missing bulk API response parameters (#55414)
Documents several parameters missing from the bulk API's response body
docs. Also moves several response-related chunks of text to the response
body section.

Relates to #55237
2020-04-22 09:48:03 -04:00
Dimitris Athanasiou 50a5afed15
[7.x][ML] Prepare parsing phase_progress from DFA process (#55580) (#55587)
Data frame analytics process currently reports progress as
an integer `progress_percent`. We parse that and report it
from the _stats API as the progress of the `analyzing` phase.
However, we want to allow the DFA process to report progress
for more than one phase. This commit prepares for this by
parsing `phase_progress` from the process, an object that
contains the `phase` name plus the `progress_percent` for that
phase.

Backport of #55580
2020-04-22 16:38:32 +03:00
Benjamin Trent 7c81cd7833
[ML] explicitly disallow partial results in datafeed extractors (#55537) (#55585)
Instead of doing our own checks against REST status, shard counts, and shard failures, this commit changes all our extractor search requests to set `.setAllowPartialSearchResults(false)`.

- Scrolls are automatically cleared when a search failure occurs with `.setAllowPartialSearchResults(false)` set.
- Code error handling is simplified

closes https://github.com/elastic/elasticsearch/issues/40793
2020-04-22 09:07:44 -04:00
David Roberts 810caf5ffe
[ML] Test that audit message is written when closing unassigned job (#55582)
Issue #55521 suggested that audit messages were not written when
closing an unassigned job.  This is not the case, but we didn't
have a test to prove it.

Backport of #55571
2020-04-22 13:23:43 +01:00
Armin Braun 250a51bca1
Fix TransportAddVotingConfigExclusionsActionTests Leaking CS Observers (#55549) (#55584)
There is no guarantee the observer and subsequent CS update will execute
before we move on to the next test here and we ahve to wait for the observer + CS update cycle to complete
before moving on to the next test.

closes #55481
2020-04-22 13:49:24 +02:00
David Roberts 2dc5586afe
[ML] Add effective max model memory limit to ML info (#55581)
The ML info endpoint returns the max_model_memory_limit setting
if one is configured.  However, it is still possible to create
a job that cannot run anywhere in the current cluster because
no node in the cluster has enough memory to accommodate it.

This change adds an extra piece of information,
limits.effective_max_model_memory_limit, to the ML info
response that returns the biggest model memory limit that could
be run in the current cluster assuming no other jobs were
running.

The idea is that the ML UI will be able to warn users who try to
create jobs with higher model memory limits that their jobs will
not be able to start unless they add a bigger ML node to their
cluster.

Backport of #55529
2020-04-22 12:28:50 +01:00
David Roberts da5aeb8be7
[ML] Return assigned node in start/open job/datafeed response (#55570)
Adds a "node" field to the response from the following endpoints:

1. Open anomaly detection job
2. Start datafeed
3. Start data frame analytics job

If the job or datafeed is assigned to a node immediately then
this field will return the ID of that node.

In the case where a job or datafeed is opened or started lazily
the node field will contain an empty string.  Clients that want
to test whether a job or datafeed was opened or started lazily
can therefore check for this.

Backport of #55473
2020-04-22 12:06:53 +01:00
David Kyle e99ef3542c Mute ModelLoadingServiceTests::testMaxCachedLimitReached 2020-04-22 11:53:07 +01:00
Fernando Briano 71672ea33d
Add skip arbitrary_key to nodes.reload_secure_settings YAML test (#55540) 2020-04-22 09:48:33 +01:00
István Zoltán Szabó 0ce3406033 [DOCS] Provides further details on aggregations in datafeeds (#55462)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-04-22 08:54:52 +02:00
Tim Vernum 8b566aea47
Fix use of password protected PKCS#8 keys for SSL (#55567)
PEMUtils would incorrectly fill the encryption password with zeros
(the '\0' character) after decrypting a PKCS#8 key.

Since PEMUtils did not take ownership of this password it should not
zero it out because it does not know whether the caller will use that
password array again. This is actually what PEMKeyConfig does - it
uses the key encryption password as the password for the ephemeral
keystore that it creates in order to build a KeyManager.

Backport of: #55457
2020-04-22 16:38:51 +10:00
Yang Wang 32e46bf552
Fix certutil http for empty password with JDK 11 and lower (#55437) (#55565)
Fix elasticseaerch-certutil http command so that it correctly accepts empty keystore password with JDK version 11 and lower.
2020-04-22 15:03:10 +10:00
James Rodewig 777ffd5801
[DOCS] Add bulk API example with failures (#55412)
Adds an example for bulk API requests that include failures.
Also documents guidance on use the `filter_path` parameter
to narrow the bulk API response for errors.

Closes #55237
2020-04-21 16:22:23 -04:00
Lee Hinman 55ff146d9b
Guard adding the index.prefer_v2_templates settings for pre-7.8 nodes (#55546)
If some of the nodes are pre-7.8 nodes, they may not support the `index.prefer_v2_templates`
setting (since it exists only on 7.8+). This commit makes sure that we only add this setting to the
index's metadata if all of the nodes are 7.8+. Otherwise we get errors like:

```
[ WARN ][o.e.i.c.IndicesClusterStateService] [v7.7.0-3] [test-snapshot-index][2] marking and sending shard failed due to [failed to create index]
  java.lang.IllegalArgumentException: unknown setting [index.prefer_v2_templates] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:595) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:549) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:176) ~[elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:484) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:246) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
  	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) [elasticsearch-7.7.0-SNAPSHOT.jar:7.7.0-SNAPSHOT]
```

Relates to #55411
Relates to #53101
Resolves #55539
2020-04-21 12:57:16 -06:00
David Kyle 8e8c6b4aee
Fix accounting in ModelLoadingServiceTests (#55307) (#55547)
In the test after the first load event is is not known which models are cached as 
loading a later one will evict an earlier one and the order is not known.
The models could have been loaded 1 or 2 times not exactly twice
2020-04-21 19:25:06 +01:00
James Baiera 2a5f1f49a9
Add enrich metricset from 7.5 (#54791) (#55356)
Co-authored-by: Julien Guay <guay_j@yahoo.fr>
2020-04-21 12:39:08 -04:00
James Rodewig b9dfd12e7e
[DOCS] Remove 'Testing' chapter (#55270) (#55532)
Removes the 'Testing' chapter from the Elasticsearch Reference guide.

This chapter was originally written for so that users using the Java HLRC client could
use the same test classes when testing Elasticsearch in their own applications.
However, this is no longer the case or recommended.

Closes #55257.
2020-04-21 10:29:58 -04:00
Armin Braun db7eb8e8ff
Remove Redundant CS Update on Snapshot Finalization (#55276) (#55528)
This change folds the removal of the in-progress snapshot entry
into setting the safe repository generation. Outside of removing
an unnecessary cluster state update, this also has the advantage
of removing a somewhat inconsistent cluster state where the safe
repository generation points at `RepositoryData` that contains a
finished snapshot while it is still in-progress in the cluster
state, making it easier to reason about the state machine of
upcoming concurrent snapshot operations.
2020-04-21 15:33:17 +02:00
Dan Hermann 402b6b1715
Identify backing indices for data streams 2020-04-21 07:43:10 -05:00
Christoph Büscher 1caa2f0515 [Tests] Fix expectations around shouldPreFilterSearchShards (#55522)
In the documentation of `pre_filter_shard_size` we state that the pre-filter
phase will be executed if the request targets more than 128 shards, yet in the
current test randomization we check that the
TransportSearchAction.shouldPreFilterSearchShards is true already for 127
targeted shards. This should be raised to 129 instead.

Closes #55514
2020-04-21 14:28:04 +02:00
David Turner be60d50452 Allow searching of snapshot taken while indexing (#55511)
Today a read-only engine requires a complete history of operations, in the
sense that its local checkpoint must equal its maximum sequence number. This is
a valid check for read-only engines that were obtained by closing an index
since closing an index waits for all in-flight operations to complete. However
a snapshot may not have this property if it was taken while indexing was
ongoing, but that's ok.

This commit weakens the check for a complete history to exclude the case of a
searchable snapshot.

Relates #50999
2020-04-21 13:21:38 +01:00
Paul Sanwald 0f7917b94b
add release notes for 7.5.2 (#51259)
Adds release notes for 7.5.2
2020-04-21 08:19:46 -04:00
Ignacio Vera e4c65b4388
mute test SSLReloadDuringStartupIntegTests.testReloadDuringStartup (#55525) 2020-04-21 14:13:13 +02:00
Jim Ferenczi 0b3bdfcc3e Fix expiration time in async search response (#55435)
This change ensures that we return the latest expiration time
when retrieving the response from the index.
This commit also fixes a bug that stops the garbage collection of saved responses if the async search index is deleted.
2020-04-21 14:04:29 +02:00
Andrei Dan f13ebc4d4b
[7.x] Validate global V2 templates don't set index.hidden (#55010) (#55519)
Validate adding global V2 templates don't configure the index.hidden setting.
This also prevents updating the component template to add the index.hidden
setting if that component template is referenced by a global index template.

(cherry picked from commit 2e768981809887649f49d265d039f056985f7e6a)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-21 12:47:59 +01:00
Yannick Welsch ba39c261e8 Use streaming reads for GCS (#55506)
To read from GCS repositories we're currently using Google SDK's official BlobReadChannel,
which issues a new request every 2MB (default chunk size for BlobReadChannel) using range
requests, and fully downloads the chunk before exposing it to the returned InputStream. This
means that the SDK issues an awfully high number of requests to download large blobs.
Increasing the chunk size is not an option, as that will mean that an awfully high amount of
heap memory will be consumed by the download process.

The Google SDK does not provide the right abstractions for a streaming download. This PR
uses the lower-level primitives of the SDK to implement a streaming download, similar to what
S3's SDK does.

Also closes #55505
2020-04-21 13:22:26 +02:00
Przemysław Witek 59d377462f
Apply default timeout in StopDataFrameAnalyticsAction.Request (#55512) (#55517) 2020-04-21 13:05:48 +02:00
Ignacio Vera 9d05913c8a
mute test TransportSearchActionTests.testShouldPreFilterSearchShards (#55516) (#55518) 2020-04-21 12:31:12 +02:00
Ignacio Vera 4783f1894c
mute test testReadRangeBlobWithRetries (#55507) (#55508) 2020-04-21 10:59:35 +02:00
Nhat Nguyen c012897667 Fix CancellableTasksIT (#55198)
We need to ensure all ban parents are removed after each test;
otherwise, the subsequent tests can fail because of the leftover.

Closes #55106
2020-04-20 22:43:05 -04:00
Nhat Nguyen 49f6dbf8e8 Fix testRecoverLocallyUpToGlobalCheckpoint (#55189)
Peer recovery fails if the primary does not see the recovering replica 
in the replication group (when the cluster state update on the primary
is delayed). To verify the local recovery stats, we have to remember 
this value in the first try because the local recovery happens once, and
its stats is reset when the recovery fails.

Closes #54829
2020-04-20 22:43:05 -04:00
Nhat Nguyen 3cc4e0dd09 Retry follow task when remote connection queue full (#55314)
If more than 100 shard-follow tasks are trying to connect to the remote 
cluster, then some of them will abort with "connect listener queue is 
full". This is because we retry on ESRejectedExecutionException, but not
on RejectedExecutionException.
2020-04-20 22:43:05 -04:00
Nhat Nguyen 9e3b813b62 Ensure not to open directory reader on transport thread (#55419)
This change adds an assertion making sure that we won't accidentally
open directory readers using transport threads.
2020-04-20 22:43:05 -04:00
Julie Tibshirani a1ec45e7e6 Remove unused method SourceLookup#extractValue. 2020-04-20 18:34:56 -07:00
Jason Tedor 1553e7e620
Encapsulate systemd extender
The systemd extender is a scheduled execution that ensures we
repeatedly let systemd know during startup that we are still starting
up. We cancel this scheduled execution once the node has successfully
started up. This extender is wrapped in a set once, which we expose
directly. This commit addresses this by putting the extender behind a
getter, which hides the implementation detail that the extener is
wrapped in a set once. This cleans up some issues in tests, that
ensures we are not making assertions about the set once, but instead
about the extender.
2020-04-20 21:17:42 -04:00
Jason Tedor 80f18ad31a
Use set once for systemd extender (#55497)
When Elasticsearch is starting up, we schedule a thread to repeatedly
let systemd know that we are still in the process of starting up. Today
we use a non-final field for this. This commit changes this to be a set
once so we can mark the field as final, and get stronger guarantees when
reasoning about the state of execution here.
2020-04-20 21:15:04 -04:00
Stuart Tettemer 93a2e9b0f9
Test: MockScoreScript can be cacheable. (#55499)
Backport: 0ed1eb5
2020-04-20 17:09:58 -06:00
Mark Tozzi ff55d761f9
Use Decimal formatter for Numeric ValuesSourceTypes (#54366) (#55470)
Co-authored-by: Igor Motov <igor@motovs.org>
2020-04-20 17:05:38 -04:00
Benjamin Trent cabff65aec
[ML] Fixing inference stats race condition (#55163) (#55486)
`updateAndGet` could actually call the internal method more than once on contention.
If I read the JavaDocs, it says:
```* @param updateFunction a side-effect-free function```
So, it could be getting multiple updates on contention, thus having a race condition where stats are double counted.

To fix, I am going to use a `ReadWriteLock`. The `LongAdder` objects allows fast thread safe writes in high contention environments. These can be protected by the `ReadWriteLock::readLock`.

When stats are persisted, I need to call reset on all these adders. This is NOT thread safe if additions are taking place concurrently. So, I am going to protect with `ReadWriteLock::writeLock`.

This should prevent race conditions while allowing high (ish) throughput in the highly contention paths in inference.

I did some simple throughput tests and this change is not significantly slower and is simpler to grok (IMO).

closes  https://github.com/elastic/elasticsearch/issues/54786
2020-04-20 16:21:18 -04:00