Commit Graph

52754 Commits

Author SHA1 Message Date
Armin Braun 6dec2cf722
Fix SLM Tests Leaking Snapshot Operation (#59150) (#59155)
Fixed an issue #59082 introduced. We have to wait for no more operations
in all tests here not just the one we were waiting in already so that the cleanup
operation from the parent class can run without failure.
2020-07-07 17:19:06 +02:00
Rene Groeschke a896df53ac
Remove misc dependency related deprecation warnings (7.x backport) (#59122)
* Fix dependency related deprecations (#58892)
* Fix classpath setup for forbiddenapi usage
2020-07-07 17:10:31 +02:00
Nik Everett eb169ae226
Fix lookup support in adjacency matrix (backport of #59099) (#59108)
This request:
```
POST /_search
{
  "aggs": {
    "a": {
      "adjacency_matrix": {
        "filters": {
          "1": {
            "terms": { "t": { "index": "lookup", "id": "1", "path": "t" } }
          }
        }
      }
    }
  }
}
```

Would fail with a 500 error and a message like:
```
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason":"async actions are left after rewrite"
      }
    ]
  }
}
```

This fixes that by moving the query rewrite phase from a synchronous
call on the data nodes into the standard aggregation rewrite phase which
can properly handle the asynchronous actions.
2020-07-07 10:28:20 -04:00
Christoph Büscher 7c64a1bd7b Muting failing ApiKeyIntegTests 2020-07-07 16:02:59 +02:00
David Turner 8f4f844e6e Add docs for filesystem health checks (#59134)
Documents the feature and settings introduced in #52680.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-07-07 14:14:58 +01:00
James Rodewig 664b546771 [DOCS] Fix anchor syntax 2020-07-07 09:02:33 -04:00
David Turner 46c8d00852
Remove nodes with read-only filesystems (#52680) (#59138)
Today we do not allow a node to start if its filesystem is readonly, but
it is possible for a filesystem to become readonly while the node is
running. We don't currently have any infrastructure in place to make
sure that Elasticsearch behaves well if this happens. A node that cannot
write to disk may be poisonous to the rest of the cluster.

With this commit we periodically verify that nodes' filesystems are
writable. If a node fails these writability checks then it is removed
from the cluster and prevented from re-joining until the checks start
passing again.

Closes #45286

Co-authored-by: Bukhtawar Khan <bukhtawar7152@gmail.com>
2020-07-07 14:00:02 +01:00
James Rodewig a8220ad51e [DOCS] Fix anchor syntax 2020-07-07 08:57:20 -04:00
Yang Wang f84b76661d
Make test more robust for API key auth 429 (#59077) (#59136)
Adds error handling when filling up the queue of the crypto thread pool. Also reduce queue size of the crypto thread pool to 10 so that the queue can be cleared out in time.

Test testAuthenticationReturns429WhenThreadPoolIsSaturated has seen failure on CI when it tries to push 1000 tasks into the queue (setup phase). Since multiple tests share the same internal test cluster, it may be possible that there are lingering requests not fully cleared out from the queue. When it happens, we will not be able to push all 1000 tasks into the queue. But since what we need is just queue saturation, so as long as we can be sure that the queue is fully filled, it is safe to ignore rejection error and just move on.

A number of 1000 tasks also take some to clear out, which could cause the test suite to time out. This PR change the queue to 10 so the tests would have better chance to complete in time.
2020-07-07 22:27:10 +10:00
Rene Groeschke e8181fc627
Fix implicit duplicate duplicatesStrategy in processResources (#58929) (#59127)
* Fix implicit duplicate duplicatesStrategy in processResources
* Fix duplicates strategy in docker distribution setup
2020-07-07 13:45:36 +02:00
Francisco Fernández Castaño 1ced3f0eb3
Extract recovery files details to its own class (#59121)
Backport of #59039
2020-07-07 12:35:57 +02:00
Ignacio Vera 5cc6457ed8
upgrade to lucene-8.6.0-snapshot-6a715e2ecc3 (#59091) (#59120) 2020-07-07 12:07:41 +02:00
Armin Braun d6d6df16bb
Share IT Infrastructure between Core Snapshot and SLM ITs (#59082) (#59119)
For #58994 it would be useful to be able to share test infrastructure.
This PR shares `AbstractSnapshotIntegTestCase` for that purpose, dries up SLM tests
accordingly and adds a shared and efficient (compared to the previous implementations)
way of waiting for no running snapshot operations to the test infrastructure to dry things up further.
2020-07-07 12:04:41 +02:00
David Roberts e217f9a1e8
[ML] Wait for shards to initialize after creating ML internal indices (#59087)
There have been a few test failures that are likely caused by tests
performing actions that use ML indices immediately after the actions
that create those ML indices.  Currently this can result in attempts
to search the newly created index before its shards have initialized.

This change makes the method that creates the internal ML indices
that have been affected by this problem (state and stats) wait for
the shards to be initialized before returning.

Backport of #59027
2020-07-07 10:52:10 +01:00
Rene Groeschke 7c8d644bbc
Fix external javadoc reference to server project (#59000) (#59117)
* Fix external javadoc reference to server project

This fixes https://github.com/elastic/infra/issues/20103
The problem here is that we rename all artifacts in the :server project to
elasticsearch. The ideal fix would be to rename the gradle project there
to elasticsearch. I'm open to that but it seems quite invasive. Would love to hear
other opinions on that.
The applied alternative minimal invasive fix provided with this PR just takes
potential renamed projects / artifacts (Currently I only see `:server`) into account
when declaring external links.

* Simplify javadoc link base name
2020-07-07 11:38:40 +02:00
David Turner ef2f0d1f67 Inline no-op IndicesModule#getEngineFactories (#59051)
This method was introduced in #31183 but it has no effect and is never
overridden so this commit removes it.
2020-07-07 09:15:20 +01:00
Francisco Fernández Castaño 0752a86fe5
Enforce higher priority for RepositoriesService ClusterStateApplier (#59040)
* Enforce higher priority for RepositoriesService ClusterStateApplier

This avoids shards allocation failures when the repository instance
comes in the same ClusterState update as the shard allocation.

Backport of #58808
2020-07-07 09:51:08 +02:00
Howard 00ed31d000 Remove IndexShardRoutingTable#primaryAsList (#59044) 2020-07-07 07:34:32 +01:00
Nik Everett be13dea113 Drop a TODO from the terms aggregator (#59100)
We did it in #56487.
2020-07-06 17:46:06 -04:00
Jake Landis 604c6dd528
7.x - Create plugin for yamlTest task (#56841) (#59090)
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 14:16:26 -05:00
Nik Everett eff5f4d234
Add pipeline aggregations to the rewrite phase (backport #58878) (#59081)
This allows pipeline aggregations to participate in the up-front rewrite
phase for searches, in particular, it allows them to load data that they
need asynchronously.

Relates to #58193

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-06 15:13:45 -04:00
Nhat Nguyen e827d2ed92 Fix testRestoreLocalHistoryFromTranslogOnPromotion (#58745)
If the global checkpoint equals max_seq_no, then we won't reset an engine 
(as all operations are safe), and max_seqno_of_updates_or_deletes 
won't advance to max_seq_no.

Closes #58163
2020-07-06 12:19:45 -04:00
Costin Leau f9c15d0fec EQL: Introduce sequencing fetch size (#59063)
The current internal sequence algorithm relies on fetching multiple results and then paginating through the dataset. Depending on the dataset and memory, setting a larger page size can yield better performance at the expense of memory.
This PR makes this behavior explicit by decoupling the fetch size from size, the maximum number of results desired.
As such, use in testing a minimum fetch size which exposed a number of bugs:

Jumping across data across queries causing valid data to be seen as a gap.
Incorrectly resuming searching across pages (again causing data to be discarded).
which have been addressed.

(cherry picked from commit 2f389a7724790d7b0bda67264d6eafcfa8b2116e)
2020-07-06 19:14:26 +03:00
Costin Leau b2e9c6f640 Update UnresolvedRelationTests
UnresolvedRelation does not care about its source during equality hence
ignore it when doing randomized mutations.

Relates #59014

(cherry picked from commit b21222e714fbf85aad0916e4d4b6a933d2b6958a)
2020-07-06 19:14:25 +03:00
Costin Leau fe775a315f EQL: Obey size request parameter (#59014)
While at it, change the default size to 10 (to align it with the search
API defaults).

(cherry picked from commit 45795939b277e736a9e4f2f008d1c3f406239075)
2020-07-06 19:14:25 +03:00
James Rodewig d66084dcaf
[DOCS] Update data stream mapping and setting docs (#58874) (#59067) 2020-07-06 11:58:43 -04:00
Adam Locke e3469bb6e2
Removing ESS icon for xpack.security.audit.enabled. (#59078) (#59079) 2020-07-06 11:20:53 -04:00
Nik Everett 2965c7fe12
Fix bug in parent and child aggregators when parent field not defined (#57089) (#59074)
Adding null check for ParentJoinFieldMapper in ChildrenAggregationBuilder.joinFieldResolveConfig

Closes #42997

Co-authored-by: ParthPunkster <parthjain.pj1994@gmail.com>
2020-07-06 10:59:47 -04:00
Yang Wang 2a1635ad69
Create API key with TransportBulkAction directly (#59046) (#59060)
Use TransportBulkAction directly to create API keys instead of going through the proxy from IndexAction to BulkAction.
2020-07-06 23:32:07 +10:00
Andrei Dan 2d516d7bcc
[7.x] Search all (_all, *) resolves data streams too (#58869) (#59058)
Part of the original PR was merged by #59028

(cherry picked from commit 2598327726124d8a86333f79cdc45bf6a4297dbc)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-07-06 14:19:15 +01:00
Dan Hermann 550dcb0ca6
[7.x] Delete data stream API accepts multiple names (#59064) 2020-07-06 08:06:10 -05:00
James Rodewig 31c71914b7
[DOCS] Clean up `Use a data stream` test snippets (#58968) (#58978) 2020-07-06 08:39:04 -04:00
Yang Wang 66c0231895
Improve threadpool usage and error handling for API key validation (#58090) (#59047)
The PR introduces following two changes:

Move API key validation into a new separate threadpool. The new threadpool is created separately with half of the available processors and 1000 in queue size. We could combine it with the existing TokenService's threadpool. Technically it is straightforward, but I am not sure whether it could be a rushed optimization since I am not clear about potential impact on the token service.

On threadpoool saturation, it now fails with EsRejectedExecutionException which in turns gives back a 429, instead of 401 status code to users.
2020-07-06 21:21:07 +10:00
Przemysław Witek 4a791e835b
Simplify parser declarations when specialist types are stored in strings (#58996) (#59056) 2020-07-06 13:05:03 +02:00
Armin Braun 722d94688b
Fix MinimumMasterNodesIT Test (#59054) (#59057)
Tiny oversight in dee9e048bdcc5ba59f20d2554e989015463df05a caused
the `otherNodes` collection to incorrectly contain `master` here.
2020-07-06 13:00:15 +02:00
Przemysław Witek f35ad0d4e1
Report peak model memory in ModelSizeStats (#59017) (#59055) 2020-07-06 12:55:12 +02:00
David Kyle c651135562
[ML] Make Inference processor field_map and inference_config optional (#59010)
Relaxes the requirement that the inference ingest processor must has a
field_map and inference_config defined even if they are empty.
2020-07-06 11:35:30 +01:00
David Kyle 0fc12194bf
[ML] Increase timeout in MlDistributedFailureIT (#58997) (#59013)
Doubles the timeout on the ensureStableClusterOnAllNodes method to 60s
to account for v slow ci
2020-07-06 11:30:41 +01:00
Armin Braun 62eabdac6e
Dry up Snapshot ITs further (#59035) (#59052)
Some more obvious cleaning up of the snapshot ITs.

follow up to #58818
2020-07-06 12:26:42 +02:00
Martijn van Groningen f0dd9b4ace
Add data stream timestamp validation via metadata field mapper (#59002)
Backport of #58582 to 7.x branch.

This commit adds a new metadata field mapper that validates,
that a document has exactly a single timestamp value in the data stream timestamp field and
that the timestamp field mapping only has `type`, `meta` or `format` attributes configured.
Other attributes can affect the guarantee that an index with this meta field mapper has a
useable timestamp field.

The MetadataCreateIndexService inserts a data stream timestamp field mapper whenever
a new backing index of a data stream is created.

Relates to #53100
2020-07-06 11:32:33 +02:00
Armin Braun 49857cc35d
Dry up Master Disconnect Disruption Tests (#58953) (#59050)
Dry up tests that use a disruption that isolates the master from all other nodes.
Also, turn disruption types that have neither parameters nor state into constants
to make things a little clearer.
2020-07-06 11:04:24 +02:00
Rene Groeschke 56136b75dc
Fix security-cli distribution packaging (#59048)
- This fixes https://github.com/elastic/elasticsearch/issues/59031
- do not use compileclasspath in distribution packaging as it uses by default
plain class files
2020-07-06 09:46:22 +02:00
Nhat Nguyen 62763b177d Implement toString for BulkByScrollTask (#59042)
We should implement "toString" of BulkByScrollTask.StatusOrException 
to have a meaningful log message when a reindex task completes.
2020-07-05 22:06:56 -04:00
Yang Wang a9151db735
Map only specific type of OIDC Claims (#58524) (#59043)
This commit changes our behavior in 2 ways:

- When mapping claims to user properties ( principal, email, groups,
name), we only handle string and array of string type. Previously
we would fail to recognize an array of other types and that would
cause failures when trying to cast to String.
- When adding unmapped claims to the user metadata, we only handle
string, number, boolean and arrays of these. Previously, we would
fail to recognize an array of other types and that would cause
failures when attempting to process role mappings.

For user properties that are inherently single valued, like
principal(username) we continue to support arrays of strings where
we select the first one in case this is being depended on by users
but we plan on removing this leniency in the next major release.

Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
2020-07-06 11:36:41 +10:00
Tanguy Leroux 49f4227837 Check acknowledged responses in FsSearchableSnapshotsIT (#59021)
Despite all my attempts I did not manage to reproduce issues like the ones 
described in #58961. My guess is that the _mount request got retried at 
some point but I wasn't able to validate this assumption.

Still, the FsSearchableSnapshotsIT can be pretty disk heavy if a small 
random chunk size and a large number of documents is picked up in the 
tests. The parent class also does not verify the acknowledged status 
of some requests.

This commit lowers down the chunk size and number of docs in tests 
(this is extensively tests in unit tests) and also adds assertions on 
acknowledged responses.

Relates #58961
2020-07-05 10:50:31 +02:00
Armin Braun 071d8b2c1c
Deduplicate Empty InternalAggregations (#58386) (#59032)
Working through a heap dump for an unrelated issue I found that we can easily rack up
tens of MBs of duplicate empty instances in some cases.
I moved to a static constructor to guard against that in all cases.
2020-07-04 14:02:16 +02:00
David Turner 2fdd8f3a2c Restores do not cause red health (#59015)
Since 2.0.0 (56a264cf6d) we have documented that restoring a snapshot
typically results in `red` cluster health. However since 5.0.0 (#19516)
this hasn't been true, we report `yellow` health for unassigned
primaries that will be recovered from a snapshot in the future. This
commit adjusts these docs to match today's behaviour.
2020-07-04 11:16:28 +01:00
Bogdan Pintea e88d71b187
[7.x] SQL: Redact credentials in connection exceptions (#58650) (#59025)
* SQL: Redact credentials in connection exceptions (#58650)

This commit adds the functionality to redact the credentials from the
exceptions generated when a connection attempt fails, preventing them
from leaking into logs, console history etc.

There are a few causes that can lead to failed connections. The most
challenging to deal with is a malformed connection string. The redaction
tries to get around it by modifying the URI to a parsable state, so that
the redaction can be applied reliably. If there's no reliability
guarantee, the redaction will bluntly replace the entire connection
string and the user informed about the option to modify it so that the
redaction won't apply. (This is done by using a caplitalized scheme,
which is legal, but otherwise never used in practice.)

The commit fixes a couple of other issues with the URI parser:
- it allows an empty hostname, or even entire connection string (as per
the existing documentation);
- it reduces the editing of the connection string in the exception
messages (so that the user easier recognize their input);
- it uses the default URI as source for the scheme and hostname.

(cherry picked from commit a0bd5929d0658c4fed44404e0c4d78eac88222fd)

* Implement String#repeat(), unavailable in Java8

Implement a client.StringUtils#repeatString() as a replacement for
String#repeat(), unavailable in Java8.
2020-07-04 11:29:06 +02:00
debadair c8e3128fe4
[DOCS] Combo version of ILM docs. (#57909) (#59029)
* [DOCS] Combo version of ILM docs.

* [DOCS] Moved tutorial from Kibana.

* Adds documentation for index lifecycle policies (#28705)

* [DOCS] Adds documentation for index lifecycle policies

* [DOCS] Updated image for policy options to show all menu items

* Update create-policy.asciidoc

* [DOCS] Incorporated review comments on hot and warm phase

* [DOCS] Additional changes to warm phase

* [DOCS] Removed the word open in the warm phase

* Adds X-Pack icon for ILM (#34178)

* Add ILM tutorial (#59502)

* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Updates for navigation redesign (#68709)

* [DOCS] Updates for navigation redesign

* Getting started

* Set up text

* Discover

* Dashboard, Graph, ML, Maps, APM, SIEM, Dev tools

* Dev Tools, Stack Monitoring, Management

* Management

* Final changes

* [DOCS] Updates for navigation redesign

* [DOCS] Updates CCR monitoring screenshots

* updates SIEM screenshot and Cases overview text

* Added Brandon's APM image

* [DOCS] Refines CCR shard screenshot

* Removed merge conflict image file

Co-authored-by: lcawl <lcawley@elastic.co>
Co-authored-by: Ben Skelker <ben.skelker@elastic.co>

* [DOCS] Put API examples in collapsible sections like ML does

* Fix include

* Added tutorial images

* Fixed images

* Add short title for FB tutorial

* Add missing files

* Incorporate review feedback

* review feedback

* Incorporated review feedback

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Melori Arellano <melori@elastic.co>
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Kaarina Tungseth <kaarina.tungseth@elastic.co>
Co-authored-by: Ben Skelker <ben.skelker@elastic.co>

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Melori Arellano <melori@elastic.co>
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Kaarina Tungseth <kaarina.tungseth@elastic.co>
Co-authored-by: Ben Skelker <ben.skelker@elastic.co>
2020-07-03 13:01:08 -07:00
Dan Hermann 7c43cbca82
[7.x] Ignore matching data streams if include_data_streams is false (#59028) 2020-07-03 14:51:32 -05:00