Commit Graph

51498 Commits

Author SHA1 Message Date
Lee Hinman 3b879b0821
[7.x] Use V2 templates when reading duplicate aliases and inge… (#54973)
When a new index is rolled over, we check to see whether there are any duplicate alias
configurations in the index template configuration. Additionally, when a new index is created from a
bulk action, we check the templates to see if there are any ingest pipelines that need to be applied
to the index that will be newly created.

Both of these actions previously checked the v1 templates for their settings, they now also check
the v2 index templates, with the v2 index templates taking precendence similar to the way they do
when creating an index.

Relates to #53101
2020-04-08 13:33:14 -06:00
Armin Braun 411dc2f607
Fix Broken Math in S3 Retries Tests (#54952) (#54972)
If we run into `length == 0` we trip an assertion in `randomIntBetween(0, length -1)`.
2020-04-08 20:32:21 +02:00
James Rodewig 964cf565c9
[DOCS] EQL: Document `between` function (#54950) 2020-04-08 13:49:15 -04:00
Lee Hinman c2c0707174
[7.x] Add allowed warnings to index template composition tests… (#54961)
We occasionally add a global template for our YAML tests, and this can cause warnings for these
template tests. This commit adds these warnings so they don't cause test failures.

Resolves #54822

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-08 10:17:32 -06:00
Jason Tedor a5391c915e
Set JAVA14_HOME in CI (#54955)
This commit updates the CI defaults so that JAVA14_HOME is set.
2020-04-08 11:09:50 -04:00
Jay Modi 3600c9862f
Reintroduce system index APIs for Kibana (#54935)
This change reintroduces the system index APIs for Kibana without the
changes made for marking what system indices could be accessed using
these APIs. In essence, this is a partial revert of #53912. The changes
for marking what system indices should be allowed access will be
handled in a separate change.

The APIs introduced here are wrapped versions of the existing REST
endpoints. A new setting is also introduced since the Kibana system
indices' names are allowed to be changed by a user in case multiple
instances of Kibana use the same instance of Elasticsearch.

Relates #52385
Backport of #54858
2020-04-08 09:08:49 -06:00
Bogdan Pintea 8d6d7b88d8
SQL: drop BASE TABLE type in favour for just TABLE (#54836) (#54951)
* Drop BASE TABLE type in favour for just TABLE

This commit drops the table type 'BASE TABLE' and replaces all
occurences with just 'TABLE', since his type is wider-used and
friendlier to the client applications that query for certain table types
in their discovery mode.

The 'TABLE' type is also explicitely mentioned by the JDBC and ODBC
standards and although other data source-specific types are permitted,
older apps will not work well with them.

* Refactor table type constants out of IndexType

Move SQL_TABLE/_ALIAS out of IndexType, so that they can also be used in
that Enum definition.

(cherry picked from commit 70241b52697ac2cf71004040042123c1ec050299)
2020-04-08 16:02:12 +02:00
Jason Tedor 6853d73e88
Require JDK 14 for compilation (#54696)
This commit bumps the minimum JDK required for compilation to JDK 14.
2020-04-08 09:26:29 -04:00
Jason Tedor 6d29da05c3
Defer node environment construction (#54919)
Today we construct the node environment relatively early in the node
construction process, before we have even constructed the final
environment, which means before the final settings are
available. Rather, we should defer constructing the node environment
until the final environment is available. This commit does that. This
helps delay node environment construction until after the node roles are
properly determined, which is important since the node environment does
some checks on the basis of whether or not the node is neither a data
nor a master node (such nodes should not have index metadata nor shard
data on disk). Note that a consequence of this is that the initial log
line that displays the node name, node ID, and cluster name does not
appear until later in startup (after we have loaded plugins). This seems
okay.
2020-04-08 09:23:19 -04:00
Théophile Helleboid - chtitux a8aa36d427 [DOCS] Fix typo in SLM retention docs (#54797) 2020-04-08 08:56:45 -04:00
Marios Trivyzas 6afd60b082
SQL: Implement DATETIME_FORMAT function for date/time formatting (#54832) (#54942)
Implement DATETIME_FORMAT(<date/datetime/time>, ) function
which allows for formatting a timestamp to the specified format. The
patterns allowed as those of java.time.format.DateTimeFormatter.

Related to #53714

(cherry picked from commit 72be0b54a9299e87e785469cdc9aafac2a48c046)
2020-04-08 13:45:47 +02:00
David Turner 0d2195191d Allocate searchable snapshots with the balancer (#54889)
Today the shards of searchable snapshots are allocated with a naive
`ExistingShardsAllocator` which selects the first valid node for each shard.
Thanks to #54729 we can now allow these shards to fall through to the balanced
shards allocator so that they are allocated in a more balanced fashion.

Relates #50999
2020-04-08 10:02:42 +01:00
István Zoltán Szabó 3a3effedc2 [DOCS] Reworks some parts of EMM API docs (#54872)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-04-08 10:20:34 +02:00
Ryan Ernst 37795d259a
Remove guava from transitive compile classpath (#54309) (#54695)
Guava was removed from Elasticsearch many years ago, but remnants of it
remain due to transitive dependencies. When a dependency pulls guava
into the compile classpath, devs can inadvertently begin using methods
from guava without realizing it. This commit moves guava to a runtime
dependency in the modules that it is needed.

Note that one special case is the html sanitizer in watcher. The third
party dep uses guava in the PolicyFactory class signature. However, only
calling a method on the PolicyFactory actually causes the class to be
loaded, a reference alone does not trigger compilation to look at the
class implementation. There we utilize a MethodHandle for invoking the
relevant method at runtime, where guava will continue to exist.
2020-04-07 23:20:17 -07:00
Tim Vernum ca20b8a828
Java8 implementation of Map.Entry (#54778)
A Java8 compatible version of Map.ofEntries() was added in #54183,
but it really needs a compat version of Map.entry as well in order to
facilitate easy backports from master.
2020-04-08 15:31:50 +10:00
Aleksandr Maus d02f774cb6
EQL: implement cidrMatch function (#54186) (#54928)
Related to https://github.com/elastic/elasticsearch/issues/54132
2020-04-07 22:07:28 -04:00
Nhat Nguyen 65713743c2 Update translog policy before the next safe commit (#54839)
IndexShardIT#testMaybeFlush relies on the assumption that the safe commit
and translog deletion policy have advanced after IndexShard#sync returns .
This assumption does not hold if there's a race with the global checkpoint sync.

Closes #52223
2020-04-07 21:55:54 -04:00
Tal Levy 254d1e3543
[7.x] Create new `geo` module and migrate geo_shape registration (#53562) (#54924)
This commit introduces a new `geo` module that is intended
to be contain all the geo-spatial-specific features in server.

As a first step, the responsibility of registering the geo_shape
field mapper is moved to this module.

Co-authored-by: Nicholas Knize <nknize@gmail.com>
2020-04-07 16:30:58 -07:00
Tim Brooks 619028c33e
Implement transport circuit breaking in aggregator (#54927)
This commit moves the action name validation and circuit breaking into
the InboundAggregator. This work is valuable because it lays the
groundwork for incrementally circuit breaking as data is received.

This PR includes the follow behavioral change:

Handshakes contribute to circuit breaking, but cannot be broken. They
currently do not contribute nor are they broken.
2020-04-07 17:10:31 -06:00
Julie Tibshirani 475b210eec
Improve guidance on removing default mappings. (#54915)
In 7.x, an index template will fail to apply if it contains a `_default_`
mapping. Several users have expressed confusion over the fact that loading the
template doesn't show any default mappings. This docs change clarifies that in
order to see all mappings in the template, you must pass `include_type_name`.
2020-04-07 15:18:13 -07:00
Aleksandr Maus de381271f1
EQL: implement stringContains function (#54380) (#54923) 2020-04-07 17:55:13 -04:00
Tim Brooks c7053ef824
Use TransportChannel in TransportHandshaker (#54921)
Currently the TransportHandshaker has a specialized codepath for sending
a response. In other work, we are going to start having handshakes
contribute to circuit breaking (while not being breakable). This commit
moves in that direction by allowing the handshaker to responding using a
standard TcpTransportChannel similar to other requests.
2020-04-07 15:37:15 -06:00
Nik Everett ce7ae4a7d1
Remove pipline aggs from agg result tree (backport of #54716) (#54920)
This removes pipeline aggregators from the aggregation result tree
except for a single field used for backwards compatibility with pre-7.8
versions of Elasticsearch. That field isn't populated unless we are
serializing to pre-7.8 Elasticsearch. So, good news! We no longer build
pipeline aggregators on the data node. Most of the time.
2020-04-07 17:22:23 -04:00
Nik Everett 100f7258c7
Improve agg reduce tests (#54910) (#54914)
This allows subclasses of `InternalAggregationTestCase` to make a `List`
of values to reduce so that it can make values that are realistic
*together*. The first use of this is with `InternalTTest` which uses it
to make results that don't cause their `sum` field to wrap. It'd likely
be useful for a ton of other aggs but just one for now.
2020-04-07 17:22:04 -04:00
Aleksandr Maus 868798e4db
EQL: implement between function (#54277) (#54913) 2020-04-07 16:52:30 -04:00
Tim Brooks 9cf2406cf1
Move network stats marking into InboundPipeline (#54908)
This is a follow-up to #48263. It moves the inbound stats tracking
inside of the InboundPipeline.
2020-04-07 13:34:05 -06:00
James Rodewig 9569a8eb13 [DOCS] Add example to "avoid scripts" advice (#54719)
Adds a detailed example to the "Avoid scripts" section of the "Tune
for search speed" docs. The detail outlines how a script used to
transform indexed data can be moved to ingest.

The update also removes an outdated reference to supported script
languages.
2020-04-07 15:25:10 -04:00
Costin Leau 8b1e87cb61 EQL: Change query folding spec from new lines to ; (#54882)
The usage of blank lines as separator between tests can be tricky to
deal with in case of merges where such lines can be added by accident.
Further more counting non-consecutive lines is non-intuitive.
The tests have been aligned to use ; at the end of the query and
exceptions so that the presence or absence of empty lines is irrelevant.
The parsing of the spec has been changed to perform validation to not
allow invalid/incomplete specs to cause exceptions.

(cherry picked from commit 192ad88d3a51e1e1f1f82830526518720ec88217)
2020-04-07 21:57:06 +03:00
Nik Everett 1798d6722b
Allow terms agg to default to depth first (#54845) (#54885)
If you didn't explictly set `global_ordinals` execution mode we were
never collecting the information that we needed to select `depth_first`
based on the request so we were always defaulting to `breadth_first`.
This fixes it so we collect the information.
2020-04-07 14:11:34 -04:00
Tanguy Leroux b8d2b952b8 Only one of azure key or token can be specified in 3rd party tests (#54876)
#54803 introduces more QA tests for Azure storage service, but 
they fail the build is one of the key or token is missing. It should i
nstead work like repository-azure:qa tests.
2020-04-07 19:36:48 +02:00
Armin Braun 37abc411dc
Remove Unused Snapshot Status Values (#54893) (#54906)
* Remove Unused Snapshot Status Values

This is a left-over from before #41940 when we used the same status enum for the shards
and the snapshots overall. The two removed values were never used on the shard level
so we can simply remove them here.
2020-04-07 19:16:13 +02:00
Jason Tedor d1d478debf
Update docs to reflect node.processors (#54855)
We namespaced the previous setting "processors" into
"node.processors". This commit updates some of the documentation to
reflect this.
2020-04-07 13:06:14 -04:00
Nik Everett faa687c0ae Fix InternalTTestTests
`testReduceRandom` was bumping up against the serialization that I added
in #54776. This makes it use random values that reduce in ways that
don't cause the randomized serialization to fail.
2020-04-07 11:51:54 -04:00
Lisa Cawley a7599031ae [DOCS] Adds tranform node to list of default types (#54850) 2020-04-07 08:49:05 -07:00
Larry Gregory 8c8baa10f4
[Backport] Add reserved_ml_user and reserved_ml_admin kibana p… (#54837)
* add reserved_ml_user and reserved_ml_admin kibana privileges

* address feedback, update dataframe roles

* fix checkstyle failure
2020-04-07 11:42:11 -04:00
Dimitris Athanasiou 9b4ac60b53
[7.x][ML] Cancel reindex task from correct thread context (#54874) (#54898)
When a data frame analytics job is stopped, if the reindexing
task was still in progress we cancel it. Cancelling it should
be done from the same context as when we executed the reindexing
task. That means from a thread context with ML origin.

Backport of #54874
2020-04-07 18:11:58 +03:00
Andrei Dan bbc57828c4
ILM fix retry delete action test (#54809) (#54895)
Asserting on the failed_step field from the explainAPI can produce flakiness
because the ILM state is moved back and forth between the (failing) step and
the ERROR step (as the workflow is retry, fail then move to ERROR step,
move back to the (failing) step,  retry, fail, etc) and the failed_step
information is only available whilst in the ERROR state.

Unmute other tests as they were collateral failures
A read-only index could not be deleted in the wipeCluster phase and caused
these failures

(cherry picked from commit 99a6d57aeb3cf11abc38b514f38a96bb1612e357)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-07 15:55:56 +01:00
Lee Hinman 72f8457f52
[7.x] Only allow retrieving a single index or component templa… (#54896)
* Only allow retrieving a single index or component template

This changes the Index Template v2 APIs to only allow retrieving a single "named" entity, where the
named entity can be nothing (return everything), a wildcard (return the ones that match), or the
name of a template.

Relates to #53101

* Throw exception when resource is not found

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-07 08:55:44 -06:00
Nik Everett 3c56e0de42
Fix scripted metric in ccs (backport of #54776) (#54888)
`scripted_metric` did not work with cross cluster search because it
assumed that you'd never perform a partial reduction, serialize the
results, and then perform a final reduction. That
serialized-after-partial-reduction step was broken.

This is also required to support #54758.
2020-04-07 10:43:00 -04:00
Nik Everett 915092dc28
More pipeline aggregation cleanup (backport of #54298) (#54890)
This replaces the last bit of validation that pipeline aggregations
performed on the data nodes with explicit checks in a few
`PipelineAggregationBuilders`. We were *already* catching these
validation errors for pipeline aggregations that require that their
parent be squentially ordered. This just adds validation for pipelines
that require *any* parent like `bucket_selector` and `bucket_sort`.
2020-04-07 10:40:34 -04:00
Ignacio Vera 076c199484
Add new point field. (#53804) (#54879)
This commit adds a new point field that is able to index arbitrary pair of values (x/y)
in the cartesian space. It only supports filtering using shape queries at the moment.
2020-04-07 15:28:50 +02:00
Tanguy Leroux 4d36917e52
Merge feature/searchable-snapshots branch into 7.x (#54803) (#54825)
This is a backport of #54803 for 7.x.

This pull request cherry picks the squashed commit from #54803 with the additional commits:

    6f50c92 which adjusts master code to 7.x
    a114549 to mute a failing ILM test (#54818)
    48cbca1 and 50186b2 that cleans up and fixes the previous test
    aae12bb that adds a missing feature flag (#54861)
    6f330e3 that adds missing serialization bits (#54864)
    bf72c02 that adjust the version in YAML tests
    a51955f that adds some plumbing for the transport client used in integration tests

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-07 13:28:53 +02:00
David Roberts 8f2ddaee1a [TEST] Allow kb or mb for data frame analytics memory estimate (#54869)
This change is to support the switch from kb to mb being made in
https://github.com/elastic/ml-cpp/pull/1126
2020-04-07 11:28:29 +01:00
Jason Tedor f3a0018175
Update link to JDK 14 compiler bug
This commit updates the link to the JDK 14 compiler bug that we have
found. At the time that we committed the workaround, we had a submission
ID, but not yet the public bug URL. This commit adds the public bug URL.
2020-04-07 06:26:14 -04:00
David Roberts df4ae79b41
[TEST] Unmute CategorizationIT.testNumMatchesAndCategoryPreference (#54868)
Should work again now that https://github.com/elastic/ml-cpp/issues/1121
is resolved.

Backport of #54768
2020-04-07 11:04:31 +01:00
Jason Tedor 91818c32c5
Fix aarch64 architecture for packages (#54856)
This commit fixes the architecture reported in the packages on aarch64.
2020-04-07 06:04:06 -04:00
Armin Braun 1039cae2cc
Fix Repository Consistency TODOs from SLM Tests (#54767) (#54860)
These TODOs don't apply any longer with the repository generation
now being tracked consistently so we can remove the workarounds.
2020-04-07 09:27:50 +02:00
Ioannis Kakavas 3560c0cbf2
Remove `_xpack` from license API example (#54698) (#54763)
Resolves #54662
2020-04-07 09:51:37 +03:00
Jim Ferenczi c7ff67ddef Preserve final response headers in asynchronous search (#54349)
This change adds the response headers of the original search request
in the stored response in order to be able to restore them when retrieving a result
from the async-search index. It also ensures that response headers are preserved for
users that retrieve a final response on a running search task.
Partial response can eventually return response headers too but this change only ensures
that they are present when the response if final.

Relates #33936
2020-04-07 08:37:03 +02:00
Jim Ferenczi d57a047ab7 Fix transport serialization of AsyncSearchUser (#54761)
This change ensures that the AsyncSearchUser is correctly (de)serialized when
an action executed by this user is sent to a remote node internally (via transport client).
2020-04-07 08:25:58 +02:00