Commit Graph

43155 Commits

Author SHA1 Message Date
Jason Tedor b030125e23
Build Docker images if the binary exists (#36459)
This commit changes the Docker assemble tasks so that they attempt to
build Docker if the Docker binaries exist, of if build.docker is set to
true. If the Docker binaries do not exist, or if build.docker is set to
false, then no attempt is made to build the Docker images.
2018-12-10 18:36:55 -05:00
Nik Everett de06769677 Docs: Fix link to transport client
The transport client was linking to a page in maven central that didn't
have the transport client in it. This is better link and happens to be
the same link that the low and high level rest client use.
2018-12-10 16:34:54 -05:00
Nhat Nguyen 2a7edca59f
Upgrade to Lucene-8.0.0-snapshot-ef61b547b1 (#36450)
Includes:

- LUCENE-8598: Improve field updates packed values
- LUCENE-8599: Use sparse bitset to store docs in SingleValueDocValuesFieldUpdates
2018-12-10 16:33:49 -05:00
Jernej Klancic d615add1b1 Add pipeline parent validation for auto date histogram (#35670)
Allow `auto_date_histogram` as a valid parent agg for derivative, 
cumulative sum, moving average, moving function and serial 
differencing pipeline aggregations.

Since all these aggs share the same requirement (sequentially
ordered parent aggs), this commit also refactors to share
the same validation code so that any newly added aggs won't
be forgotten.

Closes #35578
2018-12-10 16:02:49 -05:00
Albert Zaharovits 01afeff55d
Fix origin.type for connection_* audit events (#36410)
The `origin.type` field's permitted values are now `rest` or
`transport` (as the docs declare) instead of `ip_filter`.
2018-12-10 21:54:47 +02:00
Jason Tedor 0909a631ba
Add non-X-Pack centric rollup endpoints (#36383)
* Add non-X-Pack centric rollup endpoints

This commit adds new endpoints for rollup that do not have _xpack in
their path. The purpose for this change is to take these endpoints into
6.x as well so that they can be available in mixed cluster tests too. A
follow-up change will deprecate the use of _xpack in the rollup
endpoints. And finally, in the future, we would remove the _xpack
endpoints.

* Remove import

* Fix typo
2018-12-10 14:50:30 -05:00
Jim Ferenczi 75392adf60
[TEST] Convert SearchHitsTests to AbstractStreamableXContentTestCase (#36313)
This change adds a way to provide the content type of the rest serialization
tests when creating random instances. This is used by SearchHitsTests to ensure
that the internal members of the class are created with the same xContentType
and that equals can be used to compare an instances created from an XContent
view.
2018-12-10 20:41:20 +01:00
Tal Levy ed7afd1a9e
[ILM] TEST: fix long overflow in TimeValueScheduleTests (#36384)
Closes #35948.
2018-12-10 09:28:17 -08:00
Gordon Brown 58a5ad1f1e
Add Tribe removal to breaking changes list (#36239) 2018-12-10 10:23:58 -07:00
David Turner ca3f5c1e2e
Cancel GetDiscoveredNodesAction when bootstrapped (#36423)
Today the `GetDiscoveredNodesAction` waits, possibly indefinitely, to discover
enough nodes to bootstrap the cluster. However it is possible that the cluster
forms before a node has discovered the expected collection of nodes, in which
case the action will wait indefinitely despite the fact that it is no longer
required.

This commit changes the behaviour so that the action fails once a node receives
a cluster state with a nonempty configuration, indicating that the cluster has
been successfully bootstrapped and therefore the `GetDiscoveredNodesAction`
need wait no longer.

Relates #36380 and #36381; reverts 558f4ec278.
2018-12-10 17:23:03 +00:00
Marios Trivyzas c3dd0d393d
SQL: Simplify function registration and resolution (#36417)
Previously, we used a CamelCase to CAMEL_CASE transformation to get the
primary name of a function from its class name which led to some issues
since there are functions that we don't want to be registered this way
(e.g.: IFNULL). To simplify the logic and avoid and "magic"
transformations in the FunctionRegistry a primary name must be provided
explicitely for each function.

The same change is applied for the function resolution (when a function
is used in an SQL statement). There is no CamelCase to CAMEL_CASE
transformation but only upper-casing is applied (fuNcTiOn -> FUNCTION).
2018-12-10 18:52:09 +02:00
Andy Bristol 10938e5135
[hlrc] add index templates exist API (#36132)
This commit adds support for the index templates exist API, creating
new client-side request types for that API and the get index
templates API. Also adds links in hlrc docs to pages for supported
index template APIs
2018-12-10 08:11:57 -08:00
Ryan Ernst 1a36046421
Core: Convert FormatDateTimeFormatter to DateFormatter (#36396)
This commit creates JodaDateFormatter to replace
FormatDateTimeFormatter. It converts all uses of the old class
to DateFormatter to allow a future change to use JavaDateFormatter
when appropriate.
2018-12-10 08:06:41 -08:00
Dimitris Athanasiou 891a37adaa
[ML] Use LinkedHashMap to preserve order in job and datafeed fields (#36389) 2018-12-10 15:05:47 +00:00
Nik Everett 9626e700ce
LLRC: Make warning behavior pluggable per request (#36345)
This allows you to plug the behavior that the LLRC uses to handle
warnings on a per request basis.

We entertained the idea of allowing you to set the warnings behavior to
strict mode on a per request basis but that wouldn't allow the high
level rest client to fail when it sees an unexpected warning.

We also entertained the idea of adding a list of "required warnings" to
the `RequestOptions` but that won't work well with failures that occur
*sometimes* like those we see in mixed clusters.

Adding a list of "allowed warnings" to the `RequestOptions` would work
for mixed clusters but it'd leave many of the assertions in our tests
weaker than we'd like.

This behavior plugging implementation allows us to make a "required
warnings" option when we need it and an "allowed warnings" behavior when
we need it.

I don't think this behavior is going to be commonly used by used outside
of the Elasticsearch build, but I expect they'll be a few commendably
paranoid folks who could use this behavior.
2018-12-10 08:32:00 -05:00
Yannick Welsch f79e602d9f Mute ClusterDisruptionIT.testSendingShardFailure
Relates to #36428
2018-12-10 13:53:43 +01:00
Martijn van Groningen 38a1d4fcdb
re-enabled test with added logging to have more info when it fails. 2018-12-10 13:15:30 +01:00
Martijn van Groningen ca916f7186
[TEST] pause index following after test 2018-12-10 13:15:30 +01:00
David Pilato c9c691b0cf Change Lucene Snapshot example for 7.0.0 (#35619)
Update to a more recent version our example.
2018-12-10 12:44:39 +01:00
David Roberts 558f4ec278
Ignore zen2 discovery task in waitForPendingTasks (#36381)
Fixes #36380
2018-12-10 11:07:02 +00:00
Igor Suhorukov 0d9e3adc95 Loop replace with Collection.removeIf() (#36351) 2018-12-10 11:26:24 +01:00
Yannick Welsch 6e6e63d01d
Zen2: Move all mixed-version REST tests to Zen2 (#36398)
Moves all remaining (rolling-upgrade and mixed-version) REST tests to use Zen2. To avoid adding
extra configuration, it relies on Zen2 being set as the default discovery type. This required a few
smaller changes in other tests. I've removed AzureMinimumMasterNodesTests which tests Zen1
functionality and dates from a time where host providers were not configurable and each cloud
plugin had its own discovery.type, subclassing the ZenDiscovery class. I've also adapted a few tests
which were unnecessarily adding addTestZenDiscovery = false for the same legacy reasons. Finally,
this also moves the unconfigured-node-name REST test to Zen2, testing the auto-bootstrapping
functionality in development mode when no discovery configuration is provided.
2018-12-10 11:00:57 +01:00
Albert Zaharovits 0b1e7e96be Nitfix for SecurityDocumentationIT dad6f1c9fe 2018-12-10 11:54:50 +02:00
Albert Zaharovits dad6f1c9fe
[HLRC] Put Role (#36209)
This commit adds support for the put role API in the
java high level rest client.
2018-12-10 09:41:31 +02:00
Alpar Torok bf2c61d2d0
Forbid empty testing tasks (#36259)
Closes #34820

With this change we allow for no tests being ran in randomized testing
task, and forbid empty testing tasks from the testing conventions task.
We will no longer have to disable the task if all tests are muted.
2018-12-10 08:32:19 +02:00
Alpar Torok e160346b9a
Improve logged exec output readability (#36217)
* Improve logged exec output readability

- Split error and out streams and log them separately
- Log everything in a single call to prevent interference from
  other log messages
2018-12-10 08:23:03 +02:00
Christoph Büscher a42502df8b
[Docs] Add description of simple query string flags (#36211)
Closes #34944
2018-12-10 01:00:42 +01:00
Martijn van Groningen f4aac8d3f1
[HLRC] Added support for Follow Stats API (#36253)
This change also adds documentation for the Follow Stats API.

Relates to #33824
2018-12-09 19:59:08 +01:00
Martijn van Groningen 633ab24017
[CCR] Restructured QA modules (#36404)
Renamed the follow qa modules:
`multi-cluster-downgraded-to-basic-license` to `downgraded-to-basic-license`
`multi-cluster-with-non-compliant-license` to `non-compliant-license`
`multi-cluster-with-security` to `security`

Moved the `chain` module into the `multi-cluster` module and
changed the `multi-cluster` to start 3 clusters.

Followup from #36031
2018-12-09 19:34:48 +01:00
Jacques B bba9bb2cfa RestClient: on retry timeout add root exception (#25576) 2018-12-09 09:03:48 -05:00
Nhat Nguyen 95bafb0593 TEST: Always enable soft-deletes in ShardChangesTests 2018-12-09 02:57:13 -05:00
Yogesh Gaikwad 32c4f99238
[HLRC] Add support for put privileges API (#35679)
This commit adds support for API to create or update
application privileges in high-level rest client.
2018-12-09 16:03:28 +11:00
Nhat Nguyen 902d6f579a
Exposed engine must have all ops below gcp during rollback (#36159)
Today we expose a new engine immediately during Lucene rollback. The new
engine is started with a safe commit which might not include all
acknowledged operation. With this change, we won't expose the new engine
until it has recovered from the local translog.

Note that this solution is not complete since it's able to reserve only
acknowledged operations before the global checkpoint. This is because we
replay translog up to the global checkpoint during rollback. A per-doc
Lucene rollback would solve this issue entirely.

Relates #32867
2018-12-08 21:27:44 -05:00
Nhat Nguyen d41cf6ac9f
Use delCount of SegmentInfos to calculate numDocs (#36323)
Today, we iterate the bitset of hardLiveDocs to calculate the number of
live docs. This calculation might be expensive if we enable soft-deletes
(by default) for old indices whose soft-deletes was disabled previously
and had hard-deletes.

Once soft-deletes is enabled, we no longer hard-update or hard-delete
documents directly. We have hard-deletes in two scenarios: (1) from old
segments where soft-deletes was disabled, (2) when IndexWriter hits
non-aborted exceptions. These two cases, IW flushes SegmentInfos before
exposing the hard-deletes; thus we can use the hard-delete count of
SegmentInfos.
2018-12-08 21:24:36 -05:00
Tim Brooks cc3872d934
Reduce composite references at network level (#36400)
Creating `CompositeBytesReference` has more overhead than a single
`ByteBufferReference`. Many of our messages will be contained to a
single `ByteBuffer`. This commit avoids creating composite instances
when there is 0 or 1  underlying `ByteBuffers`.
2018-12-08 14:24:38 -07:00
Michael Basnight b5b6e37a60
Deprecate X-Pack centric watcher endpoints (#36218)
This commit is part of our plan to deprecate and ultimately remove the use of
_xpack in the REST APIs.

Relates #35958
2018-12-08 12:57:16 -06:00
David Turner 9f86e996fe
[Zen2] Support rolling upgrades from Zen1 (#35737)
We support rolling upgrades from Zen1 by keeping the master as a Zen1 node
until there are no more Zen1 nodes in the cluster, using the following
principles:

- Zen1 nodes will never vote for Zen2 nodes
- Zen2 nodes will, while not bootstrapped, vote for Zen1 nodes
- Zen2 nodes that were previously part of a mixed cluster will automatically
  (and unsafely) bootstrap themselves when the last Zen1 node leaves.
2018-12-08 07:33:35 +00:00
Ryan Ernst a27f2efca5
Core: Converge FormatDateTimeFormatter and DateFormatter apis (#36390)
This commit makes FormatDateTimeFormatter and DateFormatter apis close
to each other, so that the former can be removed in favor of the latter.
This PR does not change the uses of FormatDateTimeFormatter yet, so that
that future change can be purely mechanical.
2018-12-07 17:23:41 -08:00
Julie Tibshirani b15d1aebcf
For msearch templates, make sure to use the right name for deprecation logging. (#36344) 2018-12-07 14:50:47 -08:00
Gordon Brown 85bba0c3ae
Simplify deprecation issue levels (#36326)
This commit gets rid of the 'NONE' and 'INFO' severity levels for
deprecation issues.

'NONE' is unused and does not make much sense as a severity level.
'INFO' can be separated into two categories: Either 1) we can
definitively tell there will be a problem with the cluster/node/index
configuration that can be resolved prior to upgrade, in which case
the issue should be a WARNING, or 2) we can't, because any issues would
be at the application level, for which the user should review the
deprecation logs and/or response headers.
2018-12-07 15:45:53 -07:00
Simon Willnauer 2ba023be8a
Explicitly recommend to forceMerge before freezing (#36376)
given the benchmark results on #34352 it's important to recommend
users to `_force_merge` their indices to a single segment before freezing.
2018-12-07 23:38:35 +01:00
Tim Brooks 8a53f2b464
Implement basic `CcrRepository` restore (#36287)
This is related to #35975. It implements a basic restore functionality
for the CcrRepository. When the restore process is kicked off, it
configures the new index as expected for a follower index. This means
that the index has a different uuid, the version is not incremented, and
the Ccr metadata is installed.

When the restore shard method is called, an empty shard is initialized.
2018-12-07 15:27:04 -07:00
Ryan Ernst a998f4dec6
Core: Remove parseDefaulting from DateFormatter (#36386)
This commit removes the parseDefaulting method from DateFormatter,
bringing it more inline with the joda equivalent
FormatDateTimeFormatter. This method was only needed for the java
time implementation of DateMathParser. Instead, a DateFormatter now
returns an implementation of DateMathParser for the given format,
allowing the java time implementation to construct the appropriate date
math parser internally.
2018-12-07 12:40:14 -08:00
David Roberts 9e8cfbb40d
[ML] Deprecate X-Pack centric ML endpoints (#36315)
This commit is part of our plan to deprecate and
ultimately remove the use of _xpack in the REST APIs.

Relates #35958
2018-12-07 20:34:11 +00:00
Wilder Pereira 1d73c1712b Docs: Fix Jackson reference (#36366)
Link is totally broken.
2018-12-07 15:18:22 -05:00
Dimitris Athanasiou 7870ae81b1
[ML] Prevent stack overflow while copying ML jobs and datafeeds (#36370)
ML jobs and datafeeds wrap collections into their unmodifiable
equivalents in their constructor. However, the copying builder
does not make a copy of some of those collections resulting
in wrapping those again and again. This can eventually result
to stack overflow.

This commit addressed this issue by copying the collections in
question in the copying builder constructor.

Closes #36360
2018-12-07 20:16:25 +00:00
Nik Everett ead2b9e08b
HLRC: Add rollup search (#36334)
Relates to #29827
2018-12-07 14:39:58 -05:00
Julie Tibshirani 51e1d40dca
Small improvements related to types deprecation. (#36328)
* Make sure to use deprecatedAndMaybeLog for types deprecation messages.
* Introduce a common base class for Rest*Action tests.
2018-12-07 11:21:24 -08:00
Tim Brooks 5556204f81
Use MockNioTransport in MockTransportService (#36346)
The default transport used in the MockTransportService is the
MockTcpTransport. This commit changes that to be the
MockNioTransport.
2018-12-07 11:17:11 -07:00
Alpar Torok 03c577b477
Move VersionProperties and Version to min runtime (#36378)
Allow elasticsearch-hadoop to access these classes when running on
java 8.
2018-12-07 20:07:36 +02:00