Commit Graph

39978 Commits

Author SHA1 Message Date
Christoph Büscher e31a877a64 Fix problematic chars in javadoc
Java 11 complains about unescaped ">" characters in javadocs. Also fixed some
compiler complaints about javadoc in StringFunctionUtils.
2018-07-13 11:13:24 +02:00
Christoph Büscher 69c2b2d802 [Tests] Mute failing SearchDocumentationIT#testSearchRequestAggregations 2018-07-13 10:56:47 +02:00
Alpar Torok 23198ea551
Convert Version to Java - clusterformation part1 (#32009)
Implement buildSrc Version in java

- This allows to move all  all .java files from .groovy.
- Will prevent eclipse from tangling up in this setup
- make it possible to use Version from Java
2018-07-13 08:35:12 +00:00
Daniel Mitterdorfer f174f72fee
Circuit-break based on real memory usage
With this commit we introduce a new circuit-breaking strategy to the parent
circuit breaker. Contrary to the current implementation which only accounts for
memory reserved via child circuit breakers, the new strategy measures real heap
memory usage at the time of reservation. This allows us to be much more
aggressive with the circuit breaker limit so we bump it to 95% by default. The
new strategy is turned on by default and can be controlled  with the new cluster
setting `indices.breaker.total.userealmemory`.

Note that we turn it off for all integration tests with an internal test cluster
because it leads to spurious test failures which are of no value (we cannot
fully control heap memory usage in tests). All REST tests, however, will make
use of the real memory circuit breaker.

Relates #31767
2018-07-13 10:08:28 +02:00
David Roberts d2461643cd
[ML] Move open job failure explanation out of root cause (#31925)
When an ML job cannot be allocated to a node the exception
contained an explanation of why the job couldn't be
allocated to each node in the cluster.  For large clusters
this was not particularly easy to read and made the error
displayed in the UI look very scary.

This commit changes the structure of the error to an outer
ElasticsearchException with a high level message and an
inner IllegalStateException containing the detailed
explanation.  Because the definition of root cause is the
innermost ElasticsearchException the detailed explanation
will not be the root cause (which is what Kibana displays).

Fixes #29950
2018-07-13 08:57:33 +01:00
Tim Vernum c662565f29
Access SSL contexts using names instead of Settings (#30953)
Historically we have loaded SSL objects (such as SSLContext,
SSLIOSessionStrategy) by passing in the SSL settings, constructing a
new SSL configuration from those settings and then looking for a
cached object that matches those settings.

The primary issue with this approach is that it requires a fully
configured Settings object to be available any time the SSL context
needs to be loaded. If the Settings include SecureSettings (such as
passwords for keys or keystores) then this is not true, and the cached
SSL object cannot be loaded at runtime.

This commit introduces an alternative approach of naming every cached
ssl configuration, so that it is possible to load the SSL context for
a named configuration (such as "xpack.http.ssl"). This means that the
calling code does not need to have ongoing access to the secure
settings that were used to load the configuration.

This change also allows monitoring exporters to use SSL passwords
from secure settings, however an exporter that uses a secure SSL setting
(e.g. truststore.secure_password) may not have its SSL settings updated
dynamically (this is prevented by a settings validator).
Exporters without secure settings can continue to be defined and updated
dynamically.
2018-07-13 16:40:09 +10:00
Christoph Büscher 332c134517 [TEST] Mute SlackMessageTests.testTemplateRender 2018-07-12 22:41:35 +02:00
Nik Everett 9822a6f911 Docs: Explain closing the high level client
It looks like we weren't clear on when and why you should close the high
level client and folks were closing it after every request which is not
efficient. This explains why you should close the client and when so
this shouldn't be as common.

Closes #32001
2018-07-12 16:22:42 -04:00
Tom Veasey de2eb85aaf
[ML] Re-enable memory limit integration tests (#31328) 2018-07-12 20:56:05 +01:00
Andy Bristol 30d6fd3ca7 [test] disable packaging tests for suse boxes
For #30295
2018-07-12 11:12:37 -07:00
Tim Brooks c375d5ab23
Add nio transport to security plugin (#31942)
This is related to #27260. It adds the SecurityNioTransport to the
security plugin. Additionally, it adds support for ip filtering. And it
randomly uses the nio transport in security integration tests.
2018-07-12 11:55:38 -06:00
olcbean 334c255516 XContentTests : Insert random fields at random positions (#30867)
Currently AbstractXContentTestCase#testFromXContent appends random fields, but in
a fixed position. This PR shuffles all fields after the random fields have been appended, 
hence the random fields are actually added to random positions.
2018-07-12 19:10:51 +02:00
Igor Motov 44f280fc89
Force execution of fetch tasks (#31974)
Forces fetch tasks to queue even in the event that the queue is
already full. The reasoning is that fetch tasks may only be follow-up
to query tasks, so the number of additional fetch tasks that may enter
the threadpool is expected to be reasonable.

Closes #29442
2018-07-12 08:56:06 -07:00
Christoph Büscher 043e40adde
Fix unreachable error condition in AmazonS3Fixture (#32005)
The `else` branch where currently the error response should be thrown is not
reachable because `handler` is always non-null inside the previous outer check.
Moving error creation into an else branch on the other condition check, removing
the other superflous check for non-null handler inside the first branch.
2018-07-12 17:38:30 +02:00
Alexander Reelsen 0b7e7befdd
Tests: Fix SearchFieldsIT.testDocValueFields (#31995)
This test produced different implementations of joda time classes,
depending on if the data was serialized or not (DateTime vs
MutableDateTime). This now uses a common base class to extract the
milliseconds from the data.

Closes #31992
2018-07-12 16:06:56 +02:00
Christoph Büscher 4ae4ac08d5
Add Expected Reciprocal Rank metric (#31891)
This change adds Expected Reciprocal Rank (ERR) as a ranking evaluation metric
as descriped in:

Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P. (2009).
Expected reciprocal rank for graded relevance.
Proceeding of the 18th ACM Conference on Information and Knowledge Management.
https://doi.org/10.1145/1645953.1646033

ERR is an extension of the classical reciprocal rank to the graded relevance
case and assumes a cascade browsing model. It quantifies the usefulness of a
document at rank `i` conditioned on the degree of relevance of the items at ranks
less than `i`. ERR seems to be gain traction as an alternative to (n)DCG, so it
seems like a good metric to support. Also ERR seems to be the default optimization
metric used for training in RankLib, a widely used learning to rank library.

Relates to #29653
2018-07-12 15:50:58 +02:00
David Kyle 6fcd606536
[ML] Get ForecastRequestStats doc in RestoreModelSnapshotIT (#31973) 2018-07-12 13:59:34 +01:00
Andrei Stefan edf83c1d87
SQL: Add support for single parameter text manipulating functions (#31874)
Added support for ASCII, BIT_LENGTH, CHAR, CHAR_LENGTH, LCASE, LENGTH, LTRIM, RTRIM, SPACE, UCASE functions.
Wherever Painless scripting is necessary (WHERE conditions, ORDER BY etc), those scripts are being used.
2018-07-12 15:05:42 +03:00
Dimitris Athanasiou 2cfe703299
[ML] Ensure immutability of MlMetadata (#31957)
The test failure in #31916 revealed that updating
rules on a job was modifying the detectors list
in-place. That meant the old cluster state and the
updated cluster state had no difference and thus the
change was not propagated to non-master nodes.

This commit fixes that and also reviews all of ML
metadata in order to ensure immutability.

Closes #31916
2018-07-12 11:49:06 +01:00
Alexander Reelsen e3707efe74 Tests: Mute SearchFieldsIT.testDocValueFields()
Relates #31992
2018-07-12 12:19:39 +02:00
Martijn van Groningen ae5c70e603
muted tests due to #31940 2018-07-12 11:51:33 +02:00
Alpar Torok eb8c82a66b
Work around reported problem in eclipse (#31960)
The Gradle plugin
https://docs.gradle.org/current/userguide/java_gradle_plugin.html
added recently adds a folder to the CP, which is not created for the
eclipse import, ausing eclipse to complain.
2018-07-12 08:55:31 +00:00
Alpar Torok ba3c0f2478
Move build integration tests out of :buildSrc project (#31961)
This way building buildSrc will not be delayed for all invocations, and
will be less disruptive if integration tests fail
2018-07-12 08:41:07 +00:00
Alexander Reelsen ac4e0f1b1d
Tests: Remove use of joda time in some tests (#31922)
This also extends the dateformatters test to ensure that the printers
are acting the same in java time and joda time.
2018-07-12 09:55:17 +02:00
Tanguy Leroux 0d6b47bed9
[Test] Reactive 3rd party tests on CI (#31919)
3rd party tests are failing because the repository-s3 is expecting some 
enviromnent variables in order to test session tokens but the CI job is 
not ready yet to provide those. This pull request relaxes the constraints 
on the presence of env vars so that the 3rd party tests can still be 
executed on CI.

closes #31813
2018-07-12 09:21:10 +02:00
Costin Leau dc633e0000
SQL: Support for escape sequences (#31884)
Enhance grammar to allow JDBC/ODBC escape sequences, namely
- date, time and timestamp {d ''}, {t ''} and {ts ''}
- guid {guid ''}
- LIKE escape {escape ''}
- scalar function {fn }

Fix #31883
2018-07-11 23:36:39 +03:00
Costin Leau 6136e49a05
SQL: HAVING clause should accept only aggregates (#31872)
Improve Verifier to allow HAVING clauses only on aggregates

Close #31726
2018-07-11 23:31:46 +03:00
Jimi Ford e955ffc38d Docs: fix typo in datehistogram (#31972) 2018-07-11 15:04:57 -04:00
Nik Everett b83e99a824
Switch url repository rest tests to new style requests (#31944)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `module/repository-url` project to use the new
versions.
2018-07-11 14:52:45 -04:00
Nik Everett 939983d783
Switch reindex tests to new style requests (#31941)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `modules/reindex` project to use the new
versions.
2018-07-11 14:42:55 -04:00
Clinton Gormley aedbfc63cd Docs: Added note about cloud service to installation and getting started 2018-07-11 20:17:18 +02:00
Lisa Cawley efcfd0d827
[DOCS] Removes alternative docker pull example (#31934) 2018-07-11 09:08:32 -07:00
James Baiera 5bcdff73d7
Add Snapshots Status API to High Level Rest Client (#31515)
This PR adds the Snapshots Status API to the Snapshot Client, as 
well as additional documentation for the status api.
2018-07-11 12:07:31 -04:00
Jake Landis 51bb27a991
ingest: date_index_name processor template resolution (#31841)
This change adds support for template snippet (e.g. {{foo}}) resolution
in the date_index_name processor. The following configuration options
will now resolve a templated value if so configured:

* index_name_prefix  (e.g "index_name_prefix": "myindex-{{foo}}-")
* date_rounding (e.g. "date_rounding" : "{{bar}}")
* index_name_format (e.g."index_name_format": "{{baz}}")
2018-07-11 10:13:41 -05:00
Michael Basnight d76293f990
Test: fix null failure in watcher test (#31968)
A new commit was merged that does not allow a null attachement &&
text. This is valid for the slack API, as it does not allow this, but
our unit tests did. This commit fixes the broken unit test.

Closes #31948
2018-07-11 09:37:48 -05:00
Nik Everett 38e09a1508
Switch test framework to new style requests (#31939)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `test/framework` project to use the new
versions.
2018-07-11 10:04:17 -04:00
Nik Everett eda6d182b5
Switch low level rest tests to new style Requests (#31938)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `client/rest` project to use the new versions.
2018-07-11 09:48:47 -04:00
Nik Everett aa6a1c5ca0
Switch high level rest tests to new style requests (#31937)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `client/rest-high-level` project to use the new
versions.
2018-07-11 09:18:04 -04:00
David Kyle d268b494d7
[ML] Mute test failing due to Java 11 date time format parsing bug (#31899) 2018-07-11 10:17:44 +01:00
Yannick Welsch dadf96a840 [TEST] Mute SlackMessageTests.testTemplateRender 2018-07-11 11:12:52 +02:00
Armin Braun b4087d69d2
Fix assertIngestDocument wrongfully passing (#31913)
* Fix assertIngestDocument wrongfully passing

* Previously docA being subset of docB passed because iteration was over docA's keys only
* Scalars in nested fields were not compared in all cases
* Assertion errors were hard to interpret (message wasn't correct since it only mentioned the class type)
* In cases where two paths contained different types a ClassCastException was thrown instead of an AssertionError
* Fixes #28492
2018-07-11 10:24:21 +02:00
Christoph Büscher 4b8b831517
Remove unused reference to filePermissionsCache (#31923)
Currently Role.Builder keeps a reference to the FieldPermissionsCache that is
passed into its constructors. This seems to be unused except for passing it on
to convertFromIndicesPrivileges() in the second ctor itself, but we don't need
to keep the internal reference in that case, so it can be removed.

Relates to #31876
2018-07-11 09:56:21 +02:00
Boaz Leskes c6666fc6cb rolling upgrade should use a replica to prevent relocations while running a scroll
A scroll holds a reference to the shard store. If the cluster is moving shards
around that reference can prevent a shard from relocating back to node it used
to be on, causing test failures.

Closes #31827
2018-07-11 08:13:38 +02:00
Nik Everett 4d83a0dd5a
HLREST: Bundle the x-pack protocol project (#31904)
The `:x-pack:protocol` project is an implementation detail shared by the
xpack projects and the high level rest client and really doesn't deserve
its own maven coordinants and published javadoc. This change bundles
`:x-pack:protocol` into the high level rest client.

Relates to #29827
2018-07-10 17:59:15 -04:00
Nhat Nguyen 25cd835010 Increase logging level for testStressMaybeFlush
Relates #31629
2018-07-10 17:44:48 -04:00
Sohaib Iftikhar 88c270d844 Added lenient flag for synonym token filter (#31484)
* Added lenient flag for synonym-tokenfilter.

Relates to #30968

* added docs for synonym-graph-tokenfilter

-- Also made lenient final
-- changed from !lenient to lenient == false

* Changes after review (1)

-- Renamed to ElasticsearchSynonymParser
-- Added explanation for ElasticsearchSynonymParser::add method
-- Changed ElasticsearchSynonymParser::logger instance to static

* Added lenient option for WordnetSynonymParser

-- also added more documentation

* Added additional documentation

* Improved documentation
2018-07-10 17:11:50 -04:00
Shaunak Kashyap 3189ef49a5
[X-Pack] Beats centralized management: security role + licensing (#30520)
* Adding Beats x-pack plugin + index templates

* Adding built-in roles for Beats central management

* Fixing typo

* Refactoring: extract common code into method

* More refactoring for more code reuse

* Use a single index for Beats management

* Rename "fragment" to "block"

* Adding configuration block type

* Expand kibana_system role to include Beats management index privileges

* Fixing syntax

* Adding test

* Adding asserting for reserved role

* Fixing privileges

* Updating template

* Removing beats plugin

* Fixing tests

* Fixing role variable name

* Fixing assertions

* Switching to preferred syntax for boolean false checks

* Making class final

* Making variables final

* Updating Basic license message to be more accurate
2018-07-10 10:06:41 -07:00
Nik Everett dcbb1154bf
HLRest: Move xPackInfo() to xPack().info() (#31905)
Originally I put the X-Pack info object into the top level rest client
object. I did that because we thought we'd like to squash `xpack` from
the name of the X-Pack APIs now that it is part of the default
distribution. We still kind of want to do that, but at least for now we
feel like it is better to keep the high level rest client aligned with
the other language clients like C# and Python. This shifts the X-Pack
info API to align with its json spec file.

Relates to #31870
2018-07-10 13:01:28 -04:00
Michael Basnight e85bb734cf
Docs: add security delete role to api call table (#31907) 2018-07-10 11:17:21 -05:00
Andy Bristol 006c2c9ab0
[test] port archive distribution packaging tests (#31314)
Recreates the rest of the bats packaging tests for the tar distribution
in the java packaging test project, with support for both tar and zip
packaging, both oss and default flavors, and on Linux and Windows. Most
tests are followed fairly closely, some have either been dropped if
unnecessary or folded into others if convenient.
2018-07-10 08:21:20 -07:00