Commit Graph

46029 Commits

Author SHA1 Message Date
Jay Modi dbbdcea128
Update ciphers for TLSv1.3 and JDK11 if available (#42082)
This commit updates the default ciphers and TLS protocols that are used
when the runtime JDK supports them. New cipher support has been
introduced in JDK 11 and 12 along with performance fixes for AES GCM.
The ciphers are ordered with PFS ciphers being most preferred, then
AEAD ciphers, and finally those with mainstream hardware support. When
available stronger encryption is preferred for a given cipher.

This is a backport of #41385 and #41808. There are known JDK bugs with
TLSv1.3 that have been fixed in various versions. These are:

1. The JDK's bundled HttpsServer will endless loop under JDK11 and JDK
12.0 (Fixed in 12.0.1) based on the way the Apache HttpClient performs
a close (half close).
2. In all versions of JDK 11 and 12, the HttpsServer will endless loop
when certificates are not trusted or another handshake error occurs. An
email has been sent to the openjdk security-dev list and #38646 is open
to track this.
3. In JDK 11.0.2 and prior there is a race condition with session
resumption that leads to handshake errors when multiple concurrent
handshakes are going on between the same client and server. This bug
does not appear when client authentication is in use. This is
JDK-8213202, which was fixed in 11.0.3 and 12.0.
4. In JDK 11.0.2 and prior there is a bug where resumed TLS sessions do
not retain peer certificate information. This is JDK-8212885.

The way these issues are addressed is that the current java version is
checked and used to determine the supported protocols for tests that
provoke these issues.
2019-05-20 09:45:36 -04:00
Lisa Cawley fd2d4d761b [DOCS] Updates TLS configuration info (#41983) 2019-05-20 09:13:37 -04:00
Tomas Della Vedova 4e9bf3f18a Remove deprecated _source_exclude and _source_include from get API spec (#42188)
Support for these parameters was removed in #35097. The spec were left outdated.
2019-05-20 12:40:45 +02:00
Russ Cam 8f838198fa Remove parent query string parameter (#41098)
This commit removes the deprecated parent query string
parameter. The routing parameter should be used instead.
2019-05-20 12:08:02 +02:00
Jim Ferenczi b7599472ac Fix random failure in SearchRequestTests#testRandomVersionSerialization (#42069)
This commit fixes a test bug that ends up comparing the result of two consecutive calls to System.currentTimeMillis that can be different
on slow CIs.

Closes #42064
2019-05-20 10:14:05 +02:00
Nhat Nguyen 0ec7986049 Enable debug log in testRetentionLeasesSyncOnRecovery
Relates #39105
2019-05-19 22:07:25 -04:00
Nhat Nguyen 1362944c23 Minor improvement translog docs (#42184)
Closes #42183
2019-05-19 20:45:34 -04:00
Ed Savage 840af87a74 [ML] Temporarily muting failing tests
Muting a number of AutoDetectMemoryLimitIT tests to give CI a chance to
settle before easing in required backend changes.

relates elastic/ml-cpp#486
relates #42086
2019-05-19 08:29:50 -04:00
Ed Savage a68b04e47b [ML] Improve hard_limit audit message (#42086)
Improve the hard_limit memory audit message by reporting how many bytes
over the configured memory limit the job was at the point of the last
allocation failure.

Previously the model memory usage was reported, however this was
inaccurate and hence of limited use -  primarily because the total
memory used by the model can decrease significantly after the models
status is changed to hard_limit but before the model size stats are
reported from autodetect to ES.

While this PR contains the changes to the format of the hard_limit audit
message it is dependent on modifications to the ml-cpp backend to
send additional data fields in the model size stats message. These
changes will follow in a subsequent PR. It is worth noting that this PR
must be merged prior to the ml-cpp one, to keep CI tests happy.
2019-05-17 17:40:08 -04:00
Benjamin Trent f2447364fd
[ML] adds geo_centroid aggregation support to data frames (#42088) (#42094) 2019-05-17 16:51:05 -04:00
Igor Motov 076ca75ea5 SQL: Suppress geo tests failing on tr-TR locale (#42200)
Due to a bug in JTS WKT parser, JTS cannot parse most of WKT shapes if
the shape type is written in the lower case. For examples `point (1 2)`
is causing JTS inside H2GIS to fail on tr-TR locale as a result  of 
case-insensitive comparison.
2019-05-17 16:00:54 -04:00
Ryan Ernst ab7a7062ea
Make packaging tests use jdk downloader (#42097)
This commit removes the jdk11 download in vagrant provisioning and
converts it to using the jdk downloader for the system jdk, and sets up
a separate jdk for use by the test (which will be converted to running
gradle in a followup).
2019-05-17 14:49:29 -04:00
Ryan Ernst a6e63e6fa8 Protect logged exec spooling from no output (#42177)
This commit adds a guard around reading the spooled LoggedExec output.
It is possible the exec command did not output anything, and failed,
which would trigger a failure to read the output file.
2019-05-16 15:38:32 -04:00
Ryan Ernst c40bd31073 Use local outputstream reference (#42180)
This commit fixes the logging in LoggedExec which uses an in memory
buffer to read from a local reference, instead of with
getStandardOutput() of the Exec task. This is due to gradle internally
wrapping with a TeeOutputStream, breaking our cast.
2019-05-16 15:35:51 -04:00
David Turner 51376f98a7 Clarify rolling upgrade fallback to restart upgrade (#42161)
Adds a note that restarting half-or-more of the master-eligible nodes means
you're no longer doing a rolling upgrade, and may need to upgrade all the
things before the cluster returns to health.
2019-05-16 13:38:48 -04:00
David Roberts 226df35d96 [ML] Improve message misformation error in file structure finder (#42175)
This change replaces the extremely unfriendly message
"Number of messages analyzed must be positive" in the
case where the sample lines were incorrectly grouped
into just one message to an error that more helpfully
explains the likely root cause of the problem.
2019-05-16 18:29:38 +01:00
Hendrik Muhs 4063701f5e [DOCS] add a warning about bypassing PUT API's, update example responses (#42062)
Configurations are stored in the .data-frame-internal-1
index, but users should not add configurations directly to
the index as additional information to enable access control
is added. This adds a warning against allowing access to the
internal index.
2019-05-16 10:12:19 -04:00
Ryan Ernst fa1d1d1f57 Deprecate the native realm migration tool (#42142)
The migrate tool was added when the native realm was created, to aid
users in converting from file realms that were per node, into the
cluster managed native realm. While this tool was useful at the time,
users should now be using the native realm directly. This commit
deprecates the tool, to be removed in a followup for 8.0.
2019-05-16 09:52:31 -04:00
Ryan Ernst 8681dd9cba Hide bwc build output on success (#42102)
Previously we used LoggedExec for running the internal bwc builds.
However, this had bad performance implications as all the output was
buffered into memory, thus we changed back to normal Exec. This commit
adds a `spoolOutput` setting to LoggedExec which can be used for
commands with large amounts of output, and switches the bwc builds to
use this flag.
2019-05-16 09:49:23 -04:00
Nhat Nguyen 6ffc6ea42e Don't verify evictions in testFilterCacheStats (#42091)
If a background merge and refresh happens after a search but before a
stats query, then evictions will be non-zero.

Closes #32506
2019-05-15 18:17:53 -04:00
Nhat Nguyen a75e916078 Adjust load and timeout in testShrinkIndexPrimaryTerm (#42098)
This test can create and shuffle 2*(3*5*7) = 210 shards which is quite
heavy for our CI. This commit reduces the load, so we don't timeout on
CI.

Closes #28153
2019-05-15 18:17:46 -04:00
Marios Trivyzas 7473742e6e
SQL: Fix issue regarding INTERVAL * number (#42014)
Interval * integer number is a valid operation which previously was
only supported for foldables (literals) and not when a field was
involved. That was because:

1. There was no common type returned for that combination
2. The `BinaryArithmeticOperation` was permitting the multiplication
(called by fold()) but the BinaryArithmeticProcessor didn't allow it

Moreover the error message for invalid arithmetic operations was wrong
because of the issue with the overloading methods of
`LoggerMessageFormat.format`.

Fixes: #41239
Fixes: #41200
(cherry picked from commit 91039bab12d3ef27d6eac9cdc891a3b3ad0c694d)
2019-05-15 16:06:55 -04:00
Tim Vernum 9191b02213
Enforce transport TLS on Basic with Security (#42150)
If a basic license enables security, then we should also enforce TLS
on the transport interface.

This was already the case for Standard/Gold/Platinum licenses.

For Basic, security defaults to disabled, so some of the process
around checking whether security is actuallY enabled is more complex
now that we need to account for basic licenses.
2019-05-15 13:59:27 -04:00
Igor Motov 2f8c5ac6f8 Docs: Mark SQL Geo functionality as beta (#42138)
Adds beta marker to geosql documentation
2019-05-15 10:51:33 -04:00
David Turner 15fd233ae3 Minor cluster coordination docs fixes (#42111)
Fixes a typo and a badly-formatted warning.
2019-05-15 09:27:08 -04:00
Mark Vieira a8aa818e00
Cacheability improvements for thirdparty audit task (#42085) (#42151) 2019-05-15 08:11:55 -04:00
Igor Motov 70ea3cf847
SQL: Add initial geo support (#42031) (#42135)
Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. 

Queries that are supported as a result of this initial implementation

Metadata commands

- `DESCRIBE table`  - returns the correct column types `GEOMETRY` for geo shapes and geo points.
- `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions
- `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. 

Returning geoshapes and geopoints from elasticsearch

- `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console.
- `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation;

Using geopoints to elasticsearch

- The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. 
- `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query.

Limitations:

Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values.

Relates to #29872
Backport of #42031
2019-05-14 18:57:12 -05:00
Jay Modi 327f44e051
Concurrent tests wait for threads to be ready (#42083)
This change updates tests that use a CountDownLatch to synchronize the
running of threads when testing concurrent operations so that we ensure
the thread has been fully created and run by the scheduler. Previously,
these tests used a latch with a value of 1 and the test thread counted
down while the threads performing concurrent operations just waited.
This change updates the value of the latch to be 1 + the number of
threads. Each thread counts down and then waits. This means that each
thread has been constructed and has started running. All threads will
have a common start point now.
2019-05-14 16:29:52 -04:00
David Turner 367e027962 Log cluster UUID when committed (#42065)
Today we do not expose the cluster UUID in any logs by default, but it would be
useful to see it. For instance if a user starts multiple nodes as separate
clusters then they will silently remain as separate clusters even if they are
subsequently reconfigured to look like a single cluster. This change logs the
committed cluster UUID the first time the node encounters it.
2019-05-14 05:35:14 -04:00
James Rodewig 58f2e91684 [DOCS] Rewrite 'rewrite' parameter docs (#42018) 2019-05-13 08:43:12 -04:00
Yogesh Gaikwad 90dce0864a
Increase the sample space for random inner hits name generator (#42057) (#42072)
This commits changes the minimum length for inner hits
name to avoid name collision which sometimes failed the
test.
2019-05-12 10:32:02 +10:00
Andrei Stefan 912c6bdbff
Prevent order being lost for _nodes API filters (#42045) (#42089)
* Switch to using a list instead of a Set for the filters, so that the
order of these filters is kept.

(cherry picked from commit 74a743829799b64971e0ac5ae265f43f6c14e074)
2019-05-11 01:58:03 +03:00
Gordon Brown a85189a558
Remove toStepKeys from LifecycleAction (#41775)
The `toStepKeys()` method was only called in its own test case. The real
list of StepKeys that's used in action execution is generated from the
list of actual step objects returned by `toSteps()`.

This commit removes that method.
2019-05-10 16:06:42 -06:00
Nhat Nguyen c19ea0a6f1 Remove global checkpoint assertion in peer recovery (#41987)
If remote recovery copies an index commit which has gaps in sequence
numbers to a follower; then these assertions (introduced in #40823)
don't hold for follower replicas.

Closes #41037
2019-05-10 14:38:35 -04:00
Benjamin Trent febee07dcc
[ML] adding pivot.max_search_page_size option for setting paging size (#41920) (#42079)
* [ML] adding pivot.size option for setting paging size

* Changing field name to address PR comments

* fixing ctor usage

* adjust hlrc for field name change
2019-05-10 13:22:31 -05:00
Benjamin Trent 0931815355
[ML] properly nesting objects in document source (#41901) (#42077)
* [ML] properly nesting objects in document source

* Throw exception on agg extraction failure, cause it to fail df

* throwing error to stop df if unsupported agg is found
2019-05-10 13:22:12 -05:00
Ryan Ernst 9944fdf237 Don't create tempdir for cli scripts (#41913)
The elasticsearch-cli helper script does not use the tempdir created by
elasticsearch-env, yet the env script still creates it. This can lead to
lots of temp directories being created when running cli scripts in an
automated fashion. This commit passes a fake tmpdir to the env script to
avoid creation.

closes #34445
2019-05-10 11:17:12 -07:00
Ryan Ernst 2244697219 Fix debian-8 update (#42056)
On debian-8, when trying to apt-get update, it currently (sometimes)
fails on one of the extra repositories. This failure to update causes
keys to not be updated, which later can cause some packages to not
install due to lack of key verification. This commit removes the
troublesome repository before we attemp to update.

closes #42017
2019-05-10 11:07:46 -07:00
Ryan Ernst 69824ed908 Cleanup plugin bin directories (#41907)
This commit adds deletion of the bin directory to postrm cleanup. While
the package's bin files are cleaned up by the package manager, plugins
may have created subdirectories under bin. We already cleanup plugins,
but not the extra bin dirs their installation created.

closes #18109
2019-05-10 11:00:41 -07:00
Christoph Büscher 3e59c31a12 Change IndexAnalyzers default analyzer access (#42011)
Currently IndexAnalyzers keeps the three default as separate class members
although they should refer to the same analyzers held in the additional
analyzers map under the default names. This assumption should be made more
explicit by keeping all analyzers in the map. This change adapts the constructor
to check all the default entries are there and the getters to reach into the map
with the default names when needed.
2019-05-10 18:08:51 +02:00
Jason Tedor cd5f1b53e8
Remove reference to fs.data.spins in docs
We long ago removed fs.data.spins from the nodes stats. This commit
removes reference to this in the docs.
2019-05-10 11:49:01 -04:00
Jay Modi 80432a3552
Remove close method in PageCacheRecycler/Recycler (#41917)
The changes in #39317 brought to light some concurrency issues in the
close method of Recyclers as we do not wait for threads running in the
threadpool to be finished prior to the closing of the PageCacheRecycler
and the Recyclers that are used internally. #41695 was opened to
address the concurrent close issues but upon review, the closing of
these classes is not really needed as the instances should be become
available for garbage collection once there is no longer a reference to
the closed node.

Closes #41683
2019-05-10 08:56:05 -06:00
Alan Woodward 44c3418531 Simplify handling of keyword field normalizers (#42002)
We have a number of places in analysis-handling code where we check
if a field type is a keyword field, and if so then extract the normalizer rather
than pulling the index-time analyzer. However, a keyword normalizer is
really just a special case of an analyzer, so we should be able to simplify this
by setting the normalizer as the index-time analyzer at construction time.
2019-05-10 14:38:46 +01:00
Nhat Nguyen 809ed3b721 shouldRollGeneration should execute under read lock (#41696)
Translog#shouldRollGeneration should execute under the read lock since
it accesses the current writer.
2019-05-10 09:28:33 -04:00
David Turner 2a8a64d3f1 Remove extra `ms` from log message (#42068)
This log message logs a `TimeValue` which includes units, but also logs an
extra `ms`. This commit removes the extra `ms`.
2019-05-10 14:03:37 +01:00
David Turner 1be5bb5bfd Recognise direct buffers in heap size docs (#42070)
This commit slightly reworks the recommendations in the docs about setting the
heap size:

* the "rules of thumb" are actually instructions that should be followed

* the reason for setting `Xmx` to 50% of the heap size is more subtle than just
  leaving space for the filesystem cache

* it is normal to see Elasticsearch using more memory than `Xmx`

* replace `cutoff` and `limit` with `threshold` since all three terms are used
  interchangeably

* since we recommend setting `Xmx` equal to `Xms`, avoid talking about setting
  `Xmx` in isolation

Relates #41954
2019-05-10 13:56:47 +01:00
Alpar Torok db8fe1de00 Fix slow sync test clustres artifacts task (#42012)
* Fix slow sync test clustres artifacts task

The task was mistakenly adding a combinational explosion of task
actions  all doing the same thing.
With this PR this is fixed and each version - distribution pair is only
extracted once.
I appologieze for the SSD wear.

* Look for configurations on the root project
* Add dependency on configurations
* This should be a `copy` so we don't blow away all the other distros
* Don't copy example plugin build directory in integration tests
2019-05-10 14:28:36 +03:00
Armin Braun ea7db2bb6a
Fix testCloseOrDeleteIndexDuringSnapshot (#42007)
* This test was resulting in a `PARTIAL` instead of a `SUCCESS` state for
the case of closing an index during snapshotting on 7.x
  * The reason for this is the changed default behaviour regarding
waiting for active shards between 8.0 and 7.x
  * Fixed by adjusting the waiting behaviour on the close index request
in the test
* Closes #39828
2019-05-10 11:59:20 +02:00
Armin Braun dc444cef49
Fix Race in Closing IndicesService.CacheCleaner (#42016) (#42052)
* When close becomes true while the management pool is shut down, we run
into an unhandled `EsRejectedExecutionException` that fails tests
* Found this while trying to reproduce #32506
   * Running the IndexStatsIT in a loop is a way of reproducing this
2019-05-10 09:29:27 +02:00
Ryan Ernst 906999f1b6 Disable rhel8 in packaging tests (#41924)
This commit disables rhel 8 from being tested in vagrant packaging
tests. The vagrant image we use is beta release, but RHEL 8 was just
released, which has caused the package mirrors for the beta to stop
working.
2019-05-09 23:42:53 -07:00