Commit Graph

147 Commits

Author SHA1 Message Date
Alexander Reelsen 9493c145d7 Upgrade jcodings dependency to 1.0.44 (#43334) 2019-06-26 10:03:40 +02:00
Yogesh Gaikwad ca43cdf755
Fix for PemTrustConfigTests.testTrustConfigReloadsFileContents failure (#43539) (#43613)
The test `PemTrustConfigTests.testTrustConfigReloadsFileContents` failed
intermittently with `ArrayIndexOutOfBoundsException` while parsing
the randomly generated bytes array representing DER encoded stream.
This seems to be a bug in JDK (once confirmed we can raise the bug
in JDK bugs system).

The problem arises when the `X509Factory#parseX509orPKCS7()` tries to
[create `PKCS7` block](19fb8f93c5/src/java.base/share/classes/sun/security/provider/X509Factory.java (L460)) from der encoded stream. While constructing PKCS7
block it tries to create `ContentInfo` type but fails to do so for the
stream where the length after the DER SEQUENCE is 0.
`DerInputStream#getSequence` [may return empty array of `DerValue`](19fb8f93c5/src/java.base/share/classes/sun/security/util/DerInputStream.java (L409..L412)) but
[the code in `ContentInfo`](19fb8f93c5/src/java.base/share/classes/sun/security/pkcs/ContentInfo.java (L135)) does not check for the empty thereby throwing
`ArrayIndexOutOfBoundsException`.

Closes #42509
2019-06-26 13:32:01 +10:00
Lee Hinman 8927081981
[7.x] Add TimeValue.toHumanReadableString() to allow specifyin… (#43545)
* Enhance TimeValue.toString() to allow specifying fractional values.

This enhances the `TimeValue` class to allow specifying the number of
truncated fractional decimals when calling `toString()`. The default
remains 1, however, more (or less, such as 0) can be specified to change
the output.

This commit also re-organizes some things in `TimeValue` such as putting
all the class variables near the top of the class, and moving the
constructors to the first methods in the class, in order to follow the
structure of our other code.

* Rename `toString(...)` to `toHumanReadableString(...)`
2019-06-25 14:06:23 -06:00
Przemysław Witek 76a750a0a0
Remove unused mapStringsOrdered method (#42513) (#43585) 2019-06-25 20:43:38 +02:00
Przemysław Witek c702cd7415
[7.x] Implement XContentParser.genericMap and XContentParser.genericMapOrdered methods (#42059) (#43575) 2019-06-25 16:04:54 +02:00
Tim Brooks 38516a4dd5
Move nio ip filter rule to be a channel handler (#43507)
Currently nio implements ip filtering at the channel context level. This
is kind of a hack as the application logic should be implemented at the
handler level. This commit moves the ip filtering into a channel
handler. This requires adding an indicator to the channel handler to
show when a channel should be closed.
2019-06-24 10:03:24 -06:00
Armin Braun 5f87caa54c
Assert ServerSocketChannel is not Blocking (#43479) (#43488)
* Assert ServerSocketChannel is not Blocking

* Relates #43387 which appears to run into blocking accept calls
2019-06-21 21:28:58 +02:00
sandmannn cf610b5e81 Added parsing of erroneous field value (#42321) 2019-06-20 15:24:04 -04:00
Igor Motov 9f7d1ff2de Geo: Add coerce support to libs/geo WKT parser (#43273)
Adds support for coercing not closed polygons and ignoring Z value
to libs/geo WKT parser.

Closes #43173
2019-06-18 14:41:01 -04:00
Julie Tibshirani 4b1d8e4433 Allow big integers and decimals to be mapped dynamically. (#42827)
This PR proposes to model big integers as longs (and big decimals as doubles)
in the context of dynamic mappings.

Previously, the dynamic mapping logic did not recognize big integers or
decimals, and would an error of the form "No matching token for number_type
[BIG_INTEGER]" when a dynamic big integer was encountered. It now accepts these
numeric types and interprets them as 'long' and 'double' respectively. This
allows `dynamic_templates` to accept and and remap them as another type such as
`keyword` or `scaled_float`.

Addresses #37846.
2019-06-14 10:05:11 -07:00
Jason Tedor 5f7d5920d7
Fix IOUtils#fsync on Windows fsyncing directories (#43008)
Fsyncing directories on Windows is not possible. We always suppressed
this by allowing that an AccessDeniedException is thrown when attemping
to open the directory for reading. Yet, this suppression also allowed
other IOExceptions to be suppressed, and that was a bug (e.g., the
directory not existing, or a filesystem error and reasons that we might
get an access denied there, like genuine permissions issues). This
leniency was previously removed yet it exposed that we were suppressing
this case on Windows. Rather than relying on exceptions for flow control
and continuing to suppress there, we simply return early if attempting
to fsync a directory on Windows (we will not put this burden on the
caller).
2019-06-07 23:00:26 -04:00
Jason Tedor 45bbd7f7f1
Only ignore IOException when fsyncing on dirs (#42972)
Today in the method IOUtils#fsync we ignore IOExceptions when fsyncing a
directory. However, the catch block here is too broad, for example it
would be ignoring IOExceptions when we try to open a non-existant
file. This commit addresses that by scoping the ignored exceptions only
to the invocation of FileChannel#force.
2019-06-07 08:35:09 -04:00
Tim Brooks 667c613d9e
Remove `nonApplicationWrite` from `SSLDriver` (#42954)
Currently, when the SSLEngine needs to produce handshake or close data,
we must manually call the nonApplicationWrite method. However, this data
is only required when something triggers the need (starting handshake,
reading from the wire, initiating close, etc). As we have a dedicated
outbound buffer, this data can be produced automatically. Additionally,
with this refactoring, we combine handshake and application mode into a
single mode. This is necessary as there are non-application messages that
are sent post handshake in TLS 1.3. Finally, this commit modifies the
SSLDriver tests to test against TLS 1.3.
2019-06-06 17:44:40 -04:00
Albert Zaharovits 72eb9c2d44
Eclipse libs projects setup fix (#42852)
Fallout from #42773 for eclipse users.

(cherry picked from commit 998419c49fe51eb8343664a80f07d8d8d39abc6a)
2019-06-04 13:52:41 -07:00
Mark Vieira e44b8b1e2e
[Backport] Remove dependency substitutions 7.x (#42866)
* Remove unnecessary usage of Gradle dependency substitution rules (#42773)

(cherry picked from commit 12d583dbf6f7d44f00aa365e34fc7e937c3c61f7)
2019-06-04 13:50:23 -07:00
Alan Woodward ab6b86bac9 Add option to ObjectParser to consume unknown fields (#42491)
ObjectParser has two ways of dealing with unknown fields: ignore them entirely,
or throw an error. Sometimes it can be useful instead to gather up these unknown
fields and record them separately, for example as arbitrary entries in a map.

This commit adds the ability to specify an unknown field consumer on an ObjectParser,
called with the field name and parsed value of each unknown field encountered during
parsing. The public API of ObjectParser is largely unchanged, with a single new
constructor method and interface definition.
2019-05-31 11:33:47 +01:00
Mark Vieira c1816354ed
[Backport] Improve build configuration time (#42674) 2019-05-30 10:29:42 -07:00
Igor Motov d2f9ccbe18 Geo: Refactor libs/geo parsers (#42549)
Refactors the WKT and GeoJSON parsers from an utility class into an
instantiatable objects. This is a preliminary step in
preparation for moving out coordinate validators from Geometry
constructors. This should allow us to make validators plugable.
2019-05-29 20:07:27 -04:00
Marios Trivyzas 56677f69cf Mute testTrustConfigReloadsFileContents
Tracked by #42509
2019-05-24 14:03:46 +02:00
David Roberts 14f29de2a8 Avoid HashMap construction on Grok non-match (#42444)
This change moves the construction of the result
HashMap in Grok.captures() into the branch that
actually needs it.

This probably will not make a measurable difference
for ingest pipelines, but it is beneficial to the
ML find_file_structure endpoint, as it tries out
many Grok patterns that will fail to match.
2019-05-23 21:09:33 +01:00
Jay Modi dbbdcea128
Update ciphers for TLSv1.3 and JDK11 if available (#42082)
This commit updates the default ciphers and TLS protocols that are used
when the runtime JDK supports them. New cipher support has been
introduced in JDK 11 and 12 along with performance fixes for AES GCM.
The ciphers are ordered with PFS ciphers being most preferred, then
AEAD ciphers, and finally those with mainstream hardware support. When
available stronger encryption is preferred for a given cipher.

This is a backport of #41385 and #41808. There are known JDK bugs with
TLSv1.3 that have been fixed in various versions. These are:

1. The JDK's bundled HttpsServer will endless loop under JDK11 and JDK
12.0 (Fixed in 12.0.1) based on the way the Apache HttpClient performs
a close (half close).
2. In all versions of JDK 11 and 12, the HttpsServer will endless loop
when certificates are not trusted or another handshake error occurs. An
email has been sent to the openjdk security-dev list and #38646 is open
to track this.
3. In JDK 11.0.2 and prior there is a race condition with session
resumption that leads to handshake errors when multiple concurrent
handshakes are going on between the same client and server. This bug
does not appear when client authentication is in use. This is
JDK-8213202, which was fixed in 11.0.3 and 12.0.
4. In JDK 11.0.2 and prior there is a bug where resumed TLS sessions do
not retain peer certificate information. This is JDK-8212885.

The way these issues are addressed is that the current java version is
checked and used to determine the supported protocols for tests that
provoke these issues.
2019-05-20 09:45:36 -04:00
Tim Brooks 927013426a
Read multiple TLS packets in one read call (#41820)
This is related to #27260. Currently we have a single read buffer that
is no larger than a single TLS packet. This prevents us from reading
multiple TLS packets in a single socket read call. This commit modifies
our TLS work to support reading similar to the plaintext case. The data
will be copied to a (potentially) recycled TLS packet-sized buffer for
interaction with the SSLEngine.
2019-05-06 09:51:32 -06:00
Tim Brooks b4bcbf9f64
Support http read timeouts for transport-nio (#41466)
This is related to #27260. Currently there is a setting
http.read_timeout that allows users to define a read timeout for the
http transport. This commit implements support for this functionality
with the transport-nio plugin. The behavior here is that a repeating
task will be scheduled for the interval defined. If there have been
no requests received since the last run and there are no inflight
requests, the channel will be closed.
2019-05-02 09:48:52 -06:00
Tim Brooks df3ef66294
Remove dedicated SSL network write buffer (#41654)
This is related to #27260. Currently for the SSLDriver we allocate a
dedicated network write buffer and encrypt the data into that buffer one
buffer at a time. This requires constantly switching between encrypting
and flushing. This commit adds a dedicated outbound buffer for SSL
operations that will internally allocate new packet sized buffers as
they are need (for writing encrypted data). This allows us to totally
encrypt an operation before writing it to the network. Eventually it can
be hooked up to buffer recycling.

This commit also backports the following commit:

Handle WRAP ops during SSL read

It is possible that a WRAP operation can occur while decrypting
handshake data in TLS 1.3. The SSLDriver does not currently handle this
well as it does not have access to the outbound buffer during read call.
This commit moves the buffer into the Driver to fix this issue. Data
wrapped during a read call will be queued for writing after the read
call is complete.
2019-04-29 17:59:13 -06:00
Igor Motov 10ab838106
Geo: Add GeoJson parser to libs/geo classes (#41575) (#41657)
Adds GeoJson parser for Geometry classes defined in libs/geo.

Relates #40908 and #29872
2019-04-29 19:43:31 -04:00
Nick Knize 113b24be4b Refactor GeoHashUtils (#40869)
This commit refactors GeoHashUtils class into a new Geohash utility class located in the ES geo library. The intent is to not only better control what geo methods are whitelisted for painless scripting but to clean up the geo utility API in general.
2019-04-26 10:06:36 -05:00
Tim Vernum 13fa72cae3
Fix broken test on FIPS for specific seed (#41230)
Under random seed 4304ED44CB755610 the generated byte pattern causes
BC-FIPS to throw

    java.io.IOException: DER length more than 4 bytes: 101

Rather than simply returning an empty list (as it does for most random
values).

Backport of: #40939
2019-04-26 15:43:48 +10:00
Tim Brooks 1f8ff052a1
Revert "Remove dedicated SSL network write buffer (#41283)"
This reverts commit f65a86c258.
2019-04-25 18:39:25 -06:00
Tim Brooks f65a86c258
Remove dedicated SSL network write buffer (#41283)
This is related to #27260. Currently for the SSLDriver we allocate a
dedicated network write buffer and encrypt the data into that buffer one
buffer at a time. This requires constantly switching between encrypting
and flushing. This commit adds a dedicated outbound buffer for SSL
operations that will internally allocate new packet sized buffers as
they are need (for writing encrypted data). This allows us to totally
encrypt an operation before writing it to the network. Eventually it can
be hooked up to buffer recycling.
2019-04-25 14:30:54 -06:00
Christoph Büscher 52495843cc [Docs] Fix common word repetitions (#39703) 2019-04-25 20:47:47 +02:00
Ryan Ernst 7e3875d781 Upgrade hamcrest to 2.1 (#41464)
hamcrest has some improvements in newer versions, like FileMatchers
that make assertions regarding file exists cleaner. This commit upgrades
to the latest version of hamcrest so we can start using new and improved
matchers.
2019-04-24 23:40:03 -07:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Henning Andersen 14ee3d3f95 Unmute and fix testSubParserArray (#40626)
testSubParserArray failed, fixed and improved to not always have an
object as outer-level inside array.

Closes #40617
2019-03-29 17:39:12 +01:00
Henning Andersen 92d07e9377 Geo Point parse error fix (#40447)
When geo point parsing threw a parse exception, it did not consume
remaining tokens from the parser. This in turn meant that
indexing documents with malformed geo points into mappings with
ignore_malformed=true would fail in some cases, since DocumentParser
expects geo_point parsing to end on the END_OBJECT token.

Related to #17617
2019-03-29 17:39:12 +01:00
David Turner 1a3916a8de Optimise rejection of out-of-range `long` values (#40325)
Today if you try and insert a very large number like `1e9999999` into a long
field we first construct this number as a `BigDecimal`, convert this to a
`BigInteger` and then reject it because it is out of range. Unfortunately
making such a large `BigInteger` is rather expensive.

We can avoid this expense by performing a (weaker) range check on the
`BigDecimal` representation of incoming `long`s too.

Relates #26137
Closes #40323
2019-03-28 12:27:34 +00:00
Mayya Sharipova 49a7c6e0e8
Expose proximity boosting (#39385) (#40251)
Expose DistanceFeatureQuery for geo, date and date_nanos types

Closes #33382
2019-03-20 09:24:41 -04:00
Igor Motov 4a42e408c5 GEO: Add support for z values to libs/geo classes (#38921)
Adds support for z-values to all Geometry objects in the
libs/geo library.
2019-03-13 15:36:03 -04:00
Tim Brooks 5612ed97ca
Add log warnings for long running event handling (#39729)
Recently we have had a number of test issues related to blocking
activity occuring on the io thread. This commit adds a log warning for
when handling event takes a >150 milliseconds. This is implemented
for the MockNioTransport which is the transport used in
ESIntegTestCase.
2019-03-08 13:07:24 -07:00
Alpar Torok 813351fe26 Un-mute and fix BuildExamplePluginsIT (#38899)
* Un-mute and fix BuildExamplePluginsIT

There doesn't seem to be anything wrong with the test iteself.
I think the failure were CI performance related, but while it was muted,
some failures managed to sneak in.

Closes #38784

* PR review
2019-03-04 08:50:55 +02:00
Armin Braun da9190be0a
Add Checks for Closed Channel in Selector Loop (#39096) (#39439)
* A few warnings could be observed in test logs about `NoSuchElementException` being thrown in `InboundChannelBuffer#sliceBuffersTo`.
These were the result of calls to this method after the relevant channel and hence the buffer was closed already as a result of a failed IO operation.
  * Fixed by adding the necessary guard statements to break out in these cases. I don't think there is a need here to do any additional error handling since `eventHandler.postHandling(channelContext);` at the end of the `processKey`
call in the main selection loop handles closing channels and invoking callbacks for writes that failed to go through already.
2019-02-27 11:28:30 +01:00
Albert Zaharovits ca630bbe6f Fix DissectParserTests expecting unique keys (#39262)
Fixes a bug in DissectParserTests where the tests expected dissect
keys to be unique but were not.

Closes #39244
2019-02-22 17:16:24 +02:00
Albert Zaharovits 08ad740d48 Mute test (#39248)
Mute test DissectParserTests.testBasicMatchUnicode
2019-02-21 17:36:01 +02:00
Albert Zaharovits 5c30446bd0 Fix libs:ssl-config project setup (#39074)
The build script file for the `:libs:elasticsearch-ssl-config` and
`:libs:ssl-config-tests` projects was incorrectly named `eclipse.build.gradle`
 while the expected name was `eclipse-build.gradle`.
In addition, this also adds a missing snippet in the `build.gradle` conf file,
that fixes the project setup for Eclipse users.
2019-02-19 02:23:11 +02:00
Tim Vernum 8895befe51
Generate mvn pom for ssl-config library (#39026)
This is used by the reindex-client library which is published to maven

Relates: #37287, #37527
Backport of: #39019
2019-02-18 20:07:22 +11:00
austintp 8ebff0512b Updates the grok patterns to be consistent with logstash (#27181) 2019-02-05 12:37:02 -06:00
Jay Modi 2ca22209cd
Enable TLSv1.3 by default for JDKs with support (#38103)
This commit enables the use of TLSv1.3 with security by enabling us to
properly map `TLSv1.3` in the supported protocols setting to the
algorithm for a SSLContext. Additionally, we also enable TLSv1.3 by
default on JDKs that support it.

An issue was uncovered with the MockWebServer when TLSv1.3 is used that
ultimately winds up in an endless loop when the client does not trust
the server's certificate. Due to this, SSLConfigurationReloaderTests
has been pinned to TLSv1.2.

Closes #32276
2019-02-01 08:34:11 -07:00
Alpar Torok d417997aca
Fix eclipse config for ssl-config (#38096) 2019-02-01 10:47:54 +02:00
Henning Andersen 68ed72b923
Handle scheduler exceptions (#38014)
Scheduler.schedule(...) would previously assume that caller handles
exception by calling get() on the returned ScheduledFuture.
schedule() now returns a ScheduledCancellable that no longer gives
access to the exception. Instead, any exception thrown out of a
scheduled Runnable is logged as a warning.

This is a continuation of #28667, #36137 and also fixes #37708.
2019-01-31 17:51:45 +01:00
Igor Motov 23805fa41a
Geo: Fix Empty Geometry Collection Handling (#37978)
Fixes handling empty geometry collection and re-enables
testParseGeometryCollection test.

Fixes #37894
2019-01-30 09:20:30 -05:00
markharwood 1579ac032b
Added missing eclipse-build.gradle files (#37980)
Eclipse build files were missing so .eclipse project files were not being generated.

Closes #37973
2019-01-29 16:43:24 +00:00