Commit Graph

40512 Commits

Author SHA1 Message Date
Jack Conradson 7a1bbbc581
Painless: Clean up PainlessCast (#32754)
Renames variables and methods in PainlessCast to go along with the new naming 
scheme. Mechanical change.
2018-08-10 09:55:38 -07:00
Albert Zaharovits 1dcf80795a
[TEST] Certificate NONE not allowed in FIPS JVM (#32753)
Certificate NONE not allowed when running in a FIPS JVM
2018-08-10 19:37:05 +03:00
Dimitris Athanasiou c7b1ba33aa
[ML] Refactor ProcessCtrl into Autodetect and Normalizer builders (#32720)
This moves the helper functionality for creating the autodetect
and mormalizer processes into corresponding builders.
2018-08-10 17:28:20 +01:00
Alpar Torok ef21ab8567
Access build tools resources (#32201)
Add a buildResources 

buildResources.take('resource') writes the resources out to the filesystem so the paths can be sent int other tools.
2018-08-10 15:03:17 +00:00
Alexander Reelsen e49717393a
Tests: Disable rolling upgrade tests with system key on fips JVM (#32775)
This disables the x-pack rolling upgrade tests using a fips JVM, as
there are problems creating the keystore.

Relates #32737
2018-08-10 16:50:11 +02:00
Nik Everett 2df80ca7e3
HLRC: Ban LoggingDeprecationHandler (#32756)
LoggingDeprecationHandler requires log4j2 but we don't require log4j2 in
the client. This bans LoggingDeprecationHandler and removes all uses of
it in the high level client.

Closes #32151
2018-08-10 10:17:48 -04:00
Christoph Büscher 22f7b03430
Fix test reproducability in AbstractBuilderTestCase setup (#32403)
Currently AbstractBuilderTestCase generates certain random values in its
`beforeTest()` method annotated with @Before only the first time that a test
method in the suite is run while initializing the serviceHolder that we use for
the rest of the test. This changes the values of subsequent random values
and has the effect that when running single methods from a test suite with
"-Dtests.method=*", the random values it sees are different from when the same
test method is run as part of the whole test suite. This makes it hard to use
the reproduction lines logged on failure.

This change runs the inialization of the serviceHolder and the randomization 
connected to it using the test runners master seed, so reproduction by running
just one method is possible again.


Closes #32400
2018-08-10 15:13:44 +02:00
Alpar Torok de2897a190 Only require java<version>_home env var if needed
We accidentally started enforcing the env vars are always there.
This reverts to only do this if they are requierd.
2018-08-10 16:10:13 +03:00
Alexander Reelsen f236bb3ff6 Tests: Muted ScriptDocValuesDatesTests.testJodaTimeBwc
Relates #32779
2018-08-10 14:38:23 +02:00
Benjamin Trent 94a9b253db
Adding ML HLRC wrapper and put_job API call (#32726)
* Adding ML HLRC wrapper and put_job API call

* Changing integration test job to have consistent stucture
2018-08-10 07:16:55 -05:00
Luca Cavanna 40b0a3a014
[DOCS] Adjust high-level client x-pack docs (#32747)
- Expose x-pack usage docs page which was not linked in supported-apis page
- make watcher a top-level dir outside of x-pack directory
- move x-pack info and usage pages to miscellaneous
- add new Watcher category to supported-apis (they were under miscellaneous)
- remove x-pack prefix from watcher docs titles
2018-08-10 14:12:42 +02:00
David Roberts ae0c303dad
Move icu4j and super-csv version numbers to versions file (#32769)
The upcoming ML log structure finder functionality will use these
libraries, and it makes sense to use the same versions that are
being used elsewhere in Elasticsearch.  This is especially true
with icu4j, which is pretty big.
2018-08-10 12:19:06 +01:00
Luca Cavanna 9561a0a66e
[DOCS] Add missing link to get license API docs (#32746) 2018-08-10 10:55:25 +02:00
Alexander Reelsen 3fa3af1d11 Tests: Mute test in KerberosRealmTests
Relates #32701
2018-08-10 10:40:37 +02:00
Boaz Leskes f58ed21720
Refactor TransportShardBulkAction to better support retries (#31821)
Processing bulk request goes item by item. Sometimes during processing, we need to stop execution and wait for a new mapping update to be processed by the node. This is currently achieved by throwing a `RetryOnPrimaryException`, which is caught higher up. When the exception is caught, we wait for the next cluster state to arrive and process the request again. Sadly this is a problem because all operations that were already done until the mapping change was required are applied again and get new sequence numbers. This in turn means that the previously issued sequence numbers are never replicated to the replicas. That causes the local checkpoint of those shards to be stuck and with it all the seq# based infrastructure.

This commit refactors how we deal with retries with the goal of removing  `RetryOnPrimaryException` and `RetryOnReplicaException` (not done yet). It achieves so by introducing a class `BulkPrimaryExecutionContext` that is used the capture the execution state and allows continuing from where the execution stopped. The class also formalizes the steps each item has to go through:
1) A translation phase for updates
2) Execution phase (always index/delete)
3) Waiting for a mapping update to come in, if needed
4) Requires a retry (for updates and cases where the mapping are still not available after the put mapping call returns)
5) A finalization phase which allows updates to the index/delete result to an update result.
2018-08-10 10:15:01 +02:00
Alpar Torok af8c23eb40
Java version reproduction (#32715)
Enhance reproduction line with info about jdks

Provide the ability to control compiler and hava versions just by
passing a property. The actual java home comes from the
`JAVA<major>_HOME` env vars that we allready require.
This works better with the Gradle daemon as well.

Output is also changed a bit.

for `-Druntime.java=8 -Dcompiler.java=9`:
```
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 4.9
  OS Info               : Linux 4.17.8-1-ARCH (amd64)
  Compiler JDK Version  : 11 (Oracle Corporation 11-ea [OpenJDK 64-Bit Server VM 11-ea+22])
  Runtime JDK Version   : 11 (Oracle Corporation 11-ea [OpenJDK 64-Bit Server VM 11-ea+22])
  Gradle JDK Version    : 10 (Oracle Corporation 10.0.1 [OpenJDK 64-Bit Server VM 10.0.1+10])
  Compiler java.home    : /home/alpar/opt/jdk-11-ea22/
  Runtime java.home     : /home/alpar/opt/jdk-11-ea22/
  Gradle java.home      : /usr/lib/jvm/java-10-openjdk
  Random Testing Seed   : EA858533191E8DFB
=======================================
```

Without configuration:
```
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
  Gradle Version        : 4.9
  OS Info               : Linux 4.17.8-1-ARCH (amd64)
  JDK Version           : 10 (Oracle Corporation 10.0.1 [OpenJDK 64-Bit Server VM 10.0.1+10])
  JAVA_HOME             : /usr/lib/jvm/java-10-openjdk
  Random Testing Seed   : 4BD5B2A839C8FCA1
=======================================
```

Here's how a reproduction line will look like (test made to fail):
```
./gradlew :modules:lang-painless:test -Dtests.seed=2DA2379065A4EEAB -Dtests.class=org.elasticsearch.painless.AdditionTests -Dtests.method="testInt" -Dtests.security.manager=true -Dtests.locale=es-PE -Dtests.timezone=WET -Dcompiler.java=10 -Druntime.java=10
```
2018-08-10 08:07:43 +00:00
Alexander Reelsen 798fb546cb
Core: Create java time based DateMathParser (#32131)
This adds a java time based date math parser class in order, which will replace the joda date based one in the future. For now the class also returns the date in milliseconds since the epoch.
2018-08-10 09:38:18 +02:00
Tim Vernum 93d4f84b21
[TEST] Add "ne" as an unsupported SimpleKdc locale (#32700)
SimpleKdcServer cannot handle some locales, so we explicitly avoid
using them in tests.
Adding "ne" (Nepali) to that list.
2018-08-10 13:36:36 +10:00
Yogesh Gaikwad 0161bbc77f
[Test] Handle file permissions for Windows (#32681)
This commit modifies the test to handle file permission
tests in windows/dos environments. The test requires access
to UserPrincipal and so have modified the plugin-security policy
to access user information.

Closes#32637
2018-08-10 09:55:43 +10:00
Jack Conradson 293c8a2b24
Painless: Add an Ingest Script Processor Example (#32302)
This commit adds two pieces. The first is a small set of documentation providing 
instructions on how to get setup to run context examples. This will require a download 
similar to how Kibana works for some of the examples. The second is an ingest processor 
example using the downloaded data. More examples will follow as ideally one per PR. 
This also adds a set of tests to individually test each script as a unit test.
2018-08-09 14:24:55 -07:00
lipsill be54ba39c4 Add expected mapping type to `MapperException` (#31564)
Currently if a document cannot be indexed because it violates the defined 
mapping for the index, a MapperException is thrown. In some cases it is 
useful to expose the expected field type in the exception itself, 
so that the user can react based on the error message. This change adds 
the expected data type to the MapperException.

Closes #31502
2018-08-09 23:10:51 +02:00
Nik Everett 294ab7ee96
Core: Remove some logging constructors (#32513)
Remove a few of the logger constructors that aren't widely used or
aren't used at all and deprecate a few more logger constructors in favor
of log4j2's `LogManager`.
2018-08-09 16:11:48 -04:00
Alpar Torok 5bbed5ed9a Merge branch 'master' of github.com:elastic/elasticsearch 2018-08-09 21:58:03 +03:00
Alpar Torok c5e66cb345 mute test #32737 2018-08-09 21:54:58 +03:00
Nik Everett abda9fdac1 Logging: Fix test on windows
Windows' `\` instead of `/` strikes again!

Closes #32546
2018-08-09 14:35:30 -04:00
Ryan Ernst 9a16491ebf
Build: Add back setting artifact id of pom for rest high level client (#32731)
This commit adds back the publishing section that sets the artifact id
of the generated pom file for the high level rest client. This was
accidentally removed during a consolidationo of the shadow plugin logic.
2018-08-09 10:11:08 -07:00
Andrei Stefan 7b618f3db4
SQL: Bug fix for the optional "start" parameter usage inside LOCATE function (#32576)
The incorrect NodeInfo is created when the optional parameter is not used, leading to the incorrect constructor being used. Simplified LocateFunctionProcessorDefinition by using one constructor instead of two.
Fixes https://github.com/elastic/elasticsearch/issues/32554
2018-08-09 19:30:02 +03:00
Andrei Stefan 6750e15773
SQL: Ignore H2 comparative tests for uppercasing/lowercasing string functions (#32604)
Skip the comparative tests using lowercasing/uppercasing against H2 (which considers the Locale).
ES-SQL is, so far, ignoring the Locale.
Still, the same queries are executed against ES-SQL alone and results asserted to be correct.
2018-08-09 19:28:48 +03:00
Nicholas Knize e162127ff3 Upgrade to Lucene-7.5.0-snapshot-13b9e28f9d
The main feature is the inclusion of bkd backed geo_shape with
INTERSECT, DISJOINT, WITHIN bounding box and polygon query support.
2018-08-09 11:15:02 -05:00
Armin Braun 79375d35bb
Scripting: Replace Update Context (#32096)
* SCRIPTING: Move Update Scripts to their own context
* Added system property for backwards compatibility of change to `ctx.params`
2018-08-09 14:32:36 +02:00
Alexander Reelsen 823d40e19b
Core: Fix Java Time DateFormatter printers (#32592)
A bug in the test suite prevented to properly check that all date
formatters printed the date the same way like joda time does.

This fixes the test and thus also a fair share of formats, that
now use the strict parser for printing.
2018-08-09 10:01:40 +02:00
Jack Conradson 9b00f095b9
Painless: Move More Logic to PainlessLookup (#32689)
This moves some run-time lookups for methods and fields to the PainlessLookup.
2018-08-08 16:25:14 -07:00
Julie Tibshirani 99c3d8ab26 Fix a bug in the BWC tests where we added a field alias in unsupported versions. 2018-08-08 15:24:23 -07:00
Lee Hinman 7af28c48c3
Switch WritePipelineResponse to AcknowledgedResponse (#32722)
We previously discussed moving the classes extending `AcknowledgedResponse` to
simply use `AcknowledgedResponse`, making the class non-abstract.

This moves the first class to do this, removing `WritePipelineResponse` in the
process.

If we like the way this looks, I will switch the remaining classes over to using
`AcknowledgedResponse`.
2018-08-08 16:21:58 -06:00
Benjamin Trent d586e4cfd3
Adding `Job` and `AnalysisConfig` for HLRC (#32687)
* Adding `Job` and `AnalysisConfig` for HLRC

* Removing println used for local debugging

* Adding null checks and removing unnecessary field
2018-08-08 11:54:11 -05:00
Jim Ferenczi f1869cca35
Fix role query that can match nested documents (#32705)
This change makes sure that the role query excludes nested documents
when it is ran to select the parent documents allowed by the role.
2018-08-08 16:36:51 +02:00
Suresh N S 7fdf898518 Whitelisting / from Circuit Breaker Exception (#32325) (#32666)
When Circuit Breaker has tripped, certain diagnostic requests like
"_cluster/health" succeed where as request to / fails with
503 Service Unavailable. This behavior is observed because of this
commit f32b700 where certain API paths are whitelisted from
Circuit Breaking exception, but / is not whitelisted.
Added / to circuit breaker whitelist so that it can be used for
diagnostic purposes
2018-08-08 08:24:53 -06:00
Armin Braun 7d641ba69b
TESTS: Explicitly Fail Http Client Timeouts (#32708)
* Don't quietly ignore timeouts when waiting for HTTP responses
* Fixes #32702
2018-08-08 15:47:51 +02:00
Colin Goodheart-Smithe 781e6ad551
Fixes suggestion generics (#32706)
* Fixes suggestion generics

This solves a compile problem in Eclipse where Eclipse could not
resolve the generics for the options field in `PhraseSuggestion.Entry`.
But I think this is also a good change in general because
`PhraseSuggestion.Entry` is now declaring the specific `Option`
implementation it requires rather than `Suggest.Entry.Option` which is
more general and could lead to weird bugs. `CompletionSuggestion.Entry`
and `TermSuggestion.Entry` already declare the more specific class they
use so I think this was an oversight in `PhaseSuggestion.Entry`

* iter
2018-08-08 12:46:38 +01:00
Armin Braun 580d59e2d7
CORE: Upgrade to Jackson 2.8.11 (#32670)
* closes #30352
2018-08-08 12:04:25 +02:00
Luca Cavanna 3e437438d5
Prevent cause from being null in ShardOperationFailedException (#32640)
`ShardOperationFailedException` and corresponding implementors seem to suggest that the cause may be null, case that is also handled in a few places. Yet, it does not seem to be possible in practice for the cause to be null, hence we can clean that up and enforce the cause to be a non null value. This is best done by making `ShardOperationFailedException` an abstract class rather than an interface, which holds the basic member instance that all the subclasses have in common and can also enforce that cause, status and reason are non null.
2018-08-08 09:59:22 +02:00
Luca Cavanna 5c2ef5e869
Preserve index_uuid when creating QueryShardException (#32677)
As part of #32608 we made sure that the fully qualified index name is taken from the query shard context whenever creating a new `QueryShardException`. That change introduced a regression as instead of setting the entire `Index` object to the exception, which holds index name and index uuid, we ended up setting only the index name (including cluster alias). With this commit we make sure that the index uuid does not get lost and we try to lower the chances that a similar bug makes it in another time. That's done by making `QueryShardContext` return the fully qualified `Index` (which also holds the uuid) rather than only the fully qualified index name.
2018-08-08 09:57:11 +02:00
Tim Vernum 4cbcc1d659 [TEST] Add empty test case for FIPS
All other tests in security:cli are skipped on FIPS JVMs
2018-08-08 13:47:18 +10:00
Tim Brooks 6a7d5bce8a
Use nio driven http client for testing (#32621)
This is related to #27260. This commit replaces the netty driven http
client (Netty4HttpClient) with one that is driven by (NioHttpClient).
This client exists in the test package and is used for making http
requests.
2018-08-07 18:04:34 -06:00
Julie Tibshirani 0281846c6b Adjust the test version now that the field alias bug has been backported. 2018-08-07 16:48:41 -07:00
Julie Tibshirani d7183f8f3d
Make sure that field collapsing supports field aliases. (#32648) 2018-08-07 16:20:09 -07:00
Andy Bristol 8bfb0f3f8d
serialize suggestion responses as named writeables (#30284)
Suggestion responses were previously serialized as streamables which
made writing suggesters in plugins with custom suggestion response types
impossible. This commit makes them serialized as named writeables and
provides a facility for registering a reader for suggestion responses
when registering a suggester.

This also makes Suggestion responses abstract, requiring a suggester
implementation to provide its own types. Suggesters which do not need
anything additional to what is defined in Suggest.Suggestion should
provide a minimal subclass.

The existing plugin suggester integration tests are removed and
replaced with an equivalent implementation as an example
plugin.
2018-08-07 13:31:00 -07:00
Jack Conradson 0b7fb4e7b9
Painless: Clean up FunctionRef (#32644)
This change consolidates all the logic for generating a FunctionReference (renamed from 
FunctionRef) from several arbitrary constructors to a single static function that is used at 
both compile-time and run-time. This increases long-term maintainability as it is much 
easier to follow when and how a function reference is being generated. It moves most of 
the duplicated logic out of the ECapturingFuncRef, EFuncRef and ELambda nodes and 
Def as well.
2018-08-07 12:26:57 -07:00
Jason Tedor dcc816427e
Expose whether or not the global checkpoint updated (#32659)
It will be useful for future efforts to know if the global checkpoint
was updated. To this end, we need to expose whether or not the global
checkpoint was updated when the state of the replication tracker
updates. For this, we add to the tracker a callback that is invoked
whenever the global checkpoint is updated. For primaries this will be
invoked when the computed global checkpoint is updated based on state
changes to the tracker. For replicas this will be invoked when the local
knowledge of the global checkpoint is advanced from the primary.
2018-08-07 15:10:09 -04:00
Tim Brooks 3d5e9114e3
Reduce connections used by MockNioTransport (#32620)
The MockNioTransport (similar to the MockTcpTransport) is used for integ
tests. The MockTcpTransport has always only opened a single for all of
its work. The MockNioTransport has awlays opened the default number of
connections (13). This means that every test where two transports
connect requires 26 connections. This is more than is necessary. This
commit modifies the MockNioTransport to only require 3 connections.
2018-08-07 12:52:28 -06:00