Commit Graph

40020 Commits

Author SHA1 Message Date
Tal Levy 27d663b64b
Revert "Introduce a Hashing Processor (#31087)" (#32178)
This reverts commit 8c78fe7114.
2018-07-18 16:44:05 -07:00
Michael Basnight 63749498ea Remove empty @return from JavaDoc 2018-07-18 13:28:25 -05:00
Tim Brooks 90fcb38448
Adjust SSLDriver behavior for JDK11 changes (#32145)
This is related to #32122. A number of things changed related to adding
TLS 1.3 support in JDK11. Some exception messages and other SSLEngine
behavior changed. This commit fixes assertions on exception messages.
Additionally it identifies two bugs related to how the SSLDriver behaves
in regards to JDK11 changes. Finally, it mutes a tests until correct
behavior can be identified. There is another open issue for that muted
test (#32144).
2018-07-18 11:49:42 -06:00
Andy Bristol e20f59aa71
[test] use randomized runner in packaging tests (#32109)
Use the randomized runner from the test framework and add some basic
logging to make the packaging tests behave more similarly to how we use
junit in the rest of the project
2018-07-18 10:26:26 -07:00
Julie Tibshirani 15ff3da653
Add support for field aliases. (#32172)
* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)
2018-07-18 09:33:09 -07:00
Jack Conradson 605dc49c48
Painless: Fix caching bug and clean up addPainlessClass. (#32142)
This change cleans up the addPainlessClass methods by doing the following things:

* Rename many variable names to match the new conventions described in the JavaDocs 
for PainlessLookup
* Decouples Whitelist.Class from adding a PainlessClass directly
* Adds a second version of addPainlessClass that is intended for use to add future 
defaults in a follow PR

This change also fixes the method and field caches by storing Classes instead of Strings 
since it would technically be possible now that the whitelists are extendable to have 
different Classes with the same name. It was convenient to add this change together 
since some of the new constants are shared.

Note the changes are largely mechanical again where all the code behavior should 
remain the same.
2018-07-18 09:29:52 -07:00
Alan Woodward cfb30144c9
Call setReferences() on custom referring tokenfilters in _analyze (#32157)
When building custom tokenfilters without an index in the _analyze endpoint,
we need to ensure that referring filters are correctly built by calling
their #setReferences() method

Fixes #32154
2018-07-18 14:43:20 +01:00
Armin Braun 6de1f96cad
Fix BwC Tests looking for UUID Pre 6.4 (#32158)
* UUID field was added for #31791 and only went into 6.4 and 7.0
* Fixes #32119
2018-07-18 15:32:36 +02:00
David Turner 380b45b965
Improve docs for search preferences (#32159)
Today it is unclear what guarantees are offered by the search preference
feature, and we claim a guarantee that is stronger than what we really offer:

> A custom value will be used to guarantee that the same shards will be used
> for the same custom value.

This commit clarifies this documentation.

Forward-port of #32098 to `master`.
2018-07-18 12:58:17 +01:00
Martijn van Groningen 53ab470264
use before instead of onOrBefore 2018-07-18 13:33:57 +02:00
Martijn van Groningen 1924f5d07c
Add more contexts to painless execute api (#30511)
This change adds two contexts the execute scripts against:

* SEARCH_SCRIPT: Allows to run scripts in a search script context.
This context is used in `function_score` query's script function,
script fields, script sorting and `terms_set` query.

* FILTER_SCRIPT: Allows to run scripts in a filter script context.
This context is used in the `script` query.

In both contexts a index name needs to be specified and a sample document.
The document is needed to create an in-memory index that the script can
access via the `doc[...]` and other notations. The index name is needed
because a mapping is needed to index the document.

Examples:

```
POST /_scripts/painless/_execute
{
  "script": {
    "source": "doc['field'].value.length()"
  },
  "context" : {
    "search_script": {
      "document": {
        "field": "four"
      },
      "index": "my-index"
    }
  }
}
```

Returns:

```
{
  "result": 4
}
```

POST /_scripts/painless/_execute
{
  "script": {
    "source": "doc['field'].value.length() <= params.max_length",
    "params": {
      "max_length": 4
    }
  },
  "context" : {
    "filter_script": {
      "document": {
        "field": "four"
      },
      "index": "my-index"
    }
  }
}

Returns:

```
{
  "result": true
}
```

Also changed PainlessExecuteAction.TransportAction to use TransportSingleShardAction
instead of HandledAction, because now in case score or filter contexts are used
the request needs to be redirected to a node that has an active IndexService
for the index being referenced (a node with a shard copy for that index).
2018-07-18 12:42:07 +02:00
Vladimir Dolzhenko 8235b254ab
Add EC2 credential test for repository-s3 (#31918)
Add EC2 credential test for repository-s3

Relates to #26913
2018-07-18 12:18:00 +02:00
Boaz Leskes 5856c396dd
A replica can be promoted and started in one cluster state update (#32042)
When a replica is fully recovered (i.e., in `POST_RECOVERY` state) we send a request to the master
to start the shard. The master changes the state of the replica and publishes a cluster state to that
effect. In certain cases, that cluster state can be processed on the node hosting the replica
*together* with a cluster state that promotes that, now started, replica to a primary. This can
happen due to cluster state batched processing or if the master died after having committed the
cluster state that starts the shard but before publishing it to the node with the replica. If the master
also held the primary shard, the new master node will remove the primary (as it failed) and will also
immediately promote the replica (thinking it is started). 

Sadly our code in IndexShard didn't allow for this which caused [assertions](13917162ad/server/src/main/java/org/elasticsearch/index/seqno/ReplicationTracker.java (L482)) to be tripped in some of our tests runs.
2018-07-18 11:30:44 +02:00
Christoph Büscher ef5e8d8d8a Fix Java 11 javadoc compile problem
Java 11 complains with a "type arguments not allowed here" error when types are
used in javadoc links it seems. Simply removing it.
2018-07-18 10:36:31 +02:00
Alpar Torok 69cbdc47eb
Fix CP for namingConventions when gradle home has spaces (#31914)
* Fix CP for namingConventions when gradle home has spaces

Closes #31736.

Probably not Windows specific, just not common to have spaces on Linux.
2018-07-18 08:23:59 +00:00
Christoph Büscher 15f95a9f93
Fix `range` queries on `_type` field for singe type indices (#31756)
With the introduction of single types in 6.x, the `_type` field is no longer
indexed, which leads to certain queries that were working before throw errors
now. One such query is the `range` query, that, if performed on a single typer
index, currently throws an IAE since the field is not indexed.
This change adds special treatment for this case in the TypeFieldMapper,
comparing the range queries lower and upper bound to the one existing type and
either returns a MatchAllDocs or a MatchNoDocs query.

Relates to #31632
Closes #31476
2018-07-18 09:12:28 +02:00
Toby McLaughlin 487cfc3b3f
[DOCS] Update TLS on Docker for 6.3 (#32114)
Remove references to the `platinum` image and add a self-generated trial
licence to the example for TLS on Docker.

Fixes elastic/elasticsearch-docker#176
2018-07-18 17:07:31 +10:00
Boaz Leskes 93d7468f3a ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are
Sometimes we have a test failure that hits an `UnsupportedOperationException` in this infrastructure. When
debugging you want to know what caused this unexpected failure, but right now we're silent about it. This
commit adds some information to the `UnsupportedOperationException`

Relates to #32127
2018-07-18 08:49:16 +02:00
Nhat Nguyen df1380b8d3
Remove versionType from translog (#31945)
With the introduction of sequence number, we no longer use versionType to
resolve out of order collision in replication and recovery requests.

This PR removes removes the versionType from translog. We can only remove
it in 7.0 because it is still required in a mixed cluster between 6.x and 5.x.
2018-07-17 21:59:48 -04:00
Nik Everett 351bbb8906
Switch distribution to new style Requests (#30595)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `distribution/archives/integ-test-zip` project
to use the new versions.
2018-07-17 20:25:27 -04:00
Nik Everett 91d8371325 Build: Skip jar tests if jar disabled
The shadow plugin disables the jar task but we still attempted to extract
the jar to see if it had the right license and notice file. This skips
the extraction and those tests if the jar is built for any reason which
fixes projects that use the shadow plugin.
2018-07-17 20:14:51 -04:00
Jack Conradson 03c16cd0e3
Painless: Add PainlessClassBuilder (#32141)
Several pieces of data in PainlessClass cannot be passed in at the time the 
PainlessClass is created so it must be "frozen" after all the data is collected. This means 
PainlessClass is currently serving two functions as both a builder and a set of data. This 
separates the two pieces into clearly distinct values.

This change also removes the PainlessMethodKey in favor of a simple String. The goal is 
to have the painless method key be completely internal to the PainlessLookup eventually 
and this simplifies the way there. Note that this was added since PainlessClass and 
PainlessClassBuilder were already being changed instead of a follow up PR.
2018-07-17 13:54:49 -07:00
Ryan Ernst 6371d51866
Build: Make additional test deps of check (#32015)
This commit moves additional unit test runners from being dependencies
of the test task to dependencies of check. Without this change,
reproduce lines are incorrect due to the additional test runner not
matching any of the reproduce class/method info.

closes #31964
2018-07-17 13:14:46 -07:00
Jason Tedor d170ab3315
Disable C2 from using AVX-512 on JDK 10 (#32138)
The C2 compiler in JDK 10 appears to have an issue compiling to AVX-512
instructions (on hardware that supports such). As a workaround, this
commit adds a JVM flag on JDK 10+ to disable the use of AVX-512
instructions until a fix is introduced to the JDK. Instead, we use a
flag to enable AVX and AVX2 only.

Note: Based on my reading of the C2 code, this flag does not appear to
have any impact on hardware that does not support AVX2. I have tested
this manually on an Intel Atom C2538 processor that supports neither AVX
nor AVX2. I have also tested this manually on an Intel i5-3317U
processor that supports AVX but not AVX2.
2018-07-17 15:49:20 -04:00
Nik Everett 1b97652a4c
Build: Move shadow customizations into common code (#32014)
Moves the customizations to the build to produce nice shadow jars and
javadocs into common build code, mostly BuildPlugin with a little into
the root build.gradle file. This means that any project that applies the
shadow plugin will automatically be set up just like the high level rest
client:
* The non-shadow jar will not be built
* The shadow jar will not have a "classifier"
* Tests will run against the shadow jar
* Javadoc will include all of the shadowed classes
* Service files in `META-INF/services` will be merged
2018-07-17 14:20:41 -04:00
Jack Conradson 1c63eb1081
Painless: Fix Bug with Duplicate PainlessClasses (#32110)
When building the PainlessMethods and PainlessFields they stored a reference to a 
PainlessClass. This reference was prior to "freezing" the PainlessClass so the data was 
both incomplete and mutable. This has been replaced with a target java class instead 
since the PainlessClass is accessible through a java class now and it requires no special 
modifications to get around a chicken and egg issue.
2018-07-17 10:33:38 -07:00
Ioannis Kakavas 9cdbd1bd85 Remove empty @param from Javadoc 2018-07-17 20:25:38 +03:00
Andy Bristol 0f7d2044c3 Re-disable packaging tests on suse boxes
This reverts commit 14d7e2c7b2.
2018-07-17 09:15:11 -07:00
aptxx efb4e97cfb Docs: Fix missing example script quote (#32010) 2018-07-17 17:42:24 +02:00
David Kyle 2bd40a7927
[ML] Wait for aliases in multi-node tests (#32086) 2018-07-17 16:21:53 +01:00
David Roberts 99c2a82c04
[ML] Move analyzer dependencies out of categorization config (#32123)
The ML config classes will shortly be moved to the X-Pack protocol
library to allow the ML APIs to be moved to the high level REST
client.  Dependencies on server functionality should be removed
from the config classes before this is done.

This change is entirely about moving code between packages.  It
does not add or remove any functionality or tests.
2018-07-17 15:01:12 +01:00
Nhat Nguyen ef81c1df57
Ensure to release translog snapshot in primary-replica resync (#32045)
Previously we create a translog snapshot inside the resync method, 
and that snapshot will be closed by the resync listener. However, if 
the resync method throws an exception before the resync listener 
is initialized, the translog snapshot won't be released.

Closes #32030
2018-07-17 09:41:34 -04:00
Armin Braun ed3b44fb4c
Handle TokenizerFactory TODOs (#32063)
* Don't replace Replace TokenizerFactory with Supplier, this approach was rejected in #32063 
* Remove unused parameter from constructor
2018-07-17 14:14:02 +02:00
markharwood a7e477126f
Relax TermVectors API to work with textual fields other than TextFieldType (#31915)
This changes the field-eligibility test to check one level up in the class hierarchy to allow any subclasses of StringFieldType.
Closes #31902
2018-07-17 13:11:10 +01:00
Alpar Torok b43fe560a4
Updates the build to gradle 4.9 (#32087)
There are fixes to the dependency report, most importantly for us,
it still works even if `failOnVersionConflict` would fail the build.
2018-07-17 11:41:31 +00:00
David Kyle b655c11dbe Mute :qa:mixed-cluster indices.stats/10_index/Index - all’ 2018-07-17 10:34:56 +01:00
Luca Cavanna 049966a829
Check that client methods match API defined in the REST spec (#31825)
We have been encountering name mismatches between API defined in our
REST spec and method names that have been added to the high-level REST
client. We should check this automatically to prevent furher mismatches,
and correct all the current ones.

This commit adds a test for this and corrects the issues found by it.
2018-07-17 11:26:28 +02:00
Ioannis Kakavas 9e529d9d58
Enable testing in FIPS140 JVM (#31666)
Ensure our tests can run in a FIPS JVM

JKS keystores cannot be used in a FIPS JVM as attempting to use one
in order to init a KeyManagerFactory or a TrustManagerFactory is not
allowed.( JKS keystore algorithms for private key encryption are not
FIPS 140 approved)
This commit replaces JKS keystores in our tests with the
corresponding PEM encoded key and certificates both for key and trust
configurations.
Whenever it's not possible to refactor the test, i.e. when we are
testing that we can load a JKS keystore, etc. we attempt to
mute the test when we are running in FIPS 140 JVM. Testing for the
JVM is naive and is based on the name of the security provider as
we would control the testing infrastrtucture and so this would be
reliable enough.
Other cases of tests being muted are the ones that involve custom
TrustStoreManagers or KeyStoreManagers, null TLS Ciphers and the
SAMLAuthneticator class as we cannot sign XML documents in the
way we were doing. SAMLAuthenticator tests in a FIPS JVM can be
reenabled with precomputed and signed SAML messages at a later stage.

IT will be covered in a subsequent PR
2018-07-17 10:54:10 +03:00
Christoph Büscher 36165265ce
Fix put mappings java API documentation (#31955)
The current docs of the put-mapping Java API is currently broken. It its current
form, it creates an index and uses the whole mapping definition given as a JSON
string as the type name. Since we didn't check the index created in the
IndicesDocumentationIT so far this went unnoticed.

This change adds test to catch this error to the documentation test, changes the
documentation so it works correctly now and adds an input validation to
PutMappingRequest#buildFromSimplifiedDef() which was used internally to reject
calls where no mapping definition is given.

Closes #31906
2018-07-17 09:09:03 +02:00
Christoph Büscher 61486680a2
Add exclusion option to `keep_types` token filter (#32012)
Currently the `keep_types` token filter includes all token types specified using
its `types` parameter. Lucenes TypeTokenFilter also provides a second mode where
instead of keeping the specified tokens (include) they are filtered out
(exclude). This change exposes this option as a new `mode` parameter that can
either take the values `include` (the default, if not specified) or `exclude`.

Closes #29277
2018-07-17 09:04:41 +02:00
Yogesh Gaikwad 6717df3c2d
[Test] Modify assert statement for ssl handshake (#32072)
There have been changes in error messages for `SSLHandshakeException`.
This has caused a couple of failures in our tests.
This commit modifies test verification to assert on exception type of
class `SSLHandshakeException`.
There was another issue in Java11 which caused NPE. The bug has now
been fixed on Java11 - early access build 22.
Bug Ref: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8206355
Enable the skipped tests due to this bug.

Closes #31940
2018-07-17 15:43:03 +10:00
Armin Braun 4b5071f2d0
Add Index UUID to `/_stats` Response (#31871)
* Add "uuid" field to each index's section in the `/_stats` response
* closes #31791
2018-07-17 06:50:21 +02:00
Jack Conradson 15740d6229
Painless: Move and Rename Several Methods in the lookup package (#32105) 2018-07-16 16:13:48 -07:00
Jim Ferenczi f699cb9f55
Bypass highlight query terms extraction on empty fields (#32090)
Dealing with empty fields in the highlight phase can
slow down the query because the query terms extraction is done independently
on each field. This change shortcuts the highlighting performed by the unified highlighter
for fields that are not present in the document. In such cases there is nothing to higlight so
we don't need to visit the query to build the highligh builder.
2018-07-17 00:26:01 +02:00
Nik Everett d596447f3d
Switch non-x-pack to new style requests (#32106)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes most of the calls not in X-Pack to their new versions.
2018-07-16 17:44:19 -04:00
Zachary Tong 791b9b147c
[Rollup] Add new capabilities endpoint for concrete rollup indices (#30401)
This introduces a new GetRollupIndexCaps API which allows the user to retrieve rollup capabilities of a specific rollup index (or index pattern). This is distinct from the existing RollupCaps endpoint.

- Multiple jobs can be stored in multiple indices and point to a single target data index pattern (logstash-*). The existing API finds capabilities/config of all jobs matching that data index pattern.
- One rollup index can hold data from multiple jobs, targeting multiple data index patterns. This new API finds the capabilities based on the concrete rollup indices.
2018-07-16 17:20:50 -04:00
Andy Bristol 14d7e2c7b2 Revert "[test] disable packaging tests for suse boxes"
This reverts commit 30d6fd3ca7.
2018-07-16 13:57:17 -07:00
Costin Leau 20ea72e2f2
SQL: allow LEFT and RIGHT as function names (#32066)
Due to the way ANTLR works, any declared tokens need to be accounted for
manually inside function names (otherwise a different rule gets applied).

Fix #32046
2018-07-16 23:42:44 +03:00
ahmedakef 5b65e2ccbf DOCS: put LIMIT 10 to the SQL query (#32065)
Provides a more precise equivalent SQL query for the aggregation example in the getting started guide.
2018-07-16 16:09:56 -04:00
Andy Bristol 69c85331b0
[test] turn on host io cache for opensuse (#32053)
The hope is that this will resolve the problems with very slow io we're
seeing on this box in #30295
2018-07-16 13:02:53 -07:00