Commit Graph

4807 Commits

Author SHA1 Message Date
Armin Braun d412230cda
SCRIPTING: Support BucketAggScript return null (#32811)
* As explained in #32790, `BucketAggregationScript` must support `null` as a return value
* Closes #32790
2018-08-13 20:08:26 +02:00
Jack Conradson 47d2bcc3c4
Painless: Clean Up Whitelist Names (#32791)
Renames variables in the whitelists to match the current naming scheme. Mechanical change.
2018-08-10 14:46:31 -07:00
Jack Conradson 7a1bbbc581
Painless: Clean up PainlessCast (#32754)
Renames variables and methods in PainlessCast to go along with the new naming 
scheme. Mechanical change.
2018-08-10 09:55:38 -07:00
Christoph Büscher 22f7b03430
Fix test reproducability in AbstractBuilderTestCase setup (#32403)
Currently AbstractBuilderTestCase generates certain random values in its
`beforeTest()` method annotated with @Before only the first time that a test
method in the suite is run while initializing the serviceHolder that we use for
the rest of the test. This changes the values of subsequent random values
and has the effect that when running single methods from a test suite with
"-Dtests.method=*", the random values it sees are different from when the same
test method is run as part of the whole test suite. This makes it hard to use
the reproduction lines logged on failure.

This change runs the inialization of the serviceHolder and the randomization 
connected to it using the test runners master seed, so reproduction by running
just one method is possible again.


Closes #32400
2018-08-10 15:13:44 +02:00
Jack Conradson 293c8a2b24
Painless: Add an Ingest Script Processor Example (#32302)
This commit adds two pieces. The first is a small set of documentation providing 
instructions on how to get setup to run context examples. This will require a download 
similar to how Kibana works for some of the examples. The second is an ingest processor 
example using the downloaded data. More examples will follow as ideally one per PR. 
This also adds a set of tests to individually test each script as a unit test.
2018-08-09 14:24:55 -07:00
Nicholas Knize e162127ff3 Upgrade to Lucene-7.5.0-snapshot-13b9e28f9d
The main feature is the inclusion of bkd backed geo_shape with
INTERSECT, DISJOINT, WITHIN bounding box and polygon query support.
2018-08-09 11:15:02 -05:00
Armin Braun 79375d35bb
Scripting: Replace Update Context (#32096)
* SCRIPTING: Move Update Scripts to their own context
* Added system property for backwards compatibility of change to `ctx.params`
2018-08-09 14:32:36 +02:00
Jack Conradson 9b00f095b9
Painless: Move More Logic to PainlessLookup (#32689)
This moves some run-time lookups for methods and fields to the PainlessLookup.
2018-08-08 16:25:14 -07:00
Armin Braun 7d641ba69b
TESTS: Explicitly Fail Http Client Timeouts (#32708)
* Don't quietly ignore timeouts when waiting for HTTP responses
* Fixes #32702
2018-08-08 15:47:51 +02:00
Luca Cavanna 5c2ef5e869
Preserve index_uuid when creating QueryShardException (#32677)
As part of #32608 we made sure that the fully qualified index name is taken from the query shard context whenever creating a new `QueryShardException`. That change introduced a regression as instead of setting the entire `Index` object to the exception, which holds index name and index uuid, we ended up setting only the index name (including cluster alias). With this commit we make sure that the index uuid does not get lost and we try to lower the chances that a similar bug makes it in another time. That's done by making `QueryShardContext` return the fully qualified `Index` (which also holds the uuid) rather than only the fully qualified index name.
2018-08-08 09:57:11 +02:00
Jack Conradson 0b7fb4e7b9
Painless: Clean up FunctionRef (#32644)
This change consolidates all the logic for generating a FunctionReference (renamed from 
FunctionRef) from several arbitrary constructors to a single static function that is used at 
both compile-time and run-time. This increases long-term maintainability as it is much 
easier to follow when and how a function reference is being generated. It moves most of 
the duplicated logic out of the ECapturingFuncRef, EFuncRef and ELambda nodes and 
Def as well.
2018-08-07 12:26:57 -07:00
Armin Braun 6fa7016bbf
SCRIPTING: Move Aggregation Scripts to their own context (#32068)
* SCRIPTING: Move Aggregation Scripts to their own context
2018-08-04 10:37:07 +02:00
Jack Conradson 6ca24e13af
Painless: Use LocalMethod Map For Lookup at Runtime (#32599)
This modifies Def to use a Map<String, LocalMethod> to look up user-defined methods at runtime 
instead of writing constant methodhandles to do the reverse lookup. This creates a consistency 
between how LocalMethods are looked up at compile-time and run-time. This consistency will allow 
this code to be more maintainable moving forward. This will also allow FunctionReference to be 
cleaned up in a follow up PR.
2018-08-03 15:22:30 -07:00
Jack Conradson b938960602
Painless: Move Some Lookup Logic to PainlessLookup (#32565)
Renames existing methods in PainlessLookup. Adds lookupPainlessClass, 
lookupPainlessMethod, and lookupPainlessField to PainlessLookup. This consolidates 
the logic necessary to look these things up into a single place and begins the clean up of 
some of the nodes that were looking each of these things up individually. This also has 
the added benefit of improved consistency in error messaging.
2018-08-02 12:33:25 -07:00
Armin Braun be31cc642b
INGEST: Enable default pipelines (#32286)
* INGEST: Enable default pipelines

* Add `default_pipeline` index setting
* `_none` is interpreted as no pipeline
* closes #21101
2018-08-02 17:11:12 +02:00
Jack Conradson 2985920134
Painless: Clean Up PainlessField (#32525)
Updates PainlessField variable names to current naming scheme and removes 
extraneous variables.
2018-08-01 09:28:18 -07:00
Ryan Ernst 478f6d6cf1
Scripting: Conditionally use java time api in scripting (#31441)
This commit adds a boolean system property, `es.scripting.use_java_time`,
which controls the concrete return type used by doc values within
scripts. The return type of accessing doc values for a date field is
changed to Object, essentially duck typing the type to allow
co-existence during the transition from joda time to java time.
2018-08-01 08:58:49 -07:00
Armin Braun 4b199dde8d
NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
* Upgrade to `4.1.28` since the problem reported in #32487 is a bug in Netty itself (see https://github.com/netty/netty/issues/7337)
* Fixed other leaks in test code that now showed up due to fixes improvements in leak reporting in the newer version
* Needed to extend permissions for netty common package because it now sets a classloader at runtime after changes in 63bae0956a
* Adjusted forbidden APIs check accordingly
* Closes #32487
2018-08-01 02:34:58 +02:00
Jack Conradson 09e38f2f59
Painless: Clean up PainlessMethod (#32476)
Renames and removes variables from PainlessMethod to follow the new naming 
convention. Generates methodtypes at compile-time instead of using a method at run-
time. Moves write method to MethodWriter.
2018-07-31 16:25:53 -07:00
Ryan Ernst 2ed9782a67
Scripting: Fix painless compiler loader to know about context classes (#32385)
This commit fixes the painless compiler classloader to know about the
classes from the script context. This fixes an issue when a custom
context is used from a plugin which caused a ClassNotFoundException for
the script class and its factory classes.
2018-07-31 08:28:03 -07:00
Sohaib Iftikhar 4fa92cbf49 Changed ReindexRequest to use Writeable.Reader (#32401)
-- This is a pre-stage for adding the reindex API to the REST high-level-client
-- Follows the pattern set in #26315
2018-07-31 10:11:17 -04:00
Jack Conradson c69e62d96f
Painless: Add PainlessConstructor (#32447)
PainlessMethod was being used as both a method and a constructor, and while there are 
similarities, there are also some major differences. This allows the reflection objects to be 
stored reducing the number of other pieces of data stored in a PainlessMethod as they are 
now redundant. This temporarily increases some of the code in FunctionRef and 
PainlessDocGenerator as they now differentiate between constructors and methods, BUT 
is also makes the code more maintainable because there aren't checks in several places 
anymore to differentiate.
2018-07-30 14:46:24 -07:00
Armin Braun cf7489899a
INGEST: Clean up Java8 Stream Usage (#32059)
* GrokProcessor: Rationalize the loop over the map to save allocations and indirection
* IngestDocument: Rationalize way we append to `List`
2018-07-30 21:25:30 +02:00
Ryan Ernst 34d006f82a
Tests: Fix convert error tests to use fixed value (#32415)
The error tests for hex values previously used a random string of
digits, but this could be a valid hex value. This commit changes these
tests to use a fixed invalid hex value.

closes #32370
2018-07-30 10:00:55 -07:00
Jack Conradson e9e1095596
Painless: Add method type to method. (#32441)
MethodType can be computed at compile-time rather than run-time. This removes the 
method that collects MethodType at run-time from a PainlessMethod since is it no longer 
necessary.
2018-07-27 14:23:37 -07:00
javanna 83d007e7be [TEST] Mute failing testConvertLongHexError
See #32370
2018-07-27 11:50:13 +02:00
Jim Ferenczi 53ff06e621
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
2018-07-27 08:28:51 +02:00
Jim Ferenczi 8e5f281b27
AbstractQueryTestCase should run without type less often (#28936)
This commit changes the randomization to always create an index with a type.
It also adds a way to create a query shard context that maps to an index with
no type registered in order to explicitely test cases where there is no type.
2018-07-26 20:29:05 +02:00
Tim Brooks 7a56df7c98
Release requests in cors handler (#32364)
There are two scenarios where a http request could terminate in the cors
handler. If that occurs, the requests need to be released. This commit
releases those requests.
2018-07-26 10:06:24 -06:00
Jack Conradson df579f8bce
Painless: Clean Up PainlessClass Variables (#32380)
Removes the variables name, clazz, and type as they are unnecessary. Renames 
staticMembers -> staticFields, members -> fields, getters -> getterMethodHandles, and 
setters -> setterMethodHandles.
2018-07-26 09:02:06 -07:00
Christoph Büscher 35ae87125d
Remove some dead code (#31993)
Removing some dead code or supressing warnings where apropriate. Most of the
time the variable tested for null is dereferenced earlier or never used before.
2018-07-26 17:12:51 +02:00
Christoph Büscher bec888fa78 Rank-Eval: Reduce scope of an unchecked supression
We should only supress the unchecked warnings on ConstructingObjectParser.
2018-07-26 11:16:01 +02:00
Jack Conradson 853aa0afb4
Painless: Decouple PainlessLookupBuilder and Whitelists (#32346)
Implements a static function in PainlessLookupBuilder that contains all the logic related
to Whitelist.  PainlessLookupBuilder is available for use in loading from methods beyond
Whitelist now.
2018-07-25 10:52:01 -07:00
Dimitris Athanasiou de53f0123f [TEST] Mute ConvertProcessortTests.testConvertIntHexError
Relates #32370
2018-07-25 17:35:23 +01:00
Armin Braun 717df26fc3
Networking: Fix test leaking buffer (#32296)
* Test `handler` must release buffer the same way the replaced `org.elasticsearch.http.netty4.Netty4HttpRequestHandler#channelRead0` releases it
* Closes #32289
2018-07-24 23:04:22 +02:00
Jack Conradson 1690451a9f
Painless: Update More Methods to New Naming Scheme (#32305)
This finishes the updating the methods in the PainlessLookupBuilder to the new naming scheme. Mechanical change. Methods include the ones used for copying members in the inheritance hierarchy, calculating shortcuts, and setting the functional interface.
2018-07-24 13:08:05 -07:00
Ryan Ernst 49d4b26f16
Ingest: Support integer and long hex values in convert (#32213)
This commit adds checks for hex formatted strings in the convert
processor, allowing strings like `0x1` to be parsed as integer `1`.

closes #32182
2018-07-24 12:05:50 -07:00
Christoph Büscher 59cf600e03
Register ERR metric with NamedXContentRegistry (#32320)
This adds the ERR metric to the provided xContent parsers in the module and the
high level rest client registry. Also adding integration tests to make sure the
metric is correctly registered and usable from the client.
2018-07-24 16:05:43 +02:00
Zachary Tong 6ba144ae31
Add WeightedAvg metric aggregation (#31037)
Adds a new single-value metrics aggregation that computes the weighted 
average of numeric values that are extracted from the aggregated 
documents. These values can be extracted from specific numeric
fields in the documents.

When calculating a regular average, each datapoint has an equal "weight"; it
contributes equally to the final value.  In contrast, weighted averages
scale each datapoint differently.  The amount that each datapoint contributes 
to the final value is extracted from the document, or provided by a script.

As a formula, a weighted average is the `∑(value * weight) / ∑(weight)`

A regular average can be thought of as a weighted average where every value has
an implicit weight of `1`.

Closes #15731
2018-07-23 18:33:15 -04:00
Christoph Büscher fe6bb75eb4
Rename ranking evaluation `quality_level` to `metric_score` (#32168)
The notion of "quality" is an overloaded term in the search ranking evaluation 
context. Its usually used to decribe certain levels of "good" vs. "bad" of a 
seach result with respect to the users information need. We currently report the 
result of the ranking evaluation as `quality_level` which is a bit missleading.
This changes the response parameter name to `metric_score` which fits better.
2018-07-23 22:25:02 +02:00
Jack Conradson d3c4904fa3
Painless: Clean up add methods in PainlessLookup (#32258)
This is largely mechanical change that cleans up the addConstructor, addMethod, and 
addFields methods in PainlessLookup. Changes include renamed variables, better error 
messages, and some minor code movement to make it more maintainable long term.
2018-07-23 09:12:30 -07:00
Christoph Büscher ff87b7aba4
Remove unnecessary warning supressions (#32250) 2018-07-23 11:31:04 +02:00
Armin Braun 7aa8a0a927
INGEST: Extend KV Processor (#31789) (#32232)
* INGEST: Extend KV Processor (#31789)

Added more capabilities supported by LS to the KV processor:
* Stripping of brackets and quotes from values (`include_brackets` in corresponding LS filter)
* Adding key prefixes
* Trimming specified chars from keys and values

Refactored the way the filter is configured to avoid conditionals during execution.
Refactored Tests a little to not have to add more redundant getters for new parameters.

Relates #31786
* Add documentation
2018-07-20 22:32:50 +02:00
Armin Braun e21692e387
INGEST: Make a few Processors callable by Painless (#32170)
* INGEST: Make a few Processors callable by Painless
* Extracted a few stateless String processors as well as the json processor to static methods and whitelisted them in Painless
* provide whitelist from processors plugin
2018-07-20 21:10:35 +02:00
Nick Peihl ac63408655
Add region ISO code to GeoIP Ingest plugin (#31669) 2018-07-20 11:23:29 -07:00
Christoph Büscher 5cbd9ad177
Rename ranking evaluation response section (#32166)
Currently the ranking evaluation response contains a 'unknown_docs' section 
for each search use case in the evaluation set. It contains document ids for 
results in the search hits that currently don't have a quality rating.
This change renames it to `unrated_docs`, which better reflects its purpose.
2018-07-20 11:43:46 +02:00
Jack Conradson c7a41c501a
Painless: Simplify Naming in Lookup Package (#32177)
This removes some extraneous naming syntax and makes clear the meaning of certain
naming conventions without ambiguities (stricter) within the lookup package. Purely 
mechanical change. Note this does not cover a large portion of the 
PainlessLookupBuilder and PainlessLookup yet as there are several more follow up PRs for these incoming.
2018-07-19 16:35:03 -07:00
Julie Tibshirani 15ff3da653
Add support for field aliases. (#32172)
* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)
2018-07-18 09:33:09 -07:00
Jack Conradson 605dc49c48
Painless: Fix caching bug and clean up addPainlessClass. (#32142)
This change cleans up the addPainlessClass methods by doing the following things:

* Rename many variable names to match the new conventions described in the JavaDocs 
for PainlessLookup
* Decouples Whitelist.Class from adding a PainlessClass directly
* Adds a second version of addPainlessClass that is intended for use to add future 
defaults in a follow PR

This change also fixes the method and field caches by storing Classes instead of Strings 
since it would technically be possible now that the whitelists are extendable to have 
different Classes with the same name. It was convenient to add this change together 
since some of the new constants are shared.

Note the changes are largely mechanical again where all the code behavior should 
remain the same.
2018-07-18 09:29:52 -07:00
Alan Woodward cfb30144c9
Call setReferences() on custom referring tokenfilters in _analyze (#32157)
When building custom tokenfilters without an index in the _analyze endpoint,
we need to ensure that referring filters are correctly built by calling
their #setReferences() method

Fixes #32154
2018-07-18 14:43:20 +01:00