Commit Graph

3326 Commits

Author SHA1 Message Date
Robert Muir b9c7dbcfbd nuke overflow detection as we cannot guarantee for def. simplify life :) 2016-05-19 15:25:31 -04:00
Tanguy Leroux a01ecb20ea Port Delete By Query to Reindex infrastructure
closes #16883
2016-05-19 16:07:50 +02:00
Robert Muir 915ab16176 First steps at whitelist cleanup 2016-05-19 09:29:48 -04:00
Ali Beyad fc6df23fea Rename AggregatorBuilder and all of its subclasses to
AggregationBuilder, in keeping consistent with the Java APIs.

Closes #18377
Closes #18367
2016-05-19 09:25:29 -04:00
Robert Muir 1022123f59 Merge pull request #18400 from uschindler/painless_java9StringConcats
Use Java 9 Indy String Concats, if available
2016-05-18 08:00:20 -04:00
Uwe Schindler ea9026e6d8 painless: fix checkstyle 2016-05-18 00:31:12 +02:00
Uwe Schindler b418d01117 painless: Improve string concat tests; add state machine check after MethodWriter 2016-05-18 00:26:47 +02:00
Jack Conradson 3042a77d2b Merge pull request #18425 from jdconrad/cast
Made def variable casting consistent with invokedynamic rules
2016-05-17 13:29:49 -07:00
Jack Conradson f055ad641a Made def variable casting consistent with invokedynamic rules. 2016-05-17 13:16:49 -07:00
polyfractal 978c1e3e36 [TEST] Add missing sort processor to test
Also fixes the naming of the sort REST test
to follow the numbering convention
2016-05-17 14:52:17 -04:00
Nik Everett cfb06954ba [reindex] Add assertBusy to test
It has timing issues.
2016-05-17 14:05:35 -04:00
Nik Everett fe4823eae0 Reindex should retry on search failures
This uses the same backoff policy we use for bulk and just retries until
the request isn't rejected.

Instead of `{"retries": 12}` in the response to count retries this now
looks like `{"retries": {"bulk": 12", "search": 1}`.

Closes #18059
2016-05-17 13:58:45 -04:00
Uwe Schindler b27b0483d5 Merge branch 'master' into painless_java9StringConcats
# Conflicts:
#	modules/lang-painless/src/test/java/org/elasticsearch/painless/StringTests.java
2016-05-17 18:47:33 +02:00
Uwe Schindler 5622e9c9d3 painless: use Java 9 Indy String Concats, if available 2016-05-17 17:44:46 +02:00
Robert Muir fa5f099486 make semicolons only optional at the end 2016-05-17 10:55:40 -04:00
Robert Muir 92339c4763 painless: add method overloading based on arity
Closes #18385

Squashed commit of the following:

commit b2819df4d392d69b86e5c96d358eb03424e67e02
Author: Robert Muir <rmuir@apache.org>
Date:   Tue May 17 09:15:47 2016 -0400

    add note about tuple

commit 85fcac6a0d0674da24535121eab23e2c407d683f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon May 16 23:39:25 2016 -0400

    painless: add method overloading based on arity
2016-05-17 09:17:43 -04:00
Uwe Schindler 75f27df376 painless: Refactor WriterUtils to extends ASM GeneratorAdapter as "MethodWriter" for easy usage (and more coming later) 2016-05-17 02:25:05 +02:00
Jack Conradson 3fd3d367ef Moved ADefLink to be a marker interface instead. 2016-05-16 15:48:29 -07:00
Jack Conradson 3f23186712 Merge branch 'master' into types 2016-05-16 14:22:59 -07:00
Jack Conradson 95a1b5b2a3 Fixed a casting bug in Painless. 2016-05-16 14:18:05 -07:00
Robert Muir e69305ae21 Merge pull request #18372 from rmuir/painless_whitelist_expansion
painless whitelist expansion
2016-05-16 17:08:49 -04:00
Robert Muir 8d4c1befe5 Merge pull request #18364 from rmuir/nukeRunAsFloat
Remove LeafSearchScript.runAsFloat(): Nothing calls it.
2016-05-16 17:08:25 -04:00
Jack Conradson 1d394868d4 Merge branch 'master' into types 2016-05-16 12:43:24 -07:00
Jack Conradson 43344eee82 Add more def tests including checking bytecode for specific
optimizations.
2016-05-16 12:42:25 -07:00
Jack Conradson f664fa5f33 Merge pull request #18350 from uschindler/painless_isAssignableFrom
Use isAssignableFrom instead of relying on ClassCastException
2016-05-16 12:41:09 -07:00
Jack Conradson a2c2628a78 Merge pull request #18359 from uschindler/painless_removeStoreBoxing
Remove boxing when loading and storing values in "def" fields/arrays, remove boxing onsimple method calls of "def" methods
2016-05-16 12:11:37 -07:00
Uwe Schindler d6cbbdeb98 painless: Some reformatting in EChain to make it clear if "this" or inner "expression" is affected 2016-05-16 21:04:03 +02:00
Uwe Schindler b05ac87a5d painless: Add Jack's test 2016-05-16 20:37:51 +02:00
Uwe Schindler 3a5ef68ad0 painless: Fix issue with dup and cast 2016-05-16 20:31:27 +02:00
Robert Muir 6ac2877c32 Merge pull request #18371 from uschindler/painless_debuggerCleanup
painless: Small cleanup of Debugger class to use StringWriter
2016-05-16 09:58:50 -04:00
Robert Muir 2fc7ca46e1 expand whitelist for basic types 2016-05-16 09:56:29 -04:00
Uwe Schindler 07e4f416e6 painless: Small cleanup of Debugger class to use StringWriter 2016-05-16 15:52:14 +02:00
Nik Everett f569576c5b Switch default batch size for reindex to 1000 2016-05-16 08:19:29 -04:00
Robert Muir 8edf213492 Remove LeafSearchScript.runAsFloat(): Nothing calls it. 2016-05-15 22:59:28 -04:00
Uwe Schindler 65aca4f71c painless: make compound statement like a[1]++ work with def. There was also a bug with the size of LDefArray: fixed 2016-05-15 23:00:32 +02:00
Uwe Schindler d221cd14d2 painless: Also remove boxing for reads and method calls 2016-05-15 19:49:25 +02:00
Uwe Schindler 604bcd9320 painless: Make field stores not box; use GeneratorAdapter.invokeDynmaic for consistency with other method calls 2016-05-15 18:31:53 +02:00
Uwe Schindler 8195ef9e7a painless: Transform more try-catch-ClassCastException to isAssignableFrom 2016-05-15 01:06:29 +02:00
Uwe Schindler 7211017813 painless: Some cleanups 2016-05-14 17:23:29 +02:00
Uwe Schindler c78bf8c867 painless: Use isAssignableFrom instead of relying on ClassCastException 2016-05-14 16:04:18 +02:00
Uwe Schindler c8bd521ed3 painless: restore accidentally removed test 2016-05-14 12:00:30 +02:00
Uwe Schindler ba9dae820a Add test for wrong array index 2016-05-13 23:22:41 +02:00
Uwe Schindler 1a76329d89 Take return type from "after" field. 2016-05-13 23:17:22 +02:00
Uwe Schindler 1b10a14e8d painless: build descriptor of array and field load/store in code; fix array index to adapt type not DEF 2016-05-13 23:04:04 +02:00
Robert Muir 2028691e66 painless: improve exception stacktraces
closes #18319
2016-05-13 15:40:45 -04:00
Uwe Schindler 9731d5d314 painless: Rename the dynamic call site factory to DefBootstrap and make the inner class very short (PIC = Polymorphic Inline Cache) 2016-05-13 19:58:05 +02:00
Lee Hinman 9bcdafedda Allow only a single extension for a scripting engine
Previously multiple extensions could be provided, however, this can lead
to confusion with on-disk scripts (ie, "foo.js" and "foo.javascript")
having different content. Only a single extension is now supported.

The only language currently supporting multiple extensions was the
Javascript engine ("js" and "javascript"). It now only supports the
`.js` extension.

Relates to #10598
2016-05-13 09:54:31 -06:00
Lee Hinman d5b75491dc Merge remote-tracking branch 'dakrone/remove-script-sandbox' 2016-05-13 09:50:39 -06:00
Nik Everett 0a300320cd [test] Set logging to 11 on reindex test
It has failures we can't explain and we need logs to be able to do
anything useful with the failures:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/359/consoleFull
2016-05-13 11:48:01 -04:00
Lee Hinman efff3918d8 Remove support for mulitple languages per scripting engine 2016-05-13 09:24:31 -06:00
Lee Hinman a4060f7436 Remove vestiges of script engine sandboxing
This removes all the mentions of the sandbox from the script engine
services and permissions model. This means that the following settings
are no longer supported:

```yaml
script.inline: sandbox
script.stored: sandbox
```

Instead, only a `true` or `false` value can be specified.

Since this would otherwise break the default-allow parameter for
languages like expressions, painless, and mustache, all script engines
have been updated to have individual settings, for instance:

```yaml
script.engine.groovy.inline: true
```

Would enable all inline scripts for groovy. (they can still be
overridden on a per-operation basis).

Expressions, Painless, and Mustache all default to `true` for inline,
file, and stored scripts to preserve the old scripting behavior.

Resolves #17114
2016-05-13 09:24:31 -06:00
Adrien Grand 638da06c1d Add back support for `ip` range aggregations. #17859
This commit adds support for range aggregations on `ip` fields. However it will
only work on 5.x indices.

Closes #17700
2016-05-13 17:22:01 +02:00
Robert Muir 520697eb14 Merge pull request #18323 from uschindler/dynamiccallsite_cleanup
painless: cleanup of DynamicCallSite
2016-05-13 11:19:27 -04:00
Uwe Schindler a1d0fb5008 painless: cleanup of DynamicCallSite to have all implementation encapsulated in inner class, outer class only has bootstrap method 2016-05-13 15:50:55 +02:00
Daniel Mitterdorfer ddbfda2c68 Exclude specific transport actions from request size limit check
We add support to explicitly exclude specific transport actions
from the request size limit check.

We also exclude the following request types currently:

*MasterPingRequest
* PingRequest
2016-05-13 14:21:24 +02:00
Jack Conradson 2e25837deb Made line number available in painless nodes for debugging. 2016-05-12 10:32:06 -07:00
Jack Conradson 28164ccf2e Decouple the ANTLR AST from Painless.
Closes #18286
2016-05-12 00:46:48 -07:00
Robert Muir 113e239e0f Merge pull request #18288 from rmuir/painless_debug
painless: add debugging tool to tests
2016-05-12 01:16:32 -04:00
Jack Conradson 8871fddba7 Merge pull request #18282 from jdconrad/longs
Long priority over Float
2016-05-11 21:08:28 -07:00
Robert Muir 930a215b5c add debugger 2016-05-11 23:56:19 -04:00
Robert Muir 25dd64250b painless: remove input, support params instead 2016-05-11 21:32:10 -04:00
Robert Muir c5532d3df0 add a rest test for this that seems to work, fix the documentation. thanks @s1monw 2016-05-11 16:07:08 -04:00
Robert Muir 6b4e47bf96 this makes aggregations per-document _value fast (bypass hash put, hash get, etc) for painless.
but i have no clue how to test it, it seems this feature never worked via REST?

Should we drop the feature instead?
2016-05-11 15:39:00 -04:00
Jack Conradson 20af482dd7 Removed extra branches. Only check for long first where we already know the
second type.
2016-05-11 10:22:40 -07:00
Jack Conradson 2764b47d22 Give long priority over float in Def math calcs. Promotion of types
still works correclty.
2016-05-11 09:32:30 -07:00
Robert Muir d790c76636 painless: _score as double, not float 2016-05-11 11:19:18 -04:00
Tanguy Leroux 6d288dec11 Clean up tests in Reindex module 2016-05-11 09:54:52 +02:00
Robert Muir 7689a1af28 Add 'ctx' keyword to painless. 2016-05-11 02:31:19 -04:00
Jack Conradson d8b373cac6 Fixed an issue related to bracket shortcuts. 2016-05-10 23:11:13 -07:00
Robert Muir 632bb4c720 add reserved word tests for 'doc', too 2016-05-11 00:17:19 -04:00
Robert Muir 389ef462c4 special access to doc 2016-05-10 23:43:57 -04:00
Robert Muir abb015bcb9 add tests for reserved words. ban stores against _score 2016-05-10 22:55:06 -04:00
Robert Muir a63285aaf9 painless: retrieve _score directly from Scorer 2016-05-10 20:53:08 -04:00
Robert Muir 1d80542199 Merge pull request #18247 from rmuir/needsScore
painless: implement needsScore() correctly.
2016-05-10 19:21:09 -04:00
Robert Muir ddc2c1f682 Merge pull request #18234 from rmuir/more_indy_type_data
painless: use better typing for dynamic method calls
2016-05-10 19:17:13 -04:00
Robert Muir 1c2d3b10ef some cleanups suggested by @uschindler and break some long lines 2016-05-10 12:31:02 -04:00
Robert Muir eb1b2cf111 add test and cleanup 2016-05-10 11:34:33 -04:00
Robert Muir 7a47d33a9b Merge branch 'master' into needsScore 2016-05-10 11:23:50 -04:00
Robert Muir 546aed8390 hack 2016-05-10 11:23:24 -04:00
Uwe Schindler 243c9e77fa painless: Add synthetic length property as alias to Lists, so they can really be used like arrays 2016-05-10 15:01:49 +02:00
Uwe Schindler 2e613f49b3 painless: Array load/store and length with invokedynamic 2016-05-10 12:41:27 +02:00
Robert Muir 3aeb66885b painless: use better typing for dynamic method calls 2016-05-10 05:44:16 -04:00
Robert Muir ba2fe156e8 Switch over dynamic method calls, loads and stores to invokedynamic.
Remove performance hack for accessing a document's fields, its not needed.
Add support for accessing is-getter methods like List.isEmpty() as .empty

Closes #18201
2016-05-09 21:44:32 -04:00
Tanguy Leroux 8c52e8814b Remove ReindexResponse in favor of BulkIndexByScrollResponse 2016-05-09 17:03:16 +02:00
Chris Earle 5be79ed02c Add Failure Details to every NodesResponse
Most of the current implementations of BaseNodesResponse (plural Nodes) ignore FailedNodeExceptions.

- This adds a helper function to do the grouping to TransportNodesAction
- Requires a non-null array of FailedNodeExceptions within the BaseNodesResponse constructor
- Reads/writes the array to output
- Also adds StreamInput and StreamOutput methods for generically reading and writing arrays
2016-05-06 14:59:43 -04:00
Adrien Grand 7d8708716e QueryBuilder does not need generics. #18133
QueryBuilder has generics, but those are never used: all call sites use
`QueryBuilder<?>`. Only `AbstractQueryBuilder` needs generics so that the base
class can contain a default implementation for setters that returns `this`.
2016-05-06 08:38:20 +02:00
Robert Muir e3ce6c9048 Painless: add fielddata accessors (.value/.values/.distance()/etc)
This gives better coverage and consistency with the scripting APIs, by
whitelisting the primary search scripting API classes and using them instead
of only Map and List methods.

For example, accessing fields can now be done with `.value` instead of `.0`
because `getValue()` is whitelisted. For now, access to a document's fields in
this way (loads) are fast-pathed in the code, to avoid dynamic overhead.

Access to geo fields and geo distance functions is now supported.

TODO: date support (e.g. whitelist ReadableDateTime methods as a start)
TODO: improve docs (like expressions and groovy have for document's fields)
TODO: remove fast-path hack

Closes #18169

Squashed commit of the following:

commit ec9f24b2424891a7429bb4c0a03f9868cba0a213
Author: Robert Muir <rmuir@apache.org>
Date:   Thu May 5 17:59:37 2016 -0400

    cutover to <Def> instead of <Object> here

commit 9edb1550438acd209733bc36f0d2e0aecf190ecb
Author: Robert Muir <rmuir@apache.org>
Date:   Thu May 5 17:03:02 2016 -0400

    add fast-path for docvalues field loads

commit f8e38c0932fccc0cfa217516130ad61522e59fe5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu May 5 16:47:31 2016 -0400

    Painless: add fielddata accessors (.value/.values/.distance()/etc)
2016-05-05 18:38:41 -04:00
Jack Conradson 2cae575f53 Added single-quoted strings.
Closes #18150
2016-05-05 09:26:02 -07:00
Robert Muir 59c135b58d make internal Def methods private and add basic javadocs 2016-05-05 03:47:56 -04:00
Robert Muir 928e2b904d painless: optimize/simplify dynamic field and method access 2016-05-05 03:42:14 -04:00
Jason Tedor 784c9e5fb9 Introduce node handshake
This commit introduces a handshake when initiating a light
connection. During this handshake, node information, cluster name, and
version are received from the target node of the connection. This
information can be used to immediately validate that the target node is
a member of the same cluster, and used to set the version on the
stream. This will allow us to extend APIs that are used during initial
cluster recovery without a major version change.

Relates #15971
2016-05-04 20:06:47 -04:00
Nik Everett 230697c202 [reindex] Switch throttle to Float.POSITIVE_INFITINTY/"unlimited"
All other values are errors.

Add java test for throttling. We had a REST test but it only ran against
one node so it didn't catch serialization errors.

Add Simple round trip test for rethrottle request
2016-05-04 16:14:32 -04:00
Christoph Büscher ca21aa0cb5 Make reset() in QueryShardContext private
The query shard reset() method resets some internal state in the
query shard context, like clearing query names, the filter flag
or named queries. The problem with this method being public is
that it currently (miss?) used for modifying an existing context
for recursive invocatiob, but the contexts that have been reseted
that way cannot be properly set back to their previous state.

This PR is a step towards removing reset() entirely by first making
it only be used internally in QueryShardContext. In places where
reset() was used we can either create new QueryShardContexts or
modify the existing context because it is discarded afterwards anyway.
2016-05-03 18:56:16 +02:00
Robert Muir fff82db681 Add tests/doc for boolean fields with expressions 2016-05-02 18:13:03 -04:00
Robert Muir 693c1f6671 Support geo_point fields in lucene expressions.
Closes #18096
2016-05-02 17:49:21 -04:00
Robert Muir 28409e4509 Add support for .empty to expressions, and some docs improvements
Closes #18077
2016-05-02 09:07:25 -04:00
Martijn van Groningen 7aca1389e2 ingest: Add `date_index_name` processor.
Closes #17814
2016-04-29 17:20:48 +02:00
Ryan Ernst d12a4bb51d Merge pull request #17933 from rjernst/camelcase4
Remove camelCase support
2016-04-22 13:46:43 -07:00
Nik Everett cc1a55423c Reindex: properly mark things as child tasks
Do this by creating a Client subclass that automatically assigns the
parentTask to all requests that come through it. Code that doesn't want
to set the parentTask can call `unwrap` on the Client to get the inner
client instance that doesn't set the parentTask. Reindex uses this for
its ClearScrollRequest so that the request will run properly after the
reindex request has been canceled.
2016-04-22 14:00:11 -04:00
Ryan Ernst 55388590c1 Remove camelCase support
Now that the current uses of magical camelCase support have been
deprecated, we can remove these in master (sans remaining issues like
BulkRequest). This change removes camel case support from ParseField,
query types, analysis, and settings lookup.

see #8988
2016-04-22 09:18:10 -07:00
Nik Everett 51621f9d75 Remove ChildTaskRequest and always pass parentTaskId when building a task
Passing parentTaskId forces the caller to handle the parentTaskId.
2016-04-22 11:26:18 -04:00
Nik Everett ffeb5e38fc Remove parent-less task methods
Callers should explicitly handle parents - either using EMPTY_TASK_ID when
a parent isn't possible or piping parents from the TransportRequest when
possible.
2016-04-22 11:26:18 -04:00
Martijn van Groningen c5ad2e2865 Changed indexed scripts to be stored in the cluster state instead of the `.scripts` index.
Also added max script size soft limit for stored scripts.

Closes #16651
2016-04-22 13:42:55 +02:00
Isabel Drost-Fromm 233fe86ee4 Makes Script type writeable
Used to be Streamable. Left-over of the PROTOTYPE related refactoring by
@nik9000

Closes #17753
2016-04-21 12:40:29 +02:00
Christoph Büscher e06e122f9f Wrap xcontent parser creation in try-with-resource statement where possible 2016-04-18 16:13:56 +02:00
Christoph Büscher cdb36a2b0c Merge pull request #17417
Clean up QueryParseContext and don't hold it inside QueryRewrite/ShardContext
2016-04-18 15:13:53 +02:00
Nik Everett d3b1306069 Reindex: never report negative throttled_until
Just clamp the value at 0. It isn't useful to tell the user "this
thread should have woken 5ms ago".

Closes #17783
2016-04-15 16:53:23 -04:00
Christoph Büscher fbd558382d Clean up QueryParseContext and don't hold it inside QueryRewrite/ShardContext
This change cleans up a few things in QueryParseContext and QueryShardContext
that make it hard to reason about the state of these context objects and are
thus error prone and should be simplified.

Currently the parser that used in QueryParseContext can be set and reset any
time from the outside, which makes reasoning about it hard. This change makes
the parser a mandatory constructor argument removes ability to later set a
different ParseFieldMatcher. If none is provided at construction time, the
one set inside the parser is used. If a ParseFieldMatcher is specified at
construction time, it is implicitely set on the parser that is beeing used.
The ParseFieldMatcher is only kept inside the parser.

Also currently the QueryShardContext historically holds an inner QueryParseContext
(in the super class QueryRewriteContext), that is mainly used to hold the parser
and parseFieldMatcher. For that reason, the parser can also be reset, which leads
to the same problems as above. This change removes the QueryParseContext from
QueryRewriteContext and removes the ability to reset or retrieve it from the
QueryShardContext. Instead, `QueryRewriteContext#newParseContext(parser)` can be
used to create new parse contexts with the given parser from a shard context
when needed.
2016-04-15 17:13:01 +02:00
Christoph Büscher de036d63d8 Rename context.parseFieldMatcher() to context.getParseFieldMatcher 2016-04-15 15:15:32 +02:00
Christoph Büscher 15c59d07b3 Remove ParseFieldMatcher from AbstractXContentParser
Currently we are able to set a ParseFieldMatcher on XContentParsers,
mainly to conveniently carry it around to be available where the
actual parsing happens. This was just recently introduced together
with ObjectParser so that ObjectParser can make use of deprecation
logging and throwing errors while parsing.

This however is trappy because we create parsers in so many places in
the code and it is easy to forget setting the right ParseFieldMatcher.
Instead we should hold the ParseFieldMatcher only in the parse contexts
(e.g. QueryParseContext).

This PR removes the ParseFieldMatcher from XContentParser. ObjectParser
can still make use of it because we can make the otherwise unbounded
`context` type to extend an interface that makes sure contexts used in
ObjectParser can supply a ParseFieldMatcher. Contexts in ObjectParser
are now no longer optional, but it is sufficient to pass in a small
lambda expression in places where no other context is available.

Relates to #17417
2016-04-15 15:14:46 +02:00
Adrien Grand d84c643f58 Use the new points API to index numeric fields. #17746
This makes all numeric fields including `date`, `ip` and `token_count` use
points instead of the inverted index as a lookup structure. This is expected
to perform worse for exact queries, but faster for range queries. It also
requires less storage.

Notes about how the change works:
 - Numeric mappers have been split into a legacy version that is essentially
   the current mapper, and a new version that uses points, eg.
   LegacyDateFieldMapper and DateFieldMapper.
 - Since new and old fields have the same names, the decision about which one
   to use is made based on the index creation version.
 - If you try to force using a legacy field on a new index or a field that uses
   points on an old index, you will get an exception.
 - IP addresses now support IPv6 via Lucene's InetAddressPoint and store them
   in SORTED_SET doc values using the same encoding (fixed length of 16 bytes
   and sortable).
 - The internal MappedFieldType that is stored by the new mappers does not have
   any of the points-related properties set. Instead, it keeps setting the index
   options when parsing the `index` property of mappings and does
   `if (fieldType.indexOptions() != IndexOptions.NONE) { // add point field }`
   when parsing documents.

Known issues that won't fix:
 - You can't use numeric fields in significant terms aggregations anymore since
   this requires document frequencies, which points do not record.
 - Term queries on numeric fields will now return constant scores instead of
   giving better scores to the rare values.

Known issues that we could work around (in follow-up PRs, this one is too large
already):
 - Range queries on `ip` addresses only work if both the lower and upper bounds
   are inclusive (exclusive bounds are not exposed in Lucene). We could either
   decide to implement it, or drop range support entirely and tell users to
   query subnets using the CIDR notation instead.
 - Since IP addresses now use a different representation for doc values,
   aggregations will fail when running a terms aggregation on an ip field on a
   list of indices that contains both pre-5.0 and 5.0 indices.
 - The ip range aggregation does not work on the new ip field. We need to either
   implement range aggs for SORTED_SET doc values or drop support for ip ranges
   and tell users to use filters instead. #17700

Closes #16751
Closes #17007
Closes #11513
2016-04-14 17:56:23 +02:00
Christoph Büscher e15e7f7e6e Remove parser argument from methods where we already pass in a parse context
When we pass down both parser and QueryParseContext to a method, we cannot
make sure that the parser contained in the context and the parser that is
parsed as an argument have the same state. This removes the parser argument
from methods where we currently have both the parser and the parse context
as arguments and instead retrieves the parse from the context inside the
method.
2016-04-14 16:18:05 +02:00
Martijn van Groningen 2928fd6ef3 Cleanup query builder for inner hits construction.
* Inner hits can now only be provided and prepared via setter in the nested, has_child and has_parent query.
* Also made `score_mode` a required constructor parameter.
* Moved has_child's min_child/max_children validation from doToQuery(...) to a setter.
2016-04-14 14:43:21 +02:00
Nik Everett cca3154c43 Rename isSourceEmpty to hasSource
And add a test case for {} to reindex.
2016-04-13 08:19:58 -04:00
Nik Everett c2e745bf3b reindex: Guard against user disabling fields 2016-04-13 08:19:58 -04:00
Nik Everett 0f9804b0e2 reindex: gracefully handle when _source is disabled
Closes #17666
2016-04-13 08:19:58 -04:00
Adrien Grand 013acf9179 Remove MappedFieldType.value. #17557
This commit removes `MappedFieldType.value` and simplifies
`MappedFieldType.valueforSearch`. `valueforSearch` was used to post-process
values that come for stored fields (eg. to convert a long back to a string
representation of a date in the case of a date field) and also values that
are extracted from the source but only in the case of GET calls: it would
not be called when performing source filtering on search requests.

`valueforSearch` is now only called for stored fields, since values that are
extracted from the source should already be formatted as expected.
2016-04-12 09:12:56 +02:00
Adrien Grand a14db8e17e Remove MappedFieldType.useTermQueryWithQueryString() and isNumeric(). #17599
In both cases, what elasticsearch is really interested in is whether the field
is an analyzed string field. So it can just check `tokenized()` instead.
2016-04-12 08:45:28 +02:00
Adrien Grand 496c7fbd84 Upgrade Lucene 6 Release
* upgrades numerics to new Point format
* updates geo api changes
  * adds GeoPointDistanceRangeQuery as XGeoPointDistanceRangeQuery
  * cuts over to ES GeoHashUtils
2016-04-11 16:50:04 -05:00
Adrien Grand 0eb1a816c8 Allow the query cache to be disabled. #16268
This replaces the internal `index.queries.cache.type` setting with
a new `index.queries.cache.enabled` setting, which is documented.

Closes #15802
2016-04-11 18:06:16 +02:00
Adrien Grand 42526ac28e Remove Settings.settingsBuilder.
We have both `Settings.settingsBuilder` and `Settings.builder` that do exactly
the same thing, so we should keep only one. I kept `Settings.builder` since it
has my preference but also it is the one that we use in examples of the Java API.
2016-04-08 18:10:02 +02:00
Adrien Grand bef38a4d12 Fix test bug. 2016-04-07 09:49:27 +02:00
Adrien Grand e1bfe23c22 ExtendedStatsAggregator should also pass sigma to emtpy aggs. #17388
Because sigma is also used at reduce time, it should be passed to empty aggs.
Otherwise it causes bugs when an empty aggregation is used to perform reduction
is it would assume a sigma of zero.

Closes #17362
2016-04-07 09:34:11 +02:00
Nik Everett 16c12afabe Rework ScoreFunctionBuilder registration to remove PROTOTYPEs
This removes PROTOTYPEs from ScoreFunctionsBuilders. To do so we rework
registration so it doesn't need PROTOTYPEs and lines up with the recent
changes to query registration.
2016-04-06 13:04:11 -04:00
Nik Everett 2b6866d26b Fix references to the removed parsers
Mostly stuff is just in the builder now.
2016-04-06 11:15:22 -04:00
Adrien Grand 4c4bbb3e45 Replace FieldStatsProvider with a method on MappedFieldType. #17334
FieldStatsProvider had to perform instanceof calls to properly handle dates or
ip addresses. By moving the logic to MappedFieldType, each field type can check
whether all values are within bounds its way.

Note that this commit only keeps rewriting support for dates, which are the only
field for which the rewriting mechanism is likely to help (because of time-based
indices).
2016-04-01 10:28:58 +02:00
Nik Everett 14d37baa4b [reindex] Don't get rejected
BulkByScrollTaskTest#testDelayAndRethrottle was getting rejected exceptions
every once in a while. This was reproducible ~20% of the time for me. I
added a CyclicBarrier to prevent the test from shutting down the thread pool
before the threads get finished.
2016-03-31 14:50:14 -04:00
Nik Everett 0c762fca35 Fix test mistake 2016-03-31 12:27:35 -04:00
Nik Everett 7f794e7b77 Test for invalid scroll_size 2016-03-31 12:21:32 -04:00
Nik Everett 30a1862339 Remove PROTOTYPE from BulkItemResponse.Failure
Closes #17086
2016-03-31 09:10:36 -04:00
javanna 32b6e529f4 Merge branch 'master' into enhancement/discovery_node_one_getter 2016-03-31 10:49:26 +02:00
Jack Conradson a37e53c50f Painless clean up including fixing _score issues and improving type
error messages.

Closes #17428
2016-03-30 16:40:17 -07:00
Nik Everett 78ab6c5b7f [reindex] Dynamic throttle!
This allows the user to update the reindex throttle on the fly, with changes
that speed up the throttling being applied immediately and changes that
slow down the throttling being applied during the next batch. This means
that if a user throttles reindex in such a way that it tries to sleep for
16 years and then realizes that they've done something wrong then they
can change the throttle and reindex will wake up again. We don't apply
slow downs immediately so we never get in danger of losing the scan context.

Also, if reindex is canceled while it is sleeping (how it honor throttling)
then it'll immediately wake up and cancel itself.
2016-03-30 16:40:42 -04:00
javanna b9f9b2e3ee Merge branch 'master' into enhancement/discovery_node_one_getter 2016-03-30 17:22:40 +02:00
Nik Everett 1c16d63a9a Merge pull request #17394 from camilojd/refactor/replace-getrandom
Refactor: replace all ocurrences of ESTestCase.getRandom() with LuceneTestCase.random()
2016-03-30 08:58:21 -04:00
javanna a8bbdff3bc Remove DiscoveryNode#name in favour of existing DiscoveryNode#getName 2016-03-30 14:47:36 +02:00
Adrien Grand 068c788ec8 Disable fielddata on text fields by defaults. #17386
`text` fields will have fielddata disabled by default. Fielddata can still be
enabled on an existing index by setting `fielddata=true` in the mappings.
2016-03-30 14:35:32 +02:00
Camilo Diaz Repka 7be11a36cd Refactor: replace all ocurrences of ESTestCase.getRandom() for random().
Remove getRandom().
2016-03-29 23:18:05 -04:00
Nik Everett df08854c60 Remove PROTOTYPEs from suggesters
Also stops using guice for suggesters at all and lots of checkstyle.
2016-03-29 17:55:01 -04:00
javanna 061f09d9a4 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 20:19:33 +02:00
Tal Levy 16e888fac3 Merge pull request #17260 from talevy/fix-regex-exceptions
Handle regex parsing errors in Gsub and Grok Processors
2016-03-29 08:12:26 -07:00
Clinton Gormley 3087d2b882 Fixed bad YAML in reindex REST test: 50_routing.yaml 2016-03-29 15:03:09 +02:00
Clinton Gormley 52daed0732 Update-by-query rest tests: fixed bad yaml and deleted a client-dependent test 2016-03-29 14:58:29 +02:00
Colin Goodheart-Smithe ff3fd99074 Prevents exception being raised when ordering by an aggregation which wasn't collected
If a terms aggregation was ordered by a metric nested in a single bucket aggregator which did not collect any documents (e.g. a filters aggregation which did not match in that term bucket) an ArrayOutOfBoundsException would be thrown when the ordering code tried to retrieve the value for the metric. This fix fixes all numeric metric aggregators so they return their default value when a bucket ordinal is requested which was not collected.

Closes #17225
2016-03-29 13:28:03 +01:00
javanna 8fc9dbbb99 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 14:27:04 +02:00
Clinton Gormley 5f24581de3 The reindex body is now required, which changes the exception thrown by the REST test 2016-03-29 14:09:59 +02:00
Clinton Gormley b87beeb05f Rename update-by-query REST tests to update_by_query 2016-03-29 13:13:49 +02:00
Clinton Gormley 97606850e8 Renamed update-by-query REST spec to update_by_query 2016-03-29 11:45:20 +02:00
javanna de5cbda8e7 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 10:48:47 +02:00
Nik Everett 0e6141e675 Replace is_true: took with took >= 0
This prevents tests from failing on machines that can finish the request
less than half a millisecond.
2016-03-28 13:03:48 -04:00
javanna 27d4994aff Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-24 18:10:11 +01:00
Nik Everett 93ab4cfc99 Stop using PROTOTYPE in NamedWriteableRegistry
readFrom is confusing because it requires an instance of the type that it
is reading but it doesn't modify it. But we also have (deprecated) methods
named readFrom that *do* modify the instance. The "right" way to implement
the non-modifying readFrom is to delegate to a constructor that takes a
StreamInput so that the read object can be immutable. Now that we have
`@FunctionalInterface`s it is fairly easy to register things by referring
directly to the constructor.

This change modifying NamedWriteableRegistry so that it does that. It keeps
supporting `registerPrototype` which registers objects to be read by
readFrom but deprecates it and delegates it to a new `register` method
that allows passing a simple functional interface. It also cuts Task.Status
subclasses over to using that method.

The start of #17085
2016-03-24 11:26:44 -04:00
Nik Everett 48aaebf23d [reindex] Wait for headers
The test was checking that we'd set the headers properly but in some cases
the request had yet to come in because it was running on another thread.
Now we wait for the headers to show up before failing the test.

Closes #17299
2016-03-24 09:55:49 -04:00
Jim Ferenczi da42f199bd Enforce isolated mode for all plugins
This commit removes the isolated option, each plugin have its own classloader.
2016-03-24 09:17:33 +01:00
Nik Everett aaa4d57fff [reindex] Don't attempt to refresh on noop
If the user asks for a refresh but their reindex or update-by-query
operation touched no indexes we should just skip the resfresh call
entirely. Without this commit we refresh *all* indexes which is totally
wrong.

Closes #17296
2016-03-23 18:12:40 -04:00
Areek Zillur ed49ec437f remove suggest transport action 2016-03-23 16:37:56 -04:00
javanna 030453d320 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-23 11:25:34 +01:00
Adrien Grand e50eeeaffb Refactor fielddata mappings. #17148
The fielddata settings in mappings have been refatored so that:
 - text and string have a `fielddata` (boolean) setting that tells whether it
   is ok to load in-memory fielddata. It is true by default for now but the
   plan is to make it default to false for text fields.
 - text and string have a `fielddata_frequency_filter` which contains the same
   thing as `fielddata.filter.frequency` used to (but validated at parsing time
   instead of being unchecked settings)
 - regex fielddata filtering is not supported anymore and will be dropped from
   mappings automatically on upgrade.
 - text, string and _parent fields have an `eager_global_ordinals` (boolean)
   setting that tells whether to load global ordinals eagerly on refresh.
 - in-memory fielddata is not supported on keyword fields anymore at all.
 - the `fielddata` setting is not supported on other fields that text and string
   and will be dropped when upgrading if specified.
2016-03-23 09:48:13 +01:00
Tal Levy 534caa8927 Handle regex parsing errors in Gsub and Grok Processors
Currently, both Gsub and Grok parse regex strings during
Pipeline creation. Thrown parsing exceptions were leaking out, this
commit wraps those exceptions in ElasticsearchParseExceptions.
2016-03-22 15:06:29 -07:00
Nik Everett da96b6e41d [reindex] Add thottling support
The throttle is applied when starting the next scroll request so that its
timeout can include the throttle time.
2016-03-22 12:34:14 -04:00
javanna eebd0cfccd Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-22 10:34:40 +01:00
Simon Willnauer 47f0e6e8f4 [TEST] Disable InternalClusterInfoService in messy tests, it sends IndicesStatsRequest periodically which messes with the messy test 2016-03-22 10:12:03 +01:00
javanna bf390a935e Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-21 17:18:23 +01:00
Boaz Leskes 2d1152ebac Remove ClusterService interface, in favor of it's only production instance #17183
We current have a ClusterService interface, implemented by InternalClusterService and a couple of test classes. Since the decoupling of the transport service and the cluster service, one can construct a ClusterService fairly easily, so we don't need this extra indirection.

Closes #17183
2016-03-21 13:55:10 +01:00
Martijn van Groningen e3b7e5d75a percolator: Replace percolate api with the new percolator query
Also replaced the PercolatorQueryRegistry with the new PercolatorQueryCache.

The PercolatorFieldMapper stores the rewritten form of each percolator query's xcontext
in a binary doc values field. This make sure that the query rewrite happens only during
indexing (some queries for example fetch shapes, terms in remote indices) and
the speed up the loading of the queries in the percolator query cache.

Because the percolator now works inside the search infrastructure a number of features
(sorting fields, pagination, fetch features) are available out of the box.

The following feature requests are automatically implemented via this refactoring:

Closes #10741
Closes #7297
Closes #13176
Closes #13978
Closes #11264
Closes #10741
Closes #4317
2016-03-21 12:21:50 +01:00
Ryan Ernst f71f0d6010 Revert "Build: Switch to maven-publish plugin"
This reverts commit a90a2b34fc.
2016-03-18 17:22:25 -07:00
Ryan Ernst 6af4c43c4f Merge pull request #17128 from rjernst/maven_publish
Build: Switch to maven-publish plugin
2016-03-17 11:53:50 -07:00
Jack Conradson 800c844ebd Split up Analyzer and Writer into multiple pieces.
Closes #17158
2016-03-17 10:15:13 -07:00
Simon Willnauer e91a141233 Prevent index level setting from being configured on a node level
Today we allow to set all kinds of index level settings on the node level which
is error prone and difficult to get right in a consistent manner.
For instance if some analyzers are setup in a yaml config file some nodes might
not have these analyzers and then index creation fails.

Nevertheless, this change allows some selected settings to be specified on a node level
for instance:
 * `index.codec` which is used in a hot/cold node architecture and it's value is really per node or per index
 * `index.store.fs.fs_lock` which is also dependent on the filesystem a node uses

All other index level setting must be specified on the index level. For existing clusters the index must be closed
and all settings must be updated via the API on each of the indices.

Closes #16799
2016-03-17 14:42:18 +01:00
Christoph Büscher 6ddf9ae92f Merge branch 'master' into feature-suggest-refactoring 2016-03-16 15:27:02 +01:00
Nik Everett 7197172047 [reindex] Properly register status
Without this commit fetching the status of a reindex from a node that isn't
coordinating the reindex will fail. This commit properly registers reindex's
status so this doesn't happen. To do so it moves all task status registration
into NetworkModule and creates a method to register other statuses which the
reindex plugin calls.
2016-03-16 07:40:49 -04:00
Christoph Büscher 39667b5793 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0/java.asciidoc
2016-03-16 12:06:42 +01:00
Ryan Ernst a90a2b34fc Build: Switch to maven-publish plugin
The build currently uses the old maven support in gradle. This commit
switches to use the newer maven-publish plugin. This will allow future
changes, for example, easily publishing to artifactory.

An additional part of this change makes publishing of build-tools part
of the normal publishing, instead of requiring a separate upload step
from within buildSrc. That also sets us up for a follow up to enable
precomit checks on the buildSrc code itself.
2016-03-15 19:16:37 -07:00
Jason Tedor 618441aea3 Merge pull request #17088 from jasontedor/simplify-bootstrap-settings
Bootstrap does not set system properties
2016-03-15 19:25:16 -04:00
Jason Tedor 66ba044ec5 Use setting in integration test cluster config 2016-03-15 17:45:17 -04:00
Yannick Welsch 7fb07a9a53 Merge pull request #17112 from ywelsch/enhance/forbid-sysout-for-tests
Forbid test sources to use System.out.println and Throwable.printStackTrace
2016-03-15 16:36:34 +01:00
Yannick Welsch f5e6db4090 Remove System.out.println and Throwable.printStackTrace from tests 2016-03-15 15:40:37 +01:00
Christoph Büscher b4b874f0d8 Merge branch 'master' into feature-suggest-refactoring 2016-03-15 12:11:39 +01:00
Christoph Büscher 1f1f6861b7 Merge branch 'master' into sort-serialization-scriptsort
Conflicts:
	core/src/test/java/org/elasticsearch/search/sort/AbstractSortTestCase.java
2016-03-15 11:03:28 +01:00
Simon Willnauer 345e988bbc Merge pull request #17072 from s1monw/add_backwards_rest_tests
Add infrastructure to run REST tests on a multi-version cluster

This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Given the amount of problems this change already found I think it's worth having this run with our test suite by default. The structure of this infra will likely change over time but for now it's a step into the right direction. We will likely want to split it up into integTests and integBwcTests etc. so each plugin can have it's own bwc tests but that's left for future refactoring.
2016-03-15 09:17:43 +01:00
Ali Beyad f527a034a3 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-03-14 17:25:02 -04:00
Jack Conradson dfec4547ea Added one minor comment for expressions tests. 2016-03-14 13:19:52 -07:00
Simon Willnauer bd96075f7f Merge branch 'master' into add_backwards_rest_tests 2016-03-14 21:18:42 +01:00
Jack Conradson 7665c4fe6f Merge branch 'master' into pr/17091 2016-03-14 12:16:21 -07:00
Christoph Büscher 40f3501d7f Merge branch 'master' into feature-suggest-refactoring 2016-03-14 14:57:57 +01:00
Christoph Büscher f0074668db Merge branch 'master' into sort-serialization-scriptsort 2016-03-14 12:15:53 +01:00
Christoph Büscher 97638c95fc Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-14 11:13:47 +01:00
Simon Willnauer 31740e279f Resolve index names to Index instances early
Today index names are often resolved lazily, only when they are really
needed. This can be problematic especially when it gets to mapping updates
etc. when a node sends a mapping update to the master but while the request
is in-flight the index changes for whatever reason we would still apply the update
since we use the name of the index to identify the index in the clusterstate.
The problem is that index names can be reused which happens in practice and sometimes
even in a automated way rendering this problem as realistic.
In this change we resolve the index including it's UUID as early as possible in places
where changes to the clusterstate are possible. For instance mapping updates on a node use a
concrete index rather than it's name and the master will fail the mapping update iff
the index can't be found by it's <name, uuid> tuple.

Closes #17048
2016-03-14 11:08:48 +01:00
Alexander Kazakov 8e6b2b3909 Check that _value is used in aggregations script before setting value to specialValue #14262 2016-03-14 12:04:06 +03:00
Adrien Grand 5596e31068 Upgrade to lucene-6.0.0-f0aa4fc. #17075 2016-03-14 07:58:52 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Simon Willnauer 121e7c8ca4 Add infrastructure to run REST tests on a multi-version cluster
This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Relates to #14406
Closes #17072
2016-03-13 10:52:39 +01:00
Ryan Ernst 5bd7da5659 Addressed PR feedback
* Fix tests still referring to -E
* add comment about missing classes
* rename writer constant
2016-03-11 11:46:23 -08:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Ali Beyad 31dcb3e18b Merge pull request #16873 from abeyad/suggester-wiring-refactoring
Change internal representation of suggesters
2016-03-11 12:34:35 -05:00
Christoph Büscher bbcbba1bf5 Fixing some tests and compile problems in reindex module 2016-03-11 17:58:13 +01:00
Christoph Büscher 5107388fe9 Added enum for script sort type 2016-03-11 17:32:39 +01:00
Christoph Büscher daeffb149c Merge branch 'master' into feature-suggest-refactoring 2016-03-11 10:37:28 +01:00
Yannick Welsch 04e55ecf6b Make logging message String constant to allow static checks 2016-03-11 10:30:59 +01:00
Yannick Welsch 718876a941 Fix wrong placeholder usage in logging statements 2016-03-11 10:30:59 +01:00
Nik Everett ebc12690bc [reindex] Move refresh tests to unit test
The refresh tests were failing rarely due to refreshes happening
automatically on indexes with -1 refresh intervals. This commit moves
the refresh test into a unit test where we can check if it was attempted
so we never get false failures from background refreshes.

It also stopped refresh from being run if the reindex request was canceled.
2016-03-10 17:48:22 -05:00
Nik Everett b2eec96045 [reindex] Make search failure cause rest failure
Indexing failures have caused the reindex http request to fail for a while
now. Both search and indexing failures cause it to abort. But search
failures didn't cause a non-200 response code from the http api. This
fixes that.

Also slips in a fix to some infrequently failing rest tests.

Closes #16037
2016-03-10 13:47:49 -05:00
Nik Everett b8d931d23c [reindex] Timeout if sub-requests timeout
Sadly, it isn't easy to simulate a timeout during an integration test, you
just have to cause one. Groovy's sleep should do the job.
2016-03-10 13:05:23 -05:00
Christoph Büscher 69c83b3459 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/action/suggest/TransportSuggestAction.java
        core/src/main/java/org/elasticsearch/search/suggest/SuggestParseElement.java
	core/src/main/java/org/elasticsearch/search/suggest/SuggestionSearchContext.java
	core/src/main/java/org/elasticsearch/search/suggest/phrase/PhraseSuggester.java
2016-03-10 11:44:36 +01:00
Nik Everett 38241a5d8b [reindex] Implement CompositeIndicesRequest
Implements CompositeIndicesRequest on UpdateByQueryRequest and
ReindexRequest so that plugins can reason about the request. In both cases
this implementation is imperfect but useful because instead of listing
all requests that make up the request it instead attempts to make dummy
requests that represent the requests that it will later make.
2016-03-09 16:29:23 -05:00
Simon Willnauer 016bc3ea2b Merge pull request #17001 from s1monw/cut_over_to_uuid
Use index UUID to lookup indices on IndicesService

Today we use the index name to lookup index instances on the IndicesService
which applied to search requests but also to index deletion etc. This commit
moves the interface to expect an Index instance which is a tuple
and looks up the index by uuid rather than by name. This prevents accidental modification
of the wrong index if and index is recreated or searching from the wrong index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.
2016-03-09 19:44:36 +01:00
Simon Willnauer e72dac91b3 Use index UUID to lookup indices on IndicesService
Today we use the index name to lookup index instances on the IndicesService
which applied to search reqeusts but also to index deletion etc. This commit
moves the interface to expcet and `Index` instance which is a <name, uuid> tuple
and looks up the index by uuid rather than by name. This prevents accidential modificaiton
of the wrong index if and index is recreated or searching from the _wrong_ index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.

Closes #17001
2016-03-09 19:42:15 +01:00
Ryan Ernst 1dafead2eb Fix precommit 2016-03-08 22:55:24 -08:00
Nik Everett 6d0efae713 Teach list tasks api to wait for tasks to finish
_wait_for_completion defaults to false. If set to true then the API will
wait for all the tasks that it finds to stop running before returning. You
can use the timeout parameter to prevent it from waiting forever. If you
don't set a timeout parameter it'll default to 30 seconds.

Also adds a log message to rest tests if any tasks overrun the test. This
is just a log (instead of failing the test) because lots of tasks are run
by the cluster on its own and they shouldn't cause the test to fail. Things
like fetching disk usage from the other nodes, for example.

Switches the request to getter/setter style methods as we're going that
way in the Elasticsearch code base. Reindex is all getter/setter style.

Closes #16906
2016-03-08 11:53:57 -05:00
Christoph Büscher 1264f37a1b Merge branch 'master' into feature-suggest-refactoring 2016-03-08 11:10:08 +01:00
Christoph Büscher 5ff413074a Adding tests for `time_zone` parameter for date range aggregation 2016-03-07 15:38:24 +01:00
Christoph Büscher 7ec5075a87 Merge branch 'master' into feature-suggest-refactoring 2016-03-07 10:45:43 +01:00
Adrien Grand c69fc008d4 Merge pull request #16972 from alexshadow007/fix-16812
Build empty extended stats aggregation if no docs collected for bucket
2016-03-07 10:44:20 +01:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00
Alexander Kazakov 3674774b73 Build empty extended stats aggregation if no docs are collected for bucket #16812 2016-03-05 22:50:54 +03:00
javanna da19922489 [TEST] adapt test infra to node.client setting removal
No changes were really needed in our test infra as it didn't use `node.client`. Yet it didn't take into account ingest nodes, what we used to call client nodes in InternalTestCluster were actually ingest only nodes, which now become coordinating only nodes.
Also renamed some method to get rid of the node client terminology as much as possible in favour or coordinating only node.
2016-03-05 10:55:19 +01:00
Nik Everett 4d6cb34417 [reindex] Add ingest support 2016-03-04 10:05:13 -05:00
Christoph Büscher 30a788d87c Suggestions: Make field name mandatory ctor argument
The field name is a required argument for all suggesters, but
it was specified via a field() setter in SuggestionBuilder so far.
This changes field name to being a mandatory constructor argument
and lets suggestion builders throw an error if field name is missing
or the empty string.
2016-03-03 21:35:53 +01:00
Lee Hinman 6adbbff97c Fix organization rename in all files in project
Basically a query-replace of "https://github.com/elasticsearch/" with "https://github.com/elastic/"
2016-03-03 12:04:13 -07:00
Clinton Gormley 529f7cb42c Fixed bad YAML in msearch test 2016-03-03 14:45:02 +01:00
Christoph Büscher ef4db5c1e4 Merge branch 'master' into feature-suggest-refactoring 2016-03-03 12:18:03 +01:00
Adrien Grand 2b545df372 Fix modules/reindex to not use the string field anymore. 2016-03-03 11:11:00 +01:00
Adrien Grand eef19be072 Deprecate string in favor of text/keyword. #16877
This commit removes the ability to use string fields on indices created on or
after 5.0. Dynamic mappings now generate text fields by default for strings
but there are plans to also add a sub keyword field (in a future PR).

Most of the changes in this commit are just about replacing string with
keyword or text. Some tests have been removed because they existed because of
corner cases of string mappings like setting ignore-above on a text field or
enabling term vectors on a keyword field which are now impossible.

The plan is to remove strings entirely in 6.0.
2016-03-03 10:20:56 +01:00
Nik Everett 18e5bb83c5 Disable problematic reindex test
This should get the builds back to normal while we wait on #16914 or
something like it to fix the test properly.
2016-03-02 13:02:29 -05:00
Nik Everett 942eb70956 Revert "Silence reindex's rest tests"
This reverts commit aa0ef84f5a.
2016-03-02 09:17:06 -05:00
Nik Everett aa0ef84f5a Silence reindex's rest tests
They are failing sporadically in CI.
2016-03-02 08:32:30 -05:00
Christoph Büscher be8ed737bc Suggest: Move name of suggestion to SuggestBuilder
Currently each suggestion keeps track of its own name. This has
the disadvantage of having to pass down the parsed name property
in the suggestions fromXContent() and the serialization methods
as an argument, since we need it in the ctor.

This change moves the naming of the suggestions to the surrounding
SuggestBuilder and by this eliminates the need for passind down
the names in the parsing and serialization methods. By making
`name` a required argument in SuggestBuilder#addSuggestion() we
also make sure it is always set and prevent using the same name twice,
which wasn't possible before.
2016-03-01 17:34:47 +01:00
Christoph Büscher aecf51cb42 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java
2016-03-01 17:20:15 +01:00
Nik Everett c7c8bb357a Merge pull request #16861 from nik9000/reindex_is_ready
Reindex required some parsing changes for search requests to support
differing defaults from the regular search api.
2016-03-01 10:02:48 -05:00
javanna b7597d7aea Rename SearchServiceTransportAction to SearchTransportService
The suffix TransportAction is misleading as it may make think that it extends TransportAction, but it does not. This class makes accessible the different search operations exposed by SearchService through the transport layer. Also resolved few compiler warnings in the class itself.
2016-03-01 11:35:04 +01:00
Nik Everett aeed7ee218 Reindex: rename source to searchRequest
This makes the code easier to read for those familiar with the
Elasticsearch code base.
2016-02-29 14:57:16 -05:00
Nik Everett 16934d50b6 Reindex isn't a plugin
Remove another mention of it being a plugin.
2016-02-29 10:21:10 -05:00
Nik Everett 356b7cd12b Merge reindex to master because it is ready! 2016-02-29 10:08:29 -05:00
Nik Everett d587a74533 Fix reindex to work with master branch
Stuff changed, reindex's got to change.
2016-02-29 10:04:16 -05:00
Christoph Büscher 379bd94c2a Merge branch 'master' into feature-suggest-refactoring 2016-02-29 14:06:30 +01:00
Yannick Welsch ab4e671bfd Remove suppressAccessChecks permission for Groovy script plugin
Closes #16527
2016-02-27 18:58:00 +01:00
Yannick Welsch af918ebf2b Upgrade groovy dependency in lang-groovy module to version 2.4.6
Closes #16830
2016-02-27 17:41:15 +01:00
Nik Everett c38119bae9 Merge branch 'master' into feature/reindex 2016-02-26 16:59:54 -05:00
Christoph Büscher d888b5c267 Merge branch 'master' into feature-suggest-refactoring 2016-02-22 11:22:28 -08:00
Jack Conradson 7986770e5f Moved Painless from a plugin to a module.
Closes #16755
2016-02-21 16:50:54 -08:00
Mike McCandless 5fffede2b0 Upgrade to Lucene 5.5.0 official release 2016-02-20 17:34:16 -05:00
Ali Beyad 5320d538f1 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-02-15 11:03:33 -05:00
Colin Goodheart-Smithe ebc745211d Moved AggregatorBuilder implementations into their own class files
Also renamed histogram.AbstractBuilcer to AbstractHistogramBuilder, range.AbstractBuilder to AbstractRangeBuilder and org.elasticsearch.search.aggregations.pipeline.having to org.elasticsearch.search.aggregations.pipeline.bucketselector
2016-02-15 14:09:36 +00:00
Colin Goodheart-Smithe 1a46628daa Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java
2016-02-15 10:37:16 +00:00
Jason Tedor 1fb1ffbac6 Class permission for Groovy references
This commit adds a class permission for groovy.lang.Reference so they
can be used in scripts.

Closes #16660
2016-02-14 09:45:40 -05:00
Ali Beyad beea413820 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-02-12 11:54:44 -05:00
Simon Willnauer 297fb24112 apply fixes after review 2016-02-12 12:53:18 +01:00
Colin Goodheart-Smithe ed3f7903f4 Merge branch 'master' into feature/aggs-refactoring 2016-02-12 09:48:59 +00:00
Simon Willnauer 685bee3081 Merge branch 'master' into add_rewrite_infra 2016-02-12 09:46:01 +01:00
Ali Beyad eed557742f Refactors building query specific objects from the term suggestion
builder and merges changes between the suggestion builders to context
object implementations.
2016-02-11 18:30:58 -05:00