Commit Graph

288 Commits

Author SHA1 Message Date
javanna f2acf466aa Convert script/template objects to json format
Elasticsearch accepts multiple content-type formats, hence scripts can be stored/provided in json, yaml, cbor or smile. Yet the format that should be used internally is json. This is a problem mainly around search templates, as they only support json out of the four content-types, so instead of maintaining the content-type of the request we should rather convert the scripts/templates to json.

 Binary formats were not previously supported. If you stored a template in yaml format, you'd get back an error "No encoder found for MIME type [application/yaml]" when trying to execute it. With this commit the request content-type is independent from the template, which always gets converted to json internally. That is transparent to users and doesn't affect the content type of the response obtained when executing the template.
2017-02-22 16:20:53 +01:00
javanna 9391c6ffa9 Replace CustomMustacheFactory constant with same constant from Script (CONTENT_TYPE_OPTION) 2017-02-22 16:20:53 +01:00
Tanguy Leroux e2e5937455 Use `typed_keys` parameter to prefix suggester names by type in search responses (#23080)
This pull request reuses the typed_keys parameter added in #22965, but this time it applies it to suggesters. When set to true, the suggester names in the search response will be prefixed with a prefix that reflects their type.
2017-02-10 10:53:38 +01:00
Tanguy Leroux 3553522328 Add parameter to prefix aggs name with type in search responses (#22965)
This pull request adds a new parameter to the REST Search API named `typed_keys`. When set to true, the aggregation names in the search response will be prefixed with a prefix that reflects the internal type of the aggregation.

Here is a simple example:
```
GET /_search?typed_keys
{
    "aggs": {
        "tweets_per_user": {
            "terms": {
                "field": "user"
            }
        }
    },
    "size": 0
}
```

And the response:

```
{
    "aggs": {
        "sterms:tweets_per_user": {
            ...
        }
    }
}
```

This parameter is intended to make life easier for REST clients that could parse back the prefix and could detect the type of the aggregation to parse. It could also be implemented for suggesters.
2017-02-09 11:19:04 +01:00
Simon Willnauer ecb01c15b9 Fold InternalSearchHits and friends into their interfaces (#23042)
We have a bunch of interfaces that have only a single implementation
for 6 years now. These interfaces are pretty useless from a SW development
perspective and only add unnecessary abstractions. They also require
lots of casting in many places where we expect that there is only one
concrete implementation. This change removes the interfaces, makes
all of the classes final and removes the duplicate `foo` `getFoo` accessors
in favor of `getFoo` from these classes.
2017-02-08 14:40:08 +01:00
Jay Modi c898e8ab83 Add support for newline delimited JSON Content-Type (#22947)
This commit adds support for the newline delimited JSON Content-Type, which is how
the bulk, multi-search, and multi-search template APIs expect data to be formatted. The
`elasticsearch-js` client has also been using this content type for these types of requests.

Closes #22943
2017-02-07 09:20:06 -05:00
Jason Tedor 9a0b216c36 Upgrade checkstyle to version 7.5
This commit upgrades the checkstyle configuration from version 5.9 to
version 7.5, the latest version as of today. The main enhancement
obtained via this upgrade is better detection of redundant modifiers.

Relates #22960
2017-02-03 09:46:44 -05:00
Jay Modi 7520a107be Optionally require a valid content type for all rest requests with content (#22691)
This change adds a strict mode for xcontent parsing on the rest layer. The strict mode will be off by default for 5.x and in a separate commit will be enabled by default for 6.0. The strict mode, which can be enabled by setting `http.content_type.required: true` in 5.x, will require that all incoming rest requests have a valid and supported content type header before the request is dispatched. In the non-strict mode, the Content-Type header will be inspected and if it is not present or not valid, we will continue with auto detection of content like we have done previously.

The content type header is parsed to the matching XContentType value with the only exception being for plain text requests. This value is then passed on with the content bytes so that we can reduce the number of places where we need to auto-detect the content type.

As part of this, many transport requests and builders were updated to provide methods that
accepted the XContentType along with the bytes and the methods that would rely on auto-detection have been deprecated.

In the non-strict mode, deprecation warnings are issued whenever a request with body doesn't provide the Content-Type header.

See #19388
2017-02-02 14:07:13 -05:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id (#22206)
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
Tim Brooks a4ac29c005 Add single static instance of SpecialPermission (#22726)
This commit adds a SpecialPermission constant and uses that constant
opposed to introducing new instances everywhere.

Additionally, this commit introduces a single static method to check that
the current code has permission. This avoids all the duplicated access
blocks that exist currently.
2017-01-21 12:03:52 -06:00
Nik Everett 6265ef1c1b Deguice rest handlers (#22575)
There are presently 7 ctor args used in any rest handlers:
* `Settings`: Every handler uses it to initialize a logger and
  some other strange things.
* `RestController`: Every handler registers itself with it.
* `ClusterSettings`: Used by `RestClusterGetSettingsAction` to
  render the default values for cluster settings.
* `IndexScopedSettings`: Used by `RestGetSettingsAction` to get
  the default values for index settings.
* `SettingsFilter`: Used by a few handlers to filter returned
  settings so we don't expose stuff like passwords.
* `IndexNameExpressionResolver`: Used by `_cat/indices` to
  filter the list of indices.
* `Supplier<DiscoveryNodes>`: Used to fill enrich the response
  by handlers that list tasks.

We probably want to reduce these arguments over time but
switching construction away from guice gives us tighter
control over the list of available arguments.

These parameters are passed to plugins using
`ActionPlugin#initRestHandlers` which is expected to build and
return that handlers immediately. This felt simpler than
returning an reference to the ctors given all the different
possible args.

Breaks java plugins by moving rest handlers off of guice.
2017-01-20 11:48:51 -05:00
Tanguy Leroux f5542ed47f Simplify ElasticsearchException rendering as a XContent (#22611)
This commit tries to simplify the way ElasticsearchException are rendered to xcontent. It adds some documentation and renames and merges some methods. Current behavior is preserved, the goal is to be more readable and centralize everything in the ElasticsearchException class.
2017-01-17 15:44:49 +01:00
javanna 64c3212fdb Remove ParseFieldMatcher usages from IndexSettings 2017-01-12 14:43:35 +01:00
javanna 8072f168a3 Remove ParseFieldMatcher usages from QueryParseContext 2017-01-12 14:43:35 +01:00
Nik Everett abb7d7841f Remove SearchRequestParsers (#22538)
It is empty now that we've moved all the parsing into `namedObject`.
2017-01-11 10:28:14 -05:00
Nik Everett 3fb9254b95 Replace Suggesters with namedObject (#22491)
Removes another parser registery type thing in favor of
`XContentParser#namedObject`.
2017-01-09 16:51:08 -05:00
Nik Everett e3f77b4795 Replace AggregatorParsers with namedObject (#22397)
Removes `AggregatorParsers`, replacing all of its functionality with
`XContentParser#namedObject`.

This is the third bit of payoff from #22003, one less thing to pass
around the entire application.
2017-01-09 13:59:38 -05:00
Nik Everett f4884e0726 Replace SearchExtRegistry with namedObject (#22492)
This is one of the last things in `SearchRequestParsers`.
2017-01-09 08:35:54 -05:00
javanna ded694fc83 Make StatusToXContent extend ToXContentObject and rename it to StatusToXContentObject
This also allows to make RestToXContentListener require ToXContentObject rather than ToXContent
2017-01-06 23:31:48 +01:00
javanna d5510701a0 Make SearchResponse a ToXContentObject 2017-01-06 23:31:48 +01:00
javanna 45d4938fcc Migrate some more responses to ToXContentObject 2017-01-06 23:31:48 +01:00
javanna dea7d65439 remove ParseFieldMatcher usages from RestSearchTemplateAction 2017-01-05 19:33:04 +01:00
javanna 6102523033 remove ParseFieldMatcher usages from Script parsing code 2017-01-05 19:33:04 +01:00
Daniel Mitterdorfer 1ed64f0551 Eliminate unneccessary declaration of IOException
With this commit we remove the declaration of IOException from
assertWarnings and modify all call sites.

Checked with @javanna
2017-01-03 12:36:28 +01:00
Nik Everett f5f2149ff2 Remove much ceremony from parsing client yaml test suites (#22311)
* Remove a checked exception, replacing it with `ParsingException`.
* Remove all Parser classes for the yaml sections, replacing them with static methods.
* Remove `ClientYamlTestFragmentParser`. Isn't used any more.
* Remove `ClientYamlTestSuiteParseContext`, replacing it with some static utility methods.

I did not rewrite the parsers using `ObjectParser` because I don't think it is worth it right now.
2016-12-22 11:00:34 -05:00
Nik Everett 567c65b0d5 Replace IndicesQueriesRegistry (#22289)
* Switch query parsing to namedObject
* Remove IndicesQueriesRegistry
2016-12-21 09:05:14 -05:00
Nik Everett a04dcfb95b Introduce XContentParser#namedObject (#22003)
Introduces `XContentParser#namedObject which works a little like
`StreamInput#readNamedWriteable`: on startup components register
parsers under names and a superclass. At runtime we look up the
parser and call it to parse the object.

Right now the parsers take a context object they use to help with
the parsing but I hope to be able to eliminate the need for this
context as most what it is used for at this point is to move
around parser registries which should be replaced by this method
eventually. I make no effort to do so in this PR because it is
big enough already. This is meant to the a start down a road that
allows us to remove classes like `QueryParseContext`,
`AggregatorParsers`, `IndicesQueriesRegistry`, and
`ParseFieldRegistry`.

The goal here is to reduce the amount of plumbing required to
allow parsing pluggable things. With this you don't have to pass
registries all over the place. Instead you must pass a super
registry to fewer places and use it to wrap the reader. This is
the same tradeoff that we use for NamedWriteable and it allows
much, much simpler binary serialization. We think we want that
same thing for xcontent serialization.

The only parsing actually converted to this method is parsing
`ScoreFunctions` inside of `FunctionScoreQuery`. I chose this
because it is relatively self contained.
2016-12-20 11:05:24 -05:00
javanna 5dae10db11 [TEST] add warnings check to ESTestCase
We are currenlty checking that no deprecation warnings are emitted in our query tests. That can be moved to ESTestCase (disabled in ESIntegTestCase) as it allows us to easily catch where our tests use deprecated features and assert on the expected warnings.
2016-12-19 19:39:56 +01:00
javanna 6a27628f12 Remove support for strict parsing mode
We return deprecation warnings as response headers, besides logging them. Strict parsing mode stayed around, but was only used in query tests, though we also introduced checks for deprecation warnings there that don't need strict parsing anymore (see #20993).

 We can then safely remove support for strict parsing mode. The final goal is to remove the ParseFieldMatcher class, but there are many many users of it. This commit prepares the field for the removal, by deprecating ParseFieldMatcher and making it effectively not needed. Strict parsing is removed from ParseFieldMatcher, and strict parsing is replaced in tests where needed with deprecation warnings checks.

 Note that the setting to enable strict parsing was never ported to the new settings infra hance it cannot be set in production. It is really only used in our own tests.

 Relates to #19552
2016-12-19 19:39:56 +01:00
Nik Everett 49bdd29f91 Consolidate more parser creation into ESTestCase
This will make it easier to add the forthcoming required argument,
`NamedXContentRegistry`.
2016-12-13 20:28:41 -05:00
Nik Everett 872984d21a Continue consolidating `XContentParser` construction in tests (#22145)
Consolidate more parser creation in tests

Moves more parser creation in tests to the `createParser` methods
in `ESTestCase`.
2016-12-13 17:22:39 -05:00
Luca Cavanna 6d987a9b69 Remove support for empty queries (#22092)
Our query DSL supports empty queries (`{}`), which have a different meaning depending on the query that holds it, either ignored, match_all or match_none. We deprecated the support for empty queries in 5.0, where we log a deprecation warning wherever they are used.

The way we supported it once we moved query parsing to the coordinating node was having an Optional<QueryBuilder> return type in all of our parse methods (called fromXContent). See #17624. The central place for this was QueryParseContext#parseInnerQueryBuilder. We can now remove all the optional return types and simply throw an exception whenever an empty query is found.
2016-12-12 12:37:12 +01:00
Nik Everett 3adefb7b4a Begin centralizing XContentParser creation into RestRequest (#22041)
To get #22003 in cleanly we need to centralize as much `XContentParser` creation as possible into `RestRequest`. That'll mean we have to plumb the `NamedXContentRegistry` into fewer places.

This removes `RestAction.hasBody`, `RestAction.guessBodyContentType`, and `RestActions.getRestContent`, moving callers over to `RestRequest.hasContentOrSourceParam`, `RestRequest.contentOrSourceParam`, and `RestRequest.contentOrSourceParamParser` and `RestRequest.withContentOrSourceParamParserOrNull`. The idea is to use `withContentOrSourceParamParserOrNull` if you need to handle requests without any sort of body content and to use `contentOrSourceParamParser` otherwise.

I believe the vast majority of this PR to be purely mechanical but I know I've made the following behavioral change (I'll add more if I think of more):
* If you make a request to an endpoint that requires a request body and has cut over to the new APIs instead of getting `Failed to derive xcontent` you'll get `Body required`.
* Template parsing is now non-strict by default. This is important because we need to be able to deprecate things without requests failing.
2016-12-09 20:23:02 -05:00
Boaz Leskes fe01c0f83b fix TemplateQueryBuilderTests & Murmur3FieldMapperTests 2016-12-01 14:21:57 +01:00
Luca Cavanna 5b8bdba12e Remove subrequests method from CompositeIndicesRequest (#21873) 2016-11-30 15:03:58 +01:00
Luca Cavanna 6eaff9432d SearchTemplateRequest to implement CompositeIndicesRequest (#21865)
SearchTemplateRequest to implement CompositeIndicesRequest

Given that SearchTemplateRequest effectively delegates to search when a search is being executed, it should implement the CompositeIndicesRequest interface. The subrequests method should return a single search request. When a search is not going to be executed, because we are in simulate mode, there are no inner requests, and there are no corresponding indices to that request either.

Closes #21747
2016-11-29 20:52:43 +01:00
Ryan Ernst d14c470b89 Remove generics from ActionRequest
closes #21368
2016-11-14 15:32:01 -08:00
Jack Conradson aeb97ff412 Clean up of Script.
Closes #21321
2016-11-10 09:59:13 -08:00
Ryan Ernst 7a2c984bcc Test: Remove multi process support from rest test runner (#21391)
At one point in the past when moving out the rest tests from core to
their own subproject, we had multiple test classes which evenly split up
the tests to run. However, we simplified this and went back to a single
test runner to have better reproduceability in tests. This change
removes the remnants of that multiplexing support.
2016-11-07 15:07:34 -08:00
Christoph Büscher b3370de715 Tests: Add warning header checks to QueryBuilder tests and QueryParseContextTests
This adds checks for expected warning headers to the query builder test
infrastructure. Tests that are adding deprecation warnings to the response
headers need to check those, otherwise the abstract base class for the test
class will complain at teardown.
2016-11-02 15:45:33 +01:00
Jack Conradson 185dff7346 Cleanup ScriptType (#21179)
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
2016-10-31 13:48:51 -07:00
Jack Conradson 512a77a633 Refactor ScriptType to be a top-level class. 2016-10-26 10:21:22 -07:00
Tanguy Leroux e71c30c71d Mustache: Add {{#url}}{{/url}} function to URL encode strings (#20838)
This commit adds a new Mustache function (codename: url) and a new URLEncoder that can be used to URL encode strings in mustache templates.
2016-10-13 16:17:28 +02:00
Simon Willnauer 9c9afe3f01 Remove SearchContext#current and all it's threadlocals (#20778)
Today SearchContext expose the current context as a thread local which makes any kind of sane interface design very very hard. This PR removes the thread local entirely and instead passes the relevant context anywhere needed. This simplifies state management dramatically and will allow for a much leaner SearchContext interface down the road.
2016-10-06 19:51:54 +02:00
Simon Willnauer 587bdcef38 add extra safety when accessing scripts or now and reqeusts are cached 2016-10-05 09:41:48 +02:00
Simon Willnauer 80bf44f709 Merge branch 'master' into dont_cache_scripts 2016-10-04 21:09:40 +02:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates #20722
2016-10-04 12:45:29 -04:00
Simon Willnauer 94b7873b49 Add a #markAsNotCachable() method to context to mark requests as not cachable 2016-10-04 18:05:00 +02:00
David Pilato ed4d0881b1 Add profile and explain parameters to template API
We can now run templates using `explain` and/or `profile` parameters.
Which is interesting when you have defined a complicated profile but want to debug it in an easier way than running the full query again.

You can use `explain` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "explain": true
}
```

You can use `profile` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "profile": true
}
```
2016-09-19 17:52:13 +02:00
javanna 90ab460fcc move parsing of search ext sections to the coordinating node 2016-09-09 19:10:42 +02:00
Alexander Reelsen b6bf20c2da Dependencies: Updates to mustache 0.9.3 (#20337) 2016-09-06 13:31:59 +02:00
Jason Tedor abf8a1a3f0 Avoid allocating log parameterized messages
This commit modifies the call sites that allocate a parameterized
message to use a supplier so that allocations are avoided unless the log
level is fine enough to emit the corresponding log message.
2016-08-30 18:17:09 -04:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Ryan Ernst 743d9fd008 Merge branch 'master' into search_parser 2016-08-16 11:28:59 -07:00
Ryan Ernst 7fde410586 Internal: Consolidate search parser registries
Parsing a search request is currently split up among a number of
classes, using multiple public static methods, which take multiple
regstries of elements that may appear in the search request like query
parsers and aggregations. This change begins consolidating all this code
by collapsing the registries normally used for parsing search requests
into a single SearchRequestParsers class. It is also made available to
plugin services to enable templating of search requests.  Eventually all
of the actual parsing logic should move to the class, and the registries
should be hidden, but for now they are at least co-located to reduce the
number of objects that must be passed around.
2016-08-16 01:59:24 -07:00
Nik Everett 1452ab4b9f Squash the rest of o.e.rest.action
Squashes all the subpackages of `org.elasticsearch.rest.action` down to
the following:
* `o.e.rest.action.admin` - Administrative actions
* `o.e.rest.action.cat` - Actions that make tables for `grep`ing
* `o.e.rest.action.document` - Actions that act on documents
* `o.e.rest.action.ingest` - Actions that act on ingest pipelines
* `o.e.rest.action.search` - Actions that search

I'm tempted to merge `search` into `document` but the `document`
package feels fairly complete as is and `Suggest` isn't actually always
about documents either....

I'm also tempted to merge `ingest` into `admin.cluster` because the
latter contains the actions for dealing with stored scripts.

I've moved the `o.e.rest.action.support` into `o.e.rest.action`.

I've also added `package-info.java`s to all packges in `o.e.rest`. I
figure if the package is too small to deserve a `package-info.java` file
then it is too small to deserve to be a package....

Also fixes checkstyle in all moved classes.
2016-08-15 21:06:32 -04:00
Lee Hinman 5849c488b5 Merge remote-tracking branch 'dakrone/compliation-breaker' 2016-08-09 11:57:26 -06:00
Lee Hinman 2be52eff09 Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.

If more dynamic scripts are sent, a user will get the following
exception:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "a5V1eXcZRYiIk8lecjZ4Jw",
        "reason" : {
          "type" : "general_script_exception",
          "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
          "caused_by" : {
            "type" : "circuit_breaking_exception",
            "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
            "bytes_wanted" : 0,
            "bytes_limit" : 0
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "general_script_exception",
      "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    }
  },
  "status" : 500
}
```

This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.

Relates to #19396
2016-08-09 10:26:27 -06:00
Tanguy Leroux 841d5a210e Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects. It also adds the snakeyaml dependency that was previously shaded in jackson libs.

It also closes #18076
2016-08-05 12:26:06 +02:00
Nik Everett 1e587406d8 Fail yaml tests and docs snippets that get unexpected warnings
Adds `warnings` syntax to the yaml test that allows you to expect
a `Warning` header that looks like:
```
    - do:
        warnings:
            - '[index] is deprecated'
            - quotes are not required because yaml
            - but this argument is always a list, never a single string
            - no matter how many warnings you expect
        get:
            index:    test
            type:    test
            id:        1
```

These are accessible from the docs with:
```
// TEST[warning:some warning]
```

This should help to force you to update the docs if you deprecate
something. You *must* add the warnings marker to the docs or the build
will fail. While you are there you *should* update the docs to add
deprecation warnings visible in the rendered results.
2016-08-04 15:23:05 -04:00
Tanguy Leroux 386902903e [TEST] Kill remaining lang-groovy messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

The work started with #19280, #19302 and #19336 and this PR moves the remaining messy tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests to integration tests.

It also moves IndexLookupIT test back (even if it has good chance to be removed soon) and fixes its tests.

It also changes AbstractQueryTestCase to use custom script plugins in tests.

closes #13837
2016-08-01 16:59:47 +02:00
Nik Everett 303c9faca5 Squash o.e.rest.action.admin.cluster
In an effort to reduce the number of tiny packages we have in the
code base this moves all the files that were in subdirectories of
`org.elasticsearch.rest.action.admin.cluster` into
`org.elasticsearch.rest.action.admin.cluster`.

Also fixes line length in these packages.
2016-07-29 20:31:24 -04:00
Martijn van Groningen d1b7a335cb removed useless comment and make sure all public constructors delegate to the package protected constructor. 2016-07-27 10:01:36 +02:00
Martijn van Groningen 2fdf79d8d4 Deprecate template query.
Closes #19390
2016-07-27 09:50:44 +02:00
Nik Everett 9270e8b22b Rename client yaml test infrastructure
This makes it obvious that these tests are for running the client yaml
suites. Now that there are other ways of running tests using the REST
client against a running cluster we can't go on calling the shared
client yaml tests "REST tests". They are rest tests, but they aren't
**the** rest tests.
2016-07-26 13:53:44 -04:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes #19079

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Martijn van Groningen a9ab095b8c Moved all mustache classes into one package.
No need for multiple packages inside a small module.
2016-07-25 18:50:12 +02:00
Martijn van Groningen a784055db1 Cleaned up the tests in lang-mustache.
Messy tests with mustache were either moved to core, moved to a rest test or remained untouched if they actually tested mustache.
Also removed tests that were redundant.
2016-07-25 17:57:39 +02:00
Nik Everett 3a82c613e4 Migrate query registration from push to pull
Remove `ParseField` constants used for names where there are no deprecated
names and just use the `String` version of the registration method instead.

This is step 2 in cleaning up the plugin interface for extending
search time actions. Aggregations are next.

This is breaking for plugins because those that register a new query should
now implement `SearchPlugin` rather than `onModule(SearchModule)`.
2016-07-20 12:33:51 -04:00
Martijn van Groningen e0ebf5da1c Template cleanup:
* Removed `Template` class and unified script & template parsing logic. Templates are scripts, so they should be defined as a script. Unless there will be separate template infrastructure, templates should share as much code as possible with scripts.
* Removed ScriptParseException in favour for ElasticsearchParseException
* Moved TemplateQueryBuilder to lang-mustache module because this query is hard coded to work with mustache only
2016-07-18 10:16:01 +02:00
Martijn van Groningen 2c3165d080 Removed deprecated 1.x script and template syntax
Closes #13729
2016-07-13 15:07:36 +02:00
Nik Everett 8263873783 Switch search extension from push to pull
Switches most search behavior extensions from push (`onModule(SearchModule)`)
to pull (`implements SearchPlugin`). This effort in general gives plugin
authors a much cleaner view of how to extend Elasticsearch and starts to
set up portions of Elasticsearch as "the plugin API". This commit in
particular does that for search-time behavior like customized suggesters,
highlighters, score functions, and significance heuristics.

It also switches most such customization to being done at search module
construction time which is much, much easier to reason about from a testing
perspective. It also helps significantly in the process of de-guice-ing
Elasticsearch's startup.

There are at least two major search time extensions that aren't covered in
this commit that will simply have to wait for the next commit on the topic
because this one has already grown large: custom aggregations and custom
queries. These will likely live in the same SearchPlugin interface as well.
2016-07-11 18:49:05 -04:00
Jason Tedor 96f283c195 Rename writeThrowable to writeException
This commit renames writeThrowable to writeException. The situation here
stems from the fact that the StreamOutput method for serializing
Exceptions needs to accept Throwables too as Throwables can be the cause
of serialized Exceptions. Yet, we do not serialize Throwables in the
Error sub-hierarchy in a way that they can be deserialized into their
initial type. This leads to an asymmetry in the StreamOutput method for
serializing Exceptions and the StreamInput method for writing
Excpetions. Namely, the former will accept Throwables but the latter
will only return Exceptions. A goal with the stream methods has always
been symmetry in the method names so that serialization/deserialization
routines appear symmetrical in code. It is this asymmetry on the
input/output types for Exceptions on StreamOutput/StreamInput that
clashes with the desired symmetry of naming. Despite this, we should
favor symmetry in the naming of the methods. This commit renames
StreamOutput#writeThrowable to StreamOutput#writeException which leaves
us with Exception StreamInput#readException and void
StreamOutput#writeException(Throwable).
2016-07-05 14:37:01 -04:00
Jason Tedor 3343ceeae4 Do not catch throwable
Today throughout the codebase, catch throwable is used with reckless
abandon. This is dangerous because the throwable could be a fatal
virtual machine error resulting from an internal error in the JVM, or an
out of memory error or a stack overflow error that leaves the virtual
machine in an unstable and unpredictable state. This commit removes
catch throwable from the codebase and removes the temptation to use it
by modifying listener APIs to receive instances of Exception instead of
the top-level Throwable.

Relates #19231
2016-07-04 08:41:06 -04:00
Tanguy Leroux c8a9b28d1b Rename Search Template REST tests 2016-07-04 10:07:45 +02:00
Tanguy Leroux 8c40b2b54e Fix order of modifiers 2016-07-01 16:57:14 +02:00
Simon Willnauer 5c8164a561 Clean up BytesReference (#19196)
BytesReference should be a really simple interface, yet it has a gazillion
ways to achieve the same this. Methods like `#hasArray`, `#toBytesArray`, `#copyBytesArray`
`#toBytesRef` `#bytes` are all really duplicates. This change simplifies the interface
dramatically and makes implementations of it much simpler. All array access has been removed
and is streamlined through a single `#toBytesRef` method. Utility methods to materialize a
compact byte array has been added too for convenience.
2016-07-01 16:09:31 +02:00
Ryan Ernst 8275ab497b Merge pull request #19170 from rjernst/rest_handler_client
Changed rest handler interface to take NodeClient
2016-06-30 11:00:09 -07:00
Nik Everett f5a269b029 Start migration away from aggregation streams
We'll migrate to NamedWriteable so we can share code with the rest
of the system. So we can work on this in multiple pull requests without
breaking Elasticsearch in between the commits this change supports
*both* old style `InternalAggregations.stream` serialization and
`NamedWriteable` style serialization. As such it creates about a
half dozen `// NORELEASE` comments that will have to be removed
once the migration is complete.

This also introduces a boolean `transportClient` flag to `SearchModule`
which is used to skip inappropriate registrations for for the
transport client while still registering the things it needs. In
this case that means that the `InternalAggregation` subclasses are
registered with the `NamedWriteableRegistry` but the `AggregationBuilder`
subclasses are not.

Finally, this moves aggregation registration from guice configuration
time to `SearchModule` construction time. This will make it simpler to
work with in the future as we further clean up Elasticsearch's
extension points.
2016-06-30 12:57:34 -04:00
Ryan Ernst c762e7aa15 Merge branch 'master' into rest_handler_client 2016-06-30 08:16:25 -07:00
Ryan Ernst c77dc4a82c Merge pull request #19136 from rjernst/script_service_deps
Scripts: Remove ClusterState from compile api
2016-06-29 22:34:40 -07:00
Ryan Ernst 865b951b7d Internal: Changed rest handler interface to take NodeClient
Previously all rest handlers would take Client in their injected ctor.
However, it was only to hold the client around for runtime. Instead,
this can be done just once in the HttpService which handles rest
requests, and passed along through the handleRequest method. It also
should always be a NodeClient, and other types of Clients (eg a
TransportClient) would not work anyways (and some handlers can be
simplified in follow ups like reindex by taking NodeClient).
2016-06-29 18:02:18 -07:00
Nik Everett 8db43c0107 Move RestHandler registration to ActionModule and ActionPlugin
`RestHandler`s are highly tied to actions so registering them in the
same place makes sense.

Removes the need to for plugins to check if they are in transport client
mode before registering a RestHandler - `getRestHandlers` isn't called
at all in transport client mode.

This caused guice to throw a massive fit about the circular dependency
between NodeClient and the allocation deciders. I broke the circular
dependency by registering the actions map with the node client after
instantiation.
2016-06-29 18:31:44 -04:00
Tanguy Leroux 4820d49120 Mustache: Add util functions to render JSON and join array values
This pull request adds two util functions to the Mustache templating engine:
- {{#toJson}}my_map{{/toJson}} to render a Map parameter as a JSON string
- {{#join}}my_iterable{{/join}} to render any iterable (including arrays) as a comma separated list of values like `1, 2, 3`. It's also possible de change the default delimiter (comma) to something else.

closes #18970
2016-06-29 09:48:58 +02:00
Ryan Ernst ecf6101798 Scripts: Remove ClusterState from compile api
Stored scripts are pulled from the cluster state, and the current api
requires passing the ClusterState on each call to compile. However, this
means every user of the ScriptService needs to depend on the
ClusterService. Instead, this change makes the ScriptService a
ClusterStateListener. It also simplifies tests a lot, as they no longer
need to create fake cluster states (except when testing stored scripts).
2016-06-28 13:20:00 -07:00
Nik Everett fa4844c3f4 Pull actions from plugins
Instead of implementing onModule(ActionModule) to register actions,
this has plugins implement ActionPlugin to declare actions. This is
yet another step in cleaning up the plugin infrastructure.

While I was in there I switched AutoCreateIndex and DestructiveOperations
to be eagerly constructed which makes them easier to use when
de-guice-ing the code base.
2016-06-28 08:36:24 -04:00
Ryan Ernst 33ccc5aead Merge branch 'master' into mapper_plugin_api 2016-06-27 11:19:59 -07:00
Nik Everett 71b95fb63c Switch analysis from push to pull
Instead of plugins calling `registerTokenizer` to extend the analyzer
they now instead have to implement `AnalysisPlugin` and override
`getTokenizer`. This lines up extending plugins in with extending
scripts. This allows `AnalysisModule` to construct the `AnalysisRegistry`
immediately as part of its constructor which makes testing anslysis
much simpler.

This also moves the default analysis configuration into `AnalysisModule`
which is how search is setup.

Like `ScriptModule`, `AnalysisModule` no longer extends `AbstractModule`.
Instead it is only responsible for building `AnslysisRegistry`. We still
bind `AnalysisRegistry` but we only do so in `Node`. This is means it
is available at module construction time so we slowly remove the need to
bind it in guice.
2016-06-26 07:15:42 -04:00
Ryan Ernst 6995bde710 Merge branch 'master' into mapper_plugin_api 2016-06-24 11:15:06 -07:00
Tanguy Leroux 04da1bda0d Move templates out of the Search API, into lang-mustache module
This commit moves template support out of the Search API to its own dedicated Search Template API in the lang-mustache module. It provides a new SearchTemplateAction that can be used to render templates before it gets delegated to the usual Search API. The current REST endpoint are identical, but the Render Search Template endpoint now uses the same Search Template API with a new "simulate" option. When this option is enabled, the Search Template API only renders template and returns immediatly, without executing the search.

Closes #17906
2016-06-23 09:30:53 +02:00
Ryan Ernst e817b5daa3 Plugins: Remove guice from Mapper plugins
This changes adds a MapperPlugin interface which allows pull style
retrieval of mappers and metadata mappers added by plugins. For now, I
have kept the MapperRegistry, but this should be removed in the future
as it is just a silly container for 2 maps which could themselves be
passed around.
2016-06-21 22:50:39 -07:00
Nik Everett 8925400f67 Remove guice from ScriptService
Makes ScriptModule just a plain class that manages building the
ScriptSettings and ScriptService from plugins. When we *need*
to bind ScriptService with guice we bind it in a lambda.
2016-06-21 16:45:45 -04:00
Simon Willnauer 7fea5bd8e7 Remove obsolete Modules that can simply be inlined in node creation 2016-06-20 11:28:14 +02:00
Tanguy Leroux 98951b1203 Compile each Groovy script in its own classloader
closes #18572
2016-06-20 08:17:09 +02:00
Areek Zillur 545ffa7801 Merge branch 'master' into enhancement/rollover_api 2016-06-17 10:33:11 -04:00
Areek Zillur 6adffa6b7b Merge branch 'master' into enhancement/rollover_api 2016-06-16 17:27:32 -04:00
Ryan Ernst 8196cf01e3 Merge branch 'master' into plugin_name_api 2016-06-16 13:49:28 -07:00
Simon Willnauer b22c526b34 Cut over settings registration to a pull model (#18890)
Today we have a push model for registering basically anything. All our extension points
are defined on modules which we pass in to plugins. This is harder to maintain and adds
unnecessary dependencies on the modules itself. This change moves towards a pull model
where the plugin offers a getter kind of method to get the extensions. This will also
help in the future if we need to pass dependencies to the extension points which can
easily be defined on the method as arguments if a pull model is used.
2016-06-16 15:52:58 +02:00
Simon Willnauer 18ff051ad5 Simplify ScriptModule and script registration (#18903)
Registering a script engine or native scripts still uses Guice today
and is much more complicated than needed. This change moves to a pull
based model where script plugins have to implement a dedicated interface
`ScriptPlugin` and defines simple getter returning instances rather than
classes.
2016-06-16 09:35:13 +02:00
Ryan Ernst a4503c2aed Plugins: Remove name() and description() from api
In 2.0 we added plugin descriptors which require defining a name and
description for the plugin. However, we still have name() and
description() which must be overriden from the Plugin class. This still
exists for classpath plugins. But classpath plugins are mainly for
tests, and even then, referring to classpath plugins with their class is
a better idea. This change removes name() and description(), replacing
the name for classpath plugins with the full class name.
2016-06-15 17:12:22 -07:00
Jason Tedor da74323141 Register thread pool settings
This commit refactors the handling of thread pool settings so that the
individual settings can be registered rather than registering the top
level group. With this refactoring, individual plugins must now register
their own settings for custom thread pools that they need, but a
dedicated API is provided for this in the thread pool module. This
commit also renames the prefix on the thread pool settings from
"threadpool" to "thread_pool". This enables a hard break on the settings
so that:
 - some of the settings can be given more sensible names (e.g., the max
   number of threads in a scaling thread pool is now named "max" instead
   of "size")
 - change the soft limit on the number of threads in the bulk and
   indexing thread pools to a hard limit
 - the settings names for custom plugins for thread pools can be
   prefixed (e.g., "xpack.watcher.thread_pool.size")
 - remove dynamic thread pool settings

Relates #18674
2016-06-06 22:09:12 -04:00
Areek Zillur d96fe20e3a add named writable registry glue 2016-06-06 16:11:46 -04:00
Christoph Büscher 359f45988f Handle empty query bodies at parse time and remove EmptyQueryBuilder
Currently we support empty query clauses like the filter in

"constant_score" : {  "filter" : { } }

How these clauses are handled depends on the surrounding query.
They later are either ignored, converted to match all or no documents or
passed up further in the query hierarchy. During parsing these claues are
currently represented as EmptyQueryBuilders. When not handled anywhere else,
these special cases need to be checked for on the shard when building the
lucene query.

This is trappy, so this PR changes the parsing of compound queries. Instead
of returning QueryBuilder, the core query parsing method
QueryShardContext#parseInnerQueryBuilder() now return an Optional which can
be empty in the case of empty query clauses. This has the advantage of forcing
callers to deal with this sooner or later. When encountering empty Optionals,
compound query builders now have the choice to ignore them, pass them on or
rewrite to a different query, depending on context.
2016-06-02 11:25:56 +02:00
Simon Willnauer f74a78940c Improve phrase suggest test speed (#18633)
There is no reason to read the entire marvel hero file to test the features,
it might take several seconds to do so which is unnecessary.
This commit also splits SearchSuggestTests into core and modules/mustache 
also add @Nighlty to forbidden API to make sure we don't use it since they won't run in CI these days.
2016-05-30 17:22:03 +02:00
Robert Muir f037807117 replace ScriptException with a better one 2016-05-26 11:43:29 -04:00
Lee Hinman fdfd2a2f18 Remove ScriptMode class in favor of boolean true/false
This removes the ScriptMode class entirely, which was an enum with two
options (ON and OFF) which essentially boiled down to true and false.
Now the boolean values are used instead.
2016-05-20 15:01:30 -06:00
Martijn van Groningen 80fee8666f percolator: Removed percolator cache
Before 5.0 for it was required that the percolator queries were cached in jvm heap as Lucene queries for two reasons:
1) Performance. The percolator evaluated all percolator queries all the time. There was no pre-selecting queries that are likely to match like we have today.
2) Updates made to percolator queries were visible in realtime, Today these changes are visible in near realtime. So updating no longer requires the percolator to have the queries in jvm heap.

So having the percolator queries in jvm heap via the percolator cache is now less attractive. Especially when there are many percolator queries then these queries can consume many GBs of jvm heap.
Removing the percolator cache does make the percolate query slower compared to how the execution time in 5.0.0-alpha1 and alpha2, but it is still faster compared to 2.x and before.
2016-05-20 14:52:16 +02:00
Robert Muir 2028691e66 painless: improve exception stacktraces
closes #18319
2016-05-13 15:40:45 -04:00
Lee Hinman 9bcdafedda Allow only a single extension for a scripting engine
Previously multiple extensions could be provided, however, this can lead
to confusion with on-disk scripts (ie, "foo.js" and "foo.javascript")
having different content. Only a single extension is now supported.

The only language currently supporting multiple extensions was the
Javascript engine ("js" and "javascript"). It now only supports the
`.js` extension.

Relates to #10598
2016-05-13 09:54:31 -06:00
Lee Hinman efff3918d8 Remove support for mulitple languages per scripting engine 2016-05-13 09:24:31 -06:00
Lee Hinman a4060f7436 Remove vestiges of script engine sandboxing
This removes all the mentions of the sandbox from the script engine
services and permissions model. This means that the following settings
are no longer supported:

```yaml
script.inline: sandbox
script.stored: sandbox
```

Instead, only a `true` or `false` value can be specified.

Since this would otherwise break the default-allow parameter for
languages like expressions, painless, and mustache, all script engines
have been updated to have individual settings, for instance:

```yaml
script.engine.groovy.inline: true
```

Would enable all inline scripts for groovy. (they can still be
overridden on a per-operation basis).

Expressions, Painless, and Mustache all default to `true` for inline,
file, and stored scripts to preserve the old scripting behavior.

Resolves #17114
2016-05-13 09:24:31 -06:00
Christoph Büscher ca21aa0cb5 Make reset() in QueryShardContext private
The query shard reset() method resets some internal state in the
query shard context, like clearing query names, the filter flag
or named queries. The problem with this method being public is
that it currently (miss?) used for modifying an existing context
for recursive invocatiob, but the contexts that have been reseted
that way cannot be properly set back to their previous state.

This PR is a step towards removing reset() entirely by first making
it only be used internally in QueryShardContext. In places where
reset() was used we can either create new QueryShardContexts or
modify the existing context because it is discarded afterwards anyway.
2016-05-03 18:56:16 +02:00
Ryan Ernst 55388590c1 Remove camelCase support
Now that the current uses of magical camelCase support have been
deprecated, we can remove these in master (sans remaining issues like
BulkRequest). This change removes camel case support from ParseField,
query types, analysis, and settings lookup.

see #8988
2016-04-22 09:18:10 -07:00
Martijn van Groningen c5ad2e2865 Changed indexed scripts to be stored in the cluster state instead of the `.scripts` index.
Also added max script size soft limit for stored scripts.

Closes #16651
2016-04-22 13:42:55 +02:00
Christoph Büscher fbd558382d Clean up QueryParseContext and don't hold it inside QueryRewrite/ShardContext
This change cleans up a few things in QueryParseContext and QueryShardContext
that make it hard to reason about the state of these context objects and are
thus error prone and should be simplified.

Currently the parser that used in QueryParseContext can be set and reset any
time from the outside, which makes reasoning about it hard. This change makes
the parser a mandatory constructor argument removes ability to later set a
different ParseFieldMatcher. If none is provided at construction time, the
one set inside the parser is used. If a ParseFieldMatcher is specified at
construction time, it is implicitely set on the parser that is beeing used.
The ParseFieldMatcher is only kept inside the parser.

Also currently the QueryShardContext historically holds an inner QueryParseContext
(in the super class QueryRewriteContext), that is mainly used to hold the parser
and parseFieldMatcher. For that reason, the parser can also be reset, which leads
to the same problems as above. This change removes the QueryParseContext from
QueryRewriteContext and removes the ability to reset or retrieve it from the
QueryShardContext. Instead, `QueryRewriteContext#newParseContext(parser)` can be
used to create new parse contexts with the given parser from a shard context
when needed.
2016-04-15 17:13:01 +02:00
Adrien Grand 42526ac28e Remove Settings.settingsBuilder.
We have both `Settings.settingsBuilder` and `Settings.builder` that do exactly
the same thing, so we should keep only one. I kept `Settings.builder` since it
has my preference but also it is the one that we use in examples of the Java API.
2016-04-08 18:10:02 +02:00
Nik Everett 2b6866d26b Fix references to the removed parsers
Mostly stuff is just in the builder now.
2016-04-06 11:15:22 -04:00
Adrien Grand 4c4bbb3e45 Replace FieldStatsProvider with a method on MappedFieldType. #17334
FieldStatsProvider had to perform instanceof calls to properly handle dates or
ip addresses. By moving the logic to MappedFieldType, each field type can check
whether all values are within bounds its way.

Note that this commit only keeps rewriting support for dates, which are the only
field for which the rewriting mechanism is likely to help (because of time-based
indices).
2016-04-01 10:28:58 +02:00
Nik Everett df08854c60 Remove PROTOTYPEs from suggesters
Also stops using guice for suggesters at all and lots of checkstyle.
2016-03-29 17:55:01 -04:00
Areek Zillur ed49ec437f remove suggest transport action 2016-03-23 16:37:56 -04:00
Boaz Leskes 2d1152ebac Remove ClusterService interface, in favor of it's only production instance #17183
We current have a ClusterService interface, implemented by InternalClusterService and a couple of test classes. Since the decoupling of the transport service and the cluster service, one can construct a ClusterService fairly easily, so we don't need this extra indirection.

Closes #17183
2016-03-21 13:55:10 +01:00
Martijn van Groningen e3b7e5d75a percolator: Replace percolate api with the new percolator query
Also replaced the PercolatorQueryRegistry with the new PercolatorQueryCache.

The PercolatorFieldMapper stores the rewritten form of each percolator query's xcontext
in a binary doc values field. This make sure that the query rewrite happens only during
indexing (some queries for example fetch shapes, terms in remote indices) and
the speed up the loading of the queries in the percolator query cache.

Because the percolator now works inside the search infrastructure a number of features
(sorting fields, pagination, fetch features) are available out of the box.

The following feature requests are automatically implemented via this refactoring:

Closes #10741
Closes #7297
Closes #13176
Closes #13978
Closes #11264
Closes #10741
Closes #4317
2016-03-21 12:21:50 +01:00
Christoph Büscher 39667b5793 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0/java.asciidoc
2016-03-16 12:06:42 +01:00
Jason Tedor 66ba044ec5 Use setting in integration test cluster config 2016-03-15 17:45:17 -04:00
Christoph Büscher 97638c95fc Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-14 11:13:47 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Christoph Büscher bbcbba1bf5 Fixing some tests and compile problems in reindex module 2016-03-11 17:58:13 +01:00
Yannick Welsch 04e55ecf6b Make logging message String constant to allow static checks 2016-03-11 10:30:59 +01:00
Christoph Büscher 7ec5075a87 Merge branch 'master' into feature-suggest-refactoring 2016-03-07 10:45:43 +01:00
Christoph Büscher 30a788d87c Suggestions: Make field name mandatory ctor argument
The field name is a required argument for all suggesters, but
it was specified via a field() setter in SuggestionBuilder so far.
This changes field name to being a mandatory constructor argument
and lets suggestion builders throw an error if field name is missing
or the empty string.
2016-03-03 21:35:53 +01:00
Clinton Gormley 529f7cb42c Fixed bad YAML in msearch test 2016-03-03 14:45:02 +01:00
Christoph Büscher ef4db5c1e4 Merge branch 'master' into feature-suggest-refactoring 2016-03-03 12:18:03 +01:00
Adrien Grand eef19be072 Deprecate string in favor of text/keyword. #16877
This commit removes the ability to use string fields on indices created on or
after 5.0. Dynamic mappings now generate text fields by default for strings
but there are plans to also add a sub keyword field (in a future PR).

Most of the changes in this commit are just about replacing string with
keyword or text. Some tests have been removed because they existed because of
corner cases of string mappings like setting ignore-above on a text field or
enabling term vectors on a keyword field which are now impossible.

The plan is to remove strings entirely in 6.0.
2016-03-03 10:20:56 +01:00
Christoph Büscher be8ed737bc Suggest: Move name of suggestion to SuggestBuilder
Currently each suggestion keeps track of its own name. This has
the disadvantage of having to pass down the parsed name property
in the suggestions fromXContent() and the serialization methods
as an argument, since we need it in the ctor.

This change moves the naming of the suggestions to the surrounding
SuggestBuilder and by this eliminates the need for passind down
the names in the parsing and serialization methods. By making
`name` a required argument in SuggestBuilder#addSuggestion() we
also make sure it is always set and prevent using the same name twice,
which wasn't possible before.
2016-03-01 17:34:47 +01:00
Ali Beyad 5320d538f1 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-02-15 11:03:33 -05:00
Simon Willnauer 297fb24112 apply fixes after review 2016-02-12 12:53:18 +01:00
Ali Beyad eed557742f Refactors building query specific objects from the term suggestion
builder and merges changes between the suggestion builders to context
object implementations.
2016-02-11 18:30:58 -05:00
Simon Willnauer 8bdc7362a6 Add infrastructure to rewrite query buidlers
QueryBuilders today do all their heavy lifting in toQuery() which
can be too late for several operations. For instance if we want to fetch geo shapes
on the coordinating node we need to do all this before we create the actual lucene query
which happens on the shard itself. Also optimizations for request caching need to be done
to the query builder rather than the query which then in-turn needs to be serialized again.
This commit adds the basic infrastructure for query rewriting and moves the heavy lifting into
the rewrite method for the following queries:

 * `WrapperQueryBuilder`
 * `GeoShapeQueryBuilder`
 * `TermsQueryBuilder`
 * `TemplateQueryBuilder`

Other queries like `MoreLikeThisQueryBuilder` still need to be fixed / converted. The nice
sideeffect of this is that queries like template queries will now also match the request cache
if their non-template equivalent has been cached befoore. In the future this will allow to
add optimizataion like rewriting time-based queries into primitives like `match_all_docs` or `match_no_docs`
based on the currents shards bounds. This is especially appealing for indices that are read-only ie. never change.
2016-02-11 10:45:49 +01:00
Christoph Büscher 9e0f6e3f9c Adding method to build SuggestionContext to PhraseSuggestionBuilder
This adds a build method for the SuggestionContext to the PhraseSuggestionBuilder
and another one that creates the SuggestionSearchContext to the top level
SuggestBuilder. Also adding tests that make sure the current way of parsing
xContent to a SuggestionContext is reflected in the output the builders create.
2016-02-10 18:57:27 +01:00
Christoph Büscher 2dec129038 Merge branch 'master' into feature-suggest-refactoring 2016-02-08 09:47:06 +01:00
Simon Willnauer b185209b63 Make IndicesWarmer a private class of IndexService
There is no need for IndicesWarmer to be a global accessible class. All it needs
access to is inside IndexService. It also doesn't need to be mutable once it's not a per node
instance. This commit move IndicesWarmer to IndexWarmer and makes the default impls like field data and
norms warming an impl detail. Also the IndexShard doesn't depend on this class anymore, instead it accepts
an Engine.Warmer as a ctor argument which delegates to the actual warmer from the index.
2016-02-05 15:24:17 +01:00
Simon Willnauer 8685a003d9 fix another test 2016-02-05 11:17:53 +01:00
Christoph Büscher c0006426a7 Merge branch 'master' into feature-suggest-refactoring 2016-02-04 11:01:24 +01:00
Simon Willnauer e02d2e004e Rewrite SettingsFilter to be immutable
This change rewrites the entire settings filtering mechanism to be immutable.
All filters must be registered up-front in the SettingsModule. Filters that are comma-sparated are
not allowed anymore and check on registration.
This commit also adds settings filtering to the default settings recently added to ensure we don't render
filtered settings.
2016-02-03 20:05:55 +01:00
Ali Beyad 96d7fc858f Fixed a compiler error with the SuggestSearchTests 2016-02-03 12:42:03 -05:00
Ali Beyad 882cd5c6ee Refactored the term suggestion builder for the query refactoring effort
Added the term suggestion builder's serialization/deserialization and
equals/hashCode methods.
2016-02-03 12:00:03 -05:00
Simon Willnauer 818a9eefb2 Make settings validation strict
This commit enableds strict settings validation on node startup. All settings
passed to elasticsearch either through system properties, yaml files or any other
way to pass settings must be registered and valid. Settings that are unknown ie. due to
typos or due to deprecation or removal will cause the node to NOT start up. Plugins
have to declare all their settings on the `SettingsModule#registerSetting` and settings for
plugins that are not installed must be removed.

This commit also removes the ability to specify the nodes name via `-Des.name` or just `name` in the
configuration files. The node name must be prefixed with the node prexif like `node.name: Boom`. Left over
usage of `name` will also cause startup to fail.
2016-02-02 11:32:44 +01:00
Christoph Büscher 7cae28f96a Merge branch 'master' into feature-suggest-refactoring 2016-02-01 13:38:34 +01:00
Christoph Büscher 61c435e6a9 PhraseSuggestionBuilde: Refactor DirectCandidateGenerator
As a prerequisite for refactoring the whole PhraseSuggestionBuilder
to be able to be parsed and streamed from the coordinating node, the
DirectCandidateGenerator must implement Writeable, be able to parse
a new instance (fromXContent()) and later when transported to the
shard to generate a PhraseSuggestionContext.DirectCandidateGenerator.
Also adding equals/hashCode and tests and moving DirectCandidateGenerator
to its own DirectCandidateGeneratorBuilder class.
2016-02-01 12:22:48 +01:00
Christoph Büscher 09c8a21e83 Merge branch 'master' into feature-suggest-refactoring 2016-01-28 10:46:40 +01:00