Commit Graph

992 Commits

Author SHA1 Message Date
Daniel Mitterdorfer f174f72fee
Circuit-break based on real memory usage
With this commit we introduce a new circuit-breaking strategy to the parent
circuit breaker. Contrary to the current implementation which only accounts for
memory reserved via child circuit breakers, the new strategy measures real heap
memory usage at the time of reservation. This allows us to be much more
aggressive with the circuit breaker limit so we bump it to 95% by default. The
new strategy is turned on by default and can be controlled  with the new cluster
setting `indices.breaker.total.userealmemory`.

Note that we turn it off for all integration tests with an internal test cluster
because it leads to spurious test failures which are of no value (we cannot
fully control heap memory usage in tests). All REST tests, however, will make
use of the real memory circuit breaker.

Relates #31767
2018-07-13 10:08:28 +02:00
Jim Ferenczi 584fa261cc
Remove the ability to index or query context suggestions without context (#31007)
This is a follow up of #30712 that removes the ability to index or query
and context enabled completion field without context.

Relates #30712
2018-07-09 16:01:01 +02:00
Sohaib Iftikhar 40b822c878 Scripting: Remove support for deprecated StoredScript contexts (#31394)
Removes support for storing scripts without the usual json around the
script. So You can no longer do:
```
POST _scripts/<templatename>
{
    "query": {
        "match": {
            "title": "{{query_string}}"
        }
    }
}
```

and must instead do:
```
POST _scripts/<templatename>
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "match": {
                    "title": "{{query_string}}"
                }
            }
        }
    }
}
```

This improves error reporting when you attempt to store a script but don't
quite get the syntax right. Before, there was a good chance that we'd
think of it as a "raw" template and just store it. Now we won't do that.
Nice.
2018-07-05 09:30:08 -04:00
Daniel Mitterdorfer 3d53daeb2f
Account for XContent overhead in in-flight breaker
So far the in-flight request circuit breaker has only accounted for the
on-the-wire representation of a request. However, we convert the raw
request into XContent internally which increases the overhead.
Therefore, we increase the value of the corresponding setting
`network.breaker.inflight_requests.overhead` from one to two. While this
value is still rather conservative (we assume that the representation as
structured objects has no overhead compared to the byte[]), it is closer
to reality than the current value.

Relates #31613
2018-07-03 09:17:16 +02:00
Jonathan Little 8e4768890a Migrate scripted metric aggregation scripts to ScriptContext design (#30111)
* Migrate scripted metric aggregation scripts to ScriptContext design #29328

* Rename new script context container class and add clarifying comments to remaining references to params._agg(s)

* Misc cleanup: make mock metric agg script inner classes static

* Move _score to an accessor rather than an arg for scripted metric agg scripts

This causes the score to be evaluated only when it's used.

* Documentation changes for params._agg -> agg

* Migration doc addition for scripted metric aggs _agg object change

* Rename "agg" Scripted Metric Aggregation script context variable to "state"

* Rename a private base class from ...Agg to ...State that I missed in my last commit

* Clean up imports after merge
2018-06-25 12:01:33 +01:00
Lisa Cawley 68ec958873
[DOCS] Move migration APIs to docs (#31473) 2018-06-21 08:19:23 -07:00
Ryan Ernst c0961b79be
Docs: Add note about removing prepareExecute from the java client (#31401)
relates #30966
2018-06-19 07:21:58 -07:00
Ryan Ernst f3297ed23a
Packaging: Remove windows bin files from the tar distribution (#30596)
This commit removes windows specific files from the tar distribution.
Windows users use the zip, linux users use the tar.
2018-06-18 19:02:51 +02:00
lcawl 4918ae6fe4 [DOCS] Adds links to release notes and highlights 2018-06-15 11:04:11 -07:00
Luca Cavanna 24163d10b7
REST hl client: cluster health to default to cluster level (#31268)
With #29331 we added support for the cluster health API to the
high-level REST client. The transport client does not support the level
parameter, and it always returns all the info needed for shards level
rendering. We have maintained that behaviour when adding support for
cluster health to the high-level REST client, to ease migration, but the
correct thing to do is to default the high-level REST client to
`cluster` level, which is the same default as when going through the
Elasticsearch REST layer.
2018-06-13 15:06:13 +02:00
Luca Cavanna 92eb324776
REST high-level Client: remove deprecated API methods (#31200)
This commit removes all the API methods that accept a `Header` varargs
argument, in favour of the newly introduced API methods that accept a
`RequestOptions` argument.

Relates to #31069
2018-06-12 21:00:06 +02:00
Simon Willnauer f825a530b8
Limit the number of concurrent requests per node (#31206)
With `max_concurrent_shard_requests` we used to throttle / limit
the number of concurrent shard requests a high level search request
can execute per node. This had several problems since it limited the
number on a global level based on the number of nodes. This change
now throttles the number of concurrent requests per node while still
allowing concurrency across multiple nodes.

Closes #31192
2018-06-11 08:49:18 +02:00
Alan Woodward 852df128a5
Match phrase queries against non-indexed fields should throw an exception (#31060)
When `lenient=false`, attempts to create match phrase queries with custom analyzers against non-text fields will throw an IllegalArgumentException.

Also changes `*Match*QueryBuilderTests` so that it avoids this scenario

Fixes #31061
2018-06-04 19:12:45 +01:00
Christoph Büscher 1ea9f11b03
Change ScriptException status to 400 (bad request) (#30861)
Currently failures to compile a script usually lead to a ScriptException, which
inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does
not contain another root cause. Instead, this should be a 400 Bad Request error.
This PR changes this more generally for script compilation errors by changing 
ScriptException to return 400 (bad request) as status code.

Closes #12315
2018-05-30 14:00:07 +02:00
Jim Ferenczi f582418ada Fix missing option serialization after backport
Relates #29465
2018-05-30 12:55:31 +02:00
Vladimir Dolzhenko 81eb8ba0f0
Include size of snapshot in snapshot metadata (#29602)
Include size of snapshot in snapshot metadata

Adds difference of number of files (and file sizes) between prev and current snapshot. Total number/size reflects total number/size of files in snapshot.

Closes #18543
2018-05-25 21:04:50 +02:00
Igor Motov cf0e0606af
Use geohash cell instead of just a corner in geo_bounding_box (#30698)
Treats geohashes as grid cells instead of just points when the
geohashes are used to specify the edges in the geo_bounding_box
query. For example, if a geohash is used to specify the top_left
corner, the top left corner of the geohash cell will be used as the
corner of the bounding box.

Closes #25154
2018-05-24 14:46:15 -04:00
Tim Brooks d7040ad7b4
Reintroduce mandatory http pipelining support (#30820)
This commit reintroduces 31251c9 and 63a5799. These commits introduced a
memory leak and were reverted. This commit brings those commits back
and fixes the memory leak by removing unnecessary retain method calls.
2018-05-23 14:38:52 -06:00
Colin Goodheart-Smithe 4fd0a3e492 Revert "Make http pipelining support mandatory (#30695)" (#30813)
This reverts commit 31251c9 introduced in #30695.

We suspect this commit is causing the OOME's reported in #30811 and we will use this PR to test this assertion.
2018-05-23 10:54:46 -06:00
Tim Brooks 31251c9a6d
Make http pipelining support mandatory (#30695)
This is related to #29500 and #28898. This commit removes the abilitiy
to disable http pipelining. After this commit, any elasticsearch node
will support pipelined requests from a client. Additionally, it extracts
some of the http pipelining work to the server module. This extracted
work is used to implement pipelining for the nio plugin.
2018-05-22 09:29:31 -06:00
Tanguy Leroux 74474e99d6 [Docs] Fix broken cross link in documentation 2018-05-22 16:03:33 +02:00
Ryan Ernst 34180f2285
Scripting: Remove getDate methods from ScriptDocValues (#30690)
The getDate() and getDates() existed prior to 5.x on long fields in
scripting. In 5.x, a new Date type for ScriptDocValues was added. The
getDate() and getDates() methods were left on long fields and added to date
fields to ease the transition. This commit removes those methods for
7.0.
2018-05-18 21:26:26 -07:00
Jason Tedor d68c44b76c
Default copy settings to true and deprecate on the REST layer (#30598)
This commit defaults the copy_settings REST parameter to the shrink and
split APIs to true, and deprecates the parameter.
2018-05-18 10:12:08 -04:00
Ryan Ernst fb0aa562a5
Network: Remove http.enabled setting (#29601)
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.

closes #12792
2018-05-02 11:42:05 -07:00
Ryan Ernst fba2f00a73
Packaging: Unmark systemd service file as a config file (#29004)
Systemd overrides should happen through /etc/systemd/system, not
directly editing the service file. This commit removes marking the
service file as configuration for rpm and deb packages.
2018-05-02 09:48:49 -07:00
Jason Tedor 5de6f4ff7b Adjust copy settings on resize BWC version
This commit adjusts the BWC version for copy settings on resize
operations after the behavior was backported to 6.x.
2018-05-01 08:49:16 -04:00
Jason Tedor 50535423ff
Allow copying source settings on resize operation (#30255)
Today when an index is created from shrinking or splitting an existing
index, the target index inherits almost none of the source index
settings. This is surprising and a hassle for operators managing such
indices. Given this is the default behavior, we can not simply change
it. Instead, we start by introducing the ability to copy settings. This
flag can be set on the REST API or on the transport layer and it has the
behavior that it copies all settings from the source except non-copyable
settings (a property of a setting introduced in this
change). Additionally, settings on the request will always override.

This change is the first step in our adventure:
 - this flag is added here in 7.0.0 and immediately deprecated
 - this flag will be backported to 6.4.0 and remain deprecated
 - then, we will remove the ability to set this flag to false in 7.0.0
 - finally, in 8.0.0 we will remove this flag and the only behavior will
   be for settings to be copied
2018-05-01 08:48:19 -04:00
Jason Tedor f381e2a00c
Add migration note on thread pool API changes (#29192)
A previous change modified the output of the thread pool info contained
in the nodes info API. This commit adds a note to the migration docs for
this change.
2018-04-28 00:11:17 -04:00
Julie Tibshirani f5978d6d33
In the field capabilities API, remove support for providing fields in the request body. (#30185) 2018-04-27 16:14:11 -07:00
Julie Tibshirani 32dfb65144 In the field capabilities API, deprecate support for providing fields in the request body. (#30157)
(cherry picked from commit d8d884b29d4aa7d01070484fee5de8d3db60cb25)
2018-04-25 23:01:53 -07:00
Jason Tedor 2c3e71f116
Remove the suggest metric from stats APIs (#29635)
This metric previously existed for backwards compatibility reasons
although the suggest stats were folded into search stats. This metric
was deprecated in 6.3.0 and this commit removes them for 7.0.0.
2018-04-24 19:03:48 -04:00
Jason Tedor 5d767e449a
Remove bulk fallback for write thread pool (#29609)
The name of the bulk thread pool was renamed to "write" with "bulk" as a
fallback name. This change was made in 6.x for BWC reasons yet in 7.0.0
we are removing this fallback. This commit removes this fallback for the
write thread pool.
2018-04-19 16:59:58 -04:00
Jason Tedor 2b47d67d95
Remove the index thread pool (#29556)
Now that single-document indexing requests are executed on the bulk
thread pool the index thread pool is no longer needed. This commit
removes this thread pool from Elasticsearch.
2018-04-18 09:18:08 -04:00
Ke Li 0bfb59dcf2 Using ObjectParser in UpdateRequest (#29293)
CRUD: Parsing changes for UpdateRequest (#29293)

Use `ObjectParser` to parse `UpdateRequest` so we reject unknown fields
and drop support for the `_fields` parameter because it was deprecated
in 5.x.
2018-04-16 08:39:35 -04:00
Jim Ferenczi 1b6d5e531b
Fail _search request with trailing tokens (#29428)
This change validates that the `_search` request does not have trailing
tokens after the main object and fails the request with a parsing exception otherwise.

Closes #28995
2018-04-11 13:10:22 +02:00
Adrien Grand 4918924fae
Remove legacy mapping code. (#29224)
Some features have been deprecated since `6.0` like the `_parent` field or the
ability to have multiple types per index. This allows to remove quite some
code, which in-turn will hopefully make it easier to proceed with the removal
of types.
2018-04-11 09:41:37 +02:00
Adrien Grand aeac682869
Make purely negative queries return scores of 0. (#26015)
It would make them consistent with queries that are only made of filters.

Closes #23449
2018-04-10 14:31:06 +02:00
Christoph Büscher 231fd4eb18
Remove `delimited_payload_filter` (#27705)
From 7.0 on, using `delimited_payload_filter` should throw an error. 
It was deprecated in 6.2 in favour of `delimited_payload` (#26625).

Relates to #27704
2018-04-05 18:41:04 +02:00
Adrien Grand 569d0c0e89
Improve similarity integration. (#29187)
This improves the way similarities are plugged in in order to:
 - reject the classic similarity on 7.x indices and emit a deprecation
   warning otherwise
 - reject unkwown parameters on 7.x indices and emit a deprecation
   warning otherwise

Even though this breaks the plugin API, I'd like to backport to 7.x so
that users can get deprecation warnings when they are doing something
that will become unsupported in the future.

Closes #23208
Closes #29035
2018-04-03 16:45:25 +02:00
Jim Ferenczi c93c7f3121
Remove deprecated options for query_string (#29203)
This commit removes some parameters deprecated in 6.x (or 5.x):
`use_dismax`, `split_on_whitespace`, `all_fields` and `lowercase_expanded_terms`.

Closes #25551
2018-03-22 18:37:08 +01:00
Adrien Grand 8f9d2ee4e2
Reject updates to the `_default_` mapping. (#29165)
This will reject mapping updates to the `_default_` mapping with 7.x indices
and still emit a deprecation warning with 6.x indices.

Relates #15613
Supersedes #28248
2018-03-21 10:44:11 +01:00
joadha 08c530907a [Docs] Update api.asciidoc (#29166)
The parent page has the same title, and the URL path indicates this is about 
API changes, so include "API" in the title.
2018-03-21 10:14:26 +01:00
olcbean 3d81497f25 REST: Clear Indices Cache API remove deprecated url params (#29068)
By the time the master branch is released the deprecated url
parameters in the `/_cache/clear` API will have been deprecated
for a couple of minor releases. Since master will be the next
major release we are fine with removing these parameters.
2018-03-14 16:37:50 -04:00
Mayya Sharipova f53d159aa1
Limit analyzed text for highlighting (improvements) (#28808)
Increase the default limit of `index.highlight.max_analyzed_offset` to 1M instead of previous 10K.

Enhance an error message when offset increased to include field name, index name and doc_id.

Relates to https://github.com/elastic/kibana/issues/16764
2018-03-02 08:09:05 -08:00
Ke Li a77273fc01 Reject regex search if regex string is too long (#28542)
* Reject regex search if regex string is too long (#28344)

* Add docs

* Introduce index level setting `index.max_regex_length`
 to control the maximum length of the regular expression

Closes #28344
2018-02-23 10:41:24 -08:00
Tanguy Leroux a6a138905d
Use client settings in repository-gcs (#28575)
Similarly to what has been done for s3 and azure, this commit removes
the repository settings `application_name` and `connect/read_timeout`
in favor of client settings. It introduce a GoogleCloudStorageClientSettings
class (similar to S3ClientSettings) and a bunch of unit tests for that,
it aligns the documentation to be more coherent with the S3 one, it
documents the connect/read timeouts that were not documented at all and
also adds a new client setting that allows to define a custom endpoint.
2018-02-22 15:40:20 +01:00
Martijn van Groningen ecb1d07d00
percolator: remove deprecated map_unmapped_fields_as_string setting 2018-02-01 11:11:22 +01:00
olcbean 0c83240b5f Java Api clean up: remove deprecated `isShardsAcked` (#28311)
This PR removes previously deprecated `isShardsAcked()` method in
favour of `isShardsAcknowledged()` on `CreateIndexResponse`, `CreateIndexClusterStateUpdateResponse` and `RolloverResponse` 

Related to #27784
Follow-up of #27819
2018-01-25 14:13:20 +01:00
Adrien Grand 700d9ecc95
Remove the `update_all_types` option. (#28288)
This option is not useful in 7.x since no indices may have more than one type
anymore.
2018-01-22 12:03:07 +01:00
Mayya Sharipova dcde895f49
Introduce limit to the number of terms in Terms Query (#27968)
- Introduce index level settings to control the maximum number of terms
    that can be used in a Terms Query
- Throw an error if a request exceeds this max number

Closes #18829
2017-12-28 17:36:29 -05:00
Mayya Sharipova cbd271e497
Limit the analyzed text for highlighting (#27934)
* Limit the analyzed text for highlighting

- Introduce index level settings to control the max number of character
to be analyzed for highlighting
- Throw an error if analysis is required on a larger text

Closes #27517
2017-12-21 10:19:58 -05:00
olcbean 25c606cf09 Remove deprecated names for string distance algorithms (#27640)
#27409 deprecated the incorrectly-spelled `levenstein` in favour of `levenshtein`.
#27526 deprecated the inconsistent `jarowinkler` in favour of `jaro_winkler`.

These changes were merged into 6.2, and this change removes them entirely in 7.0.
2017-12-11 12:16:04 +00:00
Jim Ferenczi caea6b70fa
Add a new cluster setting to limit the total number of buckets returned by a request (#27581)
This commit adds a new dynamic cluster setting named `search.max_buckets` that can be used to limit the number of buckets created per shard or by the reduce phase. Each multi bucket aggregator can consume buckets during the final build of the aggregation at the shard level or during the reduce phase (final or not) in the coordinating node. When an aggregator consumes a bucket, a global count for the request is incremented and if this number is greater than the limit an exception is thrown (TooManyBuckets exception).
This change adds the ability for multi bucket aggregator to "consume" buckets in the global limit, the default is 10,000. It's an opt-in consumer so each multi-bucket aggregator must explicitly call the consumer when a bucket is added in the response.

Closes #27452 #26012
2017-12-06 09:15:28 +01:00
Deb Adair 2f9a882061 [DOCS] Fixed typos and broken attribute. 2017-12-05 11:46:40 -08:00
Mayya Sharipova c6b73239ae
Limit the number of tokens produced by _analyze (#27529)
Add an index level setting `index.analyze.max_token_count` to control
the number of generated tokens in the  _analyze endpoint.
Defaults to 10000.

Throw an error if the number of generated tokens exceeds this limit.

Closes #27038
2017-11-30 11:54:39 -05:00
Simon Willnauer f23ed6188d
Skip shard refreshes if shard is `search idle` (#27500)
Today we refresh automatically in the background by default very second.
This default behavior has a significant impact on indexing performance
if the refreshes are not needed.
This change introduces a notion of a shard being `search idle` which a
shard transitions to after (default) `30s` without any access to an
external searcher. Once a shard is search idle all scheduled refreshes
will be skipped unless there are any refresh listeners registered.
If a search happens on a `serach idle` shard the search request _park_
on a refresh listener and will be executed once the next scheduled refresh
occurs. This will also turn the shard into the `non-idle` state immediately.

This behavior is only applied if there is no explicit refresh interval set.
2017-11-27 18:16:10 +01:00
lcawley af971b3081 [DOCS] Fixed broken link in breaking changes 2017-11-24 09:16:14 -08:00
kel 4885acb048 Replace `delimited_payload_filter` by `delimited_payload` (#26625)
The `delimited_payload_filter` is renamed to `delimited_payload`, the old name is 
deprecated and should be replaced by `delimited_payload`.

Closes #21978
2017-11-24 13:03:19 +01:00
Simon Willnauer fadbe0de08
Automatically prepare indices for splitting (#27451)
Today we require users to prepare their indices for split operations.
Yet, we can do this automatically when an index is created which would
make the split feature a much more appealing option since it doesn't have
any 3rd party prerequisites anymore.

This change automatically sets the number of routinng shards such that
an index is guaranteed to be able to split once into twice as many shards.
The number of routing shards is scaled towards the default shard limit per index
such that indices with a smaller amount of shards can be split more often than
larger ones. For instance an index with 1 or 2 shards can be split 10x
(until it approaches 1024 shards) while an index created with 128 shards can only
be split 3x by a factor of 2. Please note this is just a default value and users
can still prepare their indices with `index.number_of_routing_shards` for custom
splitting.

NOTE: this change has an impact on the document distribution since we are changing
the hash space. Documents are still uniformly distributed across all shards but since
we are artificually changing the number of buckets in the consistent hashign space
document might be hashed into different shards compared to previous versions.

This is a 7.0 only change.
2017-11-23 09:48:54 +01:00
Mayya Sharipova 57e4d10007
Limit the number of nested documents (#27405)
Add an index level setting `index.mapping.nested_objects.limit` to control
the number of nested json objects that can be in a single document
across all fields. Defaults to 10000.

Throw an error if the number of created nested documents exceed this
limit during the parsing of a document.

Closes #26962
2017-11-22 10:16:28 -05:00
Mayya Sharipova 858b2c7cb8
Standardize underscore requirements in parameters (#27414)
Stardardize underscore requirements in parameters across different type of
requests:
_index, _type, _source, _id keep their underscores
params like version and retry_on_conflict will be without underscores
Throw an error if older versions of parameters are used

BulkRequest, MultiGetRequest, TermVectorcRequest, MoreLikeThisQuery
were changed

Closes #26886
2017-11-17 15:31:52 -05:00
Jim Ferenczi 29331f1127
Fail queries with scroll that explicitely set request_cache (#27342)
Queries that create a scroll context cannot use the cache.
They modify the search context during their execution so using the cache
can lead to duplicate result for the next scroll query.

This change fails the entire request if the request_cache option is explictely set
on a query that creates a scroll context (`scroll=1m`) and make sure internally that we never
use the cache for these queries when the option is not explicitely used.
For 6.x a deprecation log will be printed instead of failing the entire request and the request_cache hint
will be ignored (forced to false).
2017-11-10 16:02:06 +01:00
Mayya Sharipova abbe853f1e
Add limits for ngram and shingle settings (#27211) (#27318)
Relates to #25887
2017-11-08 10:12:57 -05:00
javanna 34666844b3 [DOCS] Clarify migrate guide and search request validation
Relates to  #26811
2017-10-31 12:36:00 +01:00
kel c3e2bdf20c Raise IllegalArgumentException if query validation failed (#26811)
Closes #26799
2017-10-31 12:17:27 +01:00
Simon Willnauer 8dda827ff4 Don't refresh on `_flush` `_force_merge` and `_upgrade` (#27000)
Today all these API calls have a sideeffect of making documents visible
to search requests. While this is sometimes desired it's an unnecessary sideeffect
and now that we have an internal (engine-private) index reader (#26972) we artificially
add a refresh call for bwc. This change removes this sideeffect in 7.0.
2017-10-16 10:16:35 +02:00
Alexander Kazakov 592ab043dd Change default value to true for transpositions parameter of fuzzy query (#26901) 2017-10-11 15:31:48 +02:00
Nhat bf4c3642b2 remove _primary and _replica shard preferences (#26791)
The shard preference _primary, _replica and its variants were useful
for the asynchronous replication. However, with the current impl, they
are no longer useful and should be removed.

Closes #26335
2017-10-08 11:03:06 -04:00
David Turner 8fe9a20982 Forbid negative values for index.unassigned.node_left.delayed_timeout (#26828)
Change delayed_timeout to be a positiveTimeSetting, and add note that this is a breaking change
2017-09-29 14:44:43 +01:00
Christoph Büscher 6189c54c84 Reject the `index_options` parameter for numeric fields (#26668)
Numeric fields no longer support the index_options parameter. This changes the parameter
to be rejected in numeric field types after it was deprecated in 6.0.

Closes #21475
2017-09-25 23:43:14 +02:00
Yannick Welsch df5c450e89 Add v6.1 BWC layer for adding wait_for_active_shards to index open command
This commit disables BWC tests while adding a v6.1 BWC layer for the PR #26682
2017-09-22 16:30:07 +02:00
David Pilato b01b1c2a58 Remove azure deprecated settings (#26099)
Follow up for #23405.

We remove azure deprecated settings in 7.0:

* The legacy azure settings which where starting with `cloud.azure.storage.` prefix have been removed.
This includes `account`, `key`, `default` and `timeout`.
You need to use settings which are starting with `azure.client.` prefix instead.

* Global timeout setting `cloud.azure.storage.timeout` has been removed.
You must set it per azure client instead. Like `azure.client.default.timeout: 10s` for example.
2017-09-12 16:51:44 +02:00
Lee Hinman cff904bf97 Enable adaptive replica selection by default (#26522)
Relates to #24915
2017-09-07 09:25:05 -06:00
Jim Ferenczi 86d97971a4 Remove the _all metadata field (#26356)
* Remove the _all metadata field

This change removes the `_all` metadata field. This field is deprecated in 6
and cannot be activated for indices created in 6 so it can be safely removed in
the next major version (e.g. 7).
2017-08-28 17:43:59 +02:00
Jim Ferenczi a48616272f #26173: Removed global_ordinals_hash and global_ordinals_low_cardinality exeuction hint deprecated in 6.1 2017-08-21 20:44:34 +02:00
Lee Hinman f18ec511ca Disallow : in cluster and index/alias names (#26247)
We use `:` for cross-cluster search (eg `cluster:index`), therefore, we should
not allow the ambiguity when allowing cluster or index names.

Relates to #23892
2017-08-17 14:57:26 -06:00
Lee Hinman cfad6688b0 Migrate migration docs from 6.0 to 7.0 (#26227)
* Migrate migration docs from 6.0 to 7.0

Since we only keep one version of migration docs and master is now on 7.0, we
should migrate these so breaking changes can be added in the right place.

* Remove release notes as well

They link to the migration guides, so they have to go.

* Add placeholder notes for 7.0 so doc build is happy
2017-08-16 13:12:44 -06:00
Zachary Tong d26becc040 Fix NPE when `values` is omitted on percentile_ranks agg (#26046)
An array of values is required because there is no default (or
reasonable way to set a default).  But validation for values
only happens if it is actually set.  If the values param is omitted
entirely than the agg builder will NPE.
2017-08-15 13:09:15 -04:00
Jason Tedor e9687622bd Rename CONF_DIR to ES_PATH_CONF
The environment variable CONF_DIR was previously inconsistently used in
our packaging to customize the location of Elasticsearch configuration
files. The importance of this environment variable has increased
starting in 6.0.0 as it's now used consistently to ensure Elasticsearch
and all secondary scripts (e.g., elasticsearch-keystore) all use the
same configuration. The name CONF_DIR is there for legacy reasons yet
it's too generic. This commit renames CONF_DIR to ES_PATH_CONF.

Relates #26197
2017-08-15 06:19:06 +09:00
Daniel Mitterdorfer 637cc872f4 Remove unused Netty-related settings (#26161)
With this commit we remove the following three previously unused 
(and undocumented) Netty 4 related settings:

* transport.netty.max_cumulation_buffer_capacity,
* transport.netty.max_composite_buffer_components and
* http.netty.max_cumulation_buffer_capacity 

from Elasticsearch.
2017-08-11 12:03:00 +02:00
Martijn van Groningen 076167fbe5
inner hits: Unfiltered nested source should keep its full path
like filtered nested source.

Closes #23090
2017-08-10 15:58:29 +02:00
Tanguy Leroux 9c8d3d3569 [Docs] Add migration notes for the high-level rest client (#25911) 2017-08-01 10:38:56 +02:00
Jason Tedor bc8dc683e4 Update config files docs
This commit updates the docs for the config files to explain the new
mechanism for customizing the configuration directory via the
environment variable CONF_DIR.

Relates #25990
2017-08-01 09:52:23 +09:00
Jim Ferenczi 4a9995145c [Docs]: Clarify query_string parser splits on operator 2017-07-24 18:36:16 +02:00
Jim Ferenczi c3784326eb Refactor field expansion for match, multi_match and query_string query (#25726)
This commit changes the way we handle field expansion in `match`, `multi_match` and `query_string` query.
 The main changes are:

- For exact field name, the new behavior is to rewrite to a matchnodocs query when the field name is not found in the mapping.

- For partial field names (with `*` suffix), the expansion is done only on `keyword`, `text`, `date`, `ip` and `number` field types. Other field types are simply ignored.

- For all fields (`*`), the expansion is done on accepted field types only (see above) and metadata fields are also filtered.

- The `*` notation can also be used to set `default_field` option on`query_string` query. This should replace the needs for the extra option `use_all_fields` which is deprecated in this change.

This commit also rewrites simple `*` query to matchalldocs query when all fields are requested (Fixes #25556). 

The same change should be done on `simple_query_string` for completeness.

`use_all_fields` option in `query_string` is also deprecated in this change, `default_field` should be set to `*` instead.

Relates #25551
2017-07-21 16:52:57 +02:00
Jack Conradson 9f7463e796 remove lang url parameter from stored script requests (#25779)
Also has updates to ScriptMetaData for allowing the old namespace format to be loaded all the way back through 5.0; however, it will throw an exception if two scripts share the same id but different languages.
2017-07-20 08:51:08 -07:00
Jason Tedor 67a4288c9a Remove support for ES_INCLUDE
Today we enable users to customize the environment through the use of
ES_INCLUDE. This made sense for legacy reasons when we did not have
nicities like jvm.options (so dumped JVM options in the default include
script) and somewhat duplicates some of the functionality that we will
need from a dedicated environment script. This commit removes support
for ES_INCLUDE as a first step towards a dedicated include script.

Relates #25804
2017-07-20 15:41:59 +09:00
Jason Tedor 3d3d99557d Expand migration note regarding default paths
This commit expands on the migration note regarding the removal of
default.path.data and default.path.logs to include a note that users
that were relying on the defaults (the common case for path.logs), and
they carry over their previous elasticsearch.yml configruation file,
then they must add explicit values for path.data and path.logs.
2017-07-19 13:40:42 +09:00
Luca Cavanna 0d8b753325 IndexClosedException to return 400 rather than 403 (#25752)
403 can be confused with security. If an API doesn't support working against closed indices and closed indices are referred to in a request, that is a bad request, hence 400 is more appropriate.
2017-07-18 10:26:32 +02:00
Christoph Büscher a6e3d356ed Change parsing of numeric `to` and `from` parameters in `date_range` aggregation (#25376)
Currently the `to` and `from` parameter in the `date_range` aggregation is not
parsed with the correct date field format from the mappings or the aggregation
if the argument is numeric, but always treated as a long value specifying
`epoch_millis`. This leads to problems e.g. when the format is `epoch_second`,
but the `to` and `from` are currently treated as millis.

With this change, we interpret these parameters according to the `format` of the target field.
If the `format` in the mappings is not compatible with numeric input values,
a compatible `format` (e.g. `epoch_millis`, `epoch_second`) must be specified in
the `date_range` aggregation itself, otherwise an error is thrown.

#Closes #17920
2017-07-18 09:45:28 +02:00
Simon Willnauer cb4eebcd6a Make `index` in TermsLookup mandatory (#25753)
This change removes the leniency of having a `null` index to fetch
terms from in 6.0 onwards. This feature will be deprecated in the 5.x series
and 6.0 nodes will require the index to be set.

Closes #25750
2017-07-17 18:50:30 +02:00
Ryan Ernst 072402463b Scripting: Remove search template actions (#25717)
The dedicated search template put/get/delete actions are deprecated in
5.6. This commit removes them from 6.0.
2017-07-14 23:12:05 -07:00
Jim Ferenczi 13da3eb53e Refactor QueryStringQuery for 6.0 (#25646)
This change refactors the query_string query to analyze the query text around logical operators of the query string the same way than a match_query/multi_match_query.
It also adds a type parameter that can be used to change the way multi fields query are built the same way than a multi_match query does.

Now that these queries share the same behavior regarding text analysis, some parameters are obsolete and have been deprecated:

split_on_whitespace: This setting is now ignored with a deprecation notice
if it is used explicitely. With this PR The query_string always splits on logical operator.
It simplifies the understanding of the other parameters that can have different meanings
depending on the value of split_on_whitespace.

auto_generate_phrase_queries: This setting is now ignored with a deprecation notice
if it is used explicitely. This setting only makes sense when the parser splits on whitespace.

use_dismax: This setting is now ignored with a deprecation notice
if it is used explicitely. The tie_breaker parameter is sufficient to handle best_fields/most_fields.

Fixes #25574
2017-07-13 15:32:17 +02:00
Jason Tedor 5a416b9922 Use config directory to find jvm.options
This commit removes the environment variable ES_JVM_OPTIONS that allows
the jvm.options file to sit separately from the rest of the config
directory. Instead, we use the CONF_DIR environment variable for custom
configuration location just as we do for the other configuration files.

Relates #25679
2017-07-12 15:29:13 -04:00
Jack Conradson d2b4f7ac5a Disallow lang to be used with Stored Scripts (#25610)
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.
2017-07-12 07:55:57 -07:00
Clinton Gormley 92849c64db Fixed bad asciidoc file name 2017-07-11 12:47:52 +02:00
Clinton Gormley ddbbe9f7cc Tidied up the breaking changes docs 2017-07-11 12:40:14 +02:00
Tal Levy e04be73ad5 remove ingest.new_date_format (#25583) 2017-07-10 13:07:50 -07:00
Colin Goodheart-Smithe 3a5a54e83e Collapses package structure for some bucket aggs (#25579)
This change collapses some of the packages for the bucket aggregations into their parent packages. This was done for the following aggregations:
* The variants of the range aggregation (geo_distance, date and ip) were moved into the `o.e.s.a.bucket.range` package
* The `o.e.s.a.bucket.terms.support` package was removed and the classes were moved to `o.e.s.a.bucket.terms`
* The filter aggregation was moved to `o.e.s.a.bucket.filter`

Since this PR is already relatively large with only the above changes subsequent PRs will do similar operations on relevant metric and pipeline aggregations

Relates to #22868
2017-07-10 15:08:15 +01:00
olcbean 2ba9fd2aec Remove deprecated created and found from index, delete and bulk (#25516)
The created and found fields in index and delete responses became obsolete after the introduction of the result field in index, update and delete responses (#19566).

After deprecating the created and found fields in 5.x (#19633), now they are removed.

Fixes #19630
2017-07-07 13:58:46 -04:00
Lee Hinman 8aa0a5c111 Improve REST error handling when endpoint does not support HTTP verb, add OPTIONS support (#24437)
* Improved REST endpoint exception handling, see #15335

Also improved OPTIONS http method handling to better conform with the
http spec.

* Tidied up formatting and comments

See #15335

* Tests for #15335

* Cleaned up comments, added section number

* Swapped out tab indents for space indents

* Test class now extends ESSingleNodeTestCase

* Capture RestResponse so it can be examined in test cases

Simple addition to surface the RestResponse object so we can run tests
against it (see issue #15335).

* Refactored class name, included feedback

See #15335.

* Unit test for REST error handling enhancements

Randomizing unit test for enhanced REST response error handling. See
issue #15335 for more details.

* Cleaned up formatting

* New constructor to set HTTP method

Constructor added to support RestController test cases.

* Refactored FakeRestRequest, streamlined test case.

* Cleaned up conflicts

* Tests for #15335

* Added functionality to ignore or include path wildcards

See #15335

* Further enhancements to request handling

Refactored executeHandler to prioritize explicit path matches. See
#15335 for more information.

* Cosmetic fixes

* Refactored method handlers

* Removed redundant import

* Updated integration tests

* Refactoring to address issue #17853

* Cleaned up test assertions

* Fixed edge case if OPTIONS method randomly selected as invalid method

In this test, an OPTIONS method request is valid, and should not return
a 405 error.

* Remove redundant static modifier

* Hook the multiple PathTrie attempts into RestHandler.dispatchRequest

* Add missing space

* Correctly retrieve new handler for each Trie strategy

* Only copy headers to threadcontext once

* Fix test after REST header copying moved higher up

* Restore original params when trying the next trie candidate

* Remove OPTIONS for invalidHttpMethodArray so a 405 is guaranteed in tests

* Re-add the fix I already added and got removed during merge :-/

* Add missing GET method to test

* Add documentation to migration guide about breaking 404 -> 405 changes

* Explain boolean response, pull into local var

* fixup! Explain boolean response, pull into local var

* Encapsulate multiple HTTP methods into PathTrie<MethodHandlers>

* Add PathTrie.retrieveAll where all matching modes can be retrieved

Then TrieMatchingMode can be package private and not leak into RestController

* Include body of error with 405 responses to give hint about valid methods

* Fix missing usageService handler addition

I accidentally removed this :X

* Initialize PathTrieIterator modes with Arrays.asList

* Use "== false" instead of !

* Missing paren :-/
2017-07-07 09:01:23 -06:00
Jim Ferenczi 31614c3ddb Remove deprecated fielddata_fields from search request (#25566)
... and inner_hits
2017-07-06 13:02:28 +02:00
Jun Ohtani 6894ef6057 [Analysis] Support normalizer in request param (#24767)
* [Analysis] Support normalizer in request param

Support normalizer param
Support custom normalizer with char_filter/filter param

Closes #23347
2017-07-04 19:16:56 +09:00
Colin Goodheart-Smithe 43efcffcc2 Adds check for negative search request size (#25397)
* Adds check for negative search request size

This change adds a check to `SearchSourceBuilder` to throw and exception if the size set on it is set to a negative value.

Closes #22530

* fix error in reindex

* update re-index tests

* Addresses review comment

* Fixed tests

* Added random negative size test

* Fixes test
2017-07-04 10:51:38 +01:00
Christoph Büscher 927111c91d Remove QueryParseContext from parsing QueryBuilders (#25448)
Currently QueryParseContext is only a thin wrapper around an XContentParser that
adds little functionality of its own. I provides helpers for long deprecated
field names which can be removed and two helper methods that can be made static
and moved to other classes. This is a first step in helping to remove
QueryParseContext entirely.
2017-06-29 17:10:20 +02:00
olcbean 3518e313b8 Unify the result interfaces from get and search in Java client (#25361)
As GetField and SearchHitField have the same members, they have been unified into
DocumentField.

Closes #16440
2017-06-29 11:35:28 +02:00
Jason Tedor be906628d5 Remove implicit 32-bit support
We previously tried to maintain (while not formally supporting) 32-bit
support, although we never tested this anywhere in CI. Since we do not
formally support this, and 32-bit usage is very low, we have elected to
no longer maintain 32-bit support. This commit removes any implication
of 32-bit support.

Relates #25435
2017-06-28 08:24:33 -04:00
Jason Tedor dfd241e0a6 Remove default path settings
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.

Relates #25408
2017-06-26 21:43:20 -04:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Boaz Leskes d963882053 Enable a long translog retention policy by default (#25294)
#25147  added the translog deletion policy but didn't enable it by default. This PR enables a default retention of 512MB (same maximum size of the current translog) and an age of 12 hours (i.e., after 12 hours all translog files will be deleted). This increases to chance to have an ops based recovery, even if the primary flushed or the replica was offline for a few hours.

In order to see which parts of the translog are committed into lucene the translog stats are extended to include information about uncommitted operations.

Views now include all translog ops and guarantee, as before, that those will not go away. Snapshotting a view allows to filter out generations that are not relevant based on a specific sequence number.

Relates to #10708
2017-06-22 17:08:14 +02:00
Martijn van Groningen a977569085
percolator: Deprecate `document_type` parameter.
The `document_type` parameter is no longer required to be specified,
because by default from 6.0 only a single type is allowed. (`index.mapping.single_type` defaults to `true`)
2017-06-22 09:55:06 +02:00
Jun Ohtani 62d1969595 Parse synonyms with the same analysis chain (#8049)
* [Analysis] Parse synonyms with the same analysis chain

Synonym Token Filter / Synonym Graph Filter tokenize synonyms with whatever tokenizer and token filters appear before it in the chain.

Close #7199
2017-06-20 21:50:33 +09:00
javanna 2fb4a0d40c [DOCS] replace '+' with `+` 2017-06-19 16:53:55 +02:00
Luca Cavanna d9ec2a23c5 Remove (deprecated) support for '+' in index expressions (#25274)
Relates to #24515
2017-06-19 15:19:17 +02:00
Luca Cavanna b5cea6980b Delete index API to work only against concrete indices (#25268)
With #23997 we have introduced a new internal index option that allows to resolve index expressions only against concrete indices while ignoring aliases. Such index option was applied to IndicesAliasesRequest, so that the index part of alias actions would only be resolved against concrete indices.

Same is done in this commit with delete index request. Deleting aliases has always been confusing as some users expect it to only remove the alias from the index (which has its own specific API). Even worse, in case of filtered aliases, deleting an alias may leave users with the expectation that only the documents that match the filter are deleted, which was never the case. To address all this confusion, delete index api works now only against concrete indices. WIldcard expressions will be only resolved against concrete index, as if aliases didn't exist. If one tries to delete against an alias, an IndexNotFoundException will be thrown regardless of whether the alias exists or not, as a concrete index with such a name doesn't exist.

Closes #2318
2017-06-16 17:46:01 +02:00
Jim Ferenczi 0036f28a6a Upgrade icu4j for the ICU analysis plugin to 59.1 (#25243)
* Upgrade icu4j for the ICU analysis plugin to 59.1

Lucene upgraded to 59.1 so we should use the same.

Closes #21425

* Add breaking change for the icu upgrade
2017-06-15 13:26:48 +02:00
Boaz Leskes a0fcfc732d Migration docs for #25080 (#25218) 2017-06-14 14:06:53 +02:00
Jim Ferenczi 8250aa4267 Remove the postings highlighter and make unified the default highlighter choice (#25028)
This change removes the `postings` highlighter. This highlighter has been removed from Lucene master (7.x) because it behaves
exactly like the `unified` highlighter when index_options is set to `offsets`:
https://issues.apache.org/jira/browse/LUCENE-7815

It also makes the `unified` highlighter the default choice for highlighting a field (if `type` is not provided).
The strategy used internally by this highlighter remain the same as before, it checks `term_vectors` first, then `postings` and ultimately it re-analyzes the text.
Ultimately it rewrites the docs so that the options that the `unified` highlighter cannot handle are clearly marked as such.
There are few features that the `unified` highlighter is not able to handle which is why the other highlighters (`plain` and `fvh`) are still available.
I'll open separate issues for these features and we'll deprecate the `fvh` and `plain` highlighters when full support for these features have been added to the `unified`.
2017-06-09 14:09:57 +02:00
olcbean 0d5f3958e7 Expand index expressions against indices only when managing aliases (#23997)
The index parameter in the update-aliases, put-alias, and delete-alias APIs no longer accepts alias names. Instead, it accepts only index names (or wildcards which will expand to matching indices).

Closes #23960
2017-06-06 11:01:38 +02:00
Lee Hinman a32d1b91fa Remove comma-separated feature parsing for GetIndicesAction
This removes the parsing of things like `GET /idx/_aliases,_mappings`, instead,
a user must choose between retriving all index metadata with `GET /idx`, or only
a specific form such as `GET /idx/_settings`.

Relates to (and is a prerequisite of) #24437
2017-06-02 14:43:38 -06:00
olcbean 6dea5f14c3 Java api: Remove unneeded getTookInMillis method (#23923)
Some response classes in the java api expose both `getTook()` which returns a `TimeValue` and `getTookInMillis` which returns a `long` value. `getTook()` is enough as one can do `getTook().millis()` to obtain the same result as `getTookInMillis()`, which can be removed.
2017-06-02 11:11:05 +02:00
olcbean e08e92d934 Deleting a document from a non-existing index creates the should not auto create it, unless using EXTERNAL* versioning (#24518)
Currently a `delete document` request against a non-existing index actually **creates** this index.

With this change the `delete document` no longer creates the previously non-existing index and throws an `index_not_found` exception instead.

However as discussed in https://github.com/elastic/elasticsearch/pull/15451#issuecomment-165772026, if an external version is explicitly used, the current behavior is preserved and the index is still created and the document is marked for deletion.

Fixes #15425
2017-05-22 10:00:22 +03:00
archana a5358f34b3 Update mappings.asciidoc
typo
2017-05-20 13:39:05 -05:00
Jack Conradson 1196dfb6bb Remove Deprecated Script Settings (#24756)
Removes all fine-grained script settings replaced by scripts.types_allowed and scripts.contexts_allowed.
2017-05-18 13:32:46 -07:00
Ryan Ernst b214b80e6c GCS Repository: Remove specifying credential file on disk (#24727)
This commit removes the ability to specify the google credential json
file on disk, which is deprecated in 5.5.0.
2017-05-18 10:22:29 -07:00
Ryan Ernst 26e2e933f5 Scripting: Remove native scripts (#24726)
Native scripts have been replaced in documentation by implementing
a ScriptEngine and they were deprecated in 5.5.0. This commit
removes the native script infrastructure for 6.0.

closes #19966
2017-05-17 14:49:24 -07:00
Ryan Ernst 97d2657e18 Remove script access to term statistics (#19462)
In scripts (at least some of the languages), the terms dictionary and
postings can be access with the special _index variable. This is for
very advanced use cases which want to do their own scoring. The problem
is segment level statistics must be recomputed for every document.
Additionally, this is not friendly to the terms index caching as the
order of looking up terms should be controlled by lucene.

This change removes _index from scripts. Anyone using it can and should
instead write a Similarity plugin, which is explicitly designed to allow
doing the calculations needed for a relevance score.

closes #19359
2017-05-16 09:10:09 -07:00
Ryan Ernst 548a5c1386 Docs: Add migration note about .yaml and .json removal (#24689)
relates #19391
relates #24633
2017-05-15 13:42:28 -07:00
Jack Conradson 43292979fd Add New Security Script Settings (#24637)
Settings are simplified to allowed_types and allowed_contexts.  If a setting is not specified the default is to enable all for that setting.
2017-05-15 13:37:46 -07:00
qwerty4030 e7d352b489 Compound order for histogram aggregations. (#22343)
This commit adds support for histogram and date_histogram agg compound order by refactoring and reusing terms agg order code. The major change is that the Terms.Order and Histogram.Order classes have been replaced/refactored into a new class BucketOrder. This is a breaking change for the Java Transport API. For backward compatibility with previous ES versions the (date)histogram compound order will use the first order. Also the _term and _time aggregation order keys have been deprecated; replaced by _key.

Relates to #20003: now that all these aggregations use the same order code, it should be easier to move validation to parse time (as a follow up PR).

Relates to #14771: histogram and date_histogram aggregation order will now be validated at reduce time.

Closes #23613: if a single BucketOrder that is not a tie-breaker is added with the Java Transport API, it will be converted into a CompoundOrder with a tie-breaker.
2017-05-11 18:06:26 +01:00
Martijn van Groningen 840da4aebf
Removed deprecated template query.
Relates to #19390
2017-05-11 14:56:45 +02:00
Ryan Ernst 0789a74055 S3 Repository: Remove deprecated settings (#24445)
These settings are deprecated in 5.5. This change removes them for 6.0.
2017-05-10 20:12:17 -07:00
Adrien Grand a72eaa8e0f Identify documents by their `_id`. (#24460)
Now that indices have a single type by default, we can move to the next step
and identify documents using their `_id` rather than the `_uid`.

One notable change in this commit is that I made deletions implicitly create
types. This helps with the live version map in the case that documents are
deleted before the first type is introduced. Otherwise there would be no way
to differenciate `DELETE index/foo/1` followed by `PUT index/foo/1` from
`DELETE index/bar/1` followed by `PUT index/foo/1`, even though those are
different if versioning is involved.
2017-05-09 16:33:52 +02:00
Clinton Gormley c9aecbb8a5 Added removal of JavaScript and Python to breaking changes 2017-05-05 12:39:50 +02:00
Clinton Gormley 6e970db533 Fixed chunking of breaking changes docs 2017-05-05 11:08:55 +02:00
Jason Tedor de65f51d34 Simplify file store
Today we go to heroic lengths to workaround bugs in the JDK or around
issues like BSD jails to get information about the underlying file
store. For example, we went to lengths to work around a JDK bug where
the file store returned would incorrectly report whether or not a path
is writable in certain situations in Windows operating
systems. Another bug prevented getting file store information on
Windows on a virtual drive on Windows. We no longer need to work
around these bugs, we could simply try to write to disk and let an I/O
exception arise if we could not write to the disk or take advantage of
the fact that these bugs are fixed in recent releases of the JDK
(e.g., the file store bug is fixed since 8u72). Additionally, we
collected information about all file stores on the system which meant
that if the user had a stale NFS mount, Elasticsearch could hang and
fail on startup if that mount point was not available. Finally, we
collected information through Lucene about whether or not a disk was a
spinning disk versus an SSD, information that we do not need since we
assume SSDs by default. This commit takes into consideration that we
simply do not need this heroic effort, we do not need information
about all file stores, and we do not need information about whether or
not a disk spins to greatly simplfy file store handling.

Relates #24402
2017-05-04 11:19:41 -04:00
Luca Cavanna 144f96eaeb Open/Close index api to allow_no_indices by default (#24401)
Open/Close index api have allow_no_indices set to false by default, while delete index has it set to true. The flag controls where a wildcard expression that matches no indices will be ignored or an error will be thrown instead. This commit aligns open/close default behaviour to that of delete index.
2017-05-03 16:22:26 +02:00
javanna a45e2efa00 fix typo in migrate_6_0/java.asciidoc 2017-05-03 15:00:44 +02:00
Luca Cavanna 92bfd16c58 Java api: ActionRequestBuilder#execute to return a PlainActionFuture (#24415)
This change makes the request builder code-path same as `Client#execute`. The request builder used to return a `ListenableActionFuture` when calling execute, which allows to associate listeners with the returned future. For async execution though it is recommended to use the `execute` method that accepts an `ActionListener`, like users would do when using `Client#execute`.

Relates to #24412
Relates to #9201
2017-05-03 11:20:53 +02:00
Nik Everett 2412574e49 Docs: Upserts no longer support version
Closes #16671
2017-04-28 08:43:46 -04:00
Christoph Büscher 026bf2e3ee Remove getCountAsString() from InternalStats and Stats interface (#24291)
The `count` value in the stats aggregation represents a simple doc count
that doesn't require a formatted version. We didn't render an "as_string"
version for count in the rest response, so the method should also be
removed in favour of just using String.valueOf(getCount()) if a string
version of the count is needed.

Closes #24287
2017-04-24 18:40:57 +02:00
Ryan Ernst 151a65ed17 Ec2 Discovery: Cleanup deprecated settings (#24150)
This commit removes the deprecated cloud.aws.* settings. It also removes
backcompat for specifying `discovery.type: ec2`, and unused aws signer
code which was removed in a previous PR.
2017-04-19 12:06:10 -07:00
Adrien Grand 4632661bc7 Upgrade to a Lucene 7 snapshot (#24089)
We want to upgrade to Lucene 7 ahead of time in order to be able to check whether it causes any trouble to Elasticsearch before Lucene 7.0 gets released. From a user perspective, the main benefit of this upgrade is the enhanced support for sparse fields, whose resource consumption is now function of the number of docs that have a value rather than the total number of docs in the index.

Some notes about the change:
 - it includes the deprecation of the `disable_coord` parameter of the `bool` and `common_terms` queries: Lucene has removed support for coord factors
 - it includes the deprecation of the `index.similarity.base` expert setting, since it was only useful to configure coords and query norms, which have both been removed
 - two tests have been marked with `@AwaitsFix` because of #23966, which we intend to address after the merge
2017-04-18 15:17:21 +02:00
Ryan Ernst 1629c9fd5c S3 Repository: Cleanup deprecated settings (#24097)
This commit removes all deprecated settings which start with
`cloud.aws`, `repositories.s3` and repository level client settings.
2017-04-17 14:18:49 -07:00
Lee Hinman 162ce85ff2 Remove customization of ES_USER and ES_GROUP
This removes the ability to configure Elasticsearch to use custom username
and/or group when it is run.

Resolves #23848
2017-04-14 13:24:21 -06:00
Lee Hinman 5cace8e48a Remove shadow replicas
Resolves #22024
2017-04-11 11:26:26 -06:00
Jason Tedor 669fcd05e6 Add note to docs on duplicate keys in config
This commit adds a note to the migration docs that duplicate keys in the
configuration file are no longer permitted.

Relates #24022
2017-04-10 10:58:59 -04:00
Jason Tedor 9056e0cb49 Remove hidden file leniency from plugin service
This commit removes some leniency from the plugin service which skips
hidden files in the plugins directory. We really want to ensure the
integrity of the plugin folder, so hasta la vista leniency.

Relates #23982
2017-04-08 18:22:44 -04:00
Ryan Ernst 83ba677e7f Discovery EC2: Remove region setting (#23991)
We have both endpoint and region settings. Region was removed from s3 to
simplify configuration. This is the ec2 equivalent.

closes #22758
2017-04-07 22:06:40 -07:00
Ryan Ernst 05e2ea1aef AWS Plugins: Remove signer type setting (#23984)
This commit removes support for s3 signer type in 6.0, and adds a note
to the migration guide.

closes #22599
2017-04-07 16:46:17 -07:00
David Pilato f5d41dfc9d Merge branch 'pr/remove-repositories-azure-settings' 2017-03-31 12:33:12 +02:00
Dimitris Athanasiou 34f116eae3 Require explicit query in _delete_by_query API (#23632)
As the query of a search request defaults to match_all,
calling _delete_by_query without an explicit query may
result in deleting all data.

In order to protect users against falling into that
pitfall, this commit adds a check to require the explicit
setting of a query.

Closes #23629
2017-03-28 15:44:57 +01:00
Clinton Gormley e37cdab87f Update scripting.asciidoc
Fixed bad asciidoc
2017-03-16 19:37:38 -07:00
Adrien Grand b388389ada Remove support for the include/pattern syntax. (#23141)
Relates #22933
2017-03-01 10:00:38 +01:00
David Pilato da907e7a7d Remove global `repositories.azure` settings
Today we have multiple ways to define settings when a user needs to create a repository:

* in `elasticsearch.yml` file using `repositories.azure` prefix
* when creating the repository itself with `PUT _snaphot/repo`

The plan is to:

* Deprecate `repositories.azure` settings in 5.x (done with #22856)
* Remove in 6.x (this PR)

Related to #22800
2017-02-20 12:22:54 +01:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
Lee Hinman b3c27a7fdd Disallow include_in_all for 6.0+ indices
Since `_all` is now deprecated and cannot be set for new indices, we should also
disallow any field that has the `include_in_all` parameter set.

Resolves #22923
2017-02-07 19:31:51 -07:00
Nik Everett 0d6e622242 Make dates be ReadableDateTimes in scripts (#22948)
Instead of longs. If you want millis since epoch you can call doc.date_field.value.millis.

Relates to #22875
2017-02-06 16:44:56 -05:00
Lee Hinman 39e7c30912 Change certain replica failures not to fail the replica shard
This changes the way that replica failures are handled such that not all
failures will cause the replica shard to be failed or marked as stale.

In some cases such as refresh operations, or global checkpoint syncs, it is
"okay" for the operation to fail without the shard being failed (because no data
is out of sync). In these cases, instead of failing the shard we should simply
fail the operation, and, in the event it is a user-facing operation, return a
5xx response code including the shard-specific failures.

This was accomplished by having two forms of the `Replicas` proxy, one that is
for non-write operations that does not fail the shard, and one that is for write
operations that will fail the shard when an operation fails.

Relates to #10708
2017-02-03 14:39:46 -07:00
Nik Everett 70e3cce904 Fix name of `enable_position_increments` (#22895)
It was accidentally renamed `enabled_position_increment` in the cleanups
for 5.0. This adds `enable_position_increment` as a deprecated alias
so it will continue to work.
2017-02-03 16:28:27 -05:00
Nicholas Knize b1a6b227e1 Remove deprecated geo query parameters, and GeoPointDistanceRangeQuery
This commit removes the following queries and parameters (which were deprecated in 5.0):

* GeoPointDistanceRangeQuery
* coerce, and ignore_malformed for GeoBoundingBoxQuery, GeoDistanceQuery, GeoPolygonQuery, and GeoDistanceSort
2017-02-03 10:08:00 -06:00
Nicholas Knize f1e1975882 [DOCS] Add sloppy_arc and factor removal to 6.0 migration docs 2017-02-03 09:49:12 -06:00
Jay Modi 7520a107be Optionally require a valid content type for all rest requests with content (#22691)
This change adds a strict mode for xcontent parsing on the rest layer. The strict mode will be off by default for 5.x and in a separate commit will be enabled by default for 6.0. The strict mode, which can be enabled by setting `http.content_type.required: true` in 5.x, will require that all incoming rest requests have a valid and supported content type header before the request is dispatched. In the non-strict mode, the Content-Type header will be inspected and if it is not present or not valid, we will continue with auto detection of content like we have done previously.

The content type header is parsed to the matching XContentType value with the only exception being for plain text requests. This value is then passed on with the content bytes so that we can reduce the number of places where we need to auto-detect the content type.

As part of this, many transport requests and builders were updated to provide methods that
accepted the XContentType along with the bytes and the methods that would rely on auto-detection have been deprecated.

In the non-strict mode, deprecation warnings are issued whenever a request with body doesn't provide the Content-Type header.

See #19388
2017-02-02 14:07:13 -05:00
David Pilato 858333246d Merge branch 'pr/remove-azure-container-auto-creation'
# Conflicts:
#	docs/reference/migration/migrate_6_0/plugins.asciidoc
2017-01-31 09:05:43 +01:00
Ryan Ernst cf7747c338 S3 Repository: Remove region setting (#22853)
This change removes the ability to set region for s3 repositories.
Endpoint should be used instead if a custom s3 location needs to be
used.

closes #22758
2017-01-30 14:34:59 -08:00
Clinton Gormley b5a0e59af4 Fixed bad asciidoc 2017-01-30 11:56:40 +01:00
David Pilato 1898dc2554 Remove auto creation of container for azure repository
Follow up of #22857 where we deprecate automatic creation of azure containers.

BTW I found that the `AzureSnapshotRestoreServiceIntegTests` does not bring any value because it runs basically a Snapshot/Restore operation on local files which we already test in core.

So instead of trying to fix it to make it pass with this PR, I simply removed it.
2017-01-30 11:47:08 +01:00
Adrien Grand dc62255ddd Document upcoming scoring changes. (#22806) 2017-01-30 11:08:49 +01:00
Ryan Ernst fe4043c8ff S3 Repository: Remove bucket auto create (#22846)
closes #22761
2017-01-28 11:13:21 -08:00
Ryan Ernst c921bebc4a S3 Repository: Remove env var and sysprop credentials support (#22842)
These are deprecated in 5.x. This commit removes support for them in 6.0.
2017-01-27 13:43:16 -08:00
Nik Everett ee264c6957 Fix parsing for `max_determinized_states` (#22749)
There was a typo in the `ParseField` declaration. I know
we want to port these parsers to `ObjectParser` eventually
but I don't have the energy for that today and want to get
this fixed.

Closes #22722
2017-01-23 11:57:43 -05:00
Daniel Mitterdorfer aece89d6a1 Make boolean conversion strict (#22200)
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
2017-01-19 07:59:18 +01:00
Boaz Leskes 1227044ddd Add a deprecation notice to shadow replicas (#22647)
Relates to #22024

On top of documentation, the PR adds deprecation loggers and deals with the resulting warning headers.

The yaml test is set exclude versions up to 6.0. This is need to make sure bwc tests pass until this is backported to 5.2.0 . Once that's done, I will change the yaml test version limits
2017-01-18 12:28:09 +01:00
Tim Brooks 16a76d9bc0 Remove blocking TCP clients and servers (#22639)
This commit removes the option to use the blocking variants of the TCP
transport server, TCP transport client, or http server.
2017-01-16 18:38:51 -06:00
Boaz Leskes f88ab76067 Revert "Add a deprecation notice to shadow replicas (#22025)"
This reverts commit 0da190234c.
2017-01-16 16:15:41 +01:00
Boaz Leskes 0da190234c Add a deprecation notice to shadow replicas (#22025)
Also adds deprecation logging.

See #22024
2017-01-16 15:40:05 +01:00
Christoph Büscher 59a48ffc41 ProfileResult and CollectorResult should print machine readable timing information (#22561)
Currently both ProfileResult and CollectorResult print the time field in a human readable string format
 (e.g. "time": "55.20315000ms"). When trying to parse this back to a long value, for example to use in 
the planned high level java rest client, we can lose precision because of conversion and rounding issues. 
This change adds a new additional field (`time_in_nanos`) to the profile response to be able to get the 
original time value in nanoseconds back. 

The old `time` field is only printed when the `?`human=true` flag in the url is set. This follow the behaviour for 
all other stats-related apis. Also the format of the `time` field is slightly changed. Instead of always formatting 
the output as a 10-digit ms value, by using the `XContentBuilder#timeValueField()` method we now print 
the largest time unit present is used (e.g. "s", "ms", "micros").
2017-01-16 14:27:55 +01:00
Lee Hinman 7a18bb50fc Disable _all by default
This change disables the _all meta field by default.

Now that we have the "all-fields" method of query execution, we can save both
indexing time and disk space by disabling it.

_all can no longer be configured for indices created after 6.0.

Relates to #20925 and #21341
Resolves #19784
2017-01-11 16:47:13 -07:00
Martijn van Groningen cb2333dacd percolator: remove deprecated percolate and mpercolate apis 2017-01-10 11:18:27 +01:00
Christoph Büscher a773d46c69 Remove deprecated `minimum_number_should_match` in BoolQueryBuilder
After deprecating getters and setters and the query DSL parameter in 5.x,
support for `minimum_number_should_match` can be removed entirely. Also
consolidated comments with the ones on 5.x branch and added an entry to the
migration docs.
2017-01-03 15:14:33 +01:00
Daniel Mitterdorfer 3ce7b119d2 Enable strict duplicate checks for all XContent types (#22225)
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default for all XContent types (not only JSON).

We have also changed the name of the system property to disable this feature
from `es.json.strict_duplicate_detection` to the now more appropriate name
`es.xcontent.strict_duplicate_detection`.

Relates elastic/elasticsearch#19614
Relates elastic/elasticsearch#22073
2016-12-19 09:29:47 +01:00
Daniel Mitterdorfer 6327e35414 Change type of ingest doc meta-data field 'TIMESTAMP' to `Date` (#22234)
With this commit we change the data type of the 'TIMESTAMP'
meta-data field from a formatted date string to a plain
`java.util.Date` instance. The main reason for this change is
that our benchmarks have indicated that this contributes
significantly to the time spent in the ingest pipeline.

The overhead in terms of indexing throughput of the ingest
pipeline is about 15% and breaks down roughly as follows:

* 5% overhead caused by the conversion from `XContent` -> `Map`
* 5% overhead caused by the timestamp formatting
* 5% overhead caused by the conversion `Map` -> `XContent`

Relates #22074
2016-12-19 09:10:58 +01:00
Daniel Mitterdorfer 7e5058037b Enable strict duplicate checks for JSON content
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.

As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.

Closes #19614
2016-12-14 09:35:53 +01:00
Lee Hinman a4e8b5d952 Throw an exception on unrecognized "match_mapping_type"
When using dynamic templates, ES will now throw an exception if a
`match_mapping_type` is used that doesn't correspond to an actual type.

Relates to #17285
2016-12-12 09:59:48 -07:00
Luca Cavanna 6d987a9b69 Remove support for empty queries (#22092)
Our query DSL supports empty queries (`{}`), which have a different meaning depending on the query that holds it, either ignored, match_all or match_none. We deprecated the support for empty queries in 5.0, where we log a deprecation warning wherever they are used.

The way we supported it once we moved query parsing to the coordinating node was having an Optional<QueryBuilder> return type in all of our parse methods (called fromXContent). See #17624. The central place for this was QueryParseContext#parseInnerQueryBuilder. We can now remove all the optional return types and simply throw an exception whenever an empty query is found.
2016-12-12 12:37:12 +01:00
Luca Cavanna 103984a4a1 Remove indices query (#21837)
The indices query is deprecated since 5.0.0 (#17710). It can now be removed in master (future 6.0 version).
2016-11-30 19:37:01 +01:00
Adrien Grand c5b9c98b99 Remove the `default` store type. (#21616)
It used to be a hybrid store between `niofs` and `mmapfs`, which we removed when
we switched to `fs` by default (which is `mmapfs` on 64-bits systems).
2016-11-30 15:33:26 +01:00
Adrien Grand 90ab477f19 The `terms` query should always map to a Lucene `TermsQuery`. (#21786)
Currently, the `terms` query is just syctactic sugar for a `bool` query when
used in a query context. This change proposes to always generate the same query
in query and filter contexts, which is less confusing.
2016-11-30 15:29:09 +01:00
Adrien Grand 6231009a8f Remove 2.x backward compatibility of mappings. (#21670)
For the record, I also had to remove the geo-hash cell and geo-distance range
queries to make the code compile. These queries already throw an exception in
all cases with 5.x indices, so that does not hurt any more.

I also had to rename all 2.x bwc indices from `index-${version}` to
`unsupported-${version}` to make `OldIndexBackwardCompatibilityIT`
happy.
2016-11-30 13:34:46 +01:00
javanna bbebc644f9 [DOCS] document breaking changes added with #21852 2016-11-29 20:05:30 +01:00
Ryan Ernst 6940b2b8c7 Remove groovy scripting language (#21607)
* Scripting: Remove groovy scripting language

Groovy was deprecated in 5.0. This change removes it, along with the
legacy default language infrastructure in scripting.
2016-11-22 19:24:12 -08:00
Luca Cavanna db8b2dceea Remove ignored type parameter in search_shards api (#21688)
The `type` parameter has always been accepted by the search_shards api, probably to make the api and its urls the same as search. Truth is that the type never had any effect, it's been ignored from day one while accepting it may make users think that we actually do something with it.

This commit removes support for the type parameter from the REST layer and the Java API. Backwards compatibility is maintained on the transport layer though.

The new added serialization test also uncovered a bug in the java API where the `ClusterSearchShardsRequest` could be created with no arguments, but the indices were required to be not null otherwise the request couldn't be serialized as `writeTo` would throw NPE. Fixed by setting a default value (empty array) for indices.
2016-11-22 17:22:33 +01:00
Adrien Grand c7fc688096 Add information about the removal of store throttling to the migration guide.
Relates to #21573
2016-11-21 15:03:07 +01:00
Lee Hinman 96122aa518 Be strict when parsing values searching for booleans (#21555)
This changes only the query parsing behavior to be strict when searching on
boolean values. We continue to accept the variety of values during index time,
but searches will only be parsed using `"true"` or `"false"`.

Resolves #21545
2016-11-15 10:36:57 -07:00
Nik Everett 7c3886769f Breaking changes docs for template index_patterns
0219a211d3 added support for templates
to have multiple patterns and renamed `template` to `index_patterns`.
This adds the breaking changes docs for that.
2016-11-11 10:13:19 -05:00
Jason Tedor b743ab0b07 Remove 5.x references from cat API migration doc
This commit removes some references to 5.x that were picked up when the
migration docs for the cat API were migrated from 5.x to master.

Relates #21342
2016-11-09 07:09:59 -05:00
Jason Tedor 168c54fa6e Add migration docs for cat API
This commit adds migration docs for the cat API, including a note
regarding the change in response in the cat thread pool API for
unbounded queue sizes.

Relates #21342
2016-11-09 07:07:52 -05:00
Luca Cavanna c2160a88b5 Remove support for controversial ignore_unavailable and allow_no_indices from indices exists api (#20712)
Exist requests are supposed to never throw an exception, but rather return true or false depending on whether some resource exists or not. Indices exists does that for indices and accepts wildcard expressions too. The way the api works internally is by resolving indices and catching IndexNotFoundException: if an exception is thrown the index does not exist hence it returns false, otherwise it returns true. That works ok only if ignore_unavailable and allow_no_indices indices options are both set to false, meaning that they are strict and any missing index or wildcard expressions that resolves to no indices will lead to an exception that can be thrown and cause false to be returned.

Unfortunately the indices options have  been configurable up until now for this request, meaning that one can set ignore_unavailable or allow_no_indices to true and have the indices exist request return true for indices that really don't exist, which makes very little sense in the context of this api.

This commit removes the indicesOptions setter from the IndicesExistsRequest and makes settable only expandWildcardsOpen and expandWildcardsClosed, hence a subset of the available indices options. This way we can guarantee more consistent behaviour of the indices exists api. We can then remove the ignore_unavailable and allow_no_indices option from indices exists api spec
2016-11-04 19:26:37 +01:00
Jun Ohtani a66c76eb44 Merge pull request #20704 from johtani/remove_request_params_in_analyze_api
Removing request parameters in _analyze API
2016-10-27 17:43:18 +09:00
Igor Motov a541f0187d Docs: add documentation about removal of cluster.routing.allocation.snapshot.relocation_enabled 2016-10-20 14:19:12 -10:00
Pascal Borreli fcb01deb34 Fixed typos (#20843) 2016-10-10 14:51:47 -06:00
Jun Ohtani 945fa499d2 Deprecating request parameters in _analyze API
Remove params in indices.analyze.json
Fix REST changes

Closes #20246
2016-10-07 16:23:24 +09:00
Jun Ohtani 99236b7627 Removing request parameters in _analyze API
Fix small English issue in breaking changes

Closes #20246
2016-10-07 16:23:24 +09:00
Jun Ohtani eca9894c5f Removing request parameters in _analyze API
Remove unused imports
Replace POST method by GET method in docs
 Add breaking changes explanation
 Fix small issue in Kuromoji docs

Closes #20246
2016-10-07 16:23:24 +09:00
Lee Hinman 85402d5220 [DOCS] Remove non-valid link to mapping migration document 2016-09-28 09:09:19 -06:00
Lee Hinman 3f77eacab1 Revert "Default `include_in_all` for numeric-like types to false"
This reverts commit 6666892038.
2016-09-28 07:07:46 -06:00
David Pilato dfd1eebdd0 Remove mapper attachments plugin
We now have in 5.0.0 `ingest-attachment` plugin.
We can remove `mapper-attachments` plugin for 6.0.

Closes #18837.
2016-09-19 09:01:16 +02:00
Lee Hinman 94625d74e4 No longer allow cluster name in data path
In 5.x we allowed this with a deprecation warning. This removes the code
added for that deprecation, requiring the cluster name to not be in the
data path.

Resolves #20391
2016-09-12 15:47:01 -06:00
Lee Hinman 49695af2ac Remove FORCE version_type
This was an error-prone version type that allowed overriding previous
version semantics. It could cause primaries and replicas to be out of
sync however, so it has been removed.

Resolves #19769
2016-09-08 15:29:47 -06:00
Jason Tedor 27ff4f327c Remove allow unquoted JSON
Previous versions of Elasticsearch permitted unquoted JSON field names even though this is against the JSON spec. This leniency was disabled by default in the 5.x series of Elasticsearch but a backwards compatibility layer was added via a system property with the intention of removing this layer in 6.0.0. This commit removes this backwards compatibility layer.

Relates #20388
2016-09-08 13:36:31 -04:00
Clinton Gormley 891a15db22 Add breaking changes for 6.0 2016-09-08 18:27:01 +02:00
Lee Hinman 6666892038 Default `include_in_all` for numeric-like types to false
This includes:

- All regular numeric types such as int, long, scaled-float, double, etc
- IP addresses
- Dates
- Geopoints and Geoshapes

Relates to #19784
2016-09-08 09:09:48 -06:00
Jason Tedor 8e7dfae7d1 Remove collect payloads parameter
The collect_payloads parameter of the span_near query was previously
deprecated with the intention to be removed. This commit removes this
parameter.

Relates #20385
2016-09-08 09:37:36 -04:00
Lee Hinman b4cc3cd35d Remove FORCE version_type
This was an error-prone version type that allowed overriding previous
version semantics. It could cause primaries and replicas to be out of
sync however, so it has been removed.

Resolves #19769
2016-09-07 13:05:18 -06:00
Martijn van Groningen 245882cde3 * Removed `script.default_lang` setting and made `painless` the hardcoded default script language.
** The default script language is now maintained in `Script` class.
* Added `script.legacy.default_lang` setting that controls the default language for scripts that are stored inside documents (for example percolator queries).  This defaults to groovy.
** Added `QueryParseContext#getDefaultScriptLanguage()` that manages the default scripting language. Returns always `painless`, unless loading query/search request in legacy mode then the returns what is configured in `script.legacy.default_lang` setting.
** In the aggregation parsing code added `ParserContext` that also holds the default scripting language like `QueryParseContext`. Most parser don't have access to `QueryParseContext`. This is for scripts in aggregations.
* The `lang` script field is always serialized (toXContent).

Closes #20122
2016-09-06 18:44:48 +02:00
Jun Ohtani c4759bcc02 Merge pull request #20285 from johtani/fix/remove_token_filter_param_in_analyze_api
Remove `token_filter` in _analyze API
2016-09-03 02:03:51 +09:00
Areek Zillur c92f82e624 Merge pull request #20169 from areek/doc/fix_completion_breaking_changes
Update breaking changes for completion suggester
2016-09-02 12:39:16 -04:00
Areek Zillur af215b528f move completion performance tips from migration docs to completion docs 2016-09-02 12:37:56 -04:00
Jun Ohtani aef2e5d90e Remove `token_filter` in _analyze API
Fix wording in docs
Refactoring RestAnalyzeActionTests using expectThrows()

Closes #20283
2016-09-02 15:08:28 +09:00
Areek Zillur c869ca18eb Update breaking changes for completion suggester
To indicate we removed completion payload option
in favour of returning document source with
completion suggestions
2016-09-01 21:48:30 -04:00
Jun Ohtani 3d9f8ed764 Remove `token_filter` in _analyze API
Remove the param and change docs

Closes #20283
2016-09-02 01:36:45 +09:00
javanna 5f299ff46f add mem section back to cluster stats
The mem section was buggy in cluster stats and removed. It is now added back with the same structure as in node stats, containing total memory, available memory, used memory and percentages. All the values are the sum of all the nodes across the cluster (or at least the ones that we were able to get the values from).
2016-09-01 11:26:03 +02:00
Simon Willnauer a0becd26b1 Optimize indexing for the autogenerated ID append-only case (#20211)
If elasticsearch controls the ID values as well as the documents
version we can optimize the code that adds / appends the documents
to the index. Essentially we an skip the version lookup for all
documents unless the same document is delivered more than once.

On the lucene level we can simply call IndexWriter#addDocument instead
of #updateDocument but on the Engine level we need to ensure that we deoptimize
the case once we see the same document more than once.

This is done as follows:

1. Mark every request with a timestamp. This is done once on the first node that
receives a request and is fixed for this request. This can be even the
machine local time (see why later). The important part is that retry
requests will have the same value as the original one.

2. In the engine we make sure we keep the highest seen time stamp of "retry" requests.
This is updated while the retry request has its doc id lock. Call this `maxUnsafeAutoIdTimestamp`

3. When the engine runs an "optimized" request comes, it compares it's timestamp with the
current `maxUnsafeAutoIdTimestamp` (but doesn't update it). If the the request
timestamp is higher it is safe to execute it as optimized (no retry request with the same
timestamp has been run before). If not we fall back to "non-optimzed" mode and run the request as a retry one
and update the `maxUnsafeAutoIdTimestamp` unless it's been updated already to a higher value

Relates to #19813
2016-09-01 10:39:40 +02:00
Jason Tedor 76ab02e002 Merge branch 'master' into log4j2
* master:
  Avoid NPE in LoggingListener
  Randomly use Netty 3 plugin in some tests
  Skip smoke test client on JDK 9
  Revert "Don't allow XContentBuilder#writeValue(TimeValue)"
  [docs] Remove coming in 2.0.0
  Don't allow XContentBuilder#writeValue(TimeValue)
  [doc] Remove leftover from CONSOLE conversion
  Parameter improvements to Cluster Health API wait for shards (#20223)
  Add 2.4.0 to packaging tests list
  Docs: clarify scale is applied at origin+offest (#20242)
2016-08-31 16:37:55 -04:00
Jason Tedor 750033dc4b Update docs for Log4j 2
This commit updates the logging docs for Elasticsearch to reflect the
migration to Log4j 2.
2016-08-31 15:51:52 -04:00
Ali Beyad 4641254ea6 Parameter improvements to Cluster Health API wait for shards (#20223)
* Params improvements to Cluster Health API wait for shards

Previously, the cluster health API used a strictly numeric value
for `wait_for_active_shards`. However, with the introduction of
ActiveShardCount and the removal of write consistency level for
replication operations, `wait_for_active_shards` is used for
write operations to represent values for ActiveShardCount. This
commit moves the cluster health API's usage of `wait_for_active_shards`
to be consistent with its usage in the write operation APIs.

This commit also changes `wait_for_relocating_shards` from a
numeric value to a simple boolean value `wait_for_no_relocating_shards`
to set whether the cluster health operation should wait for
all relocating shards to complete relocation.

* Addresses code review comments

* Don't be lenient if `wait_for_relocating_shards` is set
2016-08-31 11:58:19 -04:00
Nik Everett df73292256 Add an alias action to delete an index
While removing an index isn't actually an alias action, if we add
an alias action that deletes an index then we can delete and index
and add an alias with the same name as the index atomically, in
the same cluster state update.

Closes #20064
2016-08-30 10:15:21 -04:00
Jun Ohtani 450f47d5b5 Validate blank field name
add validation and validate only 5.0+
Add tests before 5.0

Closes #19251
2016-08-26 20:10:33 +09:00
Simon Willnauer c499427166 Use _refresh instead of reading from Translog in the RT GET case (#20102)
Today we do a lot of accounting inside the engine to maintain locations
of documents inside the transaction log. This is only needed to ensure
we can return the documents source from the engine if it hasn't been refreshed.
Aside of the added complexity to be able to read from the currently writing translog,
maintainance of pointers into the translog this also caused inconsistencies like different values
of the `_ttl` field if it was read from the tlog or not. TermVectors are totally different if
the document is fetched from the tranlog since copy fields are ignored etc.

This chance will simply call `refresh` if the documents latest version is not in the index. This
streamlines the semantics of the `_get` API and allows for more optimizations inside the engine
and on the transaction log. Note: `_refresh` is only called iff the requested document is not refreshed
yet but has recently been updated or added.

#Relates to #19787
2016-08-24 15:30:08 +02:00
Lee Hinman 3298a4ed38 Revert "Merge remote-tracking branch 'dakrone/exclude-numerics-from-all'"
This reverts commit 514585290c, reversing
changes made to 8563c8d897.
2016-08-23 09:24:33 -06:00
Jack Conradson 131e370a16 Make Painless the default scripting language.
Closes #20017
2016-08-22 17:38:02 -07:00
Lee Hinman d7e516c0b4 Default `include_in_all` for numeric-like types to false
This includes:

- All regular numeric types such as int, long, scaled-float, double, etc
- IP addresses
- Dates
- Geopoints and Geoshapes

Relates to #19784
2016-08-19 15:50:38 -06:00
Adrien Grand a4ea7e7223 Switch indices.exists_type from `{index}/{type}` to `{index}/_mapping/{type}`. #20055
This will help remove types as we will need `{index}/{id}` to tell whether a
document exists.

Relates #15613
2016-08-19 09:18:24 +02:00
Lee Hinman 6030acb43b Disallow creating indices starting with '-' or '+'
Previously this was possible, which was problematic when issuing a
request like `DELETE /-myindex`, which was interpretted as "delete
everything except for myindex".

Resolves #19800
2016-08-17 15:13:03 -06:00
Adrien Grand d894db1590 Only use `PUT` for index creation, not POST. #20001
Currently both `PUT` and `POST` can be used to create indices. This commit
removes support for `POST index_name` so that we can use it to index documents
with auto-generated ids once types are removed.

Relates #15613
2016-08-17 10:15:42 +02:00
Nicholas Knize a93af8651c add geo distance script breaking changes to migration docs 2016-08-15 19:12:24 -05:00
Jason Tedor 1f0673c9bd Default max local storage nodes to one
This commit defaults the max local storage nodes to one. The motivation
for this change is that a default value greather than one is dangerous
as users sometimes end up unknowingly starting a second node and start
thinking that they have encountered data loss.

Relates #19964
2016-08-12 09:26:20 -04:00
Nik Everett 9f8f2ea54b Remove ESIntegTestCase#pluginList
It was a useful method in 1.7 when javac's type inference wasn't as
good, but now we can just replace it with `Arrays.asList`.
2016-08-11 15:44:02 -04:00
Jason Tedor e899e8b4e0 Reword expect header bug notice
This commit rewords the expect header bug notice to provide the precise
details for the bug arising. In particular, the bug does not impact any
request over 1024 bytes, but instead impacts any request with a body
that is sent in two requests, the first with an Expect: 100-continue
header. The size is irrelevant, and requests with bodies larger than
1024 bytes are okay as long as the Expect: 100-continue header is not
also sent.

Relates #19911
2016-08-10 10:42:58 -04:00
Clinton Gormley 83532ac377 Documented netty4 Expect bug in release notes and breaking changes 2016-08-10 10:16:25 +02:00
Ali Beyad 4f70ee521f Migration Guide changes for BlobContainer (#19731)
Adds a notice in the migration guide for removing
two deleteBlobs and one writeBlob method from the
BlobContainer interface.
2016-08-03 10:41:25 -04:00
Ali Beyad a21dd80f1b Documentation changes for wait_for_active_shards (#19581)
Documentation changes and migration doc changes for introducing 
wait_for_active_shards and removing write consistency level.

Closes #19581
2016-08-02 09:15:01 -04:00
Ali Beyad a7e68e36cf Fixes header level for migration doc entry 2016-08-01 16:15:12 -04:00
Colin Goodheart-Smithe 3f13f02575 [DOCS] updated documentation for transport client changes
Updated dependency in Java API docs and added section in breaking
changes
2016-07-29 14:25:12 +01:00
Nik Everett f159156931 [docs] Deprecate found and created (#19633)
These parts of delete and index response have been replaced with the
operation field.
2016-07-28 10:20:48 -04:00
Martijn van Groningen 2fdf79d8d4 Deprecate template query.
Closes #19390
2016-07-27 09:50:44 +02:00
Boaz Leskes cd596772ee Persistent Node Names (#19456)
With #19140 we started persisting the node ID across node restarts. Now that we have a "stable" anchor, we can use it to generate a stable default node name and make it easier to track nodes over a restarts. Sadly, this means we will not have those random fun Marvel characters but we feel this is the right tradeoff.

On the implementation side, this requires a bit of juggling because we now need to read the node id from disk before we can log as the node node is part of each log message. The PR move the initialization of NodeEnvironment as high up in the starting sequence as possible, with only one logging message before it to indicate we are initializing. Things look now like this:

```
[2016-07-15 19:38:39,742][INFO ][node                     ] [_unset_] initializing ...
[2016-07-15 19:38:39,826][INFO ][node                     ] [aAmiW40] node name set to [aAmiW40] by default. set the [node.name] settings to change it
[2016-07-15 19:38:39,829][INFO ][env                      ] [aAmiW40] using [1] data paths, mounts [[ /(/dev/disk1)]], net usable_space [5.5gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2016-07-15 19:38:39,830][INFO ][env                      ] [aAmiW40] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 19:38:39,837][INFO ][node                     ] [aAmiW40] version[5.0.0-alpha5-SNAPSHOT], pid[46048], build[473d3c0/2016-07-15T17:38:06.771Z], OS[Mac OS X/10.11.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_51/25.51-b03]
[2016-07-15 19:38:40,980][INFO ][plugins                  ] [aAmiW40] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy, transport-netty], plugins []
[2016-07-15 19:38:43,218][INFO ][node                     ] [aAmiW40] initialized
```

Needless to say, settings `node.name` explicitly still works as before.

The commit also contains some clean ups to the relationship between Environment, Settings and Plugins. The previous code suggested the path related settings could be changed after the initial Environment was changed. This did not have any effect as the security manager already locked things down.
2016-07-23 22:46:48 +02:00
Jun Ohtani cebad703fe Analyze: Specify anonymous char_filters/tokenizer/token_filters in the analyze API
Add parser for anonymous char_filters/tokenizer/token_filters
Using Settings in AnalyzeRequest for anonymous definition
Add breaking changes document

Closed #8878
2016-07-21 11:06:36 +09:00
Nik Everett 3a82c613e4 Migrate query registration from push to pull
Remove `ParseField` constants used for names where there are no deprecated
names and just use the `String` version of the registration method instead.

This is step 2 in cleaning up the plugin interface for extending
search time actions. Aggregations are next.

This is breaking for plugins because those that register a new query should
now implement `SearchPlugin` rather than `onModule(SearchModule)`.
2016-07-20 12:33:51 -04:00
Martijn van Groningen e0ebf5da1c Template cleanup:
* Removed `Template` class and unified script & template parsing logic. Templates are scripts, so they should be defined as a script. Unless there will be separate template infrastructure, templates should share as much code as possible with scripts.
* Removed ScriptParseException in favour for ElasticsearchParseException
* Moved TemplateQueryBuilder to lang-mustache module because this query is hard coded to work with mustache only
2016-07-18 10:16:01 +02:00