Commit Graph

3887 Commits

Author SHA1 Message Date
Glen Smith e9dfb2a215 Fix another simulate example in ingest docs
When simulating an ingest pipeline against an existing pipeline, the
_source field is required to wrap each doc. This commit fixes another
example in the docs that is missing this.
    
Relates #25743, relates e3a0c11239
2017-07-17 15:17:42 +09:00
Glen Smith e3a0c11239 Fix simulate example in ingest docs
When simulating an ingest pipeline against an existing pipeline, the
_source field is required to wrap each doc. This commit fixes an example
in the docs that is missing this.

Relates #25742
2017-07-17 14:17:41 +09:00
Ryan Ernst 072402463b Scripting: Remove search template actions (#25717)
The dedicated search template put/get/delete actions are deprecated in
5.6. This commit removes them from 6.0.
2017-07-14 23:12:05 -07:00
Ryan Ernst b1762d69b5 Setup: Change default heap to 1G (#25695)
This commit changes the default heap size to 1 GB. Experimenting with
elasticsearch is often done on laptops, and 1 GB is much friendlier to
laptop memory. It does put more pressure on the gc, but the tradeoff is
a smaller default footprint. Users running in production can (and
should) adjust the heap size as necessary for their usecase.
2017-07-14 09:38:08 -07:00
Bodecker DellaMaria 4f0dc5bf32 Mark filtered query example as not to be used (#25661)
The Filtered Query has been deprecated in favour of the Bool Query with a filter context. However, this deleted page for the Filtered Query is often ranked highly in search results when searching for documentation on "filtered queries". Often people just copy the first code snippet they see, which in this case is the INCORRECT syntax (the correct syntax follows). I think reordering the examples would help avoid a lot of confusion (I have seen people make this same mistake 3 times now)

Adding a comment to indicate that the first example shouldn't be used
2017-07-14 11:45:21 +02:00
Martijn van Groningen c8777c4c2e
docs: Updated reference docs that `document_type` is deprecated 2017-07-14 11:07:46 +02:00
Antonio Matarrese afd9a1c1b1 [DOCS] Explain mapping explosion (#25654) 2017-07-14 09:47:41 +02:00
Neil Rickards 5189bd14f1 [Docs] Fix typo in pattern-tokenizer.asciidoc (#25626) 2017-07-13 18:43:48 +02:00
Jim Ferenczi fe383b7c27 More clarifications on the unified highlighter being the new default (#25668)
* More clarifications on the unified highlighter being the new default
2017-07-13 15:38:58 +02:00
Jim Ferenczi 13da3eb53e Refactor QueryStringQuery for 6.0 (#25646)
This change refactors the query_string query to analyze the query text around logical operators of the query string the same way than a match_query/multi_match_query.
It also adds a type parameter that can be used to change the way multi fields query are built the same way than a multi_match query does.

Now that these queries share the same behavior regarding text analysis, some parameters are obsolete and have been deprecated:

split_on_whitespace: This setting is now ignored with a deprecation notice
if it is used explicitely. With this PR The query_string always splits on logical operator.
It simplifies the understanding of the other parameters that can have different meanings
depending on the value of split_on_whitespace.

auto_generate_phrase_queries: This setting is now ignored with a deprecation notice
if it is used explicitely. This setting only makes sense when the parser splits on whitespace.

use_dismax: This setting is now ignored with a deprecation notice
if it is used explicitely. The tie_breaker parameter is sufficient to handle best_fields/most_fields.

Fixes #25574
2017-07-13 15:32:17 +02:00
Deb Adair ded9f55263 [DOCS] Incorporated feedback on the highlighting changes. 2017-07-12 16:36:33 -07:00
Ryan Ernst 70b2897bdf Scripting: Deprecate stored search template apis (#25437)
This commit deprecates the PUT, GET and DELETE search template apis.
Instead, the stored script api should be used.

closes #24596
2017-07-12 16:07:28 -07:00
Simon Willnauer e81804cfa4 Add a shard filter search phase to pre-filter shards based on query rewriting (#25658)
Today if we search across a large amount of shards we hit every shard. Yet, it's quite
common to search across an index pattern for time based indices but filtering will exclude
all results outside a certain time range ie. `now-3d`. While the search can potentially hit
hundreds of shards the majority of the shards might yield 0 results since there is not document
that is within this date range. Kibana for instance does this regularly but used `_field_stats`
to optimize the indexes they need to query. Now with the deprecation of `_field_stats` and it's upcoming removal a single dashboard in kibana can potentially turn into searches hitting hundreds or thousands of shards and that can easily cause search rejections even though the most of the requests are very likely super cheap and only need a query rewriting to early terminate with 0 results.

This change adds a pre-filter phase for searches that can, if the number of shards are higher than a the `pre_filter_shard_size` threshold (defaults to 128 shards), fan out to the shards
and check if the query can potentially match any documents at all. While false positives are possible, a negative response means that no matches are possible. These requests are not subject to rejection and can greatly reduce the number of shards a request needs to hit. The approach here is preferable to the kibana approach with field stats since it correctly handles aliases and uses the correct threadpools to execute these requests. Further it's completely transparent to the user and improves scalability of elasticsearch in general on large clusters.
2017-07-12 22:19:20 +02:00
Jason Tedor 86e9438d3c Prevent excessive disk consumption by log files
This commit enables management of the main Elasticsearch log files
out-of-the-box by the following changes:
 - compress rolled logs
 - roll logs every 128 MB
 - maintain a sliding window of logs
 - remove the oldest logs maintaining no more than 2 GB of compressed
   logs on disk

Relates #25660
2017-07-12 15:52:00 -04:00
Jason Tedor 5a416b9922 Use config directory to find jvm.options
This commit removes the environment variable ES_JVM_OPTIONS that allows
the jvm.options file to sit separately from the rest of the config
directory. Instead, we use the CONF_DIR environment variable for custom
configuration location just as we do for the other configuration files.

Relates #25679
2017-07-12 15:29:13 -04:00
Jack Conradson d2b4f7ac5a Disallow lang to be used with Stored Scripts (#25610)
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.
2017-07-12 07:55:57 -07:00
Deb Adair b5e81132cf [DOCS] Reorganized the highlighting topic so it's less confusing. 2017-07-11 21:16:14 -07:00
Jason Tedor e165c405ac Add an underscore to flood stage setting
This is a minor nitty bikeshedding change that renames the suffix of the
disk flood stage setting to "flood_stage" from "floodstage".

Relates #25659
2017-07-11 22:02:00 -04:00
James Baiera 847378a43b Add another parent value option to join documentation (#25609)
Indexing a join field on a document requires a value of type "object" and two sub fields "name" 
and "parent". The "parent" field is only required on child documents, but the "name" field which 
denotes the name of the relation is always needed. Previously, only the short-hand version of the 
join field was documented. This adds documentation for the long-hand join field data, and 
explicitly points out that just specifying the name of the relation for the field value is a 
convenience shortcut.
2017-07-11 15:36:59 -04:00
Simon Willnauer 98c91a3bd0 Limit the number of concurrent shard requests per search request (#25632)
This is a protection mechanism to prevent a single search request from
hitting a large number of shards in the cluster concurrently. If a search is
executed against all indices in the cluster this can easily overload the cluster
causing rejections etc. which is not necessarily desirable. Instead this PR adds
a per request limit of `max_concurrent_shard_requests` that throttles the number of
concurrent initial phase requests to `256` by default. This limit can be increased per request
and protects single search requests from overloading the cluster. Subsequent PRs can introduces
addiontional improvemetns ie. limiting this on a `_msearch` level, making defaults a factor of
the number of nodes or sort shards iters such that we gain the best concurrency across nodes.
2017-07-11 16:23:10 +02:00
Clinton Gormley bd7ddfa175 Removed field-stats docs 2017-07-11 15:15:25 +02:00
Clinton Gormley 92849c64db Fixed bad asciidoc file name 2017-07-11 12:47:52 +02:00
Clinton Gormley ddbbe9f7cc Tidied up the breaking changes docs 2017-07-11 12:40:14 +02:00
Herman Schaaf 977712f977 Change small typo in shards_allocation.asciidoc (#25643) 2017-07-11 11:25:49 +02:00
Tal Levy e04be73ad5 remove ingest.new_date_format (#25583) 2017-07-10 13:07:50 -07:00
Colin Goodheart-Smithe 3a5a54e83e Collapses package structure for some bucket aggs (#25579)
This change collapses some of the packages for the bucket aggregations into their parent packages. This was done for the following aggregations:
* The variants of the range aggregation (geo_distance, date and ip) were moved into the `o.e.s.a.bucket.range` package
* The `o.e.s.a.bucket.terms.support` package was removed and the classes were moved to `o.e.s.a.bucket.terms`
* The filter aggregation was moved to `o.e.s.a.bucket.filter`

Since this PR is already relatively large with only the above changes subsequent PRs will do similar operations on relevant metric and pipeline aggregations

Relates to #22868
2017-07-10 15:08:15 +01:00
Clinton Gormley e85871cfe9 Update cross-cluster-search.asciidoc
Increased the required min version of CCS in the docs to 5.5
2017-07-10 12:04:05 +02:00
DeDe Morton baa1858f56 [DOCS] Fix link (#25616) 2017-07-07 20:40:44 -07:00
DeDe Morton a4fedb213e Fix attribute reference on redirects page (#25614) 2017-07-07 20:15:42 -07:00
Jason Tedor 8148e25087 Fix disk allocator docs
This commit fixes the disk allocator docs which were broken due to the
inadvertent removal of some docs snippet markup.
2017-07-07 22:11:09 -04:00
Jason Tedor bc22c1c286 Add disk threshold settings validation
This commit adds cross-settings validation for the low/high/flood stage
disk watermark settings. This validation was enabled by the introduction
of multiple settings validation.

Relates #25600
2017-07-07 19:54:36 -04:00
matarrese 2eafbaf759 Document aggregating by day of the week (#25602)
Add documentation for aggregating by day of the week.

Closes #24660
2017-07-07 14:16:53 -04:00
olcbean 2ba9fd2aec Remove deprecated created and found from index, delete and bulk (#25516)
The created and found fields in index and delete responses became obsolete after the introduction of the result field in index, update and delete responses (#19566).

After deprecating the created and found fields in 5.x (#19633), now they are removed.

Fixes #19630
2017-07-07 13:58:46 -04:00
Lee Hinman 8aa0a5c111 Improve REST error handling when endpoint does not support HTTP verb, add OPTIONS support (#24437)
* Improved REST endpoint exception handling, see #15335

Also improved OPTIONS http method handling to better conform with the
http spec.

* Tidied up formatting and comments

See #15335

* Tests for #15335

* Cleaned up comments, added section number

* Swapped out tab indents for space indents

* Test class now extends ESSingleNodeTestCase

* Capture RestResponse so it can be examined in test cases

Simple addition to surface the RestResponse object so we can run tests
against it (see issue #15335).

* Refactored class name, included feedback

See #15335.

* Unit test for REST error handling enhancements

Randomizing unit test for enhanced REST response error handling. See
issue #15335 for more details.

* Cleaned up formatting

* New constructor to set HTTP method

Constructor added to support RestController test cases.

* Refactored FakeRestRequest, streamlined test case.

* Cleaned up conflicts

* Tests for #15335

* Added functionality to ignore or include path wildcards

See #15335

* Further enhancements to request handling

Refactored executeHandler to prioritize explicit path matches. See
#15335 for more information.

* Cosmetic fixes

* Refactored method handlers

* Removed redundant import

* Updated integration tests

* Refactoring to address issue #17853

* Cleaned up test assertions

* Fixed edge case if OPTIONS method randomly selected as invalid method

In this test, an OPTIONS method request is valid, and should not return
a 405 error.

* Remove redundant static modifier

* Hook the multiple PathTrie attempts into RestHandler.dispatchRequest

* Add missing space

* Correctly retrieve new handler for each Trie strategy

* Only copy headers to threadcontext once

* Fix test after REST header copying moved higher up

* Restore original params when trying the next trie candidate

* Remove OPTIONS for invalidHttpMethodArray so a 405 is guaranteed in tests

* Re-add the fix I already added and got removed during merge :-/

* Add missing GET method to test

* Add documentation to migration guide about breaking 404 -> 405 changes

* Explain boolean response, pull into local var

* fixup! Explain boolean response, pull into local var

* Encapsulate multiple HTTP methods into PathTrie<MethodHandlers>

* Add PathTrie.retrieveAll where all matching modes can be retrieved

Then TrieMatchingMode can be package private and not leak into RestController

* Include body of error with 405 responses to give hint about valid methods

* Fix missing usageService handler addition

I accidentally removed this :X

* Initialize PathTrieIterator modes with Arrays.asList

* Use "== false" instead of !

* Missing paren :-/
2017-07-07 09:01:23 -06:00
Viral-Inc f3569debb6 [Docs] Fix typo in bootstrap-checks.asciidoc (#25597) 2017-07-07 15:37:40 +02:00
Martijn van Groningen d0f9f425bd
parent/child: Removed ParentJoinFieldSubFetchPhase 2017-07-06 13:15:02 +02:00
Jim Ferenczi 31614c3ddb Remove deprecated fielddata_fields from search request (#25566)
... and inner_hits
2017-07-06 13:02:28 +02:00
Clinton Gormley ca12b1f2a6 Tidied up the disk allocator docs 2017-07-06 12:16:53 +02:00
Simon Willnauer 6e5cc424a8 Switch indices read-only if a node runs out of disk space (#25541)
Today when we run out of disk all kinds of crazy things can happen
and nodes are becoming hard to maintain once out of disk is hit.
While we try to move shards away if we hit watermarks this might not
be possible in many situations. Based on the discussion in #24299
this change monitors disk utilization and adds a flood-stage watermark
that causes all indices that are allocated on a node hitting the flood-stage
mark to be switched read-only (with the option to be deleted). This allows users to react on the low disk
situation while subsequent write requests will be rejected. Users can switch
individual indices read-write once the situation is sorted out. There is no
automatic read-write switch once the node has enough space. This requires
user interaction.

The flood-stage watermark is set to `95%` utilization by default.

Closes #24299
2017-07-05 22:18:23 +02:00
Adrien Grand 26de905f1e Fix the documentation to state that the `_id` field is indexed. (#25540) 2017-07-05 16:09:31 +02:00
Clinton Gormley 0170e0e8d3 Remove usage of multi-types from the docs and added a page explaining type removal (#25543)
Closes #25401
2017-07-05 12:30:19 +02:00
Russ Cam 83522ab4e5 Important note for MSI upgrade from command line (#25544)
Add an Important admonition for upgrading via the command line
using the Windows MSI Installer. This calls out the need to pass
the same command line options for an upgrade as were used for
the initial installation.
2017-07-05 11:38:21 +02:00
Wesley Merkel 7994703903 [Docs] Fix some errors in the index aliases docs (#25449) 2017-07-05 11:16:17 +02:00
Tanguy Leroux c52c3cea68 [Doc] Remove mention of generated fields in get/mget docs (#25492)
This option has been removed in #20102.
2017-07-04 18:06:00 +02:00
Jun Ohtani 6894ef6057 [Analysis] Support normalizer in request param (#24767)
* [Analysis] Support normalizer in request param

Support normalizer param
Support custom normalizer with char_filter/filter param

Closes #23347
2017-07-04 19:16:56 +09:00
Colin Goodheart-Smithe 43efcffcc2 Adds check for negative search request size (#25397)
* Adds check for negative search request size

This change adds a check to `SearchSourceBuilder` to throw and exception if the size set on it is set to a negative value.

Closes #22530

* fix error in reindex

* update re-index tests

* Addresses review comment

* Fixed tests

* Added random negative size test

* Fixes test
2017-07-04 10:51:38 +01:00
Clinton Gormley 8296618640 Include shared/attributes.asciidoc from docs master 2017-07-03 18:17:34 +02:00
Jason Tedor 1b602477ae Add doc note regarding explicit publish host
This commit adds a note to the docs regarding explicilty setting a
publish host if the network.host setting results in multiple bind
addresses.

Relates #25496
2017-06-30 12:59:02 -04:00
Russ Cam 099ccd36ee Update defaults in documentation (#25483)
Following internal discussion

Relates: https://github.com/elastic/windows-installers/pull/49
2017-06-30 16:24:53 +10:00
Christoph Büscher 927111c91d Remove QueryParseContext from parsing QueryBuilders (#25448)
Currently QueryParseContext is only a thin wrapper around an XContentParser that
adds little functionality of its own. I provides helpers for long deprecated
field names which can be removed and two helper methods that can be made static
and moved to other classes. This is a first step in helping to remove
QueryParseContext entirely.
2017-06-29 17:10:20 +02:00
olcbean 3518e313b8 Unify the result interfaces from get and search in Java client (#25361)
As GetField and SearchHitField have the same members, they have been unified into
DocumentField.

Closes #16440
2017-06-29 11:35:28 +02:00
DeDe Morton 6442d1f75e [Docs] Add link to grok debugger docs (#25412) 2017-06-28 16:14:16 -07:00
Andreas Gebhardt a156ccd80e Expand `/_cat/nodes` to return information about hard drive (#21775)
Expand `/_cat/nodes` with already present information about available disk space `diskAvail` (alias: `d`, `disk`) by:

    * `diskTotal` (alias `dt`): total disk space
    * `diskUsed` (alias `du`): used disk space (`diskTotal - diskAvail`)
    * `diskUsedPercent` (alias `dup`): used disk space percentage

Note: The available disk space is the number of bytes available to the node's Java virtual machine. The size might be smaller than the real one. That means the used disk space (percentage) is larger.

Closes #21679
2017-06-28 18:20:20 +02:00
Martijn van Groningen 9ce9c21b83
docs: added percolator script query limitation 2017-06-28 17:10:30 +02:00
Robin Clarke 1900d9c447 Docs: Fix typo for request cache (#25444) 2017-06-28 14:31:03 +02:00
Jason Tedor be906628d5 Remove implicit 32-bit support
We previously tried to maintain (while not formally supporting) 32-bit
support, although we never tested this anywhere in CI. Since we do not
formally support this, and 32-bit usage is very low, we have elected to
no longer maintain 32-bit support. This commit removes any implication
of 32-bit support.

Relates #25435
2017-06-28 08:24:33 -04:00
Marcus Wittig 960d63a3b3
[DOCS] reworded to prevent code span rendering glitch (#25442)
Changed `rescore`s to `rescore` requests as an backtick followed by the s character appears to be interpreted as an apostrophe  which then leads to an unbalanced backtick for the next code span in the remainder of the paragraph

Closes #25443
2017-06-28 11:48:31 +01:00
Russ Cam 2804fc4c29 Update MSI installer images (#25414)
Slight updating to styling
2017-06-28 09:04:23 +10:00
Lisa Cawley 03f952a838 [DOCS] Update docs to use shared attribute file (#25403)
* [DOCS] Update docs to use shared attribute file

* [DOCS] Add shared attributes to Versions.asciidoc
2017-06-27 08:33:28 -07:00
Jason Tedor dfd241e0a6 Remove default path settings
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.

Relates #25408
2017-06-26 21:43:20 -04:00
Deb Adair 2765ea41ca [DOCS] Fixed broken cross doc links to security settings. 2017-06-26 17:48:56 -07:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Alexander Kazakov 53b74348ff Fix documentation for script processor (#25299) 2017-06-26 12:14:23 -07:00
dkimdon fdb3a97152
Update percolate-query.asciidoc (#25364) 2017-06-23 10:39:57 +02:00
Boaz Leskes d963882053 Enable a long translog retention policy by default (#25294)
#25147  added the translog deletion policy but didn't enable it by default. This PR enables a default retention of 512MB (same maximum size of the current translog) and an age of 12 hours (i.e., after 12 hours all translog files will be deleted). This increases to chance to have an ops based recovery, even if the primary flushed or the replica was offline for a few hours.

In order to see which parts of the translog are committed into lucene the translog stats are extended to include information about uncommitted operations.

Views now include all translog ops and guarantee, as before, that those will not go away. Snapshotting a view allows to filter out generations that are not relevant based on a specific sequence number.

Relates to #10708
2017-06-22 17:08:14 +02:00
Martijn van Groningen a977569085
percolator: Deprecate `document_type` parameter.
The `document_type` parameter is no longer required to be specified,
because by default from 6.0 only a single type is allowed. (`index.mapping.single_type` defaults to `true`)
2017-06-22 09:55:06 +02:00
Deb Adair 0b0390aa64 [DOCS] Fixed typo. 2017-06-21 14:27:30 -07:00
Nathan Taylor 645bb9d0fb Docs: Removed duplicated line in mapping docs 2017-06-21 10:47:19 +02:00
Jun Ohtani 62d1969595 Parse synonyms with the same analysis chain (#8049)
* [Analysis] Parse synonyms with the same analysis chain

Synonym Token Filter / Synonym Graph Filter tokenize synonyms with whatever tokenizer and token filters appear before it in the chain.

Close #7199
2017-06-20 21:50:33 +09:00
Lisa Cawley c88b759b66 [DOCS] Split index-shared.asciidoc into multiple smaller files (#25302) 2017-06-19 15:14:53 -07:00
Andy Bristol 4c5bd57619 Rename simple pattern tokenizers (#25300)
Changed names to be snake case for consistency

Related to #25159, original issue #23363
2017-06-19 13:48:43 -07:00
javanna 2fb4a0d40c [DOCS] replace '+' with `+` 2017-06-19 16:53:55 +02:00
Luca Cavanna d9ec2a23c5 Remove (deprecated) support for '+' in index expressions (#25274)
Relates to #24515
2017-06-19 15:19:17 +02:00
javanna df5640efd7 [DOCS] delete index no longer supports specifying aliases 2017-06-19 10:30:05 +02:00
Russ Cam 9eca380a76 Add MSI installation to documentation (#25213)
* Add MSI installation to documentation

Move installation documentation for Windows with the .zip archive into the zip and tar installation documentation, and clearly indicate any differences for installing on macOS/Linux and Windows.

* Separate out installation with .zip on Windows
2017-06-19 14:52:47 +10:00
Jason Tedor fde6f72cb5 Fix queries in cross-cluster search docs
This commit fixes two queries in the cross-cluster search docs; they
were missing the query object wrapping the actual query.

Relates #25282
2017-06-16 21:15:35 -04:00
Luca Cavanna b5cea6980b Delete index API to work only against concrete indices (#25268)
With #23997 we have introduced a new internal index option that allows to resolve index expressions only against concrete indices while ignoring aliases. Such index option was applied to IndicesAliasesRequest, so that the index part of alias actions would only be resolved against concrete indices.

Same is done in this commit with delete index request. Deleting aliases has always been confusing as some users expect it to only remove the alias from the index (which has its own specific API). Even worse, in case of filtered aliases, deleting an alias may leave users with the expectation that only the documents that match the filter are deleted, which was never the case. To address all this confusion, delete index api works now only against concrete indices. WIldcard expressions will be only resolved against concrete index, as if aliases didn't exist. If one tries to delete against an alias, an IndexNotFoundException will be thrown regardless of whether the alias exists or not, as a concrete index with such a name doesn't exist.

Closes #2318
2017-06-16 17:46:01 +02:00
Jim Ferenczi afada69ea9 [Docs] more fix for the parent-join docs 2017-06-16 12:49:16 +02:00
Jim Ferenczi 664193185e [Docs] Fix cross reference for parent-join field 2017-06-16 11:53:16 +02:00
Adrien Grand 8c869e2a0b More advices around search speed and disk usage. (#25252)
It adds notes about:
 - how preference can help optimize cache usage
 - the fact that too many replicas can hurt search performance due to lower
   utilization of the filesystem cache
 - how index sorting can improve _source compression
 - how always putting fields in the same order in documents can improve _source
   compression
2017-06-16 11:23:40 +02:00
Jim Ferenczi ccb3c9aae7 Add documentation for the new parent-join field (#25227)
* Add documentation for the new parent-join field

This commit adds the docs for the new parent-join field.
It explains how to define, index and query this new field.

Relates #20257
2017-06-16 11:13:23 +02:00
debadair c161d90524 [DOCS] Defined es-test-dir and plugins-examples-dir in index.asciidoc. (#25232)
Use these attributes when specifying the location of included tests.
2017-06-15 08:54:10 -07:00
Jim Ferenczi 0036f28a6a Upgrade icu4j for the ICU analysis plugin to 59.1 (#25243)
* Upgrade icu4j for the ICU analysis plugin to 59.1

Lucene upgraded to 59.1 so we should use the same.

Closes #21425

* Add breaking change for the icu upgrade
2017-06-15 13:26:48 +02:00
Alexander Kazakov 64abc47ab0 [Docs] Fix documentation for percentiles bucket aggregation (#25229) 2017-06-15 10:16:32 +02:00
Adrien Grand 0c117145f6 Upgrade to lucene-7.0.0-snapshot-92b1783. (#25222)
This snapshot has faster range queries on range fields (LUCENE-7828), more
accurate norms (LUCENE-7730) and the ability to use fake term frequencies
(LUCENE-7854).
2017-06-15 09:52:07 +02:00
Boaz Leskes a0fcfc732d Migration docs for #25080 (#25218) 2017-06-14 14:06:53 +02:00
Ryan Ernst 1bd5cecc34 Docs: Add note about which secure settings are valid (#25212)
This commit adds a note to the docs to clarify that only some settings
can be used with the keystore.
2017-06-13 21:04:16 -07:00
Lisa Cawley d181761aeb [DOCS] Add ML node to node.asciidoc (#24495)
* [DOCS] Add ML node to node.asciidoc

* [DOCS] Clarify ML node in node.asciidoc

* [DOCS] Add X-Pack icon for admonition blocks

* [DOCS] Formatting X-Pack blocks in node.asciidoc

* [DOCS] Add xpack icon images to node.asciidoc

* [DOCS] Add final xpack role attributes

* [DOCS] Remove unnecssary xpackicon image

* [DOCS] Add link to X-Pack node settings

* [DOCS] Fix path to X-Pack repository

* [DOCS] Add links to X-Pack node settings

* [DOCS] Fixed text for links to X-Pack node settings

* [DOCS] Change standalone node to dedicated node
2017-06-13 14:03:42 -07:00
Andy Bristol 48696ab544 expose simple pattern tokenizers (#25159)
Expose the experimental simplepattern and 
simplepatternsplit tokenizers in the common 
analysis plugin. They provide tokenization based 
on regular expressions, using Lucene's 
deterministic regex implementation that is usually 
faster than Java's and has protections against 
creating too-deep stacks during matching.

Both have a not-very-useful default pattern of the 
empty string because all tokenizer factories must 
be able to be instantiated at index creation time. 
They should always be configured by the user 
in practice.
2017-06-13 12:46:59 -07:00
Alexander Kazakov a7dafdaa05 Add target_field parameter to gsub, join, lowercase, sort, split, trim, uppercase (#24133)
Closes #23682 #23228
2017-06-13 09:40:44 -07:00
Russ Cam f6821c41d8 Add half_float and scaled float (#22988)
to numeric datatypes
(cherry picked from commit 67ea06145a80d5ec52ba55d1f2e1e8287e1882b1)
2017-06-13 09:54:44 +10:00
Lisa Cawley 2f7de46b72 [DOC] Add X-Pack links to Elasticsearch Reference (#25164)
* [DOC] Add X-Pack links to Elasticsearch Reference

* [DOCS] Address alignment of attributes in Versions.asciidoc
2017-06-12 13:43:06 -07:00
Spencer 88591fecac [docs] include two cluster doc pages missing from index (#25180)
* [docs] include two cluster doc pages missing from index

* [rest-api-spec] update link to remote-info docs
2017-06-12 12:33:56 -07:00
Jason Tedor dcf57f296e Fix get mappings HEAD requests
Get mappings HEAD requests incorrectly return a content-length header of
0. This commit addresses this by removing the special handling for get
mappings HEAD requests, and just relying on the general mechanism that
exists for handling HEAD requests in the REST layer.

Relates #23192
2017-06-11 14:58:56 -04:00
Sergey Novikov 7c8657df0e Return the index name on a create index response
This commit modifies the create index response so that it includes the
index name.

Relates #25139
2017-06-09 13:47:47 -04:00
Ryan Ernst a03b6c2fa5 Scripting: Change keys for inline/stored scripts to source/id (#25127)
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
2017-06-09 08:29:25 -07:00
Jim Ferenczi 5e8b569255 fix highlighting docs 2017-06-09 14:42:08 +02:00
Jim Ferenczi 8250aa4267 Remove the postings highlighter and make unified the default highlighter choice (#25028)
This change removes the `postings` highlighter. This highlighter has been removed from Lucene master (7.x) because it behaves
exactly like the `unified` highlighter when index_options is set to `offsets`:
https://issues.apache.org/jira/browse/LUCENE-7815

It also makes the `unified` highlighter the default choice for highlighting a field (if `type` is not provided).
The strategy used internally by this highlighter remain the same as before, it checks `term_vectors` first, then `postings` and ultimately it re-analyzes the text.
Ultimately it rewrites the docs so that the options that the `unified` highlighter cannot handle are clearly marked as such.
There are few features that the `unified` highlighter is not able to handle which is why the other highlighters (`plain` and `fvh`) are still available.
I'll open separate issues for these features and we'll deprecate the `fvh` and `plain` highlighters when full support for these features have been added to the `unified`.
2017-06-09 14:09:57 +02:00
Pandiyan Murugan 34c3d1d5bf Fix typo in shards.asciidoc (#25143) 2017-06-09 12:45:43 +02:00
Andrey Groshev e4fd8485ce Made the same length of opening and closing lines (#23583) 2017-06-09 00:50:43 -07:00
Jim Ferenczi ad905924ae update docs that claim that classic is the default similarity 2017-06-09 09:22:48 +02:00
Deb Adair dbe2de0891 [DOCS] Fixed callout reference error. 2017-06-08 16:47:13 -07:00
Tal Levy a771912a22 Add Ingest-Processor specific Rest Endpoints & Add Grok endpoint (#25059)
This PR enables Ingest plugins to leverage processor-scoped REST
endpoints. First of which being the Grok endpoint that retrieves
Grok Patterns for users to retrieve all the built-in patterns.
Example usage: Kibana Grok Autocomplete!
2017-06-08 15:24:35 -07:00
Guillaume Le Floch 3f6d80aa66 Allow removing multiple fields in ingest processor (#24750)
* Allow removing multiple fields in ingest processor

* Iteration 2

* Few fixes
2017-06-08 13:17:44 -07:00
Jim Ferenczi 36a5cf8f35 Automatically early terminate search query based on index sorting (#24864)
This commit refactors the query phase in order to be able
to automatically detect queries that can be early terminated.
If the index sort matches the query sort, the top docs collection is early terminated
on each segment and the computing of the total number of hits that match the query is delegated to a simple TotalHitCountCollector.
This change also adds a new parameter to the search request called `track_total_hits`.
It indicates if the total number of hits that match the query should be tracked.
If false, queries sorted by the index sort will not try to compute this information and 
and will limit the collection to the first N documents per segment.
Aggregations are not impacted and will continue to see every document
even when the index sort matches the query sort and `track_total_hits` is false.

Relates #6720
2017-06-08 12:10:46 +02:00
Christian Hofstaedtler c3ec6a1714 Honor masking of systemd-sysctl.service
During package install on systemd-based systems, some sysctl settings
should be set (e.g. vm.max_map_count).

In some environments, changing sysctl settings plainly does not work;
previously a global environment variable named
ES_SKIP_SET_KERNEL_PARAMETERS was introduced to skip calling sysctl, but
this causes trouble for:
 - configuration management systems, which usually cannot apply an env
   var when running a package manager
 - package upgrades, which will not have the env var set any more, and
   thus leaving the package management system in a bad state (possibly
   half-way upgraded, can be very hard to recover)

This removes the env var again and instead of calling systemd-sysctl
manually, tells systemd to restart the wrapper unit - which itself can
be masked by system administrators or management tools if it is known
that sysctl does not work in a given environment.

The restart is not silent on systems in their default configuration, but
is ignored if the unit is masked.

Relates #24234
2017-06-06 10:44:41 -04:00
Yibin Lin fbf2e3d574 Tiny correction in inner-hits.asciidoc (#25066) 2017-06-06 13:26:37 +02:00
Clinton Gormley 8b9c201224 Added release notes for 6.0.0-alpha2 2017-06-06 11:52:18 +02:00
olcbean 0d5f3958e7 Expand index expressions against indices only when managing aliases (#23997)
The index parameter in the update-aliases, put-alias, and delete-alias APIs no longer accepts alias names. Instead, it accepts only index names (or wildcards which will expand to matching indices).

Closes #23960
2017-06-06 11:01:38 +02:00
Tal Levy e51246023a add `exclude_keys` option to KeyValueProcessor (#24876)
and modify data-structure of `include_keys` and `exclude_keys` to be
backed by a HashSet
2017-06-05 14:12:48 -07:00
Lee Hinman a32d1b91fa Remove comma-separated feature parsing for GetIndicesAction
This removes the parsing of things like `GET /idx/_aliases,_mappings`, instead,
a user must choose between retriving all index metadata with `GET /idx`, or only
a specific form such as `GET /idx/_settings`.

Relates to (and is a prerequisite of) #24437
2017-06-02 14:43:38 -06:00
Colin Goodheart-Smithe 5e7a79636d
[DOCS] Clarify behaviour of scripted-metric arg with empty parent buckets 2017-06-02 11:00:27 +01:00
Luca Cavanna 018c6c38fe [DOCS] Clarify connections and gateway nodes selection in cross cluster search docs (#24859)
Closes #24836
2017-06-02 11:13:47 +02:00
olcbean 6dea5f14c3 Java api: Remove unneeded getTookInMillis method (#23923)
Some response classes in the java api expose both `getTook()` which returns a `TimeValue` and `getTookInMillis` which returns a `long` value. `getTook()` is enough as one can do `getTook().millis()` to obtain the same result as `getTookInMillis()`, which can be removed.
2017-06-02 11:11:05 +02:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions (#24169)
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Tanguy Leroux 528bd25fa7 Add superset size to Significant Term REST response (#24865)
This commit adds a new bg_count field to the REST response of
SignificantTerms aggregations. Similarly to the bg_count that already
exists in significant terms buckets, this new bg_count field is set at
the aggregation level and is populated with the superset size value.
2017-06-02 09:45:15 +02:00
Adrien Grand bbdf50f6bd Docs: More search speed advices. (#24802) 2017-06-01 17:23:22 +02:00
Adrien Grand ebf806d38f Reorganize docs of global ordinals. (#24982)
Currently global ordinals are documented under `fielddata`. It moves them to
their own file since they also work with doc values and fielddata is on the way
out.

Closes #23101
2017-06-01 16:47:44 +02:00
Clinton Gormley 1b0c93b07c Documented the level parameter to nodes stats
Closes #24999
2017-06-01 12:11:21 +02:00
Sergey Novikov a7b21534b1 Docs: Fix typo in docker docs (#24988)
`boostrap.memory_lock` -> `bootstrap.memory_lock`
2017-05-31 13:42:47 -04:00
David Cho-Lerat 491dc1186a Add missing word to terms-query.asciidoc (#24960) 2017-05-30 09:42:07 -04:00
David Cho-Lerat c939bcb7f5 Correct some spelling in match-phrase-prefix docs (#24956) 2017-05-30 09:02:01 -04:00
Tanguy Leroux 28d97df67c Add document count to Matrix Stats aggregation response (#24776)
This commit adds a `doc_count` field to the response body of Matrix
Stats aggregation. It exposes the number of documents involved in
 the computation of statistics, a value that can already be retrieved using
  the method MatrixStats.getDocCount() in the Java API.
2017-05-30 09:39:41 +02:00
propulkit 25516868fe TCorrecting api name (#24924)
As per REST request signature for reroute, API has no underscore.
2017-05-29 13:58:31 +02:00
Clinton Gormley 0656d0236b Update context-suggest.asciidoc
Removed incorrect parameter
2017-05-26 17:41:40 +02:00
Matt Weber 601a61a91c Support Multiple Collapse Inner Hits
Support multiple named inner hits on a field collapsing
request.
2017-05-26 13:23:57 +02:00
Tal Levy dfe2ecaa28 add docs example for Ingest scripts manipulating document metadata (#24875)
It may not be clear to users that the Ingest ScriptProcessor context object `ctx` can 
manipulate document metadata like `_index` and `_type`.
2017-05-25 07:45:19 -07:00
Brian Lesperance 959990728b Docs: Fix grammar in aliases doc (#24852) 2017-05-24 10:18:25 -04:00
markharwood b7197f5e21 SignificantText aggregation - like significant_terms, but for text (#24432)
* SignificantText aggregation - like significant_terms but doesn’t require fielddata=true, recommended used with `sampler` agg to limit expense of tokenizing docs and takes optional `filter_duplicate_text`:true setting to avoid stats skew from repeated sections of text in search results.

Closes #23674
2017-05-24 13:46:43 +01:00
António Ribeiro 85a1b2b406 Fix link to perl docs (#24842)
* Fixes Elasticsearch issue #24606.

* Fixes Elasticsearch issue #24606.

* Fixes Elasticsearch issue #24606.

* Fixes Elasticsearch issue #24606.

* Issue #24606 - Changed the link text to Search::Elasticsearch::Client::5_0::Bulk and
Search::Elasticsearch::Client::5_0::Scroll.
2017-05-24 11:43:54 +02:00
Nik Everett 13a86fec99 Add magic $_path stash key to docs tests (#24724)
Adds a "magic" key to the yaml testing stash mostly for use with
documentation tests. When unstashing an object, `$_path` is the
path into the current position in the object you are unstashing.
This means that in docs tests you can use
`// TESTRESPONSEs/somevalue/$body.${_path}/` to mean "replace
`somevalue` with whatever is the response in the same position."

Compare how you must carefully mock out all the numbers in the profile
response without this change:
```
// TESTRESPONSE[s/"id": "\[2aE02wS1R8q_QFnYu6vDVQ\]\[twitter\]\[1\]"/"id": $body.profile.shards.0.id/]
// TESTRESPONSE[s/"rewrite_time": 51443/"rewrite_time": $body.profile.shards.0.searches.0.rewrite_time/]
// TESTRESPONSE[s/"score": 51306/"score": $body.profile.shards.0.searches.0.query.0.breakdown.score/]
// TESTRESPONSE[s/"time_in_nanos": "1873811"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.time_in_nanos/]
// TESTRESPONSE[s/"build_scorer": 2935582/"build_scorer": $body.profile.shards.0.searches.0.query.0.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 919297/"create_weight": $body.profile.shards.0.searches.0.query.0.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 53876/"next_doc": $body.profile.shards.0.searches.0.query.0.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "391943"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.children.0.time_in_nanos/]
// TESTRESPONSE[s/"score": 28776/"score": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.score/]
// TESTRESPONSE[s/"build_scorer": 784451/"build_scorer": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 1669564/"create_weight": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 10111/"next_doc": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "210682"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.children.1.time_in_nanos/]
// TESTRESPONSE[s/"score": 4552/"score": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.score/]
// TESTRESPONSE[s/"build_scorer": 42602/"build_scorer": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 89323/"create_weight": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 2852/"next_doc": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "304311"/"time_in_nanos": $body.profile.shards.0.searches.0.collector.0.time_in_nanos/]
// TESTRESPONSE[s/"time_in_nanos": "32273"/"time_in_nanos": $body.profile.shards.0.searches.0.collector.0.children.0.time_in_nanos/]
```

To how you can cavalierly mock all the numbers at once with this change:
```
// TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
```
2017-05-23 15:33:48 -04:00
Clinton Gormley 086abe6216 Marked `fixed_auto_queue_size` as experimental
Relates to https://github.com/elastic/elasticsearch/pull/23884
2017-05-22 10:03:31 +02:00
olcbean e08e92d934 Deleting a document from a non-existing index creates the should not auto create it, unless using EXTERNAL* versioning (#24518)
Currently a `delete document` request against a non-existing index actually **creates** this index.

With this change the `delete document` no longer creates the previously non-existing index and throws an `index_not_found` exception instead.

However as discussed in https://github.com/elastic/elasticsearch/pull/15451#issuecomment-165772026, if an external version is explicitly used, the current behavior is preserved and the index is still created and the document is marked for deletion.

Fixes #15425
2017-05-22 10:00:22 +03:00
archana a5358f34b3 Update mappings.asciidoc
typo
2017-05-20 13:39:05 -05:00
Oleksandr Chychkan 065d91bccc Typo in setup/configuration.asciidoc (#24797) 2017-05-19 10:49:56 -04:00
Jack Conradson 0aa380b770 Fix search template documentation reference to scripting security. 2017-05-18 14:27:58 -07:00
Jack Conradson 1196dfb6bb Remove Deprecated Script Settings (#24756)
Removes all fine-grained script settings replaced by scripts.types_allowed and scripts.contexts_allowed.
2017-05-18 13:32:46 -07:00
Ryan Ernst b214b80e6c GCS Repository: Remove specifying credential file on disk (#24727)
This commit removes the ability to specify the google credential json
file on disk, which is deprecated in 5.5.0.
2017-05-18 10:22:29 -07:00
Ryan Ernst 26e2e933f5 Scripting: Remove native scripts (#24726)
Native scripts have been replaced in documentation by implementing
a ScriptEngine and they were deprecated in 5.5.0. This commit
removes the native script infrastructure for 6.0.

closes #19966
2017-05-17 14:49:24 -07:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Zachary Tong a2845c86fe
CONSOLEify some more aggregation docs
Related #18160
2017-05-16 17:25:24 -04:00
Jack Conradson b7f0df626a [DOCS] Added Painless Language Spec content 2017-05-16 12:46:56 -07:00
Lee Hinman d09e64323f Add ability to automatically adjust search threadpool queue_size
This PR adds a new thread pool type: `fixed_auto_queue_size`. This thread pool
behaves like a regular `fixed` threadpool, except that every
`auto_queue_frame_size` operations (default: 10,000) in the thread pool,
[Little's Law](https://en.wikipedia.org/wiki/Little's_law) is calculated and
used to adjust the pool's `queue_size` either up or down by 50. A minimum and
maximum is taken into account also. When the min and max are the same value, a
regular fixed executor is used instead.

The `SEARCH` threadpool is changed to use this new type of thread pool. However,
the min and max are both set to 1000, meaning auto adjustment is opt-in rather
than opt-out.

Resolves #3890
2017-05-16 11:13:16 -06:00
Ryan Ernst 97d2657e18 Remove script access to term statistics (#19462)
In scripts (at least some of the languages), the terms dictionary and
postings can be access with the special _index variable. This is for
very advanced use cases which want to do their own scoring. The problem
is segment level statistics must be recomputed for every document.
Additionally, this is not friendly to the terms index caching as the
order of looking up terms should be controlled by lucene.

This change removes _index from scripts. Anyone using it can and should
instead write a Similarity plugin, which is explicitly designed to allow
doing the calculations needed for a relevance score.

closes #19359
2017-05-16 09:10:09 -07:00
Simon Willnauer 1cae850cf5 Add a cluster block that allows to delete indices that are read-only (#24678)
Today when an index is `read-only` the index is also blocked from
being deleted which sometimes is undesired since in-order to make
changes to a cluster indices must be deleted to free up space. This is
a likely scenario in a hosted environment when disk-space is limited to switch
indices read-only but allow deletions to free up space.
2017-05-16 17:34:37 +02:00
Daniel Mitterdorfer 77762fcbb0 Use correct script name in docs for Windows
With this commit we correct the name of the ES batch script to
`elasticsearch.bat` in the docs and use backslashes in path names.
2017-05-16 15:57:05 +02:00
Ryan Ernst 548a5c1386 Docs: Add migration note about .yaml and .json removal (#24689)
relates #19391
relates #24633
2017-05-15 13:42:28 -07:00
Jack Conradson 43292979fd Add New Security Script Settings (#24637)
Settings are simplified to allowed_types and allowed_contexts.  If a setting is not specified the default is to enable all for that setting.
2017-05-15 13:37:46 -07:00
Vlad Holubiev 557390d7d1 Fix typo in example (grades_count -> types_count) (#24635)
Looks like `doc.grade` was used for examples before. But not anymore - https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-aggregations-metrics-valuecount-aggregation.html
2017-05-15 14:08:46 -04:00
Jason Tedor 4e21a33689 Clarify disabling swap in docs
Our strong recommendation is disabling swap over any other alternative
to avoid the JVM from landing on disk. This commit clarifies the docs in
this regard.
2017-05-12 16:09:52 -04:00