Commit Graph

4596 Commits

Author SHA1 Message Date
Alexander Reelsen 483086220f Docs: Add search response took time explanation (#26202) 2017-08-15 08:43:26 +02:00
Jason Tedor e9687622bd Rename CONF_DIR to ES_PATH_CONF
The environment variable CONF_DIR was previously inconsistently used in
our packaging to customize the location of Elasticsearch configuration
files. The importance of this environment variable has increased
starting in 6.0.0 as it's now used consistently to ensure Elasticsearch
and all secondary scripts (e.g., elasticsearch-keystore) all use the
same configuration. The name CONF_DIR is there for legacy reasons yet
it's too generic. This commit renames CONF_DIR to ES_PATH_CONF.

Relates #26197
2017-08-15 06:19:06 +09:00
hanbj 62021147ea Update reference from DateHistogram to Histogram (#26169)
DateHistogram was renamed, but java api docs were not updated. This commit updates a reference to DateHistogram to the new general Histogram class.
2017-08-14 11:52:23 -07:00
hanbj 9a908995a8 Fix incorrect class name in deleteByQuery docs (#26151)
Class was renamed but docs weren't updated.
2017-08-11 14:55:51 -04:00
Andy Bristol 7e3cd6a019 reindex: automatically choose the number of slices (#26030)
In reindex APIs, when using the `slices` parameter to choose the number of slices, adds the option to specify `slices` as "auto" which will choose a reasonable number of slices. It uses the number of shards in the source index, up to a ceiling. If there is more than one source index, it uses the smallest number of shards among them.

This gives users an easy way to use slicing in these APIs without having to make decisions about how to configure it, as it provides a good-enough configuration for them out of the box. This may become the default behavior for these APIs in the future.
2017-08-11 08:25:25 -07:00
Martijn van Groningen 636e85e5b7
percolator: Hint what clauses are important in a conjunction query based on fields
The percolator field mapper doesn't need to extract all terms and ranges from a bool query with must or filter clauses.
In order to help to default extraction behavior, boost fields can be configured, so that fields that are known for not being
selective enough can be ignored in favor for other fields or clauses with specific fields can forcefully take precedence over other clauses.
This can help selecting clauses for fields that don't match with a lot of percolator queries over other clauses and thus improving performance of the percolate query.

For example a status like field is something that should configured as an ignore field.
Queries on this field tend to match with more documents and so if clauses for this fields
get selected as best clause then that isn't very helpful for the candidate query that the
percolate query generates to filter out percolator queries that are likely not going to match.
2017-08-11 15:32:01 +02:00
Daniel Mitterdorfer 637cc872f4 Remove unused Netty-related settings (#26161)
With this commit we remove the following three previously unused 
(and undocumented) Netty 4 related settings:

* transport.netty.max_cumulation_buffer_capacity,
* transport.netty.max_composite_buffer_components and
* http.netty.max_cumulation_buffer_capacity 

from Elasticsearch.
2017-08-11 12:03:00 +02:00
David Pilato 9c372e533e Fix wrong header level
Relates to #26113.
2017-08-10 17:19:01 +02:00
Martijn van Groningen 076167fbe5
inner hits: Unfiltered nested source should keep its full path
like filtered nested source.

Closes #23090
2017-08-10 15:58:29 +02:00
David Pilato 3fc27b0e90 Document how to import Lucene Snapshot libs when elasticsearch clients (#26113)
When using the High Level Rest Client 6.0.0-beta1, we are missing some transitive dependencies for Lucene as Lucene 7 has not been released yet. See the following `pom.xml`:

```xml
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>6.0.0-beta1</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>6.0.0-beta1</version>
        </dependency>
```

It gives:

```
[ERROR] Failed to execute goal on project fscrawler: Could not resolve dependencies for project fr.pilato.elasticsearch.crawler:fscrawler:jar:2.4-SNAPSHOT: The following artifacts could not be resolved: org.apache.lucene:lucene-analyzers-common:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-backward-codecs:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-grouping:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-highlighter:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-join:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-memory:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-misc:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-queries:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-queryparser:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-sandbox:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-spatial:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-spatial-extras:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-spatial3d:jar:7.0.0-snapshot-00142c9, org.apache.lucene:lucene-suggest:jar:7.0.0-snapshot-00142c9: Failure to find org.apache.lucene:lucene-analyzers-common:jar:7.0.0-snapshot-00142c9 in https://artifacts.elastic.co/maven/ was cached in the local repository, resolution will not be reattempted until the update interval of elastic-download-service has elapsed or updates are forced -
```

We need to add some temporary documentation on how to add the missing repository to a gradle or maven project:

```xml
        <repository>
            <id>elastic-lucene-snapshots</id>
            <name>Elastic Lucene Snapshots</name>
            <url>http://s3.amazonaws.com/download.elasticsearch.org/lucenesnapshots/00142c9</url>
            <releases><enabled>true</enabled></releases>
            <snapshots><enabled>false</enabled></snapshots>
        </repository>
```

This also applies to the transport client.

Closes #26106.
2017-08-10 14:54:07 +02:00
Nik Everett 7d5f00d1d2 Docs: Note feature missing from reindex
Reindex-from-remote doesn't support slices and I hadn't documented
that.

Closes #26114
2017-08-09 09:44:52 -04:00
Jim Ferenczi a7e1610134 Add support for auto_generate_synonyms_phrase_query in match_query, multi_match_query, query_string and simple_query_string (#26097)
* Add support for auto_generate_synonyms_phrase_query in match_query, multi_match_query, query_string and simple_query_string

This change adds a new parameter called auto_generate_synonyms_phrase_query (defaults to true).
This option can be used in conjunction with synonym_graph token filter to generate phrase queries
when multi terms synonyms are encountered.
For example, a synonym like "ny, new york" would produce the following boolean query when "ny city" is parsed:
((ny OR "new york") AND city)

Note how the multi terms synonym "new york" produces a phrase query.
2017-08-09 12:15:09 +02:00
Ian Fisk 8cb1391f40 Docs: Use correct field name in Field Value factor docs. (#26104) 2017-08-08 16:34:20 -04:00
David Pilato 80b142d218 Azure repository: Move to named configurations as we do for S3 repository
We should have the same behavior for Azure repositories as we have for S3 (see #22762).

Instead of:

```yml
cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

Support something like:

```
azure.client:
            default:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

Then instead of:

```
PUT _snapshot/my_backup3
{
    "type": "azure",
    "settings": {
        "account": "my_account2"
    }
}
```

Use:

```
PUT _snapshot/my_backup3
{
    "type": "azure",
    "settings": {
        "config": "my_account2"
    }
}
```

If someone uses:

```
PUT _snapshot/my_backup3
{
    "type": "azure"
}
```

It will use the `default` azure repository settings.

And mark as deprecated old settings.

Closes #22763.
2017-08-08 15:14:47 +02:00
markwalkom 746487c3f3 Update templates.asciidoc (#26036)
Dropped in a few links to index settings and mappings to make things easier to jump to.
2017-08-08 11:29:11 +02:00
Adrien Grand f0cba4fce5 Add a scripted similarity. (#25831)
The goal of this similarity is to help users who would like to keep the
functionality of the `tf-idf` similarity that we want to remove, or to allow
for specific usec-cases (disabling idf, disabling tf, disabling length norm,
etc.) to not have to build a custom plugin and familiarize with the low-level
Lucene API.
2017-08-08 08:55:12 +02:00
Tal Levy 872526cad3 add URL-Decode Processor to Ingest (#26045)
closes #25837

Adds a URL Decoder Processor to Ingest

this will decode urls like:

https%3a%2f%2felastic.co%2 to https://elastic.co/
2017-08-07 10:26:11 -07:00
Luca Cavanna 14ba36977e [TEST] prevent yaml tests from using raw requests (#26044)
Raw requests are supported only by the java yaml test runner and were introduced to test docs snippets. Some yaml tests ended up using them (see #23497) which causes failures for other language clients. This commit migrates those yaml tests to Java tests that send requests through the Java low-level REST client, and also moves the ability to send raw requests to a special client that's only available when testing docs snippets.

Closes #25694
2017-08-07 11:02:16 +02:00
Ryan Ernst 0266479cab Docs: Update s3 repository docs with client settings (#26033)
This commit updates the s3 repository docs to clearly mark settings as
part of the s3 client settings, as well as those that are secure and
must be stored in the elasticsearch keystore.

relates #25619
2017-08-04 11:22:49 -07:00
Martijn van Groningen b88cfe2008
docs: Use stackexchange based example to make documentation easier to understand 2017-08-04 16:04:26 +02:00
Zachary Tong 829f7cb658
CONSOLEify ip-range bucket agg docs
Related #18160
2017-08-03 17:19:54 -04:00
Zachary Tong e7eda5e1be
CONSOLEify scripted-metric agg docs
Related #18160
2017-08-03 17:19:54 -04:00
Tanguy Leroux 4f5ce5598e [Docs] Add Aggregations list to the high-level client documentation (#26025) 2017-08-03 11:50:58 +02:00
Aron Szanto 316cb42b21 Update shards_allocation.asciidoc (#26019)
Slight language and consistency updates in shard balancing heuristics
2017-08-03 11:27:02 +02:00
Zachary Tong d8414ffa29
CONSOLEify percentile and percentile-ranks docs
Related #18160
2017-08-02 17:47:27 -04:00
Zachary Tong 268923ebdc
CONSOLEify extended_stats docs
Related #18160
2017-08-02 16:13:30 -04:00
Tanguy Leroux 69f8641568 [Docs] Add documentation for search queries in high-level rest client (#25984) 2017-08-02 09:57:47 +02:00
Ryan Ernst e23919856e Docs: Remove docs for aws region and signer type (#26006)
These settings are removed in 6.0. This commit removes the corresponding
documentation.

relates #22872
relates #23984
2017-08-01 19:16:15 -04:00
Ryan Ernst f978974bc6 Docs: Remove s3 repository integ test documentation (#26005)
The s3 repository plugin has "third party" integ tests which rely
on external service and configuration setup. These tests are really
internal verification of the plugin (and should be moved to real integ
tests). Running them is not something a user should do, and the
documentation has been out of date for all of 5.x. This commit removes
the docs, removing potential confusion for users.
2017-08-01 19:13:45 -04:00
Jason Tedor 7066ec44ca Add recommendation on unicast hosts to docs
This commit adds a small note to the discovery docs to include a note
that we recommend that the unicast hosts list be maintained as the list
of master-eligible nodes in the cluster.

Relates #25991
2017-08-01 18:15:50 +09:00
Tanguy Leroux 9c8d3d3569 [Docs] Add migration notes for the high-level rest client (#25911) 2017-08-01 10:38:56 +02:00
Jason Tedor bc8dc683e4 Update config files docs
This commit updates the docs for the config files to explain the new
mechanism for customizing the configuration directory via the
environment variable CONF_DIR.

Relates #25990
2017-08-01 09:52:23 +09:00
javanna 2b0f4287b3 [DOCS] remove leftover SNAPSHOT suffix from Versions.asciidoc 2017-07-31 16:05:59 +02:00
Luca Cavanna b9ce222a6e [DOCS] add links to javadocs to clients docs (#25745)
We publish javadocs to artifacts.elastic.co (and snapshots.elastic.co) for a while. This commit adds the link to them to the transport client, low level REST client, sniffer and high level REST client pages.

Closes #23761
2017-07-31 15:41:21 +02:00
Jason Tedor fd18e3239a Remove mention of http_address in nodes info docs
This commit removes an outdated reference to http_address in the nodes
info docs. This information is available in the http object for each
node in the nodes info API response.

Relates #25980
2017-07-31 22:04:16 +09:00
Jason Tedor 540413b24a Also skip JAVA_TOOL_OPTIONS on Windows
On non-Windows platforms, we ignore the environment variable
JAVA_TOOL_OPTIONS (this is an environment variable that the JVM respects
by default for picking up extra JVM options). The primary reason that we
ignore this because of the Jayatana agent on Ubuntu; a secondary reason
is that it produces an annoying "Picked up JAVA_TOOL_OPTIONS: ..."
output message. When the elasticsearch-env batch script was introduced
for Windows, ignoring this environment variable was deliberately not
carried over as the primary reason does not apply on Windows. However,
after additional thinking, it seems that we should simply be consistent
to the extent possible here (and also avoid that annoying "Picked up
JAVA_TOOL_OPTIONS: ..." on Windows too). This commit causes the Windows
version of elasticsearch-env to also ignore JAVA_TOOL_OPTIONS.

Relates #25968
2017-07-31 21:27:42 +09:00
Jason Tedor 2ef0f8af38 Add max file size bootstrap check
This commit adds a bootstrap check for the maximum file size, and
ensures the limit is set correctly when Elasticsearch is installed as a
service on systemd-based systems.

Relates #25974
2017-07-31 21:01:47 +09:00
Adrien Grand 57e8b9ee29 Remove another reference to 64-bit systems. 2017-07-31 09:56:06 +02:00
Adrien Grand 1b34f691e5 Remove reference to 32-bit systems. (#25971)
They are not supported anymore as of #25435.
2017-07-31 09:55:09 +02:00
propulkit a2fd1b9f6d No more using 'hybrid mmapfs / niofs' (#25944)
It looks a bit ambiguous here.

ElasticSearch no more using 'hybrid mmapfs / niofs' which chooses filesystem based on the file. It is any one of the mmapfs, niofs or simplefs depending on the operating system.
As quoted here https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules-store.html

Thanks,
Pulkit Agrawal
2017-07-31 09:53:55 +02:00
Martijn van Groningen ec7ac32772
docs: document work around for the percolator if query time text analysis is expensive. 2017-07-28 15:04:15 +02:00
Martijn van Groningen 7c3735bdc4
percolator: Store the QueryBuilder's Writable representation instead of its XContent representation.
The Writeble representation is less heavy to parse and that will benefit percolate performance and throughput.

The query builder's binary format has now the same bwc guarentees as the xcontent format.

Added a qa test that verifies that percolator queries written in older versions are still readable by the current version.
2017-07-28 12:24:10 +02:00
Daniel Mitterdorfer db90455afd Update plugin-related output in reference docs (#25897)
The example output for node info and cluster stats was outdated w.r.t.
to the information that is shown for plugins. With this commit we
updated the example output and update the explanation of the respective
fields.
2017-07-28 11:27:54 +02:00
Martijn van Groningen 5cf56a846a
docs: Remove incorrect warning
Closes #25935
2017-07-28 10:53:47 +02:00
Clinton Gormley 3e568f52c1 Fixed asciidoc formatting 2017-07-27 15:55:52 +02:00
javanna 19843b50e9 [DOCS] update low level client artifact name 2017-07-27 11:19:40 +02:00
Tanguy Leroux 91dc1c5da6 [Docs] Update Java Low-Level documentation to reflect shaded deps (#25882)
Since #25208 the Java Low-Level Rest Client has shaded dependencies. This commit updates the documentation to reflect that.
2017-07-26 12:17:21 +02:00
Tanguy Leroux 90ebaaa9a8 [Docs] Add profile section to the Search API documentation (#25880) 2017-07-26 10:31:46 +02:00
Lee Hinman faee825fea Fix elvis operator documentation 2017-07-25 12:50:09 -06:00
Clinton Gormley 3759aad737 Updated doc versions to 7.0.0-alpha1 unreleased 2017-07-25 13:09:27 +02:00
Jim Ferenczi 7868373069 [Docs] remove reference to the deprecated in the docs 2017-07-25 09:41:53 +02:00
Jim Ferenczi 4a9995145c [Docs]: Clarify query_string parser splits on operator 2017-07-24 18:36:16 +02:00
Jim Ferenczi c3784326eb Refactor field expansion for match, multi_match and query_string query (#25726)
This commit changes the way we handle field expansion in `match`, `multi_match` and `query_string` query.
 The main changes are:

- For exact field name, the new behavior is to rewrite to a matchnodocs query when the field name is not found in the mapping.

- For partial field names (with `*` suffix), the expansion is done only on `keyword`, `text`, `date`, `ip` and `number` field types. Other field types are simply ignored.

- For all fields (`*`), the expansion is done on accepted field types only (see above) and metadata fields are also filtered.

- The `*` notation can also be used to set `default_field` option on`query_string` query. This should replace the needs for the extra option `use_all_fields` which is deprecated in this change.

This commit also rewrites simple `*` query to matchalldocs query when all fields are requested (Fixes #25556). 

The same change should be done on `simple_query_string` for completeness.

`use_all_fields` option in `query_string` is also deprecated in this change, `default_field` should be set to `*` instead.

Relates #25551
2017-07-21 16:52:57 +02:00
Colin Goodheart-Smithe f1f1725fcf [DOCS] improve explanation of dynamic mapping setting (#25829)
Closes #25825
2017-07-21 12:24:38 +01:00
Clinton Gormley 618ff159eb Reorganised setup docs into better order 2017-07-21 11:24:46 +02:00
Ryan Ernst 8ab0d10387 Add compatibility versions to main action response (#25799)
This commit adds the min wire/index compat versions to the main action
output. Not only will this make the compatility expected more
transparent, but it also allows to test which version others think the
compat versions are, similar to how we test the lucene version.
2017-07-20 13:01:41 -07:00
Boaz Leskes 7488877d1a Validate a joining node's version with version of existing cluster nodes (#25808)
When a node tries to join a cluster, it goes through a validation step to make sure the node is compatible with the cluster. Currently we validation that the node can read the cluster state and that it is compatible with the indexes of the cluster. This PR adds validation that the joining node's version is compatible with the versions of existing nodes. Concretely we check that:

1) The node's min compatible version is higher or equal to any node in the cluster (this prevents a too-new node from joining)
2) The node's version is higher or equal to the min compat version of all cluster nodes (this prevents a too old join where, for example, the master is on 5.6, there's another 6.0 node in the cluster and a 5.4 node tries to join).
3) The node's major version is at least as higher as the lowest node in the cluster. This is important as we use the minimum version in the cluster to stop executing bwc code for operations that require multiple nodes. If the nodes are already operating in "new cluster mode", we should prevent nodes from the previous major to join (even if they are wire level compatible). This does mean that if you have a very unlucky partition during the upgrade which partitions all old nodes which are also a minority / data nodes only, the may not be able to re-join the cluster. We feel this edge case risk is well worth the simplification it brings to BWC layers only going one way. This restriction only holds if the cluster state has been recovered (i.e., the cluster has properly formed).

 Also, the node join validation can now selectively fail specific nodes (previously the entire batch was failed). This is an important preparation for a follow up PR where we plan to have a rejected joining node die with dignity.
2017-07-20 20:11:29 +02:00
Clinton Gormley febb4bf7bc Update removal_of_types.asciidoc
Fixed `include_in_type` -> `include_type_name`
2017-07-20 19:18:51 +02:00
Jack Conradson 9f7463e796 remove lang url parameter from stored script requests (#25779)
Also has updates to ScriptMetaData for allowing the old namespace format to be loaded all the way back through 5.0; however, it will throw an exception if two scripts share the same id but different languages.
2017-07-20 08:51:08 -07:00
Jay Modi 3e4bc027eb RestClient uses system properties and system default SSLContext (#25757)
This commit calls the `useSystemProperties` method on the HttpAsyncClientBuilder so that the jvm
system properties are used. The primary reason for doing this is to ensure the builder uses the
system default SSLContext rather than the default instance created by the http client library.

Closes #23231
2017-07-20 07:36:56 -06:00
Jason Tedor 67a4288c9a Remove support for ES_INCLUDE
Today we enable users to customize the environment through the use of
ES_INCLUDE. This made sense for legacy reasons when we did not have
nicities like jvm.options (so dumped JVM options in the default include
script) and somewhat duplicates some of the functionality that we will
need from a dedicated environment script. This commit removes support
for ES_INCLUDE as a first step towards a dedicated include script.

Relates #25804
2017-07-20 15:41:59 +09:00
Boaz Leskes 9989ac69a4 Revert "Validate a joining node's version with version of existing cluster nodes (#25770)"
This reverts commit 1e1f8e6376.
2017-07-19 17:34:53 +02:00
Adrien Grand d607c3be92 Fix list of unconverted snippets. 2017-07-19 14:57:55 +02:00
Adrien Grand f1ff7f2454 Require a field when a `seed` is provided to the `random_score` function. (#25594)
We currently use fielddata on the `_id` field which is trappy, especially as we
do it implicitly. This changes the `random_score` function to use doc ids when
no seed is provided and to suggest a field when a seed is provided.

For now the change only emits a deprecation warning when no field is supplied
but this should be replaced by a strict check on 7.0.

Closes #25240
2017-07-19 14:11:15 +02:00
Clinton Gormley f69decf509 NOCONSOLE -> NOTCONSOLE in removal-of-types 2017-07-19 14:06:04 +02:00
Boaz Leskes 1e1f8e6376 Validate a joining node's version with version of existing cluster nodes (#25770)
When a node tries to join a cluster, it goes through a validation step to make sure the node is compatible with the cluster. Currently we validation that the node can read the cluster state and that it is compatible with the indexes of the cluster. This PR adds validation that the joining node's version is compatible with the versions of existing nodes. Concretely we check that:

1) The node's min compatible version is higher or equal to any node in the cluster (this prevents a too-new node from joining)
2) The node's version is higher or equal to the min compat version of all cluster nodes (this prevents a too old join where, for example, the master is on 5.6, there's another 6.0 node in the cluster and a 5.4 node tries to join).
3) The node's major version is at least as higher as the lowest node in the cluster. This is important as we use the minimum version in the cluster to stop executing bwc code for operations that require multiple nodes. If the nodes are already operating in "new cluster mode", we should prevent nodes from the previous major to join (even if they are wire level compatible). This does mean that if you have a very unlucky partition during the upgrade which partitions all old nodes which are also a minority / data nodes only, the may not be able to re-join the cluster. We feel this edge case risk is well worth the simplification it brings to BWC layers only going one way.

 Also, the node join validation can now selectively fail specific nodes (previously the entire batch was failed). This is an important preparation for a follow up PR where we plan to have a rejected joining node die with dignity.
2017-07-19 12:57:29 +02:00
Jason Tedor 3d3d99557d Expand migration note regarding default paths
This commit expands on the migration note regarding the removal of
default.path.data and default.path.logs to include a note that users
that were relying on the defaults (the common case for path.logs), and
they carry over their previous elasticsearch.yml configruation file,
then they must add explicit values for path.data and path.logs.
2017-07-19 13:40:42 +09:00
Deb Adair 23c810b334 [DOCS] Changes xrefs to cross doc links to enable building GS "mini-docs" 2017-07-18 13:52:38 -07:00
Deb Adair d9e55179f1 [DOCS] Adding index file for GS "mini book". 2017-07-18 13:44:08 -07:00
Christoph Büscher 43bfe06759 [Docs] Add sorting and source filtering section to client docs (#25767) 2017-07-18 16:58:46 +02:00
Clinton Gormley ff4a2519f2 Update experimental labels in the docs (#25727)
Relates https://github.com/elastic/elasticsearch/issues/19798

Removed experimental label from:
* Painless
* Diversified Sampler Agg
* Sampler Agg
* Significant Terms Agg
* Terms Agg document count error and execution_hint
* Cardinality Agg precision_threshold
* Pipeline Aggregations
* index.shard.check_on_startup
* index.store.type (added warning)
* Preloading data into the file system cache
* foreach ingest processor
* Field caps API
* Profile API

Added experimental label to:
* Moving Average Agg Prediction


Changed experimental to beta for:
* Adjacency matrix agg
* Normalizers
* Tasks API
* Index sorting

Labelled experimental in Lucene:
* ICU plugin custom rules file
* Flatten graph token filter
* Synonym graph token filter
* Word delimiter graph token filter
* Simple pattern tokenizer
* Simple pattern split tokenizer

Replaced experimental label with warning that details may change in the future:
* Analysis explain output format
* Segments verbose output format
* Percentile Agg compression and HDR Histogram
* Percentile Rank Agg HDR Histogram
2017-07-18 14:06:22 +02:00
Luca Cavanna 0d8b753325 IndexClosedException to return 400 rather than 403 (#25752)
403 can be confused with security. If an API doesn't support working against closed indices and closed indices are referred to in a request, that is a bad request, hence 400 is more appropriate.
2017-07-18 10:26:32 +02:00
Christoph Büscher a6e3d356ed Change parsing of numeric `to` and `from` parameters in `date_range` aggregation (#25376)
Currently the `to` and `from` parameter in the `date_range` aggregation is not
parsed with the correct date field format from the mappings or the aggregation
if the argument is numeric, but always treated as a long value specifying
`epoch_millis`. This leads to problems e.g. when the format is `epoch_second`,
but the `to` and `from` are currently treated as millis.

With this change, we interpret these parameters according to the `format` of the target field.
If the `format` in the mappings is not compatible with numeric input values,
a compatible `format` (e.g. `epoch_millis`, `epoch_second`) must be specified in
the `date_range` aggregation itself, otherwise an error is thrown.

#Closes #17920
2017-07-18 09:45:28 +02:00
Christoph Büscher 56b1250a34 [Docs] Adding highlighting section to high level client docs (#25751)
Adding a section about how to use highlighting in the SearchSourceBuilder and
how to retrieve highlighted fragments from the SearchResponse.
2017-07-17 19:30:58 +02:00
Simon Willnauer cb4eebcd6a Make `index` in TermsLookup mandatory (#25753)
This change removes the leniency of having a `null` index to fetch
terms from in 6.0 onwards. This feature will be deprecated in the 5.x series
and 6.0 nodes will require the index to be set.

Closes #25750
2017-07-17 18:50:30 +02:00
Clinton Gormley 25a89e613a Broke recipes into separate pages 2017-07-17 18:21:39 +02:00
Glen Smith e9dfb2a215 Fix another simulate example in ingest docs
When simulating an ingest pipeline against an existing pipeline, the
_source field is required to wrap each doc. This commit fixes another
example in the docs that is missing this.
    
Relates #25743, relates e3a0c11239
2017-07-17 15:17:42 +09:00
Glen Smith e3a0c11239 Fix simulate example in ingest docs
When simulating an ingest pipeline against an existing pipeline, the
_source field is required to wrap each doc. This commit fixes an example
in the docs that is missing this.

Relates #25742
2017-07-17 14:17:41 +09:00
Ryan Ernst 072402463b Scripting: Remove search template actions (#25717)
The dedicated search template put/get/delete actions are deprecated in
5.6. This commit removes them from 6.0.
2017-07-14 23:12:05 -07:00
javanna 2c38e93e96 [DOCS] Added note to high level client docs on version
The alpha2 docs is built out of master which may make users think that the high level client was already released as part of alpha2 which it was not. This note should clarify that the client will be released with 6.0.0-beta1
2017-07-15 07:50:25 +02:00
Ryan Ernst b1762d69b5 Setup: Change default heap to 1G (#25695)
This commit changes the default heap size to 1 GB. Experimenting with
elasticsearch is often done on laptops, and 1 GB is much friendlier to
laptop memory. It does put more pressure on the gc, but the tradeoff is
a smaller default footprint. Users running in production can (and
should) adjust the heap size as necessary for their usecase.
2017-07-14 09:38:08 -07:00
Christoph Büscher 5387ed00d2 [Docs] Adding suggestion sections to high level client docs (#25724)
This adds a section about how to add suggestions to the SearchSourceBuilder and
how to retrieve them from a SearchResponse.
2017-07-14 18:33:28 +02:00
Christoph Büscher f809a12493 [Docs] Adding aggregation sections to high level client docs (#25707)
This adds a section about how to add aggregations to the SearchSourceBuilder and how
to retrieve them from a SearchRepsonse to the documentation for the high level rest client.
2017-07-14 12:47:47 +02:00
Bodecker DellaMaria 4f0dc5bf32 Mark filtered query example as not to be used (#25661)
The Filtered Query has been deprecated in favour of the Bool Query with a filter context. However, this deleted page for the Filtered Query is often ranked highly in search results when searching for documentation on "filtered queries". Often people just copy the first code snippet they see, which in this case is the INCORRECT syntax (the correct syntax follows). I think reordering the examples would help avoid a lot of confusion (I have seen people make this same mistake 3 times now)

Adding a comment to indicate that the first example shouldn't be used
2017-07-14 11:45:21 +02:00
Martijn van Groningen c8777c4c2e
docs: Updated reference docs that `document_type` is deprecated 2017-07-14 11:07:46 +02:00
Antonio Matarrese afd9a1c1b1 [DOCS] Explain mapping explosion (#25654) 2017-07-14 09:47:41 +02:00
Neil Rickards 5189bd14f1 [Docs] Fix typo in pattern-tokenizer.asciidoc (#25626) 2017-07-13 18:43:48 +02:00
Jim Ferenczi fe383b7c27 More clarifications on the unified highlighter being the new default (#25668)
* More clarifications on the unified highlighter being the new default
2017-07-13 15:38:58 +02:00
Jim Ferenczi 13da3eb53e Refactor QueryStringQuery for 6.0 (#25646)
This change refactors the query_string query to analyze the query text around logical operators of the query string the same way than a match_query/multi_match_query.
It also adds a type parameter that can be used to change the way multi fields query are built the same way than a multi_match query does.

Now that these queries share the same behavior regarding text analysis, some parameters are obsolete and have been deprecated:

split_on_whitespace: This setting is now ignored with a deprecation notice
if it is used explicitely. With this PR The query_string always splits on logical operator.
It simplifies the understanding of the other parameters that can have different meanings
depending on the value of split_on_whitespace.

auto_generate_phrase_queries: This setting is now ignored with a deprecation notice
if it is used explicitely. This setting only makes sense when the parser splits on whitespace.

use_dismax: This setting is now ignored with a deprecation notice
if it is used explicitely. The tie_breaker parameter is sufficient to handle best_fields/most_fields.

Fixes #25574
2017-07-13 15:32:17 +02:00
Martijn van Groningen 02fad9ac8c
docs: updated java client api to take this into account too to take into account the p/c queries are in parent-join module
Closes #25624
2017-07-13 11:24:22 +02:00
Luca Cavanna ec66d655b5 Rename client artifacts (#25693)
It was brought up that our current client artifacts have generic names like 'rest' that may cause conflicts with other artifacts.

This commit renames:

- rest -> elasticsearch-rest-client
- sniffer -> elasticsearch-rest-client-sniffer
- rest-high-level -> elasticsearch-rest-high-level-client

A couple of small changes are also preparing the high level client for its first release.

Closes #20248
2017-07-13 09:44:25 +02:00
Deb Adair ded9f55263 [DOCS] Incorporated feedback on the highlighting changes. 2017-07-12 16:36:33 -07:00
Ryan Ernst 70b2897bdf Scripting: Deprecate stored search template apis (#25437)
This commit deprecates the PUT, GET and DELETE search template apis.
Instead, the stored script api should be used.

closes #24596
2017-07-12 16:07:28 -07:00
Simon Willnauer e81804cfa4 Add a shard filter search phase to pre-filter shards based on query rewriting (#25658)
Today if we search across a large amount of shards we hit every shard. Yet, it's quite
common to search across an index pattern for time based indices but filtering will exclude
all results outside a certain time range ie. `now-3d`. While the search can potentially hit
hundreds of shards the majority of the shards might yield 0 results since there is not document
that is within this date range. Kibana for instance does this regularly but used `_field_stats`
to optimize the indexes they need to query. Now with the deprecation of `_field_stats` and it's upcoming removal a single dashboard in kibana can potentially turn into searches hitting hundreds or thousands of shards and that can easily cause search rejections even though the most of the requests are very likely super cheap and only need a query rewriting to early terminate with 0 results.

This change adds a pre-filter phase for searches that can, if the number of shards are higher than a the `pre_filter_shard_size` threshold (defaults to 128 shards), fan out to the shards
and check if the query can potentially match any documents at all. While false positives are possible, a negative response means that no matches are possible. These requests are not subject to rejection and can greatly reduce the number of shards a request needs to hit. The approach here is preferable to the kibana approach with field stats since it correctly handles aliases and uses the correct threadpools to execute these requests. Further it's completely transparent to the user and improves scalability of elasticsearch in general on large clusters.
2017-07-12 22:19:20 +02:00
Jason Tedor 86e9438d3c Prevent excessive disk consumption by log files
This commit enables management of the main Elasticsearch log files
out-of-the-box by the following changes:
 - compress rolled logs
 - roll logs every 128 MB
 - maintain a sliding window of logs
 - remove the oldest logs maintaining no more than 2 GB of compressed
   logs on disk

Relates #25660
2017-07-12 15:52:00 -04:00
Jason Tedor 5a416b9922 Use config directory to find jvm.options
This commit removes the environment variable ES_JVM_OPTIONS that allows
the jvm.options file to sit separately from the rest of the config
directory. Instead, we use the CONF_DIR environment variable for custom
configuration location just as we do for the other configuration files.

Relates #25679
2017-07-12 15:29:13 -04:00
Christoph Büscher f3e7a1c4a4 Adding basic search request documentation for high level client (#25651) 2017-07-12 17:06:46 +02:00
Jack Conradson d2b4f7ac5a Disallow lang to be used with Stored Scripts (#25610)
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.
2017-07-12 07:55:57 -07:00
Deb Adair b5e81132cf [DOCS] Reorganized the highlighting topic so it's less confusing. 2017-07-11 21:16:14 -07:00
Jason Tedor e165c405ac Add an underscore to flood stage setting
This is a minor nitty bikeshedding change that renames the suffix of the
disk flood stage setting to "flood_stage" from "floodstage".

Relates #25659
2017-07-11 22:02:00 -04:00
James Baiera 847378a43b Add another parent value option to join documentation (#25609)
Indexing a join field on a document requires a value of type "object" and two sub fields "name" 
and "parent". The "parent" field is only required on child documents, but the "name" field which 
denotes the name of the relation is always needed. Previously, only the short-hand version of the 
join field was documented. This adds documentation for the long-hand join field data, and 
explicitly points out that just specifying the name of the relation for the field value is a 
convenience shortcut.
2017-07-11 15:36:59 -04:00
Adrien Grand de99610c4e Remove reference to field-stats docs. 2017-07-11 18:38:25 +02:00
Simon Willnauer 98c91a3bd0 Limit the number of concurrent shard requests per search request (#25632)
This is a protection mechanism to prevent a single search request from
hitting a large number of shards in the cluster concurrently. If a search is
executed against all indices in the cluster this can easily overload the cluster
causing rejections etc. which is not necessarily desirable. Instead this PR adds
a per request limit of `max_concurrent_shard_requests` that throttles the number of
concurrent initial phase requests to `256` by default. This limit can be increased per request
and protects single search requests from overloading the cluster. Subsequent PRs can introduces
addiontional improvemetns ie. limiting this on a `_msearch` level, making defaults a factor of
the number of nodes or sort shards iters such that we gain the best concurrency across nodes.
2017-07-11 16:23:10 +02:00
Clinton Gormley bd7ddfa175 Removed field-stats docs 2017-07-11 15:15:25 +02:00
Clinton Gormley 92849c64db Fixed bad asciidoc file name 2017-07-11 12:47:52 +02:00
Clinton Gormley ddbbe9f7cc Tidied up the breaking changes docs 2017-07-11 12:40:14 +02:00
Herman Schaaf 977712f977 Change small typo in shards_allocation.asciidoc (#25643) 2017-07-11 11:25:49 +02:00
Tal Levy e04be73ad5 remove ingest.new_date_format (#25583) 2017-07-10 13:07:50 -07:00
Colin Goodheart-Smithe 3a5a54e83e Collapses package structure for some bucket aggs (#25579)
This change collapses some of the packages for the bucket aggregations into their parent packages. This was done for the following aggregations:
* The variants of the range aggregation (geo_distance, date and ip) were moved into the `o.e.s.a.bucket.range` package
* The `o.e.s.a.bucket.terms.support` package was removed and the classes were moved to `o.e.s.a.bucket.terms`
* The filter aggregation was moved to `o.e.s.a.bucket.filter`

Since this PR is already relatively large with only the above changes subsequent PRs will do similar operations on relevant metric and pipeline aggregations

Relates to #22868
2017-07-10 15:08:15 +01:00
Clinton Gormley e85871cfe9 Update cross-cluster-search.asciidoc
Increased the required min version of CCS in the docs to 5.5
2017-07-10 12:04:05 +02:00
DeDe Morton baa1858f56 [DOCS] Fix link (#25616) 2017-07-07 20:40:44 -07:00
DeDe Morton a4fedb213e Fix attribute reference on redirects page (#25614) 2017-07-07 20:15:42 -07:00
Jason Tedor 8148e25087 Fix disk allocator docs
This commit fixes the disk allocator docs which were broken due to the
inadvertent removal of some docs snippet markup.
2017-07-07 22:11:09 -04:00
Jason Tedor bc22c1c286 Add disk threshold settings validation
This commit adds cross-settings validation for the low/high/flood stage
disk watermark settings. This validation was enabled by the introduction
of multiple settings validation.

Relates #25600
2017-07-07 19:54:36 -04:00
matarrese 2eafbaf759 Document aggregating by day of the week (#25602)
Add documentation for aggregating by day of the week.

Closes #24660
2017-07-07 14:16:53 -04:00
olcbean 2ba9fd2aec Remove deprecated created and found from index, delete and bulk (#25516)
The created and found fields in index and delete responses became obsolete after the introduction of the result field in index, update and delete responses (#19566).

After deprecating the created and found fields in 5.x (#19633), now they are removed.

Fixes #19630
2017-07-07 13:58:46 -04:00
Luca Cavanna 4f4f9e0af1 [DOCS] revise high level client Search Scroll API docs (#25599)
Moved the full example at the end of the page, reduced the number of bullet points for it, and added smaller examples at the beginning of the page.
2017-07-07 17:48:58 +02:00
Lee Hinman 8aa0a5c111 Improve REST error handling when endpoint does not support HTTP verb, add OPTIONS support (#24437)
* Improved REST endpoint exception handling, see #15335

Also improved OPTIONS http method handling to better conform with the
http spec.

* Tidied up formatting and comments

See #15335

* Tests for #15335

* Cleaned up comments, added section number

* Swapped out tab indents for space indents

* Test class now extends ESSingleNodeTestCase

* Capture RestResponse so it can be examined in test cases

Simple addition to surface the RestResponse object so we can run tests
against it (see issue #15335).

* Refactored class name, included feedback

See #15335.

* Unit test for REST error handling enhancements

Randomizing unit test for enhanced REST response error handling. See
issue #15335 for more details.

* Cleaned up formatting

* New constructor to set HTTP method

Constructor added to support RestController test cases.

* Refactored FakeRestRequest, streamlined test case.

* Cleaned up conflicts

* Tests for #15335

* Added functionality to ignore or include path wildcards

See #15335

* Further enhancements to request handling

Refactored executeHandler to prioritize explicit path matches. See
#15335 for more information.

* Cosmetic fixes

* Refactored method handlers

* Removed redundant import

* Updated integration tests

* Refactoring to address issue #17853

* Cleaned up test assertions

* Fixed edge case if OPTIONS method randomly selected as invalid method

In this test, an OPTIONS method request is valid, and should not return
a 405 error.

* Remove redundant static modifier

* Hook the multiple PathTrie attempts into RestHandler.dispatchRequest

* Add missing space

* Correctly retrieve new handler for each Trie strategy

* Only copy headers to threadcontext once

* Fix test after REST header copying moved higher up

* Restore original params when trying the next trie candidate

* Remove OPTIONS for invalidHttpMethodArray so a 405 is guaranteed in tests

* Re-add the fix I already added and got removed during merge :-/

* Add missing GET method to test

* Add documentation to migration guide about breaking 404 -> 405 changes

* Explain boolean response, pull into local var

* fixup! Explain boolean response, pull into local var

* Encapsulate multiple HTTP methods into PathTrie<MethodHandlers>

* Add PathTrie.retrieveAll where all matching modes can be retrieved

Then TrieMatchingMode can be package private and not leak into RestController

* Include body of error with 405 responses to give hint about valid methods

* Fix missing usageService handler addition

I accidentally removed this :X

* Initialize PathTrieIterator modes with Arrays.asList

* Use "== false" instead of !

* Missing paren :-/
2017-07-07 09:01:23 -06:00
Viral-Inc f3569debb6 [Docs] Fix typo in bootstrap-checks.asciidoc (#25597) 2017-07-07 15:37:40 +02:00
Tanguy Leroux b06a744b05 [Docs] Document Scroll API for Java High Level REST Client (#25554)
This commit adds documentation for _search/scroll and clear scroll methods of the high level Java REST client
2017-07-07 12:19:33 +02:00
Tanguy Leroux d9bc0f48b4 [Docs] Document Bulk Processor for Java High Level REST Client (#25572) 2017-07-06 17:05:10 +02:00
Martijn van Groningen d0f9f425bd
parent/child: Removed ParentJoinFieldSubFetchPhase 2017-07-06 13:15:02 +02:00
Jim Ferenczi 31614c3ddb Remove deprecated fielddata_fields from search request (#25566)
... and inner_hits
2017-07-06 13:02:28 +02:00
Clinton Gormley ca12b1f2a6 Tidied up the disk allocator docs 2017-07-06 12:16:53 +02:00
Luca Cavanna 26bc900058 [DOCS] Rewrite low-level REST client docs and verify snippets compile (#25559)
Using the infra that we now have in place, we can convert the low-level REST client docs so that they extract code snippets from real Java classes. This way we make sure that all the snippets properly compile. Compared to the high level REST client docs, in this case we don't run the tests themselves, as that would require depending on test-framework which requires java 8 while the low-level REST client is compatible with java 7. I think that compiling snippets is enough for now.
2017-07-06 10:05:50 +02:00
Simon Willnauer 6e5cc424a8 Switch indices read-only if a node runs out of disk space (#25541)
Today when we run out of disk all kinds of crazy things can happen
and nodes are becoming hard to maintain once out of disk is hit.
While we try to move shards away if we hit watermarks this might not
be possible in many situations. Based on the discussion in #24299
this change monitors disk utilization and adds a flood-stage watermark
that causes all indices that are allocated on a node hitting the flood-stage
mark to be switched read-only (with the option to be deleted). This allows users to react on the low disk
situation while subsequent write requests will be rejected. Users can switch
individual indices read-write once the situation is sorted out. There is no
automatic read-write switch once the node has enough space. This requires
user interaction.

The flood-stage watermark is set to `95%` utilization by default.

Closes #24299
2017-07-05 22:18:23 +02:00
Adrien Grand 26de905f1e Fix the documentation to state that the `_id` field is indexed. (#25540) 2017-07-05 16:09:31 +02:00
Clinton Gormley 0170e0e8d3 Remove usage of multi-types from the docs and added a page explaining type removal (#25543)
Closes #25401
2017-07-05 12:30:19 +02:00
Tanguy Leroux fefcae3d45 [Docs] Document Update API for Java High Level REST Client (#25536)
This commit adds documentation for Java High Level REST Client's Update API.
2017-07-05 12:16:42 +02:00
javanna 7899dbef3b [DOCS] High Level Client: fix index items order
Also group APIs under single document, multi-document and search
2017-07-05 12:07:09 +02:00
Luca Cavanna 6f8c0453bc [DOCS] add docs for high level client get method (#25538)
Document high level client get method
2017-07-05 11:57:57 +02:00
Russ Cam 83522ab4e5 Important note for MSI upgrade from command line (#25544)
Add an Important admonition for upgrading via the command line
using the Windows MSI Installer. This calls out the need to pass
the same command line options for an upgrade as were used for
the initial installation.
2017-07-05 11:38:21 +02:00
Wesley Merkel 7994703903 [Docs] Fix some errors in the index aliases docs (#25449) 2017-07-05 11:16:17 +02:00
Tanguy Leroux 3da3632021 [Docs] Document Bulk API for Java High Level REST Client (#25532)
This commit adds documentation for Java High Level REST Client's Bulk API.
2017-07-05 09:26:26 +02:00
Tanguy Leroux c52c3cea68 [Doc] Remove mention of generated fields in get/mget docs (#25492)
This option has been removed in #20102.
2017-07-04 18:06:00 +02:00
Jun Ohtani 6894ef6057 [Analysis] Support normalizer in request param (#24767)
* [Analysis] Support normalizer in request param

Support normalizer param
Support custom normalizer with char_filter/filter param

Closes #23347
2017-07-04 19:16:56 +09:00
Colin Goodheart-Smithe 43efcffcc2 Adds check for negative search request size (#25397)
* Adds check for negative search request size

This change adds a check to `SearchSourceBuilder` to throw and exception if the size set on it is set to a negative value.

Closes #22530

* fix error in reindex

* update re-index tests

* Addresses review comment

* Fixed tests

* Added random negative size test

* Fixes test
2017-07-04 10:51:38 +01:00
Luca Cavanna 3c5eb6cd1b [DOCS] restructure java clients docs pages (#25517)
This commit converts the low level client and high level client chapters into two parts, which allows each high level client supported api to be on a separate page and show up in the index on the right.
2017-07-04 10:58:57 +02:00
Clinton Gormley 8296618640 Include shared/attributes.asciidoc from docs master 2017-07-03 18:17:34 +02:00
Clinton Gormley 091b1b0765 Fixed page breaks for ICU Collation Keyword Fields 2017-07-03 17:49:28 +02:00
Luca Cavanna 99fef2490a [DOCS] add docs for REST high level client index method (#25501)
This commit restructures the existing high level client docs, adapts the existing delete method docs and adds docs for the index method.
2017-07-03 10:48:16 +02:00
Jason Tedor 1b602477ae Add doc note regarding explicit publish host
This commit adds a note to the docs regarding explicilty setting a
publish host if the network.host setting results in multiple bind
addresses.

Relates #25496
2017-06-30 12:59:02 -04:00
Russ Cam 099ccd36ee Update defaults in documentation (#25483)
Following internal discussion

Relates: https://github.com/elastic/windows-installers/pull/49
2017-06-30 16:24:53 +10:00
Lisa Cawley 7e1a5c68db [DOCS] Add docs-dir to Painless (#25482) 2017-06-29 16:31:54 -07:00
Lisa Cawley 57c752000c [DOCS] Update shared attributes for Elasticsearch (#25479)
* [DOCS] Update shared attributes for Elasticsearch

* [DOCS] Moved shared attributes to Versions.asciidoc

* [DOCS] More more book URLs to shared attributes
2017-06-29 13:48:58 -07:00
Christoph Büscher 927111c91d Remove QueryParseContext from parsing QueryBuilders (#25448)
Currently QueryParseContext is only a thin wrapper around an XContentParser that
adds little functionality of its own. I provides helpers for long deprecated
field names which can be removed and two helper methods that can be made static
and moved to other classes. This is a first step in helping to remove
QueryParseContext entirely.
2017-06-29 17:10:20 +02:00
olcbean 3518e313b8 Unify the result interfaces from get and search in Java client (#25361)
As GetField and SearchHitField have the same members, they have been unified into
DocumentField.

Closes #16440
2017-06-29 11:35:28 +02:00
DeDe Morton 6442d1f75e [Docs] Add link to grok debugger docs (#25412) 2017-06-28 16:14:16 -07:00
Andreas Gebhardt a156ccd80e Expand `/_cat/nodes` to return information about hard drive (#21775)
Expand `/_cat/nodes` with already present information about available disk space `diskAvail` (alias: `d`, `disk`) by:

    * `diskTotal` (alias `dt`): total disk space
    * `diskUsed` (alias `du`): used disk space (`diskTotal - diskAvail`)
    * `diskUsedPercent` (alias `dup`): used disk space percentage

Note: The available disk space is the number of bytes available to the node's Java virtual machine. The size might be smaller than the real one. That means the used disk space (percentage) is larger.

Closes #21679
2017-06-28 18:20:20 +02:00
Martijn van Groningen 9ce9c21b83
docs: added percolator script query limitation 2017-06-28 17:10:30 +02:00