Commit Graph

3800 Commits

Author SHA1 Message Date
Adrien Grand 68b0e395b2 Add recommendations about getting consistent scores despite shards and replicas. (#21167)
This is a topic that has triggered many questions recently so it would be good
to have these recommendations documented.
2016-11-02 10:50:38 +01:00
Jim Ferenczi 9d6fac809c Expose splitOnWhitespace in `Query String Query` (#20965)
This change adds an option called `split_on_whitespace` which prevents the query parser to split free text part on whitespace prior to analysis. Instead the queryparser would parse around only real 'operators'. Default to true.
For instance the query `"foo bar"` would let the analyzer of the targeted field decide how the tokens should be splitted.
Some options are missing in this change but I'd like to add them in a follow up PR in order to be able to simplify the backport in 5.x. The missing options (changes) are:
* A `type` option which similarly to the `multi_match` query defines how the free text should be parsed when multi fields are defined.
* Simple range query with additional tokens like ">100 50" are broken when `split_on_whitespace` is set to false. It should be possible to preserve this syntax and make the parser aware of this special syntax even when `split_on_whitespace` is set to false.
* Since all this options would make the `query_string_query` very similar to a match (multi_match) query we should be able to share the code that produce the final Lucene query.
2016-11-02 10:00:40 +01:00
Alexander Lin f86d784bbc Add extra note about Java8 requirement to sub-install docs (#21190)
Closes #20005
2016-11-01 15:52:09 -06:00
Adriel Dean-Hall b72a708c0d Add docs with up to date instructions on updating default similarity (#21242)
* Add docs with up to date instructions on updating default similarity

The default similarity can no longer be set in the configuration file
(you will get an error on startup). Update the docs with the method
that works.

* Add instructions for changing similarity on index creation
2016-11-01 16:14:20 -04:00
Craig Squire 1f1daf59bc Documentation updates for scroll API size parameter (#21229)
* Document size parameter for scroll API

* Fix size parameter behavior description for scroll
2016-11-01 15:55:09 -04:00
Nik Everett a612e5988e Bump reindex-from-remote's buffer to 200mb
It was 10mb and that was causing trouble when folks reindex-from-remoted
with large documents.

We also improve the error reporting so it tells folks to use a smaller
batch size if they hit a buffer size exception. Finally, adds some docs
to reindex-from-remote mentioning the buffer and giving an example of
lowering the size.

Closes #21185
2016-11-01 13:19:28 -04:00
LakumiNarayanan 5af6deb5b5 Fix typo in keyword.asciidoc (#21237) 2016-11-01 10:15:12 -04:00
Jason Tedor f81d28d0a0 Fix transport host setting
This commit fixes a reference to the transport.host setting which was
mistakenly identified as transport.tcp.host.
2016-10-31 20:48:55 -04:00
Jason Tedor 5fab17c7bc Fix markup for bootstrap checks docs
This commit adds a missing backtick to the bootstrap check docs.
2016-10-31 17:33:15 -04:00
Jack Conradson 185dff7346 Cleanup ScriptType (#21179)
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
2016-10-31 13:48:51 -07:00
Jason Tedor 1d8d8ea55f Clarify production mode for bootstrap checks
This commit clarifies that production mode for the bootstrap checks is
only tripped if transport is bound to an external interface.

Relates #21220
2016-10-31 16:34:21 -04:00
Christoph Büscher 1f5adaa824 Docs: Adding Ukrainian analyzer 2016-10-31 18:20:39 +01:00
Lee Hinman 6a8bad8a06 [DOCS] Document all date formats (#21164)
Resolves #21046
2016-10-31 09:15:36 -06:00
Stanislav Mamontov 7738af27e8 Fix malformed JSON in Delete API example (#21168)
Obviously, there should be

    "result": "deleted"

instead of

    "result: deleted"
2016-10-31 09:13:46 -06:00
Nik Everett b5f86f6f05 Add note to getting-started doc about CONSOLE
Link to the documentation for CONSOLE in getting-started so the
layout of the snippet makes some sense.

Relates to #21158
2016-10-30 00:09:09 -04:00
Nik Everett 3a7a218e8f Support negative array ofsets in painless
Adds support for indexing into lists and arrays with negative
indexes meaning "counting from the back". So for if
`x = ["cat", "dog", "chicken"]` then `x[-1] == "chicken"`.

This adds an extra branch to every array and list access but
some performance testing makes it look like the branch predictor
successfully predicts the branch every time so there isn't a
in execution time for this feature when the index is positive.
When the index is negative performance testing showed the runtime
is the same as writing `x[x.length - 1]`, again, presumably thanks
to the branch predictor.

Those performance metrics were calculated for lists and arrays but
`def`s get roughly the same treatment though instead of inlining
the test they need to make a invoke dynamic so we don't screw up
maps.

Closes #20870
2016-10-29 16:12:40 -04:00
Boaz Leskes 75ee2bb61d Update resiliency page for the release of v5 (#21177) 2016-10-28 18:46:54 +02:00
Adrien Grand bafedc93fa Fix Lucene version in the docs. 2016-10-28 14:50:46 +02:00
Adrien Grand b3cc54cf0d Upgrade to lucene-6.3.0-snapshot-ed102d6 (#21150)
Lucene 6.3 is expected to be released in the next weeks so it'd be good to give
it some integration testing. I had to upgrade randomized-testing too so that
both Lucene and Elasticsearch are on the same version.
2016-10-28 14:47:15 +02:00
Adrien Grand 9cbbddb6dc Add support for `quote_field_suffix` to `simple_query_string`. (#21060)
Closes #18641
2016-10-28 09:11:57 +02:00
Jun Ohtani a66c76eb44 Merge pull request #20704 from johtani/remove_request_params_in_analyze_api
Removing request parameters in _analyze API
2016-10-27 17:43:18 +09:00
Colin Goodheart-Smithe c1a9833445 Correct similarity default for 5.0 (#21144) 2016-10-27 09:33:21 +01:00
Igor Motov 17ad88d539 Makes search action cancelable by task management API
Long running searches now can be cancelled using standard task cancellation mechanism.
2016-10-25 12:27:34 -10:00
Nik Everett 44c3b04bef Convert more docs to // CONSOLE
Converts docs for `_cat/segments`, `_cat/plugins` and `_cat/repositories`
from `curl` to `// CONSOLE` so they are tested as part of the build and
are cleaner to use in Console. They should work fine with `curl` with
the `COPY AS CURL` link.

Also swaps the `source` type of the response from `js` to `txt` because
that is more correct. The syntax highlighter doesn't care. It looks at
the text to figure out the language. So it looks a little funny for `_cat`
responses regardless.

Relates to #18160
2016-10-25 11:17:24 -04:00
Jason Tedor 900ee0536e Strengthen handling of unavailable cgroup stats
On some systems, cgroups will be available but not configured. And in
some cases, cgroups will be configured, but not for the subsystems that
we are expecting (e.g., cpu and cpuacct). This commit strengthens the
handling of cgroup stats on such systems.

Relates #21094
2016-10-24 16:36:51 -04:00
Praveen Shukla 2e18f2e818 [DOCS] clarifies master nodes to be master eligible nodes 2016-10-24 08:58:44 -04:00
Jason Tedor 3d642ab0eb Add basic cgroup CPU metrics
This commit adds basic cgroup CPU metrics to the node stats API.

Relates #21029
2016-10-24 08:26:56 -04:00
Jason Tedor 904dcc7127 Remove timeout parameter from plugin script docs
Support for this parameter was removed but the docs were not
updated. This commit removes this stale parameter from the docs.

Relates #21068
2016-10-21 10:56:46 -04:00
Lukáš Vlček d88e6c62aa [DOC] Fix HttpHost constructor arguments (#21056)
Protocol is the last argument.
Oficial API doc: https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpHost.html
Your own use here: 7560101ec7/client/rest/src/test/java/org/elasticsearch/client/RequestLoggerTests.java (L54)
2016-10-21 09:53:08 +02:00
Igor Motov a541f0187d Docs: add documentation about removal of cluster.routing.allocation.snapshot.relocation_enabled 2016-10-20 14:19:12 -10:00
Jim Ferenczi d0bbe89c16 Optimize query with types filter in the URL (t/t/_search) (#20979)
This change adds a TypesQuery that checks if the disjunction of types should be rewritten to a MatchAllDocs query. The check is done only if the number of terms is below a threshold (16 by default and configurable via max_boolean_clause).
2016-10-20 12:33:32 +02:00
Nik Everett acf7c7430b Add "simple match" support for reindex-from-remote whitelist
This allows you to whitelist `localhost:*` or `127.0.10.*:9200`.
It explicitly checks for patterns like `*` in the whitelist and
refuses to start if the whitelist would match everything. Beyond
that the user is on their own designing a secure whitelist.
2016-10-18 21:47:21 -04:00
Tal Levy 38c650f376 make painless the default scripting language for ScriptProcessor (#20981)
- fixes a bug in the docs that mentions `lang` as optional
- now `lang` defaults to "painless"
2016-10-18 16:22:01 -07:00
Chris Earle c2894d2362 Add missing '// CONSOLE' to code snippet 2016-10-18 12:57:03 -04:00
Chris Earle 9cf7214380 [DOCS] Add "version" to template and pipeline docs (#20407)
* [DOCS] Add "version" to template and pipeline docs

This adds details about the "version" to both the template and pipeline pages.
2016-10-18 11:56:18 -04:00
Joshua Rich 8133c5e85d Merge pull request #20958 from joshuar/joshuar-docs/clearer-shard-allocation-filtering
Docs: Cluster Allocation Filtering
2016-10-18 14:24:41 +11:00
Joshua Rich cdb156e691 Merge pull request #20794 from joshuar/doc/fix_highlighter_ambiguities
[DOCS] Use a better name for fields in examples to avoid ambiguity
2016-10-18 14:23:27 +11:00
Clinton Gormley 5ec2ba3166 Update scripted-metric-aggregation.asciidoc
Removed docs for `reduce_params`

Closes #20917
2016-10-17 19:31:30 +02:00
David Pilato 1be7bdf721 Merge pull request #20975 from zhenxing914/2.3
node.client() should not be used in the documentation
2016-10-17 18:28:57 +02:00
Clinton Gormley 43ad8de75b Fixed typo in index rollover docs
Closes #20935
2016-10-17 17:56:54 +02:00
Jason Tedor 96aa5e33ce Fix slowlog docs
This commit fixes two issues with the slow log docs:
 - clarifies that these settings are per index
 - updates index slow log configuration for Log4j 2

Relates #20976
2016-10-17 10:50:32 -04:00
Quinn Shanahan 1bef6c7fee Update regexp-syntax.asciidoc (#20973) 2016-10-17 16:32:17 +02:00
Joshua Rich 63f484ffa3 Docs: Cluster Allocation Filtering
Put more emphasis on the fact that multiple values can be specified
and move examples after explanation of settings.
2016-10-17 11:02:33 +11:00
Nik Everett 4fbe1a8819 CONSOLEify _cat/pending_tasks docs
Relates to #18160
2016-10-14 14:12:35 -04:00
Jason Tedor 370253f95a Add doc note regarding processors bound
This commit expands the thread pool docs regarding the processor
setting.

Relates #20895
2016-10-14 10:32:31 -04:00
Nik Everett 68ed183381 CONSOLEify a few more _cat docs
`_cat/master`, `_cat/nodeattrs`, `_cat/nodes`.
2016-10-13 16:43:06 -04:00
Nik Everett 279baa0284 Add a flush to test in _cat/indices.asciidoc
We test that sorting by `store.size` works but sometimes the
sizes aren't what we expect. At least in CI:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=opensuse/101/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-unix-compatibility/os=centos/100/console

I haven't been able to reproduce it locally but adding a `_flush`
won't hurt and might make the inconsistency vanish.
2016-10-13 13:21:57 -04:00
Nik Everett 3bba7dbe07 Docs: note about snapshot version compatibility (#20896)
It is important that folks understand that snapshot/restore isn't
for archiving. It is appropriate for backup and disaster recovery
but not for archival over long periods of time because of version
incompatibility.

Closes #20866
2016-10-13 10:49:32 -04:00
Uli Fahrer 85094d9190 Fix wrong heading
Relates #20906
2016-10-13 07:06:53 -04:00
Clinton Gormley 101ea0549a Add note about retention periods to reindex-upgrade docs 2016-10-13 12:13:55 +02:00
Thibaud BARDIN 1bcd26627c [DOCS] Fix typo in "Wait For Active Shards" part (#20900)
Add missing closing backtick
2016-10-13 08:53:30 +02:00
Robin Clarke bbe6555b7a Docs: your -> you're (#20883) 2016-10-12 11:09:34 -04:00
Nik Everett 42a7a554b1 Don't mind pending cluster tasks in docs build
This removes an assertion that the cluster doesn't have any
pending cluster state tasks from the `_cat/health` docs.

Relates to #18160
2016-10-11 17:44:50 -04:00
Nik Everett 298cf1cf21 CONSOLEify _cat/indices docs
Relates to #18160
Uses the new sorting (#20658) in the `_cat` API to support all use
cases natively. We can still resort to piping things through `sort`
if we need to, but we don't have to for basic stuff like sorting!
2016-10-11 14:55:37 -04:00
Thibaud BARDIN 82881ef99a [DOCS] Fix typo in "Cluster Health" part (#20864)
Replace "we can see and total of ..." by "we can see a total of ..."
2016-10-11 13:53:11 -04:00
Lee Hinman f0a2726dcd [DOCS] Remove documentation for `force` version-type
This option should not be recommended to anyone, and should never be
used, upon chance of primary/replica divergence.

Relates to #19769
2016-10-11 11:21:32 -06:00
Alexander Lin 10ddcc4616 Adding built-in sorting capability to _cat apis. (#20658)
* Adding built-in sorting capability to _cat apis.

Closes #16975

* addressing pr comments

* changing value types back to original implementation and fixing cosmetic issues

* Changing compareTo, hashCode of value types to a better implementation

* Changed value compareTos to use Double.compare instead of if statements + fixed some failed unit tests
2016-10-11 12:29:22 -04:00
Nik Everett 3528a514e2 Fix docs build
Removes a non-existant file from the non-`// CONSOLE` list. If you
remove all the non-`// CONSOLE` snippets from a file it must be
removed from the whitelist for the build to pass. Removing the file
entirely counts as removing the snippets.

Related to #18160
2016-10-11 08:58:51 -04:00
Clinton Gormley 82e2f6e747 Document the ctx._now variable in the update API
Relates to #20835
2016-10-11 13:13:03 +02:00
Clinton Gormley 5b5306ea82 Add an indices-upgrade section to the docs redirects. 2016-10-11 12:39:02 +02:00
Clinton Gormley a7d8de14bb Removed the upgrade API docs
Relates to #20675
2016-10-11 12:21:46 +02:00
Clinton Gormley 02a739d3c9 Added upgrade docs explaining how to reindex in place or reindex from remote
Closes #20675
2016-10-11 12:14:35 +02:00
Pascal Borreli fcb01deb34 Fixed typos (#20843) 2016-10-10 14:51:47 -06:00
Nik Everett c414faf2fb Reword note about windows and FDs
Make it clear why this isn't a thing in windows and remove value
judgement about file descriptors.

Relates to #20737
2016-10-10 08:47:25 -04:00
Nik Everett 741ecf80ff Loosen assertion in getting-started.asciidoc
We were asserting that the size of an index had a decimal point but it
doesn't always:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.0+java9-periodic/240/console
2016-10-09 13:47:59 -04:00
Lucas Bremgartner 4bf45e53a5 Documentation: comments in index templates (#20701)
Fixes #20513
2016-10-08 12:38:45 +02:00
Clinton Gormley 22f1acde94 Docs: Pattern analyzer does not support a max_token_length parameter
Closes #20713
2016-10-08 12:27:33 +02:00
David Pilato 2e1539b879 Fix URL for plugins
Reported at https://discuss.elastic.co/t/ingest-geoip-for-beta1/62505
2016-10-07 23:33:13 +02:00
Nik Everett 06049283a0 CONSOLEify some _cat docs
`/_cat/count`, `/_cat/fielddata`, and `/_cat/health`.

Three more files down, 141 to go.

Relates to #18160
2016-10-07 16:30:45 -04:00
Shane Connelly 78ae19ae5e Merge pull request #20784 from eskibars/master
Add a note that reindex does not set up mappings, etc
2016-10-07 09:39:10 -07:00
Clinton Gormley 6cf7a93837 Better formatting of date-match URL encoding 2016-10-07 16:33:30 +02:00
Clinton Gormley eb62f90578 Document date math use in the rollover API (#20780)
* Document date math use in the rollover API

Relates to #20709
2016-10-07 15:49:28 +02:00
Adrien Grand 7a403f640b Clarify some docs about geo-distance sorting. (#20735)
This also improves formatting a bit.
2016-10-07 15:26:34 +02:00
Jason Tedor d01a62908a Change separator for shards preference
The shards preference on a search request enables specifying a list of
shards to hit, and then a secondary preference (e.g., "_primary") can be
added. Today, the separator between the shards list and the secondary
preference is ';'. Unfortunately, this is also a valid separtor for URL
query parameters. This means that a preference like "_shards:0;_primary"
will be parsed into two URL parameters: "_shards:0" and "_primary". With
the recent change to strict URL parsing, the second parameter will be
rejected, "_primary" is not a valid URL parameter on a search
request. This means that this feature has never worked (unless the ';'
is escaped, but no one does that because our docs do not that, and there
was no indication from Elasticsearch that this did not work). This
commit changes the separator to '|'.

Relates #20786
2016-10-07 07:17:01 -05:00
Simon Willnauer 194a6b1df0 Remove LocalTransport in favor of MockTcpTransport (#20695)
This change proposes the removal of all non-tcp transport implementations. The
mock transport can be used by default to run tests instead of local transport that has
roughly the same performance compared to TCP or at least not noticeably slower.

This is a master only change, deprecation notice in 5.x will be committed as a
separate change.
2016-10-07 11:27:47 +02:00
Jun Ohtani 945fa499d2 Deprecating request parameters in _analyze API
Remove params in indices.analyze.json
Fix REST changes

Closes #20246
2016-10-07 16:23:24 +09:00
Jun Ohtani 99236b7627 Removing request parameters in _analyze API
Fix small English issue in breaking changes

Closes #20246
2016-10-07 16:23:24 +09:00
Jun Ohtani eca9894c5f Removing request parameters in _analyze API
Remove unused imports
Replace POST method by GET method in docs
 Add breaking changes explanation
 Fix small issue in Kuromoji docs

Closes #20246
2016-10-07 16:23:24 +09:00
Jun Ohtani 370f0b885e Removing request parameters in _analyze API
Remove request params in _analyze API without index param
Change rest-api-test using JSON
Change docs using JSON

Closes #20246
2016-10-07 16:23:24 +09:00
Joshua Rich e06a40ccbd [DOCS] Use a better name for fields in examples to avoid ambiguity
Previously, this doc was using a field called "content". This is
confusing, especially when the doc starts talking about the content of
the content field.  This change makes the field name "comment" which
is less ambiguous and also changes some related field names in the doc
to make a consistent example theme of editing docs around blog posts.
2016-10-07 14:46:55 +11:00
Clinton Gormley e83e8e890e Fixed released_state docs switch to build correctly and include missing links 2016-10-06 21:42:11 +02:00
Shane Connelly 3164917fd4 Adds a note that reindex does not set up mappings, etc. Closes #20783 2016-10-06 12:27:08 -07:00
Clinton Gormley d822a642b0 Document how to run a single docs test 2016-10-06 21:10:49 +02:00
Nik Everett 6d42e197b8 CONSOLEify remaining getting-started docs
Moves some test data used by those docs into the Elasticsearch
repository so we can use it when we test the docs during the build.

Relates to #18160
2016-10-06 14:06:48 -04:00
Nik Everett d9781bd069 Fix broken regex in doc tests
Regexes are hard.
2016-10-06 13:53:20 -04:00
Nik Everett d7d5df8863 CONSOLEify some _cat docs
Added `// NOTCONSOLE` to some `_cat` docs that rely on `sort` or
are otherwise too difficult for us to test at this point.

Relates to #20717
2016-10-06 13:37:21 -04:00
Clinton Gormley 40bf2c206c Add placeholder to the docs to distinguish released from unreleased from prerelease versions (#20644)
Closes https://github.com/elastic/stack/issues/18
2016-10-06 17:58:03 +02:00
Clinton Gormley 43a851f1f6 Most date-math chars in index names need to be percent encoded
Updated docs to provide encodings
2016-10-06 15:44:42 +02:00
Nik Everett 41d6529d06 CONSOLEify scroll docs
This causes the snippets to be tested during the build and gives
helpful links to the reader to open the docs in console or copy them
as curl commands.

Relates to #18160
2016-10-05 11:21:54 -04:00
Anatolii Stepaniuk f895abcf40 Fix grammar issues in some docs
This commit fixes some grammar issues in various docs.

Closes #20751
Closes #20752
Closes #20754
Closes #20755
2016-10-05 11:20:45 -04:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates #20722
2016-10-04 12:45:29 -04:00
Nik Everett 2b3760f78b File descriptors limit doesn't apply to Windows
On Windows the JDK uses `CreateFileW` which has a stupidly high
limit for the number of `Handle`s it can make - `16 * 1024 * 1024`.
So this isn't really a problem on Windows at all.

Closes #20732
2016-10-04 11:32:08 -04:00
Simon Willnauer 56f35baf47 Add date-math support to `_rollover` (#20709)
today it's not possible to use date-math efficiently with the `_rollover`
API. This change adds support for date-math in the target index as well as
support for preserving the math logic when an existing index that was created with
a date math expression all subsequent indices are created with the same expression.
2016-10-03 16:52:33 +02:00
Nik Everett 9271c0302f CONSOLEify some aggs docs
Cleans up the example result in `children-aggregation` so that
it matches the example data.

Relates to #18160
2016-10-03 09:22:56 -04:00
Tanguy Leroux 91835400b5 [Docs] Adapt //TESTREPONSE for Windows platforms
This commit adapts a //TESTRESPONSE check in the getting-started.asciidoc because the Load Average is not available on Windows platforms.
2016-10-03 11:30:23 +02:00
Nik Everett 7cfc2f126c CONSOLEify the first half of getting-started
Adds a `// TESTRESPONSE[_cat]` syntax that sets up some substitutions
useful for asserting that _cat APIs match.
2016-10-01 12:56:23 +02:00
Boaz Leskes 27eab74510 merge from master 2016-09-30 17:19:30 +02:00
Martijn van Groningen 55dce523c2 docs: marked `foreach` processor as experimental
Closes #19602
2016-09-30 12:23:42 +02:00
Simon Willnauer 74184cb1b0 Stabelize tests in phrase-suggest.asciidoc 2016-09-29 11:13:17 +02:00
Simon Willnauer f2e6862803 Add a hard limit for `index.number_of_shard` (#20682)
this change adds a hard limit to `index.number_of_shard` that prevents
indices from being created that have more than 1024 shards. This is still
a huge limit and can only be changed via settings a system property.
2016-09-29 11:03:30 +02:00
Jason Tedor 8879360f66 Fix failing doc tests in feature/seq_no
This commit fixes failing doc tests in feature/seq_no after merging
master into this branch.
2016-09-29 03:58:02 +02:00
Tal Levy f3a5ee671b [docs] [fix] `field` is no longer an option for the script processor (#20614) 2016-09-29 03:04:43 +02:00
Jason Tedor 25fd9e26c4 Merge branch 'master' into feature/seq_no
* master: (1199 commits)
  [DOCS] Remove non-valid link to mapping migration document
  Revert "Default `include_in_all` for numeric-like types to false"
  test: add a test with ipv6 address
  docs: clearify that both ip4 and ip6 addresses are supported
  Include complex settings in settings requests
  Add production warning for pre-release builds
  Clean up confusing error message on unhandled endpoint
  [TEST] Increase logging level in testDelayShards()
  change health from string to enum (#20661)
  Provide error message when plugin id is missing
  Document that sliced scroll works for reindex
  Make reindex-from-remote ignore unknown fields
  Remove NoopGatewayAllocator in favor of a more realistic mock (#20637)
  Remove Marvel character reference from guide
  Fix documentation for setting Java I/O temp dir
  Update client benchmarks to log4j2
  Changes the API of GatewayAllocator#applyStartedShards and (#20642)
  Removes FailedRerouteAllocation and StartedRerouteAllocation
  IndexRoutingTable.initializeEmpty shouldn't override supplied primary RecoverySource (#20638)
  Smoke tester: Adjust to latest changes (#20611)
  ...
2016-09-29 00:22:31 +02:00
Tal Levy 550a0449bc [docs] [fix] `field` is no longer an option for the script processor (#20614) 2016-09-28 21:50:32 +02:00
Lee Hinman 85402d5220 [DOCS] Remove non-valid link to mapping migration document 2016-09-28 09:09:19 -06:00
Lee Hinman 3f77eacab1 Revert "Default `include_in_all` for numeric-like types to false"
This reverts commit 6666892038.
2016-09-28 07:07:46 -06:00
Martijn van Groningen a7ad11c2e7 docs: clearify that both ip4 and ip6 addresses are supported 2016-09-28 10:03:43 +02:00
Lee Hinman 27496d6b92 Merge remote-tracking branch 'dakrone/breaker-done-resiliency' 2016-09-26 08:39:55 -06:00
Nik Everett 560fba1b28 Document that sliced scroll works for reindex
Surprise! You can use sliced scroll to easily parallelize reindex
and friend. They support it because they use the same infrastructure
as a regular search to parse the search request. While we would like
to make an "automatic" option for parallelizing reindex, this manual
option works right now and is pretty convenient!
2016-09-26 05:27:44 +02:00
Pius 660d9d0de3 Remove Marvel character reference from guide
Relates #20654
2016-09-25 11:28:13 -04:00
Jason Tedor a7c777f58a Fix documentation for setting Java I/O temp dir
This commit fixes the documentation for configuring the Java I/O temp
dir which incorrectly suggested using the -D flag as a parameter on the
command line; these flags have been removed and should now be specified
as arguments to the JVM using either the ES_JAVA_OPTS environment
variable or using the jvm.options configuration file.

Closes #20652
2016-09-24 21:13:22 -04:00
Clinton Gormley e3b7b4f032 Reorganised docs for mapping safeguard settings 2016-09-22 14:58:17 +02:00
Martijn van Groningen ad7c22198c docs: describe more explicitly what happens when indexing queries that fetch terms 2016-09-22 10:00:11 +00:00
Nik Everett 3ed3e5e660 Convert more docs to CONSOLE
* plugins/discovery-azure-class.asciidoc
* reference/cluster.asciidoc
* reference/modules/cluster/misc.asciidoc
* reference/modules/indices/request_cache.asciidoc

After this is merged there will be no unconvereted snippets outside
of `reference`.

Related to #18160
2016-09-21 09:36:21 -04:00
Clinton Gormley c753a0bfe0 Change APT/YUM urls to point to the prerelease repositories
Closes #20593
2016-09-21 14:49:14 +02:00
Lee Hinman 45d4d08f32 [DOCS] Mark combinatorial explosion in aggs as 'done'
This marks the "Prevent combinatorial explosion in aggregations from
causing OOM" task as done in 5.0.0.

Relates to #8081 and #19394
2016-09-20 16:07:43 -06:00
Nik Everett c97f4f38e7 Weaken overspecified tests in response snippets
The docs in api-conventions contained some very specific response
snippets that made the tests a bit fragile. This removes some of
that fragility.
2016-09-20 13:53:07 -04:00
Nik Everett af285f1aa4 Convert more docs to CONSOLE
* reference/api-conventions.asciidoc
* reference/redirects.asciidoc

Relates to #18160
2016-09-20 13:12:12 -04:00
Clinton Gormley 57d40a4691 Fix JVM heap size docs for 5.0
Closes #20587
2016-09-20 18:51:56 +02:00
David Pilato c2d621dfdf Merge branch 'doc/plugins-offline-url' 2016-09-20 14:53:05 +02:00
Tanguy Leroux 33348a9c7c [DOC] Add /cat/templates to list of excluded docs 2016-09-20 11:02:55 +02:00
Alexander Lin d31a8e6558 Provides a cat api endpoint for templates. (#20545)
Adds a cat api endpoint: /_cat/templates and its more specific version, /_cat/templates/{name}.

It looks something like:

$ curl "localhost:9200/_cat/templates?v"
name                  template     order version
sushi_california_roll *avocado*    1     1
pizza_hawaiian        *pineapples* 1
pizza_pepperoni       *pepperoni*  1

The specified version (only allows * globs) looks like:

$ curl "localhost:9200/_cat/templates/pizza*"
name            template     order version
pizza_hawaiian  *pineapples* 1
pizza_pepperoni *pepperoni*  1

Partially specified columns:

$ curl "localhost:9200/_cat/templates/pizza*?v=true&h=name,template"
name            template
pizza_hawaiian  *pineapples*
pizza_pepperoni *pepperoni*

The help text:

$ curl "localhost:9200/_cat/templates/pizza*?help"
name     | n | template name
template | t | template pattern string
order    | o | template application order number
version  | v | version

Closes #20467
2016-09-20 10:40:23 +02:00
Tanguy Leroux 24fda8a928 [DOC] Remove 'reference/docs/get.asciidoc' from list of excluded docs 2016-09-20 10:23:09 +02:00
Jim Ferenczi f98d5b6261 Add CONSOLE tests for snippets in get and bulk API docs (#20473)
* Add CONSOLE tests for snippets in get and bulk API docs

This change adds tests for the snippets in the get and bulk API documentation.
2016-09-20 09:18:08 +02:00
Clinton Gormley 95bca7c390 Remove redundant .x from apt/rpm repo docs
Relates to #20574
2016-09-20 08:11:02 +02:00
Ryan Ernst b77c16086c Convert old download links to unified release urls (#20574)
With the unified release process across the elastic stack, download
links for all products are changing. This change updates docs referring
to the old download and packages urls.

Note that this change also updates the plugin installation command as
the url for downloads is being changed to be consistent with that for
packages (both plural).
2016-09-19 18:10:45 -07:00
Jason Tedor 05b4e0c0e3 Add serial collector bootstrap check
The serial collector is not suitable for running with a server
application like Elasticsearch and can decimate performance and lead to
cluster instability. This commit adds a bootstrap check to prevent usage
of the serial collector when Elasticsearch is running in production
mode.

Relates #20558
2016-09-19 20:25:50 -04:00
Nik Everett 156393be0e Fail build if new doc snippets aren't `// CONSOLE`
This tracks the snippets that probably should be converted to
`// CONSOLE` or `// TESTRESPONSE` and fails the build if the list
of files with such snippets doesn't match the list in `docs/build.gradle`.
Setting the file looks like
```
/* List of files that have snippets that probably should be converted to
 * `// CONSOLE` and `// TESTRESPONSE` but have yet to be converted. Try and
 * only remove entries from this list. When it is empty we'll remove it
 * entirely and have a party! There will be cake and everything.... */
buildRestTests.expectedUnconvertedCandidates = [
  'plugins/discovery-azure-classic.asciidoc',
...
  'reference/search/suggesters/completion-suggest.asciidoc',
]
```

This list is in `build.gradle` because we expect it to be fairly
temporary. In a few months we'll have converted all of the docs and won't
ned it any more.

From now on if you add now docs that contain a snippet that shows an
interaction with elasticsearch you have three choices:
1. Stick `// CONSOLE` on the interactions and `// TESTRESPONSE` on the
responses. The build (specifically (`gradle docs:check`) will test that
these interactions "work". If there isn't a `// TESTRESPONSE` snippet
then "work" just means "Elasticsearch responds with a 200-level response
code and no `WARNING` headers. This is way better than nothing.

2. Add `// NOTCONSOLE` if the snippet isn't actually interacting with
Elasticsearch. This should only be required for stuff like javascript
source code or `curl` against an external service like AWS or GCE. The
snippet will not get "OPEN IN CONSOLE" or "COPY AS CURL" buttons or be
tested.

3. Add `// TEST[skip:reason]` under the snippet. This will just skip the
snippet in the test phase. This should really be reserved for snippets
where we can't test them because they require an external service that
we don't have at testing time.

Please, please, please, please don't add more things to the list. After
all, it sais there'll be cake when we remove it entirely!

Relates to #18160
2016-09-19 16:43:43 -04:00
David Pilato ed4d0881b1 Add profile and explain parameters to template API
We can now run templates using `explain` and/or `profile` parameters.
Which is interesting when you have defined a complicated profile but want to debug it in an easier way than running the full query again.

You can use `explain` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "explain": true
}
```

You can use `profile` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "profile": true
}
```
2016-09-19 17:52:13 +02:00
David Pilato 4c9a77e6c0 Add link to "offline install" and show the full URL 2016-09-19 15:04:29 +02:00
David Pilato 9e58ca835c Merge branch 'master' into doc/plugins-offline-url
# Conflicts:
#	docs/plugins/mapper-attachments.asciidoc
2016-09-19 14:51:55 +02:00
David Pilato a530399ae5 Merge branch 'pr/remove-mapper-attachments' 2016-09-19 14:47:14 +02:00
Jason Tedor 7874463db0 Merge pull request #20553 from jasontedor/node-attributes
Node attributes and REST API spec README
2016-09-19 08:31:09 -04:00
Alexander Lin 7cd0316b51 Fix minhash docs level
Relates #20547
2016-09-19 07:54:04 -04:00
Jason Tedor b456823434 Fix leftover node attributes usage
Previously node attributes could be set via node.* but this now requires
using node.attr.*. This commit fixes some leftover usages of the old
way.
2016-09-19 07:45:31 -04:00
Tanguy Leroux 656596c2a9 [DOC] Remove obsolete node names from documentation
Funny node names have been removed in #19456 and replaced by UUID. This commit removes these obsolete node names and replace them by real UUIDs in the documentation.

closes #20065
2016-09-19 11:56:28 +02:00
Tanguy Leroux 1894489832 [DOC] Update /_cat/nodes doc
closes #20162
2016-09-19 09:31:48 +02:00
David Pilato dfd1eebdd0 Remove mapper attachments plugin
We now have in 5.0.0 `ingest-attachment` plugin.
We can remove `mapper-attachments` plugin for 6.0.

Closes #18837.
2016-09-19 09:01:16 +02:00
Alexander Guz c1e8b6a8ba Fixed aggregation by "gender" request example.
`gender.keyword` should be used instead of just `gender` or we
get an error `Fielddata is disabled on text fields by default. Set
fielddata=true on [gender] in order to load fielddata in memory by
uninverting the inverted index. Note that this can however use

Closes #20535
significant memory.`
2016-09-17 17:23:27 -04:00
Ali Beyad f608e6c6cf Improves the documentation for the (#20531)
`cluster.routing.allocation.cluster_concurrent_rebalance` setting,
clarifying in which shard allocation situations the rebalance limit
takes effect.

Closes #20529
2016-09-16 16:06:18 -04:00
Alexander Guz 8ec94a4ba0 Edited response structure on indexing a document (#20517)
Added "_shards" and "result" keys to the response.
2016-09-16 10:16:40 -06:00
Boaz Leskes 577dcb3237 Add current cluster state version to zen pings and use them in master election (#20384)
During a networking partition, cluster states updates (like mapping changes or shard assignments)
are committed if a majority of the masters node received the update correctly. This means that the current master has access to enough nodes in the cluster to continue to operate correctly. When the network partition heals, the isolated nodes catch up with the current state and get the changes they couldn't receive before. However, if a second partition happens while the cluster
is still recovering from the previous one *and* the old master is put in the minority side, it may be that a new master is elected which did not yet catch up. If that happens, cluster state updates can be lost.

This commit fixed 95% of this rare problem by adding the current cluster state version to `PingResponse` and use them when deciding which master to join (and thus casting the node's vote). 

Note: this doesn't fully mitigate the problem as a cluster state update which is issued concurrently with a network partition can be lost if the partition prevents the commit message (part of the two phased commit of cluster state updates) from reaching any single node in the majority side *and* the partition does allow for the master to acknowledge the change. We are working on a more comprehensive fix but that requires considerate work  and is targeted at 6.0.
2016-09-15 23:39:11 +02:00
Areek Zillur 0556c93920 Merge pull request #20488 from areek/docs/clarify_plugins_installation
[DOC] Add note requiring full cluster restart for installing plugins with custom metadata
2016-09-15 17:05:27 -04:00
Areek Zillur 9b17242b5d [DOC] Add note for full cluster restart for installing plugins using custom metadata
Currently, we check if a node has the same set of custom metadata as the master
before joining the cluster. This implies freshly installing a plugin that has its
custom metadata requires a full cluster restart.
2016-09-15 17:04:45 -04:00
Nik Everett e4c80c94e9 Convert more search docs to CONSOLE
`profile.asciidoc` now runs all of its command but it doesn't validate
all of the results. Writing the validation is time consuming so I only
did some of it.
2016-09-15 11:58:21 -04:00
Luca Cavanna 37489c3274 Add clusterUUID to RestMainAction output (#20503)
Add clusterUUID to RestMainAction output

GET / now returns the clusterUUID as well as part of its output for monitoring purposes
2016-09-15 16:25:17 +02:00
gfyoung b03c807368 Rename service.bat to elasticsearch-service.bat (#20496)
Closes gh-17528.
2016-09-15 15:53:44 +02:00
Clinton Gormley 960efe6202 Fixed typo in cat indices docs 2016-09-14 18:36:58 +01:00
Nik Everett 2d568ece2d CONSOLEify some search docs
* search/search.asciidoc
* search/request-body.asciidoc
* search/explain.asciidoc
* search/search-shards.asciidoc
2016-09-14 13:06:37 -04:00
Tobias Günther 3a7a437594 Update rescoring docs in respect to sort (#20477)
* Update rescoring docs in respect to sort

If sort is present in a query the rescore query is not executed. As long as this feature is neither implemented (see discussion in #6788) nor  the combination of sort and rescoring raises an error, we should warn the user in the documentation about this.

* Missed a dot
2016-09-14 17:07:10 +01:00
Jason Tedor 412c61c402 Update logger names in docs
In 7560101ec7, the Elasticsearch logger
names were modified to be their fully-qualified class name (with some
exceptions for special loggers like the slow logs and the transport
tracer). This commit updates the docs accordingly.

Relates #20475
2016-09-14 08:08:49 -04:00
Tal Levy 4704efaef4 [ingest-geoip] do not insert null-valued fields in geoip response
update geoip to not include null-valued results from database

Originally, the plugin would still insert all the requested fields, but
assign null to each one. This fixes that by not writing the fields at
all. Makes for a better experience when the null fields conflict with
the typical geo_point field mapping.
2016-09-13 18:12:02 -07:00
Ali Beyad 730f9058a1 Documentation for file-based discovery plugin (#20462)
Documentation for file-based discovery plugin.

Closes #20458
2016-09-13 20:53:10 -04:00
debadair ee4798d852 Docs: Added redirect for geohash_cell query topic. 2016-09-13 17:40:24 -07:00
Nicholas Knize 598bab93ae [DOC] Cleanup dangling references to deprecated geo parameters
With the cut over to LatLonPoint the geohash, geohash_precision, lat_lon, and geohash_prefix parameters have been removed. This commit fixes the doc build by removing the remaining dangling references to these removed parameters.
2016-09-13 16:38:38 -05:00
Jim Ferenczi 1764ec56b3 Fixed naming inconsistency for fields/stored_fields in the APIs (#20166)
This change replaces the fields parameter with stored_fields when it makes sense.
This is dictated by the renaming we made in #18943 for the search API.

The following list of endpoint has been changed to use `stored_fields` instead of `fields`:
* get
* mget
* explain

The documentation and the rest API spec has been updated to cope with the changes for the following APIs:
* delete_by_query
* get
* mget
* explain

The `fields` parameter has been deprecated for the following APIs (it is replaced by _source filtering):
* update: the fields are extracted from the _source directly.
* bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields.

Some APIs still have the `fields` parameter for various reasons:
* cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed.
* indices.clear_cache: used to indicate which fielddata fields should be cleared.
* indices.get_field_mapping: used to filter fields in the mapping.
* indices.stats: get stats on fields (stored or not stored).
* termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise.
* mtermvectors:
* nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all.

Fixes #20155
2016-09-13 20:54:41 +02:00
Nicholas Knize 1a60e1c3d2 Update docs for LatLonPoint cut over
This commit removes documentation for:

* geohash cell query
* lat_lon parameter
* geohash parameter
* geohash_precision parameter
* geohash_prefix parameter

It also updates failing tests that reference these parameters for backcompat.
2016-09-13 12:18:21 -05:00
Tanguy Leroux 6090c51fc5 Add quiet option to disable console logging (#20422)
This commit adds a -q/--quiet option to Elasticsearch so that it does not log anything in the console and closes stdout & stderr streams. This is useful for SystemD to avoid duplicate logs in both journalctl and /var/log/elasticsearch/elasticsearch.log while still allows the JVM to print error messages in stdout/stderr if needed.

closes #17220
2016-09-13 14:08:24 +02:00
Jason Tedor c7bfbe3e69 Add health status parameter to cat indices API
This commit adds a health status parameter to the cat indices API for
filtering on indices that match the specified status (green|yellow|red).

Relates #20393
2016-09-13 07:57:18 -04:00
Yevhen Bobrov 786508be08 Documentation for field_masking_span query (#20395)
* Documentation for field_masking_span query. Fixes #20293

* After review fixes
2016-09-13 12:27:33 +01:00
Isabel Drost-Fromm 0e707f241e Add docs to template support for _msearch (#17382)
Add docs to template support for _msearch

Relates to #10885
Relates to #15674

* Reference those docs from the rest api spec for _msearch/template support.
2016-09-13 13:19:25 +02:00
Lee Hinman 94625d74e4 No longer allow cluster name in data path
In 5.x we allowed this with a deprecation warning. This removes the code
added for that deprecation, requiring the cluster name to not be in the
data path.

Resolves #20391
2016-09-12 15:47:01 -06:00
Nik Everett 69bf08f6c6 Disable regexes by default in painless
Adds a new node level, non-dynamic setting, `script.painless.regex.enabled`
can be used to enable regexes.

Closes #20397
2016-09-12 14:09:43 -04:00
David Pilato 472c679981 Add offline install instructions for plugins
Follow up of https://github.com/elastic/elasticsearch/issues/15925#issuecomment-171250150

This commit adds offline install instructions for plugins.
2016-09-12 15:36:16 +02:00
Tal Levy 9f1f5fdedc introduce the JSON Processor (#20128)
introduce the JSON Processor
2016-09-09 14:34:32 -07:00
Tal Levy dda32545bb add ignore_missing option to relevant processors (#20194) 2016-09-09 12:20:18 -07:00
Clinton Gormley 9c3007496c Update search-template.asciidoc
Fixed typo
2016-09-09 13:06:49 +02:00
Clinton Gormley 2f6d0119f1 Added warning messages about the dangers of pathological regexes to:
* pattern-replace charfilter
* pattern-capture and pattern-replace token filters
* pattern tokenizer
* pattern analyzer

Relates to #20038
2016-09-09 09:53:07 +02:00
Clinton Gormley add2fbd7b2 Update painless.asciidoc
Asciidoc typo
2016-09-09 09:30:04 +02:00
Lee Hinman 49695af2ac Remove FORCE version_type
This was an error-prone version type that allowed overriding previous
version semantics. It could cause primaries and replicas to be out of
sync however, so it has been removed.

Resolves #19769
2016-09-08 15:29:47 -06:00
Jason Tedor 27ff4f327c Remove allow unquoted JSON
Previous versions of Elasticsearch permitted unquoted JSON field names even though this is against the JSON spec. This leniency was disabled by default in the 5.x series of Elasticsearch but a backwards compatibility layer was added via a system property with the intention of removing this layer in 6.0.0. This commit removes this backwards compatibility layer.

Relates #20388
2016-09-08 13:36:31 -04:00
Clinton Gormley 100cdf0ba9 Use a comment block to comment out release notes 2016-09-08 18:30:37 +02:00
Clinton Gormley 9974e3f3d8 Bumped doc versions to 6.0.0-alpha1 2016-09-08 18:29:18 +02:00
Clinton Gormley 891a15db22 Add breaking changes for 6.0 2016-09-08 18:27:01 +02:00
Clinton Gormley b9d20a039f Comment out release notes section 2016-09-08 18:17:27 +02:00
Clinton Gormley 1f11a5d93c Remove links to release notes 2016-09-08 18:07:39 +02:00
Jason Tedor 21e1ae8611 Remove 5.x release notes
This commit removes the series 5.x release notes from master as that
branch is currently on the 6.x series.
2016-09-08 11:31:28 -04:00
Lee Hinman 40b088d728 Rework documentation example for _all to be less ambigious with numerics 2016-09-08 09:09:48 -06:00
Lee Hinman 6666892038 Default `include_in_all` for numeric-like types to false
This includes:

- All regular numeric types such as int, long, scaled-float, double, etc
- IP addresses
- Dates
- Geopoints and Geoshapes

Relates to #19784
2016-09-08 09:09:48 -06:00
Jason Tedor 8e7dfae7d1 Remove collect payloads parameter
The collect_payloads parameter of the span_near query was previously
deprecated with the intention to be removed. This commit removes this
parameter.

Relates #20385
2016-09-08 09:37:36 -04:00
Lee Hinman b4cc3cd35d Remove FORCE version_type
This was an error-prone version type that allowed overriding previous
version semantics. It could cause primaries and replicas to be out of
sync however, so it has been removed.

Resolves #19769
2016-09-07 13:05:18 -06:00
Nik Everett bebdec570f [docs] Mark percolator response snippets properly
Now the docs tests will catch any errors in the responses. This
would have caught the error fixed in
https://github.com/elastic/elasticsearch/pull/20351
2016-09-07 09:45:50 -04:00
antonisppn e77f4710e4 [docs] Percolator samples are not working. Mapping is wrong.
Hi all, 

I was trying to run the percolate examples, but I figured that because of the "type":"keyword" , the code wasn't working.
In the saerch query the "message" : "A new bonsai tree in the office" is a pure string. 

I changed it to "text".
2016-09-07 08:15:20 -04:00
Alexander Lin f825e8f4cb Exposing lucene 6.x minhash filter. (#20206)
Exposing lucene 6.x minhash tokenfilter

Generate min hash tokens from an incoming stream of tokens that can
be used to estimate document similarity.

Closes #20149
2016-09-07 09:38:12 +02:00
Lee Hinman fd3392aef8 [DOCS] Mark cluster allocation explain API as experimental in docs 2016-09-06 11:29:33 -06:00
Martijn van Groningen 245882cde3 * Removed `script.default_lang` setting and made `painless` the hardcoded default script language.
** The default script language is now maintained in `Script` class.
* Added `script.legacy.default_lang` setting that controls the default language for scripts that are stored inside documents (for example percolator queries).  This defaults to groovy.
** Added `QueryParseContext#getDefaultScriptLanguage()` that manages the default scripting language. Returns always `painless`, unless loading query/search request in legacy mode then the returns what is configured in `script.legacy.default_lang` setting.
** In the aggregation parsing code added `ParserContext` that also holds the default scripting language like `QueryParseContext`. Most parser don't have access to `QueryParseContext`. This is for scripts in aggregations.
* The `lang` script field is always serialized (toXContent).

Closes #20122
2016-09-06 18:44:48 +02:00
Nik Everett e03fb602cd Add CONSOLE places where it is obviously missing
These places already have other annotations like `// TEST` and
`// TESTSETUP` so they are already in console format.
2016-09-06 10:48:19 -04:00
Nik Everett 5cff2a046d Remove most of the need for `// NOTCONSOLE`
and be much more stingy about what we consider a console candidate.

* Add `// CONSOLE` to check-running
* Fix version in some snippets
* Mark groovy snippets as groovy
* Fix versions in plugins
* Fix language marker errors
* Fix language parsing in snippets

  This adds support for snippets who's language is written like
  `[source, txt]` and `["source","js",subs="attributes,callouts"]`.

  This also makes language required for snippets which is nice because
  then we can be sure we can grep for snippets in a particular language.
2016-09-06 10:32:54 -04:00
Martijn van Groningen 6f6d17dc9c ingest: Add `dot_expander` processor that can turn fields with dots in the field name into object fields. 2016-09-05 07:28:38 +02:00
Nik Everett bfd072bc10 Switch more docs to CONSOLE
Related to #18160
2016-09-02 18:22:30 -04:00
Jun Ohtani c4759bcc02 Merge pull request #20285 from johtani/fix/remove_token_filter_param_in_analyze_api
Remove `token_filter` in _analyze API
2016-09-03 02:03:51 +09:00
Areek Zillur c92f82e624 Merge pull request #20169 from areek/doc/fix_completion_breaking_changes
Update breaking changes for completion suggester
2016-09-02 12:39:16 -04:00
Areek Zillur af215b528f move completion performance tips from migration docs to completion docs 2016-09-02 12:37:56 -04:00
Clinton Gormley 4885709e10 Update rollover-index.asciidoc
Fixed weirdly formatted callouts in rollover docs
2016-09-02 16:08:34 +02:00
Greg Ichneumon Brown 639b7278d9 Docs: clarify calculation of sigma and lambda in function_score (#20267)
- Using log() to indicate natural log can add some confusion when trying to further adjust/tweak scores. Other parts of the API (field_value_factor on this same page) use 'ln' and 'log', so this change should be more consistent
- Fixes #20027
- I generated the images using http://latex2png.com/ at a resolution of 150 which seemed to be about the same size as before
2016-09-02 14:41:07 +02:00
Adrien Grand cdc27b75b8 Add more information to the how-to docs. #20297
- use auto-generated ids for indexing #20211
 - use rounded dates in queries #20115
2016-09-02 14:28:47 +02:00
Clinton Gormley 234ca6b1fd Fix bad link in plugin docs 2016-09-02 10:54:05 +02:00
Luca Cavanna faa03ad9fa Merge pull request #20255 from javanna/enhancement/cluster_stats_available_memory
Add mem section back to cluster stats
2016-09-02 10:19:51 +02:00
Jun Ohtani aef2e5d90e Remove `token_filter` in _analyze API
Fix wording in docs
Refactoring RestAnalyzeActionTests using expectThrows()

Closes #20283
2016-09-02 15:08:28 +09:00
Areek Zillur c869ca18eb Update breaking changes for completion suggester
To indicate we removed completion payload option
in favour of returning document source with
completion suggestions
2016-09-01 21:48:30 -04:00
nrichers e5bf02b155 Fix broken link reportedby Twitter user (#20291) 2016-09-01 17:41:44 -07:00
Jack Conradson 7c552f8127 Merge branch 'master' into deprecate 2016-09-01 14:53:49 -07:00
Nik Everett c8b984aee9 Add CONSOLE to more docs
Relates to #18160
2016-09-01 17:08:18 -04:00
Jason Tedor 5fe4cb6adc Size limit deprecation logs
This commit configures the deprecation logs to be size-limited to 1 GB,
and compress these logs when they roll. The default configuration will
preserve up to four rolled logs.

Relates #20287
2016-09-01 14:25:04 -04:00
Florian Hopf 359e76f7e7 Fixed wording 2016-09-01 11:22:44 -06:00
Nik Everett 4b219d15d2 Add CONSOLE to a few snippets in reference docs
This allows them to be run in Console and adds them to the list of
docs that are automatically tested as part of the build.

Relates to #18160
2016-09-01 13:08:36 -04:00
Jun Ohtani 3d9f8ed764 Remove `token_filter` in _analyze API
Remove the param and change docs

Closes #20283
2016-09-02 01:36:45 +09:00
Jack Conradson 71d8ee5eac Merge branch 'master' into deprecate 2016-09-01 08:51:29 -07:00
javanna 5f299ff46f add mem section back to cluster stats
The mem section was buggy in cluster stats and removed. It is now added back with the same structure as in node stats, containing total memory, available memory, used memory and percentages. All the values are the sum of all the nodes across the cluster (or at least the ones that we were able to get the values from).
2016-09-01 11:26:03 +02:00
Simon Willnauer a0becd26b1 Optimize indexing for the autogenerated ID append-only case (#20211)
If elasticsearch controls the ID values as well as the documents
version we can optimize the code that adds / appends the documents
to the index. Essentially we an skip the version lookup for all
documents unless the same document is delivered more than once.

On the lucene level we can simply call IndexWriter#addDocument instead
of #updateDocument but on the Engine level we need to ensure that we deoptimize
the case once we see the same document more than once.

This is done as follows:

1. Mark every request with a timestamp. This is done once on the first node that
receives a request and is fixed for this request. This can be even the
machine local time (see why later). The important part is that retry
requests will have the same value as the original one.

2. In the engine we make sure we keep the highest seen time stamp of "retry" requests.
This is updated while the retry request has its doc id lock. Call this `maxUnsafeAutoIdTimestamp`

3. When the engine runs an "optimized" request comes, it compares it's timestamp with the
current `maxUnsafeAutoIdTimestamp` (but doesn't update it). If the the request
timestamp is higher it is safe to execute it as optimized (no retry request with the same
timestamp has been run before). If not we fall back to "non-optimzed" mode and run the request as a retry one
and update the `maxUnsafeAutoIdTimestamp` unless it's been updated already to a higher value

Relates to #19813
2016-09-01 10:39:40 +02:00
Jack Conradson 1cdfc3ccfe Merge branch 'master' into deprecate 2016-08-31 15:58:14 -07:00
Jack Conradson 3b3baa6e6c Made deprecation of Groovy, Javascript, and Python more explicit. 2016-08-31 15:56:31 -07:00
Jason Tedor 76ab02e002 Merge branch 'master' into log4j2
* master:
  Avoid NPE in LoggingListener
  Randomly use Netty 3 plugin in some tests
  Skip smoke test client on JDK 9
  Revert "Don't allow XContentBuilder#writeValue(TimeValue)"
  [docs] Remove coming in 2.0.0
  Don't allow XContentBuilder#writeValue(TimeValue)
  [doc] Remove leftover from CONSOLE conversion
  Parameter improvements to Cluster Health API wait for shards (#20223)
  Add 2.4.0 to packaging tests list
  Docs: clarify scale is applied at origin+offest (#20242)
2016-08-31 16:37:55 -04:00
Jason Tedor 750033dc4b Update docs for Log4j 2
This commit updates the logging docs for Elasticsearch to reflect the
migration to Log4j 2.
2016-08-31 15:51:52 -04:00
Nik Everett 1128776b3c [docs] Remove coming in 2.0.0
2.0.0's been out for a long time.

Closes #20268
2016-08-31 14:43:04 -04:00
Nik Everett dcaed58f90 [doc] Remove leftover from CONSOLE conversion 2016-08-31 13:14:47 -04:00
Ali Beyad 4641254ea6 Parameter improvements to Cluster Health API wait for shards (#20223)
* Params improvements to Cluster Health API wait for shards

Previously, the cluster health API used a strictly numeric value
for `wait_for_active_shards`. However, with the introduction of
ActiveShardCount and the removal of write consistency level for
replication operations, `wait_for_active_shards` is used for
write operations to represent values for ActiveShardCount. This
commit moves the cluster health API's usage of `wait_for_active_shards`
to be consistent with its usage in the write operation APIs.

This commit also changes `wait_for_relocating_shards` from a
numeric value to a simple boolean value `wait_for_no_relocating_shards`
to set whether the cluster health operation should wait for
all relocating shards to complete relocation.

* Addresses code review comments

* Don't be lenient if `wait_for_relocating_shards` is set
2016-08-31 11:58:19 -04:00
Greg Ichneumon Brown 92c54aa4a1 Docs: clarify scale is applied at origin+offest (#20242)
- fixes #19832
2016-08-31 17:02:59 +02:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Jason Tedor 1a805bb675 Increase visibility of deprecation logger
The deprecation logger is an important way to make visible features of
Elasticsearch that are deprecated. Yet, the default logging makes the
log messages for the deprecation logger invisible. We want these log
messages to be visible, so the default logging for the deprecation
logger should enable these log messages. This commit changes the log
level of deprecation log message to warn, and configures the deprecation
logger so that these log messages are visible out of the box.

Relates #20254
2016-08-31 10:51:17 -04:00
Jack Conradson 2b9c3a1a98 Fixed a doc test. 2016-08-30 16:51:37 -07:00
Jack Conradson c8f57df5cf Merge branch 'master' into deprecate 2016-08-30 09:11:18 -07:00
Jack Conradson 7930233527 Deprecate Groovy, Python, and Javascript scripts. 2016-08-30 09:06:18 -07:00
Chris Earle 6ad92c0f9d [DOCS] Document performRequest being renamed to performRequestAsync
This updates the docs to reflect that the asynchronous variants were renamed to have "Async" at the end.
2016-08-30 11:29:52 -04:00
Nik Everett df73292256 Add an alias action to delete an index
While removing an index isn't actually an alias action, if we add
an alias action that deletes an index then we can delete and index
and add an alias with the same name as the index atomically, in
the same cluster state update.

Closes #20064
2016-08-30 10:15:21 -04:00
Clinton Gormley 70f4d718f8 Update percolate.asciidoc
Tidy up percolator docs
2016-08-30 13:05:02 +02:00
Jim Ferenczi accb636824 Merge pull request #20213 from jimferenczi/painless_list_add
Fix docs that uses += to add an element in a list even though painless does not accept it.
2016-08-30 09:33:59 +02:00
Tanguy Leroux b4245c7ad9 Add exclusion filters support to filter_path
This commit adds the support for exclusion filter to the response filtering (filter_path) feature. It changes the XContentBuilder APIs so that it now accepts two types of filters: inclusive and exclusive. Filters are no more String arrays but sets of String instead.
2016-08-30 09:08:30 +02:00
Nik Everett 9c3f6d58ac Support downgrading keyword/text into string
This changes Elasticsearch to automatically downgrade `text` and
`keyword` fields into appropriate `string` fields when changing the
mapping of indexes imported from 2.x. This allows users to use the
modern, documented syntax against 2.x indexes. It also makes it clear
that reindexing in order to recreate the index in 5.0 is required for
any long lived indexes. This change is useful for the times when you
can't (cluster is just starting, not stable enough for reindex) or
shouldn't (index will only live 90 days or something).
2016-08-29 11:27:37 -04:00
Jim Ferenczi dc663a432b Fix docs that uses += to add an element in a list even though painless does not accept it. 2016-08-29 16:00:11 +02:00
markwalkom a8e7d57a84 Update api-conventions.asciidoc (#20203) 2016-08-29 15:10:20 +02:00
Nik Everett c24ad1df4e Mark lots of stuff `// NOTCONSOLE` 2016-08-29 08:28:16 -04:00
Nik Everett 5357d9d9ce Add `// TESTRESPONSE` to mapper-attachements docs 2016-08-29 08:28:16 -04:00
Munish Goyal 81b815ff76 Correct grammar in parent field doc 2016-08-29 07:51:39 -04:00
Jun Ohtani 2a00c9dc46 Merge pull request #19860 from johtani/fix/validate_empty_field_name
Validate blank field name
2016-08-29 11:52:18 +09:00
Nik Everett 3fe42beb64 Cleanup some docs
Mark one `// NOTCONSOLE`, mark another `[source,painless]`, and
another `// TESTRESPONSE` and fix a bug in it.
2016-08-26 16:01:07 -04:00
Nik Everett 52f23918c2 Use `painless` as language for painless snippets (#20185)
The syntax highlighter does a decent job when you do this. This lets
us `grep` for painless snippets in the docs.

Closes #20025
2016-08-26 15:39:44 -04:00
Nicolas Ruflin 4ab1093564 Add reindex example on how to reindex daily indices (#18654)
This can be a common case with beats in case the template changes between two versions and the old data should be reindex with the new templates.
2016-08-26 13:08:52 -04:00
Josh Becker 3c24ea43fd Docs: Remove extra word from phrase-suggester 2016-08-26 13:02:54 -04:00
Jun Ohtani 450f47d5b5 Validate blank field name
add validation and validate only 5.0+
Add tests before 5.0

Closes #19251
2016-08-26 20:10:33 +09:00
Martijn van Groningen 7c9af98a3c docs: add sort workaround 2016-08-26 10:55:42 +02:00
Dominik Stadler f0db4d9942 Add an example call of how to stop a snapshot or restore operation (#20153) 2016-08-25 13:01:04 +02:00
Igor Motov b36fbc4452 Add support for parameters to the script ingest processor
The script processor should support `params` to be consistent with all other script consumers.
2016-08-24 16:49:48 -04:00
Jim Ferenczi 9bedbbaa6a Fixed doc links 2016-08-24 22:37:59 +02:00
Jim Ferenczi 4682fc34ae Add the ability to disable the retrieval of the stored fields entirely
This change adds a special field named _none_ that allows to disable the retrieval of the stored fields in a search request or in a TopHitsAggregation.

To completely disable stored fields retrieval (including disabling metadata fields retrieval such as _id or _type) use _none_ like this:

````
POST _search
{
   "stored_fields": "_none_"
}
````
2016-08-24 16:40:08 +02:00
Clinton Gormley 9cce45d4af Add cloud to S3/EC2 plugin docs 2016-08-24 15:38:43 +02:00
Simon Willnauer c499427166 Use _refresh instead of reading from Translog in the RT GET case (#20102)
Today we do a lot of accounting inside the engine to maintain locations
of documents inside the transaction log. This is only needed to ensure
we can return the documents source from the engine if it hasn't been refreshed.
Aside of the added complexity to be able to read from the currently writing translog,
maintainance of pointers into the translog this also caused inconsistencies like different values
of the `_ttl` field if it was read from the tlog or not. TermVectors are totally different if
the document is fetched from the tranlog since copy fields are ignored etc.

This chance will simply call `refresh` if the documents latest version is not in the index. This
streamlines the semantics of the `_get` API and allows for more optimizations inside the engine
and on the transaction log. Note: `_refresh` is only called iff the requested document is not refreshed
yet but has recently been updated or added.

#Relates to #19787
2016-08-24 15:30:08 +02:00
Jason Tedor 90c2aae28b Remove minimum master nodes bootstrap docs
The minimum master nodes bootstrap check was removed in
069fc22696 but the docs were left behind.
This commit removes these stale docs.

Relates #20127
2016-08-23 14:31:58 -04:00
ddddn 6228b002c5 Update index_.asciidoc (#20125) 2016-08-23 20:08:52 +02:00
Nik Everett 5b34bec92a Add deprecation warnings to docs for geohash
Relates to #20126
2016-08-23 13:43:35 -04:00