Commit Graph

5691 Commits

Author SHA1 Message Date
lcawl 8a341a3ea5 [DOCS] Fix link to ML node description 2019-06-13 13:56:06 -07:00
Lisa Cawley 7b90ceae0c [DOCS] Update node descriptions for default distribution (#42812) 2019-06-13 13:55:56 -07:00
Jason Tedor 5bc3b7f741
Enable node roles to be pluggable (#43175)
This commit introduces the possibility for a plugin to introduce
additional node roles.
2019-06-13 15:15:48 -04:00
Ryan Ernst c3ce3f6891 Add native code info to ML info api (#43172)
The machine learning feature of xpack has native binaries with a
different commit id than the rest of code. It is currently exposed in
the xpack info api. This commit adds that commit information to the ML
info api, so that it may be removed from the info api.
2019-06-13 11:38:58 -07:00
Luca Cavanna a28569462f Add 6.8 to the remote clusters compatibility table (#42389)
The table does not include 6.8 as it was written before we knew we were releasing it. This commit adds it.
2019-06-13 11:30:35 +02:00
Mirek Svoboda afbb791969 Document wildcard for network interfaces (#28839)
With this commit we mention how Elasticsearch behaves when
either `0` or `0.0.0.0` is used for `network.host`.
2019-06-13 10:18:49 +02:00
Lisa Cawley 7c9acdb0ac [DOCS] Adds ML release highlights (#43169) 2019-06-12 13:44:59 -07:00
James Baiera 51618af056
shrink may full copy when using multi data paths (#42913) (#42961)
Additional scenario for full segment copy if hard link
cannot work across disks.
2019-06-12 14:34:31 -04:00
Lisa Cawley 7f2f0b7620 [DOCS] Adds dataframe authorization details (#43009) 2019-06-12 10:17:24 -07:00
Shaunak Kashyap 5ae2460782
[7.x] Metricbeat monitoring Elasticsearch: Reorder/remove steps (#42917) (#43130) 2019-06-12 06:25:30 -07:00
Luca Cavanna 4da0fadedc [DOCS] Clarify phrase suggester docs smoothing parameter (#42947)
Closes #28512
2019-06-12 11:25:03 +02:00
Luca Cavanna e538592652 Update max_concurrent_shard_request parameter docs (#42227)
Some of the docs were outdated as they did not mention that the limit is
not per node. Also, The default value changed.

Relates to #31206
2019-06-12 11:25:03 +02:00
markharwood a75964d8fd Docs change for exists query. (#43092)
Now emphasises the test is for indexed values.
Previous documentation only mentioned the state of the input JSON doc (null values) but this is only one of several reasons why an indexed value may not exist.

Closes #24256
2019-06-12 09:28:18 +01:00
Ryan Ernst 172cd4dbfa Remove description from xpack feature sets (#43065)
The description field of xpack featuresets is optionally part of the
xpack info api, when using the verbose flag. However, this information
is unnecessary, as it is better left for documentation (and the existing
descriptions describe anything meaningful). This commit removes the
description field from feature sets.
2019-06-11 09:22:58 -07:00
markharwood b17fbe2933 Docs enhancement for quote_field_suffix. (#43093)
* Docs enhancement for quote_field_suffix.
Mentions the use of a fall-back field when specified field is missing.
Closes #40778
2019-06-11 16:33:12 +01:00
Andrei Stefan 8de65daa45 Rename TESTRESPONSE[_cat] to TESTRESPONSE[non_json] (#43087)
(cherry picked from commit 897b24e0563f59c03e85096fdb64cbc1dd1a5d60)
2019-06-11 12:40:00 +03:00
Andrei Stefan 5b35ec1d9b Restructure the SQL Language section to have proper sub-sections (#43007)
Rest docs page update
- have the section be on separate pages
- add an Overview page
- add other formats examples

(cherry picked from commit 309bd691ff3f8625f67ca09fc1dd8e265f7e6c92)
2019-06-11 12:39:59 +03:00
Andrei Stefan 4a3287836d SQL: Clarify that the connections the jdbc driver creates are not pooled (#42992)
(cherry picked from commit 406d5281bdfe682fb7ec9fefcdb61cce1b9e7270)
2019-06-11 12:39:58 +03:00
Benjamin Trent 79052050bf
[ML] Adding support for geo_shape, geo_centroid, geo_point in datafeeds (#42969) (#43069)
* [ML] Adding support for geo_shape, geo_centroid, geo_point in datafeeds

* only supporting doc_values for geo_point fields

* moving validation into GeoPointField ctor
2019-06-10 21:52:53 -05:00
James Rodewig 5913723788 [DOCS] Change `// TESTRESPONSE[_cat]` to `// TESTRESPONSE[non_json]` (#43006) 2019-06-10 09:53:05 -04:00
Sachin Frayne 44aedcf97a Correct the description of generate_word_parts (#43026) 2019-06-10 11:36:31 +01:00
Sam Mingo 12962ee0a7 Update search-settings.asciidoc (#43016)
Grammar and spelling fixes
2019-06-10 10:14:03 +01:00
Shubham Vipul Majmudar b2e7045b50 Update regexp-syntax.asciidoc (#43021)
Corrects a typo.
2019-06-10 10:13:54 +01:00
Jason Tedor b96ed1f9f7
Add note to CCR docs about mapping/alias updates
This commit adds a note to the docs clarifying that it is not possible
to manually update the mapping nor the aliases of a follower index.
2019-06-09 22:57:23 -04:00
Jason Tedor 25ca315d78
Add note to CCR docs regarding alias replication
This commit adds a note to the docs regarding the automatic replication
of aliases by a follower index from its leader index.
2019-06-09 22:55:20 -04:00
James Rodewig 5342616a23 [DOCS] Add explicit `articles_case` parameter to Elision Token Filter example (#42987) 2019-06-07 11:24:43 -04:00
Henning Andersen dea935ac31
Reindex max_docs parameter name (#42942)
Previously, a reindex request had two different size specifications in the body:
* Outer level, determining the maximum documents to process
* Inside the source element, determining the scroll/batch size.

The outer level size has now been renamed to max_docs to
avoid confusion and clarify its semantics, with backwards compatibility and
deprecation warnings for using size.
Similarly, the size parameter has been renamed to max_docs for
update/delete-by-query to keep the 3 interfaces consistent.

Finally, all 3 endpoints now support max_docs in both body and URL.

Relates #24344
2019-06-07 12:16:36 +02:00
James Rodewig 2de919e3a8 [DOCS] Move 'Scripting' section to top-level navigation. (#42939) 2019-06-06 10:46:02 -04:00
James Rodewig ed186b4485 [DOCS] Rewrite terms query (#42889) 2019-06-06 08:33:52 -04:00
David Roberts b202a59f88 [ML] Add earliest and latest timestamps to field stats (#42890)
This change adds the earliest and latest timestamps into
the field stats for fields of type "date" in the output of
the ML find_file_structure endpoint.  This will enable the
cards for date fields in the file data visualizer in the UI
to be made to look more similar to the cards for date
fields in the index data visualizer in the UI.
2019-06-06 08:58:35 +01:00
Gordon Brown 6eb4600e93
Add custom metadata to snapshots (#41281)
Adds a metadata field to snapshots which can be used to store arbitrary
key-value information. This may be useful for attaching a description of
why a snapshot was taken, tagging snapshots to make categorization
easier, or identifying the source of automatically-created snapshots.
2019-06-05 17:30:31 -06:00
Christoph Büscher 99542e66a6 [Docs] Clarify caveats for phonetic filters replace option (#42807)
The `replace` option in the phonetic token filter can have suprising side
effects, e.g. such as described in #26921. This PR adds a note to be mindful
about such scenarios and offers alternatives to using the `replace` option.

Closes #26921
2019-06-05 22:03:54 +02:00
Lisa Cawley 757c6a45a0 [DOCS] Adds discovery.type (#42823)
Co-Authored-By: David Turner <david.turner@elastic.co>
2019-06-05 12:37:17 -07:00
Dimitrios Liappis 00f01aaece Clarify heap setting in Docker docs (#42754)
Add note in the Docker docs that even when container memory is limited,
we still require specifying -Xms/-Xmx using one of the supported
methods.
2019-06-05 09:44:43 +03:00
Jason Tedor 117df87b2b
Replicate aliases in cross-cluster replication (#42875)
This commit adds functionality so that aliases that are manipulated on
leader indices are replicated by the shard follow tasks to the follower
indices. Note that we ignore write indices. This is due to the fact that
follower indices do not receive direct writes so the concept is not
useful.

Relates #41815
2019-06-04 20:36:24 -04:00
James Rodewig 783159dcbc [DOCS] Fix typo in bucket script aggregation link 2019-06-04 09:40:38 -04:00
James Rodewig d050c52fd1 [DOCS] Fix broken bucket script agg link 2019-06-04 08:43:38 -04:00
Christoph Büscher d9c582e66b [Docs] Add to preference parameter docs (#42797)
Adding notes to the existing docs about how using `preference` might increase
request cache utilization but also add warning about the downsides.

Closes #24278
2019-06-04 14:38:18 +02:00
Benjamin Trent 32eae0dfe9
[ML] [Data Frame] Adding supported aggs in docs (#42728) (#42842)
* [ML] [Data Frame] Adding supported aggs in docs

* [DOCS] Moves pivot to definitions list
2019-06-04 07:19:58 -05:00
David Turner 9f470c20ed More improvements to cluster coordination docs (#42799)
This commit addresses a few more frequently-asked questions:

* clarifies that bootstrapping doesn't happen even after a full cluster
  restart.

* removes the example that uses IP addresses, to try and further encourage the
  use of node names for bootstrapping.

* clarifies that auto-bootstrapping might form different clusters on different
  hosts, and gives a process for starting again if this wasn't what you wanted.

* adds the "do not stop half-or-more of the master-eligible nodes" slogan that
  was notably absent.

* reformats one of the console examples to a narrower width
2019-06-04 08:25:41 +01:00
Marios Trivyzas eab88354f2
[Docs] Fix reference to `boost` and `slop` params (#42803)
For `multi_match` query: link `boost` param to the generic reference
for query usage and `slop` to the `match_phrase` query where its usage
is documented.

Fixes: #40091
(cherry picked from commit 69993049a8bd9e7f042935729fe69a8266d95a0a)
2019-06-03 22:57:19 +02:00
Marios Trivyzas 3b42dde64f
[Docs] Add note for date patterns used for index search. (#42810)
Add an explanatory NOTE section to draw attention to the difference
between small and capital letters used for the index date patterns.
e.g.: HH vs hh, MM vs mm.

Closes: #22322
(cherry picked from commit c8125417dc33215651f9bb76c9b1ffaf25f41caf)
2019-06-03 22:27:19 +02:00
Marios Trivyzas 6c50246a58
SQL: [Docs] Fix links syntax (#42806)
Fix a couple of wrong links because of the order of the anchor
and the usage of backquotes.

(cherry picked from commit 4e0c6525153b60a57202937c2ae57968c8e35285)
2019-06-03 17:51:19 +02:00
David Roberts b61202b0a8 [ML] Add a limit on line merging in find_file_structure (#42501)
When analysing a semi-structured text file the
find_file_structure endpoint merges lines to form
multi-line messages using the assumption that the
first line in each message contains the timestamp.
However, if the timestamp is misdetected then this
can lead to excessive numbers of lines being merged
to form massive messages.

This commit adds a line_merge_size_limit setting
(default 10000 characters) that halts the analysis
if a message bigger than this is created.  This
prevents significant CPU time being spent subsequently
trying to determine the internal structure of the
huge bogus messages.
2019-06-03 13:45:51 +01:00
Christian Kotzbauer 929215c0d5
Update release-notes.asciidoc (#42779) 2019-06-01 08:18:00 -04:00
Julie Tibshirani 3a00d08c50 Clarify that inner_hits must be used to access nested fields. (#42724)
This PR updates the docs for `docvalue_fields` and `stored_fields` to clarify
that nested fields must be accessed through `inner_hits`. It also tweaks the
nested fields documentation to make this point more visible.

Addresses #23766.
2019-05-31 10:06:11 -07:00
James Rodewig f51f8ed04c [DOCS] Remove unneeded options from `[source,sql]` code blocks (#42759)
In AsciiDoc, `subs="attributes,callouts,macros"` options were required
to render `include-tagged::` in a code block.

With elastic/docs#827, Elasticsearch Reference documentation migrated
from AsciiDoc to Asciidoctor.

In Asciidoctor, the `subs="attributes,callouts,macros"` options are no
longer needed to render `include-tagged::` in a code block. This commit
removes those unneeded options.

Resolves #41589
2019-05-31 13:05:13 -04:00
James Rodewig 0a37dd7a86 [DOCS] Remove unneeded `ifdef::asciidoctor[]` conditionals (#42758)
Several `ifdef::asciidoctor` conditionals were added so that AsciiDoc
    and Asciidoctor doc builds rendered consistently.

    With https://github.com/elastic/docs/pull/827, Elasticsearch Reference
    documentation migrated completely to Asciidoctor. We no longer need to
    support AsciiDoc so we can remove these conditionals.

    Resolves #41722
2019-05-31 11:08:54 -04:00
James Rodewig 478919c0bb [DOCS] Remove unneeded `ifdef::asciidoctor[]` conditionals (#42758)
Several `ifdef::asciidoctor` conditionals were added so that AsciiDoc
and Asciidoctor doc builds rendered consistently.

With https://github.com/elastic/docs/pull/827, Elasticsearch Reference
documentation migrated completely to Asciidoctor. We no longer need to
support AsciiDoc so we can remove these conditionals.

Resolves #41722
2019-05-31 11:05:44 -04:00
Marios Trivyzas 01446ff4bd
[Docs] Mention search related deprecations (#42751)
Add deprecation entries for 7.3 regarding `common` query
and `cutoff_frequency` parameter.

Follows: #42691
2019-05-31 12:56:07 +02:00
Alex Pang 5f9382acc2 Fix docs typo in the certutil CSR mode (#42593)
Changes the mention of `cert` to `csr`.

Co-Authored-By: Alex Pang <pangyikhei+github@gmail.com>
2019-05-31 01:03:43 +03:00
Lisa Cawley d83b91d56a [DOCS] Disable Metricbeat system module (#42601) 2019-05-30 12:19:48 -07:00
Julie Tibshirani 1bb505c70d Clarify the settings around limiting nested mappings. (#42686)
* Previously, we mentioned multiple times that each nested object was indexed as its own document. This is repetitive, and is also a bit confusing in the context of `index.mapping.nested_fields.limit`, as that applies to the number of distinct `nested` types in the mappings, not the number of nested objects. We now just describe the issue once at the beginning of the section, to illustrate why `nested` types can be expensive.
* Reference the ongoing example to clarify the meaning of the two settings.

Addresses #28363.
2019-05-30 10:36:38 -07:00
Marios Trivyzas ce30afcd01
Deprecate CommonTermsQuery and cutoff_frequency (#42619) (#42691)
Since the max_score optimization landed in Elasticsearch 7,
the CommonTermsQuery is redundant and slower. Moreover the
cutoff_frequency parameter for MatchQuery and MultiMatchQuery
is redundant.

Relates to #27096

(cherry picked from commit 04b74497314eeec076753a33b3b6cc11549646e8)
2019-05-30 18:04:47 +02:00
Mayya Sharipova 5a76f46ac6 Fix error with mapping in docs
Related to #39630
2019-05-30 10:28:09 -04:00
Peter Dyson b84b5525e1 [DOCS] path_hierarchy tokenizer examples (#39630)
Closes #17138
2019-05-30 09:17:55 -04:00
James Rodewig 67326252d8 [DOCS] Rewrite 'wildcard' query (#42670) 2019-05-30 08:31:27 -04:00
Mayya Sharipova 5e02dc6878 Add warning scores are floats (#42667) 2019-05-29 16:49:04 -04:00
lcawl 78f280de9c [DOCS] Adds more monitoring tagged regions 2019-05-29 11:21:13 -07:00
James Rodewig 3193dfa8e6 [DOCS] Set explicit anchors for TLS/SSL settings (#42524) 2019-05-29 08:25:37 -04:00
Hendrik Muhs 345ff21ae5 [ML-DataFrame] rewrite start and stop to answer with acknowledged (#42589)
rewrite start and stop to answer with acknowledged

fixes #42450
2019-05-29 11:14:32 +02:00
Julie Tibshirani 8b325164f9 Fix a callout in the field alias docs. 2019-05-28 17:49:44 -07:00
James Rodewig e54e74852a [DOCS] Fix X-Pack tag for Asciidoctor (#42443) 2019-05-28 15:19:31 -04:00
James Rodewig 54d194409e [DOCS] Set explicit anchors for Asciidoctor (#42521) 2019-05-28 14:21:00 -04:00
James Rodewig ee1e4db266
[DOCS] Set literal anchors for Asciidoctor (#42462) 2019-05-28 14:16:18 -04:00
Lisa Cawley 77fc7b2107 [DOCS] Reorg monitoring configuration for re-use (#42547) 2019-05-28 09:13:00 -07:00
lcawl 8ff37e99f5 [DOCS] Removes coming tags 2019-05-28 08:58:41 -07:00
Benjamin Trent d06618a70d
[ML] adding delayed_data_check_config to datafeed update docs (#42095) (#42626)
* [ML] adding delayed_data_check_config to datafeed update docs

* [DOCS] Edits delayed data configuration details
2019-05-28 11:36:30 -04:00
James Rodewig 31d2bdca37
[DOCS] Fix Moving Avg Aggregation `deprecated` macro for Asciidoctor (#42405) 2019-05-28 08:56:50 -04:00
James Rodewig b30ca8da28 [DOCS] Fix API Quick Reference rollup attribute for Asciidoctor (#42403) 2019-05-28 08:53:20 -04:00
James Rodewig 3079d2d295 [DOCS] Escape cross-ref link comma for Asciidoctor (#42402) 2019-05-28 08:47:51 -04:00
Travis Steel 381e100217 Fixed typo in docker.asciidoc (#42455) 2019-05-27 11:54:56 +02:00
bellengao 380f296631 Update script-fields.asciidoc (#42490) 2019-05-27 11:48:37 +02:00
Julie Tibshirani 3a6c2525ca
Deprecate support for chained multi-fields. (#42330)
This PR contains a straight backport of #41926, and also updates the
migration documentation and deprecation info API for 7.x.
2019-05-24 15:55:06 -07:00
James Rodewig d521a88e19 [DOCS] Move callouts to end of line for Asciidoctor migration (#42356) 2019-05-24 15:03:46 -04:00
David Roberts 09e8910b0f [DOCS] Adding ML-specific prerequisites to setup docs (#42529) 2019-05-24 10:49:41 -07:00
James Rodewig 43dd081e22 [DOCS] Fix nested def list for Asciidoctor (#42353) 2019-05-24 13:39:49 -04:00
Simon Willnauer 46ccfba808 Remove IndexStore and DirectoryService (#42446)
Both of these classes are basically a bloated wrapper around a simple
construct that can simply be a DirectoryFactory interface. This change
removes both classes and replaces them with a simple stateless interface
that creates a new `Directory` per shard. The concept of `index.store` is preserved
since it makes sense from a configuration perspective.
2019-05-24 12:14:56 +02:00
David Roberts f472186b9f [ML] Improve file structure finder timestamp format determination (#41948)
This change contains a major refactoring of the timestamp
format determination code used by the ML find file structure
endpoint.

Previously timestamp format determination was done separately
for each piece of text supplied to the timestamp format finder.
This had the drawback that it was not possible to distinguish
dd/MM and MM/dd in the case where both numbers were 12 or less.
In order to do this sensibly it is best to look across all the
available timestamps and see if one of the numbers is greater
than 12 in any of them.  This necessitates making the timestamp
format finder an instantiable class that can accumulate evidence
over time.

Another problem with the previous approach was that it was only
possible to override the timestamp format to one of a limited
set of timestamp formats.  There was no way out if a file to be
analysed had a timestamp that was sane yet not in the supported
set.  This is now changed to allow any timestamp format that can
be parsed by a combination of these Java date/time formats:
yy, yyyy, M, MM, MMM, MMMM, d, dd, EEE, EEEE, H, HH, h, mm, ss,
a, XX, XXX, zzz
Additionally S letter groups (fractional seconds) are supported
providing they occur after ss and separated from the ss by a dot,
comma or colon.  Spacing and punctuation is also permitted with
the exception of the question mark, newline and carriage return
characters, together with literal text enclosed in single quotes.

The full list of changes/improvements in this refactor is:

- Make TimestampFormatFinder an instantiable class
- Overrides must be specified in Java date/time format - Joda
  format is no longer accepted
- Joda timestamp formats in outputs are now derived from the
  determined or overridden Java timestamp formats, not stored
  separately
- Functionality for determining the "best" timestamp format in
  a set of lines has been moved from TextLogFileStructureFinder
  to TimestampFormatFinder, taking advantage of the fact that
  TimestampFormatFinder is now an instantiable class with state
- The functionality to quickly rule out some possible Grok
  patterns when looking for timestamp formats has been changed
  from using simple regular expressions to the much faster
  approach of using the Shift-And method of sub-string search,
  but using an "alphabet" consisting of just 1 (representing any
  digit) and 0 (representing non-digits)
- Timestamp format overrides are now much more flexible
- Timestamp format overrides that do not correspond to a built-in
  Grok pattern are mapped to a %{CUSTOM_TIMESTAMP} Grok pattern
  whose definition is included within the date processor in the
  ingest pipeline
- Grok patterns that correspond to multiple Java date/time
  patterns are now handled better - the Grok pattern is accepted
  as matching broadly, and the required set of Java date/time
  patterns is built up considering all observed samples
- As a result of the more flexible acceptance of Grok patterns,
  when looking for the "best" timestamp in a set of lines
  timestamps are considered different if they are preceded by
  a different sequence of punctuation characters (to prevent
  timestamps far into some lines being considered similar to
  timestamps near the beginning of other lines)
- Out-of-the-box Grok patterns that are considered now include
  %{DATE} and %{DATESTAMP}, which have indeterminate day/month
  ordering
- The order of day/month in formats with indeterminate day/month
  order is determined by considering all observed samples (plus
  the server locale if the observed samples still do not suggest
  an ordering)

Relates #38086
Closes #35137
Closes #35132
2019-05-24 09:10:08 +01:00
Adrien Grand f3c33d6d96 Add 7.1.1 release notes. 2019-05-24 09:26:04 +02:00
Costin Leau 9fdf4215dd Docs: Documentation for the upcoming SQL support of frozen indices (#41863)
(cherry picked from commit a3cc03eb1503df24c1706a721fcc9af38c3b2873)
(cherry picked from commit f42dcf2ffd7bd25f3f91aa6127515f393cd1860f)
2019-05-23 21:16:16 +03:00
Yannick Welsch f57fdc57e9
Deprecate max_local_storage_nodes (#42426)
Allows this setting to be removed in 8.0, see #42428
2019-05-23 15:59:55 +02:00
swstepp 4181c5ccf5 Fix grammar problem in stemming reference. (#42148) 2019-05-22 09:50:30 -07:00
Julie Tibshirani a3caed2bee Fix a rendering issue in the geo envelope docs. (#42332)
Previously the formatting information didn't display in the docs, and the
sentence just rendered as "bounding rectangle in the format :".
2019-05-22 09:49:58 -07:00
Alpar Torok eb1639c5fc TestClusters: Convert docs (#42100)
* TestClusters: Convert docs
2019-05-22 14:44:08 +03:00
Jack Conradson 813db163d8 Reorganize Painless doc structure (#42303) 2019-05-21 10:50:21 -07:00
Glen Smith a6204a5eaf Remove stray back tick that's messing up table format (#41705) 2019-05-21 09:00:06 -04:00
Mayya Sharipova 216c74d10a Add experimental and warnings to vector functions (#42205) 2019-05-21 06:39:05 -04:00
David Turner 7abeaba8bb Prevent in-place downgrades and invalid upgrades (#41731)
Downgrading an Elasticsearch node to an earlier version is unsupported, because
we do not make any attempt to guarantee that a node can read any of the on-disk
data written by a future version. Yet today we do not actively prevent
downgrades, and sometimes users will attempt to roll back a failed upgrade with
an in-place downgrade and get into an unrecoverable state.

This change adds the current version of the node to the node metadata file, and
checks the version found in this file against the current version at startup.
If the node cannot be sure of its ability to read the on-disk data then it
refuses to start, preserving any on-disk data in its upgraded state.

This change also adds a command-line tool to overwrite the node metadata file
without performing any version checks, to unsafely bypass these checks and
recover the historical and lenient behaviour.
2019-05-21 08:04:30 +01:00
Jake Landis df8fef3c1a
fix assumption that 6.7 is last 6.x release (#42255) 2019-05-20 14:35:28 -05:00
Jake Landis 87bff89500
7.1.0 release notes forward port (#42252)
Forward port of #42208
2019-05-20 14:39:17 -04:00
Zachary Tong 6ae6f57d39
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.

This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed.  And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit).  This arrangement is very
error-prone for users.

This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.

The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.

The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Jay Modi dbbdcea128
Update ciphers for TLSv1.3 and JDK11 if available (#42082)
This commit updates the default ciphers and TLS protocols that are used
when the runtime JDK supports them. New cipher support has been
introduced in JDK 11 and 12 along with performance fixes for AES GCM.
The ciphers are ordered with PFS ciphers being most preferred, then
AEAD ciphers, and finally those with mainstream hardware support. When
available stronger encryption is preferred for a given cipher.

This is a backport of #41385 and #41808. There are known JDK bugs with
TLSv1.3 that have been fixed in various versions. These are:

1. The JDK's bundled HttpsServer will endless loop under JDK11 and JDK
12.0 (Fixed in 12.0.1) based on the way the Apache HttpClient performs
a close (half close).
2. In all versions of JDK 11 and 12, the HttpsServer will endless loop
when certificates are not trusted or another handshake error occurs. An
email has been sent to the openjdk security-dev list and #38646 is open
to track this.
3. In JDK 11.0.2 and prior there is a race condition with session
resumption that leads to handshake errors when multiple concurrent
handshakes are going on between the same client and server. This bug
does not appear when client authentication is in use. This is
JDK-8213202, which was fixed in 11.0.3 and 12.0.
4. In JDK 11.0.2 and prior there is a bug where resumed TLS sessions do
not retain peer certificate information. This is JDK-8212885.

The way these issues are addressed is that the current java version is
checked and used to determine the supported protocols for tests that
provoke these issues.
2019-05-20 09:45:36 -04:00
Lisa Cawley fd2d4d761b [DOCS] Updates TLS configuration info (#41983) 2019-05-20 09:13:37 -04:00
Nhat Nguyen 1362944c23 Minor improvement translog docs (#42184)
Closes #42183
2019-05-19 20:45:34 -04:00
David Turner 51376f98a7 Clarify rolling upgrade fallback to restart upgrade (#42161)
Adds a note that restarting half-or-more of the master-eligible nodes means
you're no longer doing a rolling upgrade, and may need to upgrade all the
things before the cluster returns to health.
2019-05-16 13:38:48 -04:00
Hendrik Muhs 4063701f5e [DOCS] add a warning about bypassing PUT API's, update example responses (#42062)
Configurations are stored in the .data-frame-internal-1
index, but users should not add configurations directly to
the index as additional information to enable access control
is added. This adds a warning against allowing access to the
internal index.
2019-05-16 10:12:19 -04:00
Ryan Ernst fa1d1d1f57 Deprecate the native realm migration tool (#42142)
The migrate tool was added when the native realm was created, to aid
users in converting from file realms that were per node, into the
cluster managed native realm. While this tool was useful at the time,
users should now be using the native realm directly. This commit
deprecates the tool, to be removed in a followup for 8.0.
2019-05-16 09:52:31 -04:00
Igor Motov 2f8c5ac6f8 Docs: Mark SQL Geo functionality as beta (#42138)
Adds beta marker to geosql documentation
2019-05-15 10:51:33 -04:00
David Turner 15fd233ae3 Minor cluster coordination docs fixes (#42111)
Fixes a typo and a badly-formatted warning.
2019-05-15 09:27:08 -04:00
Igor Motov 70ea3cf847
SQL: Add initial geo support (#42031) (#42135)
Adds an initial limited implementations of geo features to SQL. This implementation is based on the [OpenGIS® Implementation Standard for Geographic information - Simple feature access](http://www.opengeospatial.org/standards/sfs), which is the current standard for GIS system implementation. This effort is concentrate on SQL option AKA ISO 19125-2. 

Queries that are supported as a result of this initial implementation

Metadata commands

- `DESCRIBE table`  - returns the correct column types `GEOMETRY` for geo shapes and geo points.
- `SHOW FUNCTIONS` - returns a list that includes supported `ST_` functions
- `SYS TYPES` and `SYS COLUMNS` display correct types `GEO_SHAPE` and `GEO_POINT` for geo shapes and geo points accordingly. 

Returning geoshapes and geopoints from elasticsearch

- `SELECT geom FROM table` - returns the geoshapes and geo_points as libs/geo objects in JDBC or as WKT strings in console.
- `SELECT ST_AsWKT(geom) FROM table;` and `SELECT ST_AsText(geom) FROM table;`- returns the geoshapes ang geopoints in their WKT representation;

Using geopoints to elasticsearch

- The following functions will be supported for geopoints in queries, sorting and aggregations: `ST_GeomFromText`, `ST_X`, `ST_Y`, `ST_Z`, `ST_GeometryType`, and `ST_Distance`. In most cases when used in queries, sorting and aggregations, these function are translated into script. These functions can be used in the SELECT clause for both geopoints and geoshapes. 
- `SELECT * FROM table WHERE ST_Distance(ST_GeomFromText(POINT(1 2), point) < 10;` - returns all records for which `point` is located within 10m from the `POINT(1 2)`. In this case the WHERE clause is translated into a range query.

Limitations:

Geoshapes cannot be used in queries, sorting and aggregations as part of this initial effort. In order to fully take advantage of geoshapes we would need to have access to geoshape doc values, which is coming in #37206. `ST_Z` cannot be used on geopoints in queries, sorting and aggregations since we don't store altitude in geo_point doc values.

Relates to #29872
Backport of #42031
2019-05-14 18:57:12 -05:00
James Rodewig 58f2e91684 [DOCS] Rewrite 'rewrite' parameter docs (#42018) 2019-05-13 08:43:12 -04:00
Jason Tedor cd5f1b53e8
Remove reference to fs.data.spins in docs
We long ago removed fs.data.spins from the nodes stats. This commit
removes reference to this in the docs.
2019-05-10 11:49:01 -04:00
David Turner 1be5bb5bfd Recognise direct buffers in heap size docs (#42070)
This commit slightly reworks the recommendations in the docs about setting the
heap size:

* the "rules of thumb" are actually instructions that should be followed

* the reason for setting `Xmx` to 50% of the heap size is more subtle than just
  leaving space for the filesystem cache

* it is normal to see Elasticsearch using more memory than `Xmx`

* replace `cutoff` and `limit` with `threshold` since all three terms are used
  interchangeably

* since we recommend setting `Xmx` equal to `Xms`, avoid talking about setting
  `Xmx` in isolation

Relates #41954
2019-05-10 13:56:47 +01:00
James Rodewig 732ef15f0d [DOCS] Adds placeholder for 7.1.0 release notes (#42024) 2019-05-09 13:17:04 -04:00
James Rodewig ea5019665a [DOCS] Replace table with def list for ids query (#41865) 2019-05-09 09:52:20 -04:00
Alexander Reelsen 8e33a5292a Add HTML strip processor (#41888)
This processor uses the lucene HTMLStripCharFilter class to remove HTML
entities from a field. This adds to the char filter, so that there is
possibility to store the stripped version as well.

Note, that the characeter filter replaces tags with a newline, so that
the produced HTML will look slightly different than the incoming HTML
with regards to newlines.
2019-05-09 13:01:07 +02:00
Flavio Pompermaier 83fef23fd1
Fix wrong property name (#40636) 2019-05-09 08:53:05 +02:00
Gordon Brown 4358cc6ac8
Add note about ILM action ordering (#41771)
Adds a note clarifying that actions are ordered automatically.
2019-05-08 16:42:50 -06:00
David Turner 60f84a2eb2 Remove mention of bulk threadpool in examples (#41935)
The `bulk` threadpool is now called `write`, but `bulk` is still
used in some examples. This commit fixes that.

Also, the only way `threadpool.bulk.write: 30` is a valid increase in the size
of this threadpool is if you have 29 processors, which is an odd number of
processors to have. This commit removes the "more threads" bit.
2019-05-08 12:14:23 +01:00
David Turner 99b5a27ea0 Node names in bootstrap config have no ports (#41569)
In cases where node names and transport addresses can be muddled, it is unclear
that `cluster.initial_master_nodes: master-a:9300` means to look for a node
called `master-a:9300` rather than a node called `master-a` with transport port
`9300`. This commit adds docs to that effect.
2019-05-08 10:38:40 +01:00
Yannick Welsch 818e05c05f Highlight the use of single-node discovery in docker docs (#41241)
Relates to https://discuss.elastic.co/t/es-7-and-docker-single-node-cluster/176585
2019-05-08 09:38:37 +02:00
David Turner 4c909e93bb
Reject port ranges in `discovery.seed_hosts` (#41905)
Today Elasticsearch accepts, but silently ignores, port ranges in the
`discovery.seed_hosts` setting:

```
discovery.seed_hosts: 10.1.2.3:9300-9400
```

Silently ignoring part of a setting like this is trappy. With this change we
reject seed host addresses of this form.

Closes #40786
Backport of #41404
2019-05-08 08:34:32 +01:00
Tim Vernum e04953a2bf
Clarify settings in default SSL/TLS (#41930)
The settings listed under the "Default values for TLS/SSL settings"
heading are not actual settings, rather they are common suffixes that
are used for settings that exist in a variety of contexts.

This commit changes the way they are presented to reduce this
confusion.

Backport of: #41779
2019-05-08 16:07:21 +10:00
Marios Trivyzas d5b0badeb7
SQL: Remove CircuitBreaker from parser (#41835)
The CircuitBreaker was introduced as means of preventing a
`StackOverflowException` during the build of the AST by the parser.

The ANTLR4 grammar causes a weird behaviour for a Parser Listener.
The `enterEveryRule()` method is often called with a different parsing
context than the respective `exitEveryRule()`. This makes it difficult
to keep track of the tree's depth, and a custom Map was used as an
attempt of matching the contextes as they are encounter during `enter`
and during `exit` of the rules.

This approach had 2 important drawbacks:
1. It's hard to maintain this custom Map as the grammar changes.
2. The CircuitBreaker could often lead to false positives which caused
valid queries to return an Exception and prevent them from executing.

So, this removes completely the CircuitBreaker which is replaced be
a simple handling of the `StackOverflowException`

Fixes: #41471
(cherry picked from commit 1559a8e2dbd729138b52e89b7e80264c9f4ad1e7)
2019-05-07 23:25:37 +03:00
James Rodewig 77f634ba25 [DOCS] Rewrite `exists` query docs (#41868) 2019-05-07 09:23:20 -04:00
James Rodewig f5bb1cc65a [DOCS] Fix typo referring to multi search API 2019-05-07 09:08:36 -04:00
Jason Tedor d7fd51a84e
Provide names for all artifact repositories (#41857)
This commit adds a name for each Maven and Ivy repository used in the
build.
2019-05-07 06:35:28 -04:00
Alan Woodward 3a35427b6d Improvements to docs around multiplexer and synonyms (#41645)
This commit fixes a multiplexer doc error concerning synonyms, and adds
suggestions on how to combine the two filters.
2019-05-07 09:10:14 +01:00
Julie Tibshirani a90aac1c71 Clarify that path_match also considers object fields. (#41658)
The `path_match` and `path_unmatch` parameters in dynamic templates match on
object fields in addition to leaf fields. This is not obvious and can cause
surprising errors when a template is meant for a leaf field, but there are
object fields that match. This PR adds a note to the docs to describe the
current behavior.
2019-05-06 14:48:08 -07:00
Julie Tibshirani eb9bce3930 Clarify _doc is a permanent part of certain document APIs. (#41727)
We received some feedback that it is not completely clear why `_doc` is present
in the typeless document APIs:

> The new index APIs are PUT {index}/_doc/{id} in case of explicit ids and POST
{index}/_doc for auto-generated ids."_ Isn't this contradicting? Specifying
*types in requests is deprecated*, but we are supposed to still mention *_doc*
in write requests?

This PR updates the 'removal of types' documentation to try to clarify that
`_doc` now represents the endpoint name, as opposed to a type.
2019-05-06 10:43:50 -07:00
Marios Trivyzas 228d23de6d
SQL: [Docs] Add example for custom bucketing with CASE (#41787)
Add a TIP on how to use CASE to achieve custom bucketing
with GROUP BY.

Follows: #41349

(cherry picked from commit eb5f5d45533c5f81e57dd0221d902a73ec400098)
2019-05-06 18:05:03 +03:00
James Rodewig 2306531815 [DOCS] Rewrite `term` query docs for new format (#41498)
* [DOCS] Restructure `term` query docs.
2019-05-06 10:36:53 -04:00
David Kyle 4421744184 [ML DataFrame] Update Preview docs for source config change (#41761) 2019-05-03 16:34:10 +01:00
Gordon Brown 9a4cab9f77
Recommend use of ln1p/log1p to avoid negative scores (#41610)
As negative scores will now cause an error, and it is easy to
accidentally produce negative scores with some of the built-in modifiers
(especially `ln` and `log`), this adjusts the documentation to more
strongly recommend the use of `ln1p` and `log1p` instead.

Also corrects some awkward formatting on the note sections following the
table.
2019-05-03 09:10:29 -06:00
David Turner c942277822 Fix discovery config in docker-compose docs (#41753)
Today's `docker-compose` docs are missing the `discovery.seed_nodes` config on
one of the nodes. With today's configuration the cluster can still form the
first time it is started, because `cluster.initial_master_nodes` requires both
nodes to bootstrap the cluster which ensures that each discover the other.
However if `es02` is elected master it will remove `es01` from the voting
configuration and then when restarted it will form a cluster on its own without
needing to do any discovery. Meanwhile `es01` doesn't know how to find `es02`
after a restart so will be unable to join this cluster.

This commit fixes this by adding the missing configuration.

Relates #41394, which fixes a different `docker-compose.yml` in the same way.
2019-05-03 10:46:48 +02:00
lcawl 9f77ea26a9 [DOCS] Adds placeholder for 7.2 breaking changes 2019-05-02 15:36:31 -07:00
James Rodewig 33945463f3 [DOCS] Add `:` to render multiple inline macros in Asciidoctor (#41615) 2019-05-01 15:54:19 -04:00
lcawl 5fd3774d43 [DOCS] Adds missing 7.2.0 highlights page 2019-05-01 12:02:34 -07:00
Jason Tedor 7f3ab4524f
Bump 7.x branch to version 7.2.0
This commit adds the 7.2.0 version constant to the 7.x branch, and bumps
BWC logic accordingly.
2019-05-01 13:38:57 -04:00
James Rodewig c86f797df8 [DOCS] Reword `type` query deprecation note for Asciidoctor migration (#41595) 2019-05-01 09:29:05 -04:00
James Rodewig b65ceb36bc [DOCS] Escape quotes to avoid smart quotes in Asciidoctor (#41603) 2019-04-30 16:31:20 -04:00
James Rodewig 721787fa76 [DOCS] Remove soft line breaks (+) for Asciidoctor migration (#41600) 2019-04-30 16:22:50 -04:00
James Rodewig 3e68164825 [DOCS] Move beta[] tag for Asciidoctor migration (#41599) 2019-04-30 16:03:24 -04:00
James Rodewig 9506e3f1c5 [DOCS] Escape commas in deprecated[] for Asciidoctor migration (#41598) 2019-04-30 15:52:57 -04:00
James Rodewig d329186972 [DOCS] Escape depcreated[] note for Asciidoctor migration (#41608) 2019-04-30 15:19:43 -04:00
James Rodewig d46f55f013 [DOCS] Add attribute to escape minimal pt token link in Asciidoctor (#41613) 2019-04-30 14:11:48 -04:00
James Rodewig 005296dac6 [DOCS] Allow attribute substitution in titleabbrevs for Asciidoctor migration (#41574)
* [DOCS] Replace attributes in titleabbrevs for Asciidoctor migration

* [DOCS] Add [subs="attributes"] so attributes render in Asciidoctor

* Revert "[DOCS] Replace attributes in titleabbrevs for Asciidoctor migration"

This reverts commit 98f130257a7c71e9f6cddf5157af7886418338d8.

* [DOCS] Fix merge conflict
2019-04-30 13:46:45 -04:00
James Rodewig 53702efddd [DOCS] Add anchors for Asciidoctor migration (#41648) 2019-04-30 10:20:17 -04:00
James Rodewig c26b8eb4de [DOCS] Remove line break from deprecated[] macro (#41616) 2019-04-30 09:58:58 -04:00
James Rodewig cc39233ff8 [DOCS] Add soft line breaks to maintain nested definition list (#41617) 2019-04-30 09:25:43 -04:00
James Rodewig fc4f401214 [DOCS] Document 200 http code response for partial results (#40367) 2019-04-29 08:30:17 -04:00
Benjamin Trent a0990ca239
[ML] cleanup + adding description field to transforms (#41554) (#41605)
* [ML] cleanup + adding description field to transforms

* making description length have a max of 1k
2019-04-26 16:50:59 -05:00
Alan Woodward d7ab86db9c Document restrictions on fuzzy matching when using synonyms (#40783)
Relates to #25518 #41592
2019-04-26 10:10:32 -07:00
James Rodewig 06827a495f [DOCS] Fix delimited blocks for Asciidoctor migration (#41583) 2019-04-26 11:43:41 -04:00
James Rodewig 4adf7963c7 [DOCS] Escape commas in experimental[] for Asciidoctor migration (#41578) 2019-04-26 11:35:47 -04:00
James Rodewig f8081e8558 [DOCS] Fix ID for Asciidoctor migration (#41577) 2019-04-26 11:21:32 -04:00
James Rodewig 6a7459ff11 [DOCS] Clarify Recovery Settings for Shard Relocation (#40329)
* Clarify that peer recovery settings apply to shard relocation

* Fix awkward wording of 1st sentence

* [DOCS] Remove snapshot recovery reference.
Call out link to [[cat-recovery]].
Separate expert settings.
2019-04-26 10:24:14 -04:00
Mayya Sharipova 4127d6889b Remove experimental label froms script_score query (#41572) 2019-04-26 08:51:29 -04:00
James Rodewig 08c5d3b912 [DOCS] Explicitly set section IDs for Asciidoctor migration (#41547)
* [DOCS] Explicitly set section ID for faster phrase queries

* [DOCS] Explicitly set section ID for faster prefix queries
2019-04-25 15:07:52 -04:00