Commit Graph

5698 Commits

Author SHA1 Message Date
James Rodewig 999462f460 [DOCS] Document limits for JSON objects with `ignore_malformed` mapping setting (#40976) 2019-04-10 14:47:26 -04:00
James Rodewig a0b54a9a92 [DOCS] Restructure `ids` to new query docs format (#41077) 2019-04-10 14:36:50 -04:00
Jason Tedor 24446ceae0
Add packaging to cluster stats response (#41048)
This commit adds a packaging_types field to the cluster stats response
that outlines the build flavors and types present in a cluster.
2019-04-10 13:47:19 -04:00
Zachary Tong 0a85d1fe03 Add missing "long form" 7.0 breaking changes (#41049)
These are breaking changes that were present in the release notes but the PR didn't include any "narrative long form" description for the docs.
2019-04-10 09:44:45 -04:00
Zachary Tong 9aa8ab58ee Add 7.0.0 Release Notes (#41029) 2019-04-10 09:44:45 -04:00
Adrien Grand 683cf56982 Update headline of the "removal of types" doc page to match changes in 7.0. (#40868)
Currently it describes what broke in 6.0.
2019-04-10 11:34:29 +02:00
Alexander Reelsen 3b9ab5da04 Fix order of request body search parameter names in documentation (#40777)
The order was random, which made it super hard to find anything. This
changes the order to be alphabetically.
2019-04-09 16:35:45 +02:00
Henning Andersen c5a77e5d8c Node repurpose tool docs (#40525)
Added documentation for node repurpose tool and included documentation on how to repurpose nodes safely. Adjusted order of tools in `elasticsearch-node` tool since the repurpose tool is most likely to be used.

Co-Authored-By: David Turner <david.turner@elastic.co>
2019-04-09 15:07:37 +02:00
Christoph Büscher 40638d7b28 [Docs] Delete explanation for completion suggester default analyzer choice (#36720)
The explanation given in the completion suggester documentation why we use the
"simple" analyzer as the default is no longer valid. Since we still use "simple"
as the default, we should just delete the explanation that doesn't fit anymore.

Closes #36715
2019-04-09 13:50:29 +02:00
Lisa Cawley 1ed4f215ac
[DOCS] Add notable-breaking-changes tags (#40991) 2019-04-08 18:54:29 -07:00
James Rodewig 9e271b9ec2 [DOCS] Fix broken link to Elasticsearh Docker source code (#40979)
* [DOCS] Fix broken link to Elasticsearch Docker source code

* [DOCS] Link to Dockerfile in elastic/elasticsearch repo

* [DOCS] Link to Docker source files in elastic/elasticsearch repo
2019-04-08 16:12:28 -04:00
Shaunak Kashyap 5990c54801
[7.x] Docs: Simplifying setup by using module configuration variant syntax (#40879) (#40974)
Backports the following commits to 7.x:
 - Docs: Simplifying setup by using module configuration variant syntax  (#40879)
2019-04-08 09:31:26 -07:00
debadair a69ff8221f [DOCS] Added settings page for ILM. (#40880)
* [DOCS] Added settings page for ILM.

* [DOCS] Adding ILM settings file

* [DOCS] Moved the ILM settings to a separate section

* [DOCS] Linked to the rollover docs.

* [DOCS] Tweaked the "required" wording.
2019-04-05 16:39:18 -07:00
DeDe Morton b0ce3e0a10 [Docs] Remove extraneous text (#40914)
Removes text that was likely introduced by copy/paste error.
2019-04-05 15:32:25 -07:00
Jason Tedor ac58b9bded
Fix date index name processor default date_formats (#40915)
This commit is a correction of a doc bug in the docs for the ingest
date-index-name processor. The correct pattern is
yyyy-MM-dd'T'HH:mm:ss.SSSXX. This is due to the transition from Joda
time to Java time where Z does not mean the same thing between the two.
2019-04-05 17:45:57 -04:00
lcawl f4348843ba [DOCS] Adds placeholder for 7.0.0 release notes 2019-04-05 14:27:05 -07:00
DeDe Morton 360d8d40be [Docs] Change example to show col headers (#40822)
Command needs `?v` so user can see the column headers. Otherwise the instructions in the note about checking the init and relo columns don't make sense
2019-04-05 14:58:09 -04:00
Alexander Reelsen 669d72e47a Fix dense/sparse vector limit documentation (#40852)
The documentation stated a wrong limit of dense/sparse vector sizes.
This was changed in #40597 but the documentation was not fixed.
2019-04-05 09:35:30 +02:00
Deb Adair 8f72d72c4c [DOCS] Removed redundant (not quite right) information about upgrades. 2019-04-04 16:19:34 -07:00
James Rodewig b7497bf2df [DOCS] Remind users to include @ symbol when applying license file (#40688) 2019-04-04 11:50:20 -04:00
Taylor Scott 38f4dda2be Fix grammatical error in analysis.asciidoc (#40827) 2019-04-04 17:33:02 +02:00
Nhat Nguyen 2756a3936b Reject illegal flush parameters (#40213)
This change rejects an illegal combination of flush parameters where
force is true, but wait_if_ongoing is false. This combination is trappy
and should be forbidden.

Closes #36342
2019-04-04 09:02:31 -04:00
James Rodewig 11f2da9d5e [DOCS] Rewrite query def (#40746) 2019-04-04 08:57:39 -04:00
James Rodewig 7c669026ac
[DOCS] Document index.load_fixed_bitset_filters_eagerly (#40780) (#40842) 2019-04-04 08:48:37 -04:00
Marios Trivyzas c8047c0644 SQL: [Docs] Small fixes for CURRENT_TIMESTAMP docs (#40792)
- Added square brackets for the optional argument of precision
- Fixed character to lower case after comma

(cherry picked from commit d2f6f3b9ce36875e2eb6145c50464b4d72f2b1df)
2019-04-04 11:45:20 +02:00
Marios Trivyzas f3c207d27b SQL: Implement CURRENT_TIME/CURTIME functions (#40662)
After `TIME` SQL data type is introduced, implement
`CURRENT_TIME/CURTIME` functions similarly to CURRENT_TIMESTAMP
that return the system's current time (only, without the date part).

Closes: #40468
(cherry picked from commit 9feede781409d0e264ce45951a25b28ff129b187)
2019-04-04 11:45:20 +02:00
Alan Woodward 8ca7325f65 Document restrictions on fuzzy matching when using synonyms (#40783)
Relates to #25518
2019-04-04 09:10:44 +01:00
James Rodewig 347e059fdc
[DOCS] Consistently document dynamic monitoring collection settings (#40598) (#40808) 2019-04-03 14:55:25 -04:00
James Rodewig 050c0ec21e [DOCS] Note index rollover is not automatically monitored (#40464) 2019-04-03 13:51:22 -04:00
Jake Landis 41a2b37a55
release notes for 7.0.0-rc2 (#40796) 2019-04-03 11:48:59 -05:00
michaelbaamonde fd6c5db938
Add release notes for 7.0.0-rc1. (#40479)
* Add release notes for 7.0.0-rc1.

* [DOCS] Fixes broken link to breaking changes

* [DOCS] Removed old ML PRs; edited titles

* Remove superseded PR.

* Clean up Lucene upgrade PRs/issues.
2019-04-03 11:48:58 -05:00
Marios Trivyzas 952c4d9653
SQL: Fix display size for DATE/DATETIME (#40669)
A full format for a DATETIME would be:
`2019-03-30T10:20:30.123+10:00` which is 29 chars long.

For DATE a full format would be: `2019-03-30T00:00:00.000+10:00`
which is also 29 chars long.


(cherry picked from commit 6be83964ed025528778bca8d35692762e166983b)
2019-04-03 13:29:04 +02:00
Jim Ferenczi 4c8c4e5951 remove experimental label from search_as_you_type documentation (#40744) 2019-04-03 09:42:20 +02:00
Jason Tedor 91a74ad799
Add migration note for Ubuntu 14.04
This commit adds a migration note to the docs for Ubuntu 14.04 seeing as
it will not be supported after April 30, 2019 by upstream.
2019-04-02 08:54:54 -04:00
Adrien Grand 965e311094 Update indexing speed recommendations around the refresh interval. (#40690)
We now need to update recommendations now that we have introduced the concept
of "search idle" shards.
2019-04-02 11:19:22 +02:00
Christoph Büscher dfc70e6ef0 Correct indention in synonym docs (#40711)
The stopword filter should be on the same level as the synonym filter in the
example request. Correcting this for better readability.
2019-04-02 01:44:24 +02:00
Marios Trivyzas 899ed2bf81 SQL: Introduce SQL TIME data type (#39802)
Support ANSI SQL's TIME type by introductin a runtime-only
ES SQL time type.

Closes: #38174
(cherry picked from commit 046ccd4cf0a251b2a3ddff6b072ab539a6711900)
2019-04-01 23:57:27 +02:00
Andy Bristol e88b862192 fix summary of phrase_prefix scoring (#40567)
The language here implies that phrase_prefix scoring works like
most_fields, but it actually works like best_fields
2019-04-01 12:04:59 -07:00
lcawl 7aa3cf5445 [DOCS] Adds placeholder for 7.0.0-rc2 release notes 2019-04-01 12:02:13 -07:00
lcawl edb0d42b41 [DOCS] Add notable release highlights for 7.0 2019-04-01 10:40:44 -07:00
Andrei Stefan f8d3d685e5 SQL: Documentation for LIKE and RLIKE operators (#40623)
(cherry picked from commit 9536c5f7b7fecda4592f6b169adef180eadbba54)
2019-04-01 18:15:47 +03:00
Andrei Stefan 8cd96ee160 Add JDBC trustore type to the docs (#40592)
(cherry picked from commit cafde31129d85d20f9e445be131c9e85163bed9d)
2019-04-01 18:15:46 +03:00
James Rodewig 2b4ebcab53
[DOCS] Add 'time value' links to several monitor settings (#40633) (#40685) 2019-04-01 08:47:12 -04:00
Jason Tedor e596b33ce8
Remove mention of soft deletes from getting started (#40668)
We no longer need to mention soft deletes in the getting started guide
now that retention leases exist and default to 12h. This commit removes
mention of soft deletes from the getting started guide, to simplify that
content.
2019-03-31 22:11:09 -04:00
Henning Andersen 567e8f8b63 Reindex conflicts clarification (docs) (#40442)
Made it more clear that conflicts : proceed only affects version
conflicts.
2019-03-31 14:14:39 +02:00
Mayya Sharipova 1e90b29c00 Add information about the default sort mode (#40657) 2019-03-30 10:58:47 -04:00
Marios Trivyzas 6edb802716
SQL: [Docs] Fix doc errors regarding CURRENT_DATE. (#40649)
Some parts wrongly refered to CURRENT_TIMESTAMP.

(cherry picked from commit 3dd0384d683940871f5d061f153b70c0420150fa)
2019-03-30 12:08:38 +01:00
Christoph Büscher 4f5a83dba2 Clarify using time_zone and date math in range query (#40655)
Currently, the docs correctly state that using `now` in range queries will not
be affected by the `time_zone` parameter. However, using date math roundings
like e.g. `now\d` will be affected by the `time_zone`. Adding this example
because it seems to be a frequently asked question and source of confusion.

Relates to #40581
2019-03-29 23:39:18 +01:00
David Roberts cbe7d335ff [DOCS] Use "source" instead of "inline" in ML docs (#40635)
Specifying an inline script in an "inline" field
was deprecated in 5.x.  The new field name is
"source".

(Since 6.x still accepts "inline" I will only backport
this docs change as far as 7.0.)
2019-03-29 17:30:28 +00:00
aladdam f7e3f748b6 [DOCS] match description with example in allocation docs page (#39606) 2019-03-29 17:01:14 +01:00
avshiav 1edebfc834 [DOCS] Fixed missing space (#38305) 2019-03-29 17:01:14 +01:00
Ryan Ernst 7e8f1c2da7
Add docs for bundled jdk (#40487)
This commit changes the note in docs about required java version to note
the existence of the bundled jdk and how to bring your own java. It also
reorganizes the zip/targz docs as zip is no longer suitable on
Linux/MacOS.
2019-03-29 09:08:32 -04:00
Jason Tedor 585f38787c
Add usage indicators for the bundled JDK (#40616)
This commit adds indications whether or not a distribution is from the
bundled JDK, and whether or not we are using the bundled JDK.
2019-03-29 08:25:32 -04:00
Jim Ferenczi ae65f77add Update docs for the DFR similarity (#40579)
The basic models `b, de, p` and the after effect `no`
are not available anymore in Lucene 8 but they are still
listed in the >7x documentation. This change removes these
references that should also be listed in the breaking change
of es 7.0.

Closes #40264
2019-03-29 10:44:24 +01:00
Tim Vernum c012c59115
Improve certutil --pass documentation about empty password (#40574)
Improve the documentation of parameter --pass of elasticsearch-certutil

Backport of: #40137

Co-Authored-By: Diego Cardozo Sandrim <diegocsandrim@users.noreply.github.com>
Co-Authored-By: Vigneash Sundar <vikene@users.noreply.github.com>
2019-03-29 10:11:00 +11:00
Mayya Sharipova 24755209b4 Add randomScore function in script_score query (#40186)
To make script_score query to have the same features
as function_score query, we need to add randomScore
function.

This function produces different
random scores on different index shards.
It is also able to produce random scores
based on the internal Lucene Document Ids.
2019-03-28 13:23:47 -04:00
lcawl 85848af8cf [DOCS] Fixes formatting in breaking changes 2019-03-28 10:05:02 -07:00
Andrei Stefan fe32ce6009 Include functions' aliases in the list of functions (#40584)
(cherry picked from commit 4034f8f31616353d2f5d09a90753d75e2870ee18)
2019-03-28 14:39:19 +02:00
David Turner 073b13f5b0 Add docs for cluster.remote.*.proxy setting (#40281)
In #33062 we introduced the `cluster.remote.*.proxy` setting for proxied
connections to remote clusters, but left it deliberately undocumented since it
needed followup work so that it could work with SNI. However, since #32517 is
now closed we can add this documentation and remove the comment about its lack
of documentation.
2019-03-28 12:11:24 +00:00
Andrei Stefan 89d97905e5 SQL: add "fuzziness" option to QUERY and MATCH function predicates (#40529)
* Remove unused "locale" and "lowercase_expanded_terms" options from QUERY.

(cherry picked from commit c122fc6edddbb99c73ce25168d1152409c0b7892)
2019-03-28 10:37:09 +02:00
Costin Leau de5f924daa SQL: Adjust the precision and scale for drivers (#40467)
Fix #40357

(cherry picked from commit 1557d77b07bbeef99604857c1ca824e20771c283)
2019-03-28 10:08:09 +02:00
Andy Bristol 23395a9b9f
search as you type fieldmapper (#35600)
Adds the search_as_you_type field type that acts like a text field optimized
for as-you-type search completion. It creates a couple subfields that analyze
the indexed terms as shingles, against which full terms are queried, and a
prefix subfield that analyze terms as the largest shingle size used and
edge-ngrams, against which partial terms are queried

Adds a match_bool_prefix query type that creates a boolean clause of a term
query for each term except the last, for which a boolean clause with a prefix
query is created.

The match_bool_prefix query is the recommended way of querying a search as you
type field, which will boil down to term queries for each shingle of the input
text on the appropriate shingle field, and the final (possibly partial) term
as a term query on the prefix field. This field type also supports phrase and
phrase prefix queries however
2019-03-27 13:29:13 -07:00
David Turner f1eca21081
Mention legacy discovery settings in migration doc (#40528)
Today the upgrade assistant identifies that discovery configuration is required
in production mode, but links to docs saying to add one of these settings:

- `discovery.seed_hosts`
- `discovery.seed_providers`
- `cluster.initial_master_nodes`

However these settings do not exist in 6.7 so this is unhelpful advice. This
commit adjusts the docs to give more useful advice to users arriving at this
page from the upgrade assistant.

Relates https://discuss.elastic.co/t/174102
2019-03-27 15:56:02 +00:00
Andrei Stefan 99dca30197 SQL: MATCH and QUERY documentation; one list of functions (#40494)
* Document MATCH and QUERY function predicates.
* Polish the functions pages and add a list of functions to the main Functions & Operators page.

(cherry picked from commit 4cec0ae1b962ec7ea011a290aec72740386eb808)
2019-03-27 17:37:10 +02:00
James Rodewig cfe85c73bf Fix typo in rollup_index definition (#40520) 2019-03-27 10:04:49 -04:00
Bogdan Pintea 6182b464a3 SQL: ODBC: document extra connection string parameters (#40476)
* document ODBC's extra connection string parameters

Document the connection string parameters that are currently not
configurable from the GUI.

Add a note about SmartScreen possible warning on driver MSI
installation.

Add a note about the TLS certificate file not being supported as bundled
or password-protected.

* rephrasing and restructuring the list of params

- addressing PR review notes

* rephrasings and reference linking

- addressing PR review notes

(cherry picked from commit ca90190226c0a8c63f51c2603b27df299ce8d503)
2019-03-27 14:20:23 +01:00
Boaz Leskes 6efe748e2e Only recommend disabling allocation of replicas in a rolling upgrade (#40299)
The current documentation recommends disabling allocations of all shards. This prevents shards of
new indices from being allocated as well. That means that during a rolling upgrade, operations like
roll over and index shrinking will operate correctly. This, in turn, can cause issues for data ingestion
and ILM.
2019-03-27 14:18:17 +01:00
Benjamin Trent 12943c5d2c
[ML] Add data frame task state object and field (#40169) (#40490)
* [ML] Add data frame task state object and field

* A new state item is added so that the overall task state can be
accoutned for
* A new FAILED state and reason have been added as well so that failures
can be shown to the user for optional correction

* Addressing PR comments

* adjusting after master merge

* addressing pr comment

* Adjusting auditor usage with failure state

* Refactor, renamed state items to task_state and indexer_state

* Adding todo and removing redundant auditor call

* Address HLRC changes and PR comment

* adjusting hlrc IT test
2019-03-27 06:53:58 -05:00
Costin Leau 38dc005b8d SQL: Update JDBC class name in client screenshots
(cherry picked from commit 79a3b93deb858961ecbe4516ed9d8f209016666f)
2019-03-27 12:21:53 +02:00
lcawl 81510746d6 [DOCS] Adds anchors and x-pack icons for frozen indices 2019-03-26 15:53:18 -07:00
alex101101 fb8ad0cf30 Add a soft limit to the field name length (#40309)
Adds an optional limit to the length of field names, throws an IllegalArgumentException if the limit is breached. 
Closes #33651
2019-03-26 17:58:32 +01:00
Lisa Cawley 6f03a6c546 [DOCS] Simplify ML upgrade step (#40006) 2019-03-25 10:26:29 -07:00
Costin Leau 61f49af497 SQL: Spec tests now use classpath discovery (#40388)
To avoid having to specify each spec by hand (which can miss specs to be
added), the test infrastructure now performs classpath discovery so that
each spec added, is automatically considered.

Relates #40358

(cherry picked from commit d0f60b4425c731509aa8ca765d55f563f866ef90)
2019-03-25 15:22:52 +02:00
James Rodewig e8a93b8915
Update Windows command to support prettified json licenses (#40407) 2019-03-25 09:00:03 -04:00
lcawl 421d7654a7 [DOCS] Adds 7.1 breaking changes 2019-03-22 16:37:47 -07:00
Costin Leau 496070fda6 SQL: CAST supports both SQL and ES types (#40365)
Extend CAST to support all data types notations (whether SQL or ES
specific)

Fix #40282

(cherry picked from commit eb2ee8a344da946920598839a5db76c8bb9bc3fe)
2019-03-22 23:55:51 +02:00
lcawl eb10afabb8 [DOCS] Fixes comments in release highlights 2019-03-22 14:40:59 -07:00
lcawl bc8c9f358d [DOCS] Adds 7.1.0 release highlights 2019-03-22 13:36:37 -07:00
lcawl 15a1e65e48 [DOCS] Adds placeholder for 7.0.0-rc1 release notes 2019-03-22 11:34:50 -07:00
Costin Leau 970cd57368 DOC: polish client docs
(cherry picked from commit 72e413df26791b90475ce2f22c31d29e79bd52b9)
2019-03-22 19:43:56 +02:00
Costin Leau 980ee14f57 DOC: Expand section on ORDER BY aggs (#40332)
(cherry picked from commit 99d2f6fc9864ab972259ef5692129ab49e4a7ab8)
2019-03-22 10:04:52 +02:00
Tim Brooks b71066129b
Expand `following` documentation in ccr overview (#39936)
This commit expands the ccr overview page to include more information
about the lifecycle of following an index. It adds information linking
to the remote recovery documentation. And describes how an index can
fall-behind and how to fix it when this happens.
2019-03-21 12:19:46 -06:00
Lisa Cawley e6799849d1 [DOCS] Adds placeholder for start and stop data frame transform APIs (#40278) 2019-03-21 09:39:10 -07:00
Lisa Cawley caa0129d44 [DOCS] Adds placeholder for create and delete data frame transform APIs (#40233) 2019-03-21 09:13:50 -07:00
RomainGeffraye f4b819a578 Fix typo (#40273) 2019-03-21 07:57:13 -07:00
Lisa Cawley ff2bcc9d11 [DOCS] Adds placeholder for get data frame transform APIs (#40283) 2019-03-21 07:57:01 -07:00
James Rodewig 244e6758cd
[DOCS] Add 'recovery' to glossary for issue #7264 (#40275) (#40311) 2019-03-21 08:58:26 -04:00
James Rodewig 95f1c8bea0
Add links to Freeze/Unfreeze index APIs (#40225) (#40268) 2019-03-20 15:09:36 -04:00
Lisa Cawley 75a788ec5d [DOCS] Adds placeholder for data frame preview API (#40232) 2019-03-20 10:30:02 -07:00
Alan Woodward 83d2870308 Add `use_field` option to intervals query (#40157)
This is the equivalent of the `field_masking_span` query, allowing users to
merge intervals from multiple fields - for example, to search for stemmed tokens
near unstemmed tokens.
2019-03-20 16:26:04 +00:00
Jim Ferenczi 3400483af4
Add date and date_nanos conversion to the numeric_type sort option (#40199) (#40224)
This change adds an option to convert a `date` field to nanoseconds resolution
 and a `date_nanos` field to millisecond resolution when sorting.
The resolution of the sort can be set using the `numeric_type` option of the
field sort builder. The conversion is done at the shard level and is restricted
to dates from 1970 to 2262 for the nanoseconds resolution in order to avoid
numeric overflow.
2019-03-20 16:50:28 +01:00
David Turner 5eb33f2df4
Document the reduction in fault detection timeouts (#40200)
The new cluster coordination subsystem introduced in 7.0 will only keep an
unresponsive node in the cluster for 30 seconds, whereas in earlier versions it
might have remained in the cluster for 90 seconds. This commit adds a note to
the migration documentation to that effect.
2019-03-20 15:49:50 +00:00
Mayya Sharipova 49a7c6e0e8
Expose proximity boosting (#39385) (#40251)
Expose DistanceFeatureQuery for geo, date and date_nanos types

Closes #33382
2019-03-20 09:24:41 -04:00
David Turner 522f33e9eb Reword frozen indices introduction (#40160)
Clarifies that we don't move anything extra to persistent storage when freezing
an index, and gives a bit more context for less experienced users.
2019-03-20 11:59:36 +00:00
Florian Kelbert 4f28d769ca Correction to ShingleTokenFilter in migration docs (#40208)
Changed "Limit to the difference between max_shingle_size and min_shingle_size in ShingleTokenFilter"
2019-03-19 12:04:43 -07:00
Luca Cavanna a11f1c88b3 [DOCS] add details on version compatibility and remote gateway selection (#40056)
This commit clarifies how the gateway selection works when configuring
remote clusters for CCR or CCS. Specifically, it clarifies compatibility
between different versions which is a very common question.
2019-03-19 14:43:38 +01:00
Alex Doerr 740020dfe8 Clarify version compatibility in snapshot/restore docs (#39329) 2019-03-19 14:43:38 +01:00
Gordon Brown c8a4a7fc9d
Remove Migration Upgrade and Assistance APIs (#40075)
The Migration Assistance API has been functionally replaced by the
Deprecation Info API, and the Migration Upgrade API is not used for the
transition from ES 6.x to 7.x, and does not need to be kept around to
repair indices that were not properly upgraded before upgrading the
cluster, as was the case in 6.
2019-03-18 13:46:56 -06:00
Lisa Cawley 7325b2a3a7 [DOCS] Moves monitoring settings (#40167) 2019-03-18 11:13:26 -07:00
Costin Leau 076a68007c SQL: Add multi_value_field_leniency inside FieldHitExtractor (#40113)
For cases where fields can have multi values, allow the behavior to be
customized through a dedicated configuration field.
By default this will be enabled on the drivers so that existing datasets
work instead of throwing an exception.
For regular SQL usage, the behavior is false so that the user is aware
of the underlying data.

Fix #39700

(cherry picked from commit 2b351571961f172fd59290ee079126bbd081ceaf)
2019-03-18 14:56:03 +02:00
David Turner 8a3f87bdcd Note that GET /_cluster/state is unstable (#40104)
The `GET /_cluster/state` API returns an internal representation of the cluster
state that does change from version to version. It's useful for debugging, but
it is not intended for regular use by clients.

This change adjusts the documentation of `GET /_cluster/state` to clarify that
this API yields an internal representation that should not be expected to
remain stable between versions.

Relates #40061, #40016
2019-03-18 09:28:00 +00:00
Jim Ferenczi 5b73a1bc7d
Add an option to force the numeric type of a field sort (#38095) (#40084)
This change adds an option to the `FieldSortBuilder` that allows to transform the type
of a numeric field into another. Possible values for this option are `long` that transforms
the source field into an integer and `double` that transforms the source field into a floating point.
This new option is useful for cross-index search when the sort field is mapped differently on some
indices. For instance if a field is mapped as a floating point in one index and as an integer in another
it is possible to align the type for both indices using the `numeric_type` option:

```
{
   "sort": {
    "field": "my_field",
    "numeric_type": "double" <1>
   }
}
```

<1> Ensure that values for this field are transformed to a floating point if needed.
2019-03-18 09:32:45 +01:00
Daniel Mitterdorfer 95024798c0
Document monitoring node stats collection timeout (#39846) (#40140)
With this commit we document the setting
`xpack.monitoring.collection.node.stats.timeout` that has been missing
so far in the docs.

Supersedes #31043
2019-03-18 08:59:02 +01:00
Julie Tibshirani 6ffa8a040d Document the limitation around field aliases and percolator. (#40073)
Currently if a field alias is updated, any percolator queries that contain the
alias will still refer to its old target. This PR documents the issue while we
look into addressing it.

Relates to #37212.
2019-03-15 10:54:09 -07:00
Lisa Cawley c92476f591 [DOCS] Replaces CCS terms with attributes (#40076) 2019-03-15 07:57:51 -07:00
Henning Andersen 95e61d4bb1
Blob Store compress default false (#40054)
Fixed documentation to comply with code (compress=false
is default until 8.0).
2019-03-15 12:31:03 +01:00
Costin Leau 3960374a6f SQL: Introduce MAD (MedianAbsoluteDeviation) aggregation (#40048)
Add Median Absolute Deviation aggregation

Fix #39597

(cherry picked from commit 4f09613942a9249d06c74da64ad7e6f362e97f56)
2019-03-15 11:45:15 +02:00
Lisa Cawley e050d50759 [DOCS] Removes X-Pack settings section (#39870) 2019-03-14 14:37:37 -07:00
Lisa Cawley 52b0b9cb55 [DOCS] Add reindex steps for internal indices (#40019) 2019-03-14 10:30:13 -07:00
debadair a3ef565a25 [DOCS] First pass at upgrade updates for 7.0. (#39944)
* [DOCS] First pass at upgrade updates for 7.0.

* [DOCS] Updates X-Pack terminology

* [DOCS] Incorporated feedback from lcawl.
2019-03-13 14:43:32 -07:00
Adrien Grand 9731ba4338
Make the `type` parameter optional when percolating existing documents. (#39987) (#39989)
`document_type` is the type to use for parsing the document to percolate, which
is already optional and deprecated. However `percotale` queries also have the
ability to percolate existing documents, identified by an index, an id and a
type. This change makes the latter optional and deprecated.

Closes #39963
2019-03-13 15:04:41 +01:00
Jason Tedor 16e4499b97
Fix CCR forget follower docs example
This example was missing sample values in the forget follower API
call. This commit addresses this.
2019-03-13 10:03:22 -04:00
Martijn van Groningen 8f7ab84c5c
Add minimal docs around upgrading clusters with ccr enabled (#38037) 2019-03-13 10:11:14 +01:00
Lisa Cawley 26983d1fdf [DOCS] Replaces CCR terms with attributes (#39516) 2019-03-12 14:35:22 -07:00
Lisa Cawley 77e46b4f53 [DOCS] Added upgrade step for realm settings (#39672) 2019-03-12 13:04:10 -07:00
Tal Levy 9ab2410436
Adding an example in the Set processor documentation to address #30604 (#39941) (#39969)
* Added an example of using set to copy values from one field to another.

* Modified the document type to match the test.
2019-03-12 11:14:41 -07:00
Andrey Ershov aa967edd73 Add elasticsearch-node tool docs (#37812)
This commit, mostly authored by @DaveCTurner,
adds documentation for elasticsearch-node tool #37696.

(cherry picked from commit 09425d5a5158c2d3fdad794411b3bbc4bba47b15)
2019-03-12 12:47:10 +01:00
Adrien Grand 66b3a3a546
Warn about the fact that the terms index is moving off-heap. (#39918)
Lucene 8.0 includes a [change](https://issues.apache.org/jira/browse/LUCENE-8635)
that moves the terms index off-heap for all fields but ID fields. I'm
including this in the migration notes so that users who have queries that match
lots of terms won't be surprised in case of slowdown.
2019-03-12 10:16:41 +01:00
Przemyslaw Gomulka 675489d54c
Date_math_expression doc to inform about format backport#39840 #39903
The format since 7 should be compatible with java-time. Users should be informed about this on this page
backport#39840
2019-03-12 09:19:44 +01:00
Adrien Grand 466864710a Update the how-to section of the docs for 7.0: (#37717)
- new `rank_feature`/`script_score` queries
 - new `index_phrases`/`index_prefixes` options
 - disabling `_field_names` doesn't help anymore
 - adaptive replica selection is on by default
2019-03-12 08:24:39 +01:00
Adrien Grand 5528554aac
Generate release notes for 7.0.0-alpha1. (#39899)
We were missing release notes for 7.0.0-alpha1. I generated them by running
the release-notes script with a quick hack that filtered out pull requests
that had been closed on or after 2018-11-15.

Some breaking changes had been documented in the release notes rather than
the migration guide so I moved them.
2019-03-12 08:10:18 +01:00
Lisa Cawley ea16015bc4
[DOCS] Fixes breaking changes for low level client (#39930) 2019-03-11 10:38:49 -07:00
MK Swanson 7c7933b0aa
[DOCS] Edited intro sentence for clarity. Closes #39865 (#39868) 2019-03-11 12:36:43 -04:00
Yannick Welsch 368b5482fa Add note about cluster state diffs (#39847)
Mentions cluster state diffs in CS publishing docs.
2019-03-11 15:40:07 +01:00
David Kyle 48788269b0
[ML] Correct small inconsistencies in ml APIs spec and docs (#39907) 2019-03-11 14:02:50 +00:00
Mayya Sharipova 671a209ed9 Correct errors in min_hash filter documentation
Related to #39671
2019-03-08 16:21:24 -05:00
lcawl 3246d3e355 [DOCS] Adds tagged region for notable breaking changes 2019-03-08 11:14:27 -08:00
Jake Landis 797d6b8a66
Execute ingest node pipeline before creating the index (#39607) (#39796)
Prior to this commit (and after 6.5.0), if an ingest node changes
the _index in a pipeline, the original target index would be created.
For daily indexes this could create an extra, empty index per day.

This commit changes the TransportBulkAction to execute the ingest node
pipeline before attempting to create the index. This ensures that the 
only index created is the original or one set by the ingest node pipeline. 
This was the execution order prior to 6.5.0 (#32786). 

The execution order was changed in 6.5 to better support default pipelines. 
Specifically the execution order was changed to be able to read the settings
from the index meta data. This commit also includes a change in logic such 
that if the target index does not exist when ingest node pipeline runs, it 
will now pull the default pipeline (if one exists) from the settings of the 
best matched of the index template. 

Relates #32786
Relates #32758 
Closes #36545
2019-03-07 13:31:41 -06:00
Jason Tedor 0250d554b6
Introduce forget follower API (#39718)
This commit introduces the forget follower API. This API is needed in cases that
unfollowing a following index fails to remove the shard history retention leases
on the leader index. This can happen explicitly through user action, or
implicitly through an index managed by ILM. When this occurs, history will be
retained longer than necessary. While the retention lease will eventually
expire, it can be expensive to allow history to persist for that long, and also
prevent ILM from performing actions like shrink on the leader index. As such, we
introduce an API to allow for manual removal of the shard history retention
leases in this case.
2019-03-07 11:08:45 -05:00
Mayya Sharipova 54d41afac1 Add documentation for min_hash filter (#39671)
Closes #20757
2019-03-07 08:49:48 -05:00
Ryan Ernst 7da62d3b79
Add note about negative timestamps to migration guide (#39734)
This commit adds a note to the migration guide for 7.0 about negative
epoch timestamps no longer being supported.

closes #39375
2019-03-06 12:17:12 -08:00
David Roberts 5f8f91c03b
[ML] Use scaling thread pool and xpack.ml.max_open_jobs cluster-wide dynamic (#39736)
This change does the following:

1. Makes the per-node setting xpack.ml.max_open_jobs
   into a cluster-wide dynamic setting
2. Changes the job node selection to continue to use the
   per-node attributes storing the maximum number of open
   jobs if any node in the cluster is older than 7.1, and
   use the dynamic cluster-wide setting if all nodes are on
   7.1 or later
3. Changes the docs to reflect this
4. Changes the thread pools for native process communication
   from fixed size to scaling, to support the dynamic nature
   of xpack.ml.max_open_jobs
5. Renames the autodetect thread pool to the job comms
   thread pool to make clear that it will be used for other
   types of ML jobs (data frame analytics in particular)

Backport of #39320
2019-03-06 12:29:34 +00:00
Ian 95409d3a7e Correct date in daterange-aggregation.asciidoc (#39727) 2019-03-06 11:29:32 +01:00
Jason Tedor 4a3fa5ac7d
Remove beta label from CCR (#39722)
This commit removes the beta label from CCR.
2019-03-05 22:19:53 -05:00
jimczi ecb6df137c fix typo in synonym graph filter docs 2019-03-05 18:20:14 +01:00
Tim Brooks ee7c01988f
Add documentation on remote recovery (#39483)
This is related to #35975. It adds documentation on the remote recovery
process. Additionally, it adds documentation about the various settings
that can impact the process.
2019-03-05 10:17:25 -07:00
Samuel Cifuentes García ca83408542 Improved Terms Aggregation documentation (#38892)
Added a note after the first query example talking about fielddata.
2019-03-05 10:44:59 -05:00
Yannick Welsch 936dbb00e3
Isolate Zen1 (#39470)
Cherry-picks a few commits from #39466 to align 7.x with master branch.
2019-03-04 15:51:17 +01:00
Lisa Cawley f1a7166708 [DOCS] Adds link to list of built-in users (#39529) 2019-03-01 10:32:49 -08:00
Andrei Stefan ba44f28340 SQL: ignore UNSUPPORTED fields for JDBC and ODBC modes in 'SYS COLUMNS' (#39518)
* SYS COLUMNS will skip UNSUPPORTED field types in ODBC and JDBC, as well.
NESTED and OBJECT types were already skipped in ODBC mode, now they are
skipped in JDBC mode, as well.

(cherry picked from commit 9e0df64b2d36c9069dfa506570468f0522c86417)
2019-03-01 15:26:31 +02:00
Lisa Cawley e6c2dae250 [DOCS] Fix image warnings in CCR documentation (#39430) 2019-02-27 07:37:33 -08:00
Jason Tedor 6c5bf3ac13
Remove outdated DNS caching docs from HTTP exporter (#39394)
These docs are out of date, now that we override the infinite DNS cache
within Elasticsearch. This commit completely removes this content, as
specific guidance is no longer needed here.
2019-02-27 08:08:44 -05:00
Andrei Stefan 542e2c55f6 SQL: change the default precision for CURRENT_TIMESTAMP function (#39391)
(cherry picked from commit dbb93310b083226c96e4bde3eef0079eb01cbca9)
2019-02-27 09:49:42 +02:00
Andrei Stefan 4deb69e9e4 SQL: introduce the columnar option for REST requests (#39287)
* Add "columnar" option for REST requests (but be lenient for non-"plain"
modes) for json, yaml, smile and cbor formats.
* Updated documentation

(cherry picked from commit 5b7e0de237fb514d14a61a347bc669d4b4adbe56)
2019-02-27 09:37:28 +02:00
Neeraj Jain 2f7206ada7 Use pkill to shutdown elasticsearch using pid file (#39135)
While running these commands from alias, facing issues using kill `cat pid`, In some situations, the more compact:
```
pkill -F /var/run/myProcess.pid
```
is the way to go.
2019-02-26 16:28:36 +01:00
Lee Hinman 7b8178c839
Remove Hipchat support from Watcher (#39374)
* Remove Hipchat support from Watcher (#39199)

Hipchat has been shut down and has previously been deprecated in
Watcher (#39160), therefore we should remove support for these actions.

* Add migrate note
2019-02-25 15:08:46 -07:00
James Baiera e6a124c118
[Backport 7.x] Fix the OS sensing code in ClusterFormationTasks (#38457)
This fixes a bug in the sensing of the current OS family in the test cluster
formation code. Previously all builds would assume every environment 
was windows and would jump to using the windows zip build. This fixes 
the OS sensing code as well as updates some tests to account for 
different build flavors.

Backport of #38457
2019-02-25 14:39:34 -05:00
Luca Cavanna b6734b412d [DOCS] Fix typo in network-host.asciidoc 2019-02-25 16:55:06 +01:00
Darren Meiss 1d902cce88 Edits to text in Optimistic Concurrency Ctrl doc (#39191) 2019-02-25 16:54:23 +01:00
Darren Meiss 8f0d864ae1 Minor edits to text in Reindex API doc (#39137) 2019-02-25 16:54:17 +01:00
Mayya Sharipova e80284231d
Backport distance functions vectors (#39330)
Distance functions for dense and sparse vectors

Backport for #37947, #39313
2019-02-23 11:52:43 -05:00
Tim Brooks 44df76251f
Rebuild remote connections on profile changes (#39146)
Currently remote compression and ping schedule settings are dynamic.
However, we do not listen for changes. This commit adds listeners for
changes to those two settings. Additionally, when those settings change
we now close existing connections and open new ones with the settings
applied.

Fixes #37201.
2019-02-21 14:00:39 -07:00
Christoph Büscher 4b77d0434a Remove `nGram` and `edgeNGram` token filter names (#39070)
In #30209 we deprecated the camel case `nGram` filter name in favour of `ngram` and
did the same for `edgeNGram` and `edge_ngram` and we are removing those names in
8.0. This change disallows using the deprecated names for new indices created in 7.0 by
throwing an error if these filters are used.

Relates to #38911
2019-02-21 16:55:40 +01:00
Tal Levy b854c92696
add missing `test2` index in alias example (#39212) (#39214)
* missing 'test2' index example (#39055)

If I got the idea of aliases properly, I think that the index "test2" should have
a reference in the example above of the following sentence:

" ... we associate the alias `alias1` to both `test` and `test2` ... "

* add PUT test2

* Update aliases.asciidoc

swap which is write/read

Co-authored-by: Guilherme Ferreira <guilhermeaferreira_t@yahoo.com.br>
2019-02-20 17:45:24 -08:00
Lisa Cawley bfc9a00d9f [DOCS] Unset machine learning upgrade mode (#39149) 2019-02-20 12:06:27 -08:00
Tal Levy cb7e3708bc
Rollup jobs should be cleaned up before indices are deleted (#38930) (#39144)
Rollup jobs should be stopped + deleted before the indices are removed.
It's possible for an active rollup job to issue a bulk request, the test
ends and the cleanup code deletes all indices.  The in-flight bulk
request will then stall + error because the index no-longer exists...
but this process might take longer than the StopRollup timeout.

Which means the test fails, and often fails several other tests since
the job is still active (e.g. other tests cannot create the same-named
job, or fail to stop the job in their cleanup because it's still stalled).

This tends to knock over several tests before the bulk finally times
out and the job shuts down.

Instead, we need to simply stop jobs first.  Inflight bulks will resolve
quickly, and we can carry on with deleting indices after the jobs are
confirmed inactive.

stop-job.asciidoc tended to trigger this issue because it executed
an async stop API and then exited, which setup the above situation. In
can and did happen with other tests though.  As an extra precaution,
the doc test was modified to substitute in wait_for_completion
to help head off these issues too.
2019-02-20 11:12:01 -08:00
Lisa Cawley d74e25a778 [DOCS] Edits the remote clusters documentation (#38996) 2019-02-20 09:01:16 -08:00
Darren Meiss eae2c9dd5c Edits to text in Phrase Suggester doc (#38966) 2019-02-20 12:35:21 +01:00
Darren Meiss 27f7ff157b Fix a typo in the cat shards API docs (#38536) 2019-02-20 11:36:46 +01:00
Darren Meiss 988ce844e1 Edits to text in Bulk API doc (#39083) 2019-02-20 11:17:54 +01:00
Darren Meiss a070cd6822 Edits to text in Multi Get API doc (#39082) 2019-02-20 11:17:54 +01:00
Darren Meiss ad792669be Edits to text in Update By Query API doc (#39078) 2019-02-20 11:17:54 +01:00
Darren Meiss 07ebb8c49a Edits to text in Update API doc (#39069) 2019-02-20 11:17:54 +01:00
Darren Meiss 2f520c663c Edits to text in Delete By Query API doc (#39017) 2019-02-20 11:17:54 +01:00
Darren Meiss 1037aa0665 Edits to text in Index API doc (#39010) 2019-02-20 11:17:54 +01:00
Darren Meiss dc1ed80c73 Edits to text in API Conventions docs (#39001) 2019-02-20 11:17:54 +01:00
Andrei Stefan 92206c8567 Added "validate.properties" property to JDBC's list of allowed properties. (#39050)
This defaults to "true" (current behavior) and will throw an exception
if there is a property that cannot be recognized. If "false", it will
ignore anything unrecognizable.

(cherry picked from commit 38fbf9792bcf4fe66bb3f17589e5fe6d29748d07)
2019-02-20 11:29:01 +02:00
Lee Hinman 80540a2fcc Document 'max_size' parameter as shard size for rollover (#38750)
It was not clear that this is *primary* shard size, not the entire shard size.

Resolves #37981
2019-02-19 11:17:54 -07:00
Jim Ferenczi 83402b1320 Remove beta marker from the synonym_graph docs (#38185) 2019-02-19 10:49:49 +01:00
Jim Ferenczi 199155f5fb
Enforce Completion Context Limit (#38675) (#39075)
This change adds a limit to the number of completion contexts that a completion field can define.

Closes #32741
2019-02-19 08:52:24 +01:00
Jason Tedor 2d8f6b6501
Introduce retention lease state file (#39004)
This commit moves retention leases from being persisted in the Lucene
commit point to being persisted in a dedicated state file.
2019-02-18 16:53:46 -05:00
Darren Meiss 231a892161 Fix wording in Get API docs (#39012) 2019-02-18 17:11:33 +01:00
Darren Meiss 007aaf716a Fix punctuation in replication docs (#39009) 2019-02-18 17:11:31 +01:00
Darren Meiss 231c8c6d40 Fix wording in Delete API docs (#39013) 2019-02-18 17:11:28 +01:00
Alan Woodward ab4d5f404f Add overlapping, before, after filters to intervals query (#38999)
Lucene recently added `overlapping`, `before` and `after` filters to the intervals package. This
commit exposes them in elasticsearch.
2019-02-18 15:06:24 +00:00
Clinton Gormley d59ec89726 Update track-total-hits.asciidoc
Added missing `
2019-02-18 13:33:23 +01:00
Hendrik Muhs 4f662bd289
Add data frame feature (#38934) (#39029)
The data frame plugin allows users to create feature indexes by pivoting a source index. In a
nutshell this can be understood as reindex supporting aggregations or similar to the so called entity
centric indexing.

Full history is provided in: feature/data-frame-transforms
2019-02-18 11:07:29 +01:00
Luca Cavanna a1a49f201d Tie break search shard iterator comparisons on cluster alias (#38853)
`SearchShardIterator` inherits its `compareTo` implementation from `PlainShardIterator`. That is good in most of the cases, as such comparisons are based on the shard id which is unique, even when searching against indices with same names across multiple clusters (thanks to the index uuid being different). In case though the same cluster is registered multiple times with different aliases, the shard id is exactly the same, hence remote results will be returned before local ones with same shard id objects. That is because remote iterators are added before local ones, and we use a stable sorting method in `GroupShardIterators` constructor.

This PR enhances `compareTo` for `SearchShardIterator` to tie break on cluster alias and introduces consistent `equals` and `hashcode` methods. This allows to remove a TODO in `SearchResponseMerger` which otherwise has to handle this special case specifically. Also, while at it I added missing tests around equals/hashcode and compareTo and expanded existing ones.
2019-02-16 09:41:03 +01:00
Tal Levy 92756288b4 relax ML Info Docs expected response (#38993)
the get-ml-info API documentation tested that the
response show that ML's `upgrade_mode` was false.
For reasons that may be true due to other tests running in
parallel or not cleaning themselves up, this may not be
guaranteed. Since the actual value here is not of importance,
this commit relaxes the requirement that upgrade_mode be
static.
2019-02-15 16:31:01 -08:00
Guilherme Ferreira 9fbfe77bb0 Fix typo in Index API doc (#38961) 2019-02-15 18:17:11 -05:00
Darren Meiss dc0e657091 Edits to text & formatting in Term Suggester doc (#38963) 2019-02-15 16:00:28 -05:00
Darren Meiss 56997bf53d Edits to text in Completion Suggester doc (#38980) 2019-02-15 15:47:54 -05:00
Costin Leau c5dce42667 SQL: doc polishing 2019-02-15 22:14:36 +02:00
Costin Leau 79bc6aba79 SQL: Polish the rest chapter (#38971)
Organize the text a bit and add tip on triple quotes in Kibana Console
2019-02-15 22:14:36 +02:00
lcawl e78f70e5d8 [DOCS] Fixes broken formatting 2019-02-15 11:18:00 -08:00
Lisa Cawley d300048cd5
[DOCS] Updates methods for upgrading machine learning (#38876) (#38967) 2019-02-15 09:29:45 -08:00
Hannes Van De Vreken 27cf7e27e7 Fix typo in DateRange docs (yyy → yyyy) (#38883) 2019-02-15 10:19:54 -05:00
Costin Leau 9357c17288 SQL: Doc on syntax (identifiers in particular) (#38662)
Add section on syntax, identifiers and literals and on single vs double quotes.

(cherry picked from commit aafdb598082e451f36294bd174d0887a276d8c7f)
2019-02-15 15:24:33 +02:00
debadair d9c255dbbf [DOCS] Added include and reference to beta1 RNs (#38905) 2019-02-14 07:43:54 -08:00
Christoph Büscher 52a4ca5962 Remove mentioning of types from bulk API docs (#38896)
The docs on master still mention types in the context of conflicts with
documents and also mentions the deprecated endpoint including types.
2019-02-14 16:32:57 +01:00
taku333 8bb2a2a405 SQL: change JDBC setup URL in the documentation (#38564)
(cherry picked from commit 103786ea27da72b2fccd3cf511b3143dae0fc530)
2019-02-14 17:12:08 +02:00
Alexander Reelsen 8e5e48319e
Add documentation about breaking java time changes (#38886)
In addition remove joda time mentions across the docs, make 
sure links are updated to java time javadocs.

Forward port of #38720
2019-02-14 10:18:12 +01:00
Julie Tibshirani 4ad4bc7f5f Update the removal of types docs with the new 6.7 behavior. (#38869)
Follow-up to #38825, where we made a tweak to the deprecation behavior.
2019-02-13 14:45:17 -08:00
Jake Landis 46bb663a09
Make 7.x like 6.7 user agent ecs, but default to true (#38828)
Forward port of https://github.com/elastic/elasticsearch/pull/38757

This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object (#38115)"
This reverts commit 5b008a34aa.

* Revert "Add ECS schema for user-agent ingest processor (#37727) (#37984)"
This reverts commit cac6b8e06f.

* cherry-pick 5dfe1935345da3799931fd4a3ebe0b6aa9c17f57 
Add ECS schema for user-agent ingest processor (#37727)

* cherry-pick ec8ddc890a34853ee8db6af66f608b0ad0cd1099 
Ingest node - user agent, move device to an object (#38115) (#38121)
  
* cherry-pick f63cbdb9b426ba24ee4d987ca767ca05a22f2fbb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor (#38362)

* make true the default for the ecs option, and update 7.0 references and tests
2019-02-13 10:28:01 -06:00
Darren Meiss 34b6383c47 Edits to text of Profile API documentation (#38742)
Minor edits of text.
2019-02-13 10:08:50 +01:00
Jim Ferenczi 251ced79ee Add release notes for 7.0.0-beta1 (#38729)
Add release notes for 7.0.0-beta1
2019-02-13 09:54:34 +01:00
Mayya Sharipova 6eec065353
Describe what _source.includes/excludes do (#38319) (#38794) 2019-02-12 11:09:15 -05:00
Marios Trivyzas 032bcf99d6
SQL: Implement `::` cast operator (#38774)
`<expression>::<dataType>` is a simplified altenative syntax to
`CAST(<expression> AS <dataType> which exists in PostgreSQL and
provides an improved user experience and possibly more compact
SQL queries.

Fixes: #38717
2019-02-12 16:54:14 +02:00
Darren Meiss f8426d9b76 Fix typos in Field-Caps documentation (#38580)
Fix typo in Field-Caps documentation

Reworded because asciidoc was formatting the ellipse/space as a numbered list.
2019-02-11 20:58:31 +01:00
Mayya Sharipova 0e1b1959fe
Correct rebuilt persian analyzer (#38724) (#38744)
Make substitution of \u200C with a space explicit

The problem with this symbol `\u200C` in a test string, 
that **SHOULD** be substituted with space in the rebuilt Persian analyzer, but it is not.

Correcting this line `"mappings": [ "\\u200C=> "] <1>` to
 `"mappings": [ "\\u200C=>\\u0020"] <1>` in solves the problem.
This change explicitly says to substitute ZWNJ with a space.

Closes #38188
2019-02-11 14:17:18 -05:00
Tim Vernum fb1c9c25fa
Add docs for new reindex.ssl settings (#38686)
Reindex from remote now supports configurable SSL/TLS (node level)
settings. This change adds documentation relating to those settings

Relates: #37527
Backport of: #38486
2019-02-11 17:16:55 +11:00
Costin Leau 5b112b1d9d SQL: remove beta marker from documentation (#38661)
(cherry picked from commit fb6e7a30c9eed1e8b83496aaf1efe7e2288f9dd8)
2019-02-10 00:09:58 +02:00
Jake Landis 46bd04959e
fix dissect doc "ip" --> "clientip" (#38544)
Forward port of #38512.
2019-02-08 16:51:58 -06:00
Tal Levy 193464d8c8
add geotile_grid ref to asciidoc (#38632) (#38639)
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2019-02-08 09:40:53 -08:00
David Roberts 02f57b1e29
[DOCS] Add warning about bypassing ML PUT APIs (#38605)
Now that ML configurations are stored in the .ml-config
index rather than in cluster state there is a possibility
that some users may try to add configurations directly to
the index.  Allowing this creates a variety of problems
including possible data exflitration attacks (depending on
how security is set up), so this commit adds warnings
against allowing writes to the .ml-config index other than
via the ML APIs.

Backport of #38509
2019-02-08 11:35:37 +00:00
Lee Hinman 70956f6f34
bad formatted JSON object (#38515) (#38526)
It just need to replace the wrong " , " to " : "

Backport of #38515
2019-02-06 13:01:45 -07:00
David Turner 5a3c452480
Align docs etc with new discovery setting names (#38492)
In #38333 and #38350 we moved away from the `discovery.zen` settings namespace
since these settings have an effect even though Zen Discovery itself is being
phased out. This change aligns the documentation and the names of related
classes and methods with the newly-introduced naming conventions.
2019-02-06 11:34:38 +00:00
Yogesh Gaikwad 6ff4a8cfd5
Add API key settings documentation (#38490)
This commit adds missing
API key service settings documentation.
2019-02-06 20:58:22 +11:00
Luca Cavanna a7046e001c
Remove support for maxRetryTimeout from low-level REST client (#38085)
We have had various reports of problems caused by the maxRetryTimeout
setting in the low-level REST client. Such setting was initially added
in the attempts to not have requests go through retries if the request
already took longer than the provided timeout.

The implementation was problematic though as such timeout would also
expire in the first request attempt (see #31834), would leave the
request executing after expiration causing memory leaks (see #33342),
and would not take into account the http client internal queuing (see #25951).

Given all these issues, it seems that this custom timeout mechanism 
gives little benefits while causing a lot of harm. We should rather rely 
on connect and socket timeout exposed by the underlying http client 
and accept that a request can overall take longer than the configured 
timeout, which is the case even with a single retry anyways.

This commit removes the `maxRetryTimeout` setting and all of its usages.
2019-02-06 08:43:47 +01:00
Lee Hinman 2b6b85815b
Update ilm-api.asciidoc, point to REMOVE policy (#38235) (#38463) 2019-02-05 14:42:31 -07:00
Boaz Leskes 033ba725af
Remove support for internal versioning for concurrency control (#38254)
Elasticsearch has long [supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning) compare and set (a.k.a optimistic concurrency control) operations using internal document versioning. Sadly that approach is flawed and can sometime do the wrong thing. Here's the relevant excerpt from the resiliency status page:

> When a primary has been partitioned away from the cluster there is a short period of time until it detects this. During that time it will continue indexing writes locally, thereby updating document versions. When it tries to replicate the operation, however, it will discover that it is partitioned away. It won’t acknowledge the write and will wait until the partition is resolved to negotiate with the master on how to proceed. The master will decide to either fail any replicas which failed to index the operations on the primary or tell the primary that it has to step down because a new primary has been chosen in the meantime. Since the old primary has already written documents, clients may already have read from the old primary before it shuts itself down. The version numbers of these reads may not be unique if the new primary has already accepted writes for the same document 

We recently [introduced](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/optimistic-concurrency-control.html) a new sequence number based approach that doesn't suffer from this dirty reads problem. 

This commit removes support for internal versioning as a concurrency control mechanism in favor of the sequence number approach.

Relates to #1078
2019-02-05 20:53:35 +01:00
David Turner f2dd5dd6eb
Remove DiscoveryPlugin#getDiscoveryTypes (#38414)
With this change we no longer support pluggable discovery implementations. No
known implementations of `DiscoveryPlugin` actually override this method, so in
practice this should have no effect on the wider world. However, we were using
this rather extensively in tests to provide the `test-zen` discovery type. We
no longer need a separate discovery type for tests as we no longer need to
customise its behaviour.

Relates #38410
2019-02-05 17:42:24 +00:00
Marios Trivyzas c9701be1e8
SQL: Implement CURRENT_DATE (#38175)
Since DATE data type is now available, this implements the
`CURRENT_DATE/CURRENT_DATE()/TODAY()` similar to `CURRENT_TIMESTAMP`.

Closes: #38160
2019-02-05 18:15:26 +02:00
David Turner 3b2a0d7959
Rename no-master-block setting (#38350)
Replaces `discovery.zen.no_master_block` with `cluster.no_master_block`. Any
value set for the old setting is now ignored.
2019-02-05 08:47:56 +00:00
David Turner 2d114a02ff
Rename static Zen1 settings (#38333)
Renames the following settings to remove the mention of `zen` in their names:

- `discovery.zen.hosts_provider` -> `discovery.seed_providers`
- `discovery.zen.ping.unicast.concurrent_connects` -> `discovery.seed_resolver.max_concurrent_resolvers`
- `discovery.zen.ping.unicast.hosts.resolve_timeout` -> `discovery.seed_resolver.timeout`
- `discovery.zen.ping.unicast.hosts` -> `discovery.seed_addresses`
2019-02-05 08:46:52 +00:00
Przemyslaw Gomulka 89feaa0e23
Migration doc for audit json log file (#38165)
The migration documentation for an audit logging changes. Removal of plaintext logs and rename of json log file

relates #32850
2019-02-05 09:09:15 +01:00
Tal Levy 48f09471f8
add docs saying mixed-cluster ILM is not supported (#37954)
Closes #37085.
2019-02-04 17:00:40 -08:00
Gordon Brown 292e0f6fb7
Deprecate `_type` in simulate pipeline requests (#37949)
As mapping types are being removed throughout Elasticsearch, the use of
`_type` in pipeline simulation requests is deprecated. Additionally, the
default `_type` used if one is not supplied has been changed to `_doc` for
consistency with the rest of Elasticsearch.
2019-02-04 16:11:44 -07:00
Nik Everett 5d949dddfb
Docs: Drop inline callout from scroll example (#38340)
Coalesces two calls into one in a scroll example so all callouts are at
the end of the line. This is the only sort of callouts that are
supported by asciidoctor and we'd like to start building our docs with
asciidoctor.

At present we don't have any mechanism to stop folks adding more inline
callouts but we ought to be able to have one in a few weeks. For now,
though, removing these inline callouts is a step in the right direction.

Relates to #38335
2019-02-04 14:57:38 -05:00
Yannick Welsch ece8c659c5
Decrease leader and follower check timeout (#38298)
Reduces the leader and follower check timeout to 3 * 10 = 30s instead of 3 * 30 = 90s, with 30s still
being a very long time for a node to be completely unresponsive.
2019-02-04 15:11:12 +01:00
Alexander Reelsen 87f3579125
Add nanosecond field mapper (#37755)
This adds a dedicated field mapper that supports nanosecond resolution -
at the price of a reduced date range.

When using the date field mapper, the time is stored as milliseconds since the epoch
in a long in lucene. This field mapper stores the time in nanoseconds
since the epoch - which means its range is much smaller, ranging roughly from
1970 to 2262.

Note that aggregations will still be in milliseconds.
However docvalue fields will have full nanosecond resolution

Relates #27330
2019-02-04 11:31:16 +01:00
Costin Leau 783c9ed372
SQL: Allow sorting of groups by aggregates (#38042)
Introduce client-side sorting of groups based on aggregate
functions. To allow this, the Analyzer has been extended to push down
to underlying Aggregate, aggregate function and the Querier has been
extended to identify the case and consume the results in order and sort
them based on the given columns.
The underlying QueryContainer has been slightly modified to allow a view
of the underlying values being extracted as the columns used for sorting
might not be requested by the user.

The PR also adds minor tweaks, mainly related to tree output.

Close #35118
2019-02-02 01:38:25 +02:00
Jason Tedor f181e17038
Introduce retention leases versioning (#37951)
Because concurrent sync requests from a primary to its replicas could be
in flight, it can be the case that an older retention leases collection
arrives and is processed on the replica after a newer retention leases
collection has arrived and been processed. Without a defense, in this
case the replica would overwrite the newer retention leases with the
older retention leases. This commit addresses this issue by introducing
a versioning scheme to retention leases. This versioning scheme is used
to resolve out-of-order processing on the replica. We persist this
version into Lucene and restore it on recovery. The encoding of
retention leases is starting to get a little ugly. We can consider
addressing this in a follow-up.
2019-02-01 17:19:19 -05:00
Benjamin Trent a70f54fc77
Adding ml_settings entry to HLRC and Docs for deprecation_info (#38118) 2019-02-01 12:45:28 -06:00
Nick Knize 603cdf40f1
Update geo_shape docs to include unsupported features (#38138)
There are a two major features that are not yet supported by BKD Backed geo_shape: MultiPoint queries, and CONTAINS relation. It is important we are explicitly clear in the documentation that using the new approach may not work for users that depend on these features. This commit adds an IMPORTANT NOTE section to geo_shape docs that explicitly highlights these missing features and what should be done if they are an absolute necessity.
2019-02-01 10:41:41 -06:00
David Roberts 1fa413a16d
[ML] Remove "8" prefixes from file structure finder timestamp formats (#38016)
In 7.x Java timestamp formats are the default timestamp format and
there is no need to prefix them with "8".  (The "8" prefix was used
in 6.7 to distinguish Java timestamp formats from Joda timestamp
formats.)

This change removes the "8" prefixes from timestamp formats in the
output of the ML file structure finder.
2019-02-01 15:36:04 +00:00
Jay Modi 2ca22209cd
Enable TLSv1.3 by default for JDKs with support (#38103)
This commit enables the use of TLSv1.3 with security by enabling us to
properly map `TLSv1.3` in the supported protocols setting to the
algorithm for a SSLContext. Additionally, we also enable TLSv1.3 by
default on JDKs that support it.

An issue was uncovered with the MockWebServer when TLSv1.3 is used that
ultimately winds up in an endless loop when the client does not trust
the server's certificate. Due to this, SSLConfigurationReloaderTests
has been pinned to TLSv1.2.

Closes #32276
2019-02-01 08:34:11 -07:00
Marios Trivyzas 19dccf8f3e
SQL: [Docs] Add limitation for aggregate functions on scalars (#38186)
Currently aggregate functions can operate only directly on fields.
They cannot be used on top of scalar functions as painless scripting
is currently not supported.
2019-02-01 16:13:51 +02:00
Luca Cavanna e18cac3659
Add finalReduce flag to SearchRequest (#38104)
With #37000 we made sure that fnial reduction is automatically disabled
whenever a localClusterAlias is provided with a SearchRequest.

While working on #37838, we found a scenario where we do need to set a
localClusterAlias yet we would like to perform a final reduction in the
remote cluster: when searching on a single remote cluster.

Relates to #32125

This commit adds support for a separate finalReduce flag to
SearchRequest and makes use of it in TransportSearchAction in case we
are searching against a single remote cluster.

This also makes sure that num_reduce_phases is correct when searching
against a single remote cluster: it makes little sense to return
`num_reduce_phases` set to `2`, which looks especially weird in case
the search was performed against a single remote shard. We should
perform one reduction phase only in this case and `num_reduce_phases`
should reflect that.

* line length
2019-02-01 12:11:42 +01:00
Jim Ferenczi 6fa93ca493
Forbid negative field boosts in analyzed queries (#37930)
This change forbids negative field boost in the `query_string`, `simple_query_string`
and `multi_match` queries.
Negative boosts are not allowed in Lucene 8 (scores must be positive).
The backport of this change to 6x will turn the error into a deprecation warning
in order to raise the awareness of this breaking change in 7.0.

Closes #33309
2019-02-01 11:41:40 +01:00
Tim Vernum 6fcbd07420
Remove heuristics that enable security on trial licenses (#38075)
In 6.3 trial licenses were changed to default to security
disabled, and ee added some heuristics to detect when security should
be automatically be enabled if `xpack.security.enabled` was not set.

This change removes those heuristics, and requires that security be
explicitly enabled (via the `xpack.security.enabled` setting) for
trial licenses.

Relates: #38009
2019-02-01 17:59:13 +11:00
Yuri Astrakhan f3cde06a1d
geotile_grid implementation (#37842)
Implements `geotile_grid` aggregation

This patch refactors previous implementation https://github.com/elastic/elasticsearch/pull/30240

This code uses the same base classes as `geohash_grid` agg, but uses a different hashing
algorithm to allow zoom consistency.  Each grid bucket is aligned to Web Mercator tiles.
2019-01-31 19:11:30 -05:00
Jake Landis 5b008a34aa
Ingest node - user agent, move device to an object (#38115)
When the ingest node user agent parses the device field, it
will result in a string value. To match the ecs schema
this commit moves the value of the parsed device to an
object with an inner field named 'name'. There are not
any passivity concerns since this modifies an unreleased change.

closes #38094
relates #37329
2019-01-31 13:54:34 -06:00
Julie Tibshirani 91b79ebed4
Update 'removal of types' docs to reflect the new plan. (#38003) 2019-01-31 10:26:24 -08:00
Henning Andersen ff7620fd7f
Fail start on obsolete indices documentation (#37786)
Added breaking changes documentation for node start up obsolete indices
detection.

Issue #27073
2019-01-31 15:40:06 +01:00
Marios Trivyzas 4710a7472f
SQL: Implement FIRST/LAST aggregate functions (#37936)
FIRST and LAST can be used with one argument and work similarly to MIN
and MAX but they are implemented using a Top Hits aggregation and
therefore can also operate on keyword fields. When a second argument is
provided then they return the first/last value of the first arg when its
values are ordered ascending/descending (respectively) by the values of
the second argument. Currently because of the usage of a Top Hits
aggregation FIRST and LAST cannot be used in the HAVING clause of a
GROUP BY query to filter on the results of the aggregation.

Closes: #35639
2019-01-31 16:33:05 +02:00
Luca Cavanna 622fb7883b
Introduce ability to minimize round-trips in CCS (#37828)
With #37566 we have introduced the ability to merge multiple search responses into one. That makes it possible to expose a new way of executing cross-cluster search requests, that makes CCS much faster whenever there is network latency between the CCS coordinating node and the remote clusters. The coordinating node can now send a single search request to each remote cluster, which gets reduced by each one of them. from + size results are requested to each cluster, and the reduce phase in each cluster is non final (meaning that buckets are not pruned and pipeline aggs are not executed). The CCS coordinating node performs an additional, final reduction, which produces one search response out of the multiple responses received from the different clusters.

This new execution path will be activated by default for any CCS request unless a scroll is provided or inner hits are requested as part of field collapsing. The search API accepts now a new parameter called ccs_minimize_roundtrips that allows to opt-out of the default behaviour.

Relates to #32125
2019-01-31 15:12:14 +01:00
Andrei Stefan 22d3290078
SQL: Added SSL configuration options tests (#37875)
* Added SSL configuration options tests
Removed the allow.self.signed option from the documentation since we allow
by default self signed certificates as well.

* Added more tests
2019-01-31 10:52:49 +02:00
Adrien Grand 3332e332c7
Fix typo in docs. (#38018)
This has been introduced in #37871.
2019-01-31 09:51:03 +01:00
David Turner 81c443c9de
Deprecate minimum_master_nodes (#37868)
Today we pass `discovery.zen.minimum_master_nodes` to nodes started up in
tests, but for 7.x nodes this setting is not required as it has no effect.
This commit removes this setting so that nodes are started with more realistic
configurations, and deprecates it.
2019-01-30 20:09:15 +00:00
Lee Hinman cac6b8e06f
Add ECS schema for user-agent ingest processor (#37727) (#37984)
* Add ECS schema for user-agent ingest processor (#37727)

This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs).
So rather than something like this:

```
{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}
```

The structure is now like this:

```
{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}
```

This is now the default for 7.0. The deprecated `ecs` setting in 6.x is not
supported.

Resolves #37329

* Remove `ecs` setting from docs
2019-01-30 11:24:18 -07:00
Benjamin Trent 8280a20664
ML: Add upgrade mode docs, hlrc, and fix bug (#37942)
* ML: Add upgrade mode docs, hlrc, and fix bug

* [DOCS] Fixes build error and edits text

* adjusting docs

* Update docs/reference/ml/apis/set-upgrade-mode.asciidoc

Co-Authored-By: benwtrent <ben.w.trent@gmail.com>

* Update set-upgrade-mode.asciidoc

* Update set-upgrade-mode.asciidoc
2019-01-30 06:51:11 -06:00
Martijn van Groningen 3865435a01
Docs test fix, wait for shards active.
(a restore needs to be complete, which happens in the background and
by default the ccr put follow api doesn't wait for this)

(this was a recent change and the pr that added this docs test,
did not include this change)

Relates to #37917
2019-01-30 11:02:16 +01:00
Martijn van Groningen e959dbaa99
Revert "Revert "Documented default values for index follow request parameters. (#37917)""
This reverts commit 7205833f92.
2019-01-30 10:49:42 +01:00
Adrien Grand 7205833f92 Revert "Documented default values for index follow request parameters. (#37917)"
This reverts commit 4da7a44648.
2019-01-30 10:48:01 +01:00
Adrien Grand c8af0f4bfa
Use mappings to format doc-value fields by default. (#30831)
Doc-value fields now return a value that is based on the mappings rather than
the script implementation by default.

This deprecates the special `use_field_mapping` docvalue format which was added
in #29639 only to ease the transition to 7.x and it is not necessary anymore in
7.0.
2019-01-30 10:31:51 +01:00
Adrien Grand b63b50b945
Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. (#37871)
Currently if you mix typed templates and typeless index creation or typeless
templates and typed index creation then you will end up with an error because
Elasticsearch tries to create an index that has multiple types: `_doc` and
the explicit type name that you used.

This commit proposes to give precedence to the index creation call so that
the type from the template will be ignored if the index creation call is
typeless while the template is typed, and the type from the index creation
call will be used if there is a typeless template.

This is consistent with the fact that index creation already "wins" if a field
is defined differently in the index creation call and in a template: the
definition from the index creation call is used in such cases.

Closes #37773
2019-01-30 10:28:24 +01:00
Martijn van Groningen 4da7a44648
Documented default values for index follow request parameters. (#37917) 2019-01-30 09:37:41 +01:00
Martijn van Groningen f51bc00fcf
Added ccr to xpack usage infrastructure (#37256)
* Added ccr to xpack usage infrastructure

Closes #37221
2019-01-30 07:58:26 +01:00
Ryan Ernst 8e5f9c4b14
Add OS/architecture classifier to distributions (#37881)
This commit adds classifiers to the distributions indicating the
OS (for archives) and platform. The current OSes are for windows, darwin (ie
macos) and linux. This change will allow future OS/architecture specific
changes to the distributions. Note the docs using distribution links
have been updated, but will be reworked in a followup to make OS
specific instructions for the archives.
2019-01-29 11:18:30 -08:00
Albert Zaharovits 697b2fbe52
Remove implicit index monitor privilege (#37774)
Restricted indices (currently only .security-6 and .security) are special
internal indices that require setting the `allow_restricted_indices` flag
on every index permission that covers them. If this flag is `false`
(default) the permission will not cover these and actions against them
will not be authorized.
However, the monitoring APIs were the only exception to this rule.

This exception is herein forfeited and index monitoring privileges have to be
granted explicitly, using the `allow_restricted_indices` flag on the permission,
as is the case for any other index privilege.
2019-01-29 21:10:03 +02:00
Julie Tibshirani 9ca26b7e63
Remove more references to type in docs. (#37946)
* Update the top-level 'getting started' guide.
* Remove custom types from the painless getting started documentation.
* Fix an incorrect references to '_doc' in the cardinality query docs.
* Update the _update docs to use the typeless API format.
2019-01-29 10:51:07 -08:00
Tim Brooks 00ace369af
Use `CcrRepository` to init follower index (#35719)
This commit modifies the put follow index action to use a
CcrRepository when creating a follower index. It routes 
the logic through the snapshot/restore process. A 
wait_for_active_shards parameter can be used to configure
how long to wait before returning the response.
2019-01-29 11:47:29 -07:00
Boaz Leskes 218df3009a
Move update and delete by query to use seq# for optimistic concurrency control (#37857)
The delete and update by query APIs both offer protection against overriding concurrent user changes to the documents they touch. They currently are using internal versioning. This PR changes that to rely on sequences numbers and primary terms.

Relates #37639 
Relates #36148 
Relates #10708
2019-01-29 10:23:05 -05:00
Boaz Leskes 65a9b61a91
Add Seq# based optimistic concurrency control to UpdateRequest (#37872)
The update request has a lesser known support for a one off update of a known document version. This PR adds an a seq# based alternative to power these operations.

Relates #36148 
Relates #10708
2019-01-29 09:18:05 -05:00
Przemyslaw Gomulka 4f4113e964
Rename security audit.log to _audit.json (#37916)
in order to keep json logs consistent the security audit logs are renamed from .log to .json
relates #32850
2019-01-29 14:53:55 +01:00
Yannick Welsch 504a89feaf
Step down as master when configured out of voting configuration (#37802)
Abdicates to another master-eligible node once the active master is reconfigured out of the voting
configuration, for example through the use of voting configuration exclusions.

Follow-up to #37712
2019-01-29 12:43:04 +01:00
Przemyslaw Gomulka 891320f5ac
Elasticsearch support to JSON logging (#36833)
In order to support JSON log format, a custom pattern layout was used and its configuration is enclosed in ESJsonLayout. Users are free to use their own patterns, but if smooth Beats integration is needed, they should use ESJsonLayout. EvilLoggerTests are left intact to make sure user's custom log patterns work fine.

To populate additional fields node.id and cluster.uuid which are not available at start time, 
a cluster state update will have to be received and the values passed to log4j pattern converter.
A ClusterStateObserver.Listener is used to receive only one ClusteStateUpdate. Once update is received the nodeId and clusterUUid are set in a static field in a NodeAndClusterIdConverter. 

Following fields are expected in JSON log lines: type, tiemstamp, level, component, cluster.name, node.name, node.id, cluster.uuid, message, stacktrace
see ESJsonLayout.java for more details and field descriptions

Docker log4j2 configuration is now almost the same as the one use for ES binary. 
The only difference is that docker is using console appenders, whereas ES is using file appenders.

relates: #32850
2019-01-29 07:20:09 +01:00
Gordon Brown 49bd8715ff
Inject Unfollow before Rollover and Shrink (#37625)
We inject an Unfollow action before Shrink because the Shrink action
cannot be safely used on a following index, as it may not be fully
caught up with the leader index before the "original" following index is
deleted and replaced with a non-following Shrunken index. The Unfollow
action will verify that 1) the index is marked as "complete", and 2) all
operations up to this point have been replicated from the leader to the
follower before explicitly disconnecting the follower from the leader.

Injecting an Unfollow action before the Rollover action is done mainly
as a convenience: This allow users to use the same lifecycle policy on
both the leader and follower cluster without having to explictly modify
the policy to unfollow the index, while doing what we expect users to
want in most cases.
2019-01-28 14:09:12 -07:00
Lisa Cawley 19529da2db
[DOCS] Delayed data annotations (#37939) 2019-01-28 13:04:38 -08:00
Benjamin Trent 7e4c0e6991
ML: Adds set_upgrade_mode API endpoint (#37837)
* ML: Add MlMetadata.upgrade_mode and API

* Adding tests

* Adding wait conditionals for the upgrade_mode call to return

* Adding tests

* adjusting format and tests

* Adjusting wait conditions for api return and msgs

* adjusting doc tests

* adding upgrade mode tests to black list
2019-01-28 09:07:30 -06:00
David Turner 2a610abef2
Document that auto_create_index is dynamic (#37903)
We changed the `action.auto_create_index` setting to be a dynamic cluster-level
setting in #20274 but today the reference manual indicates that it is still a
static node-level setting. This commit addresses this, and clarifies the
semantics of patterns that may both permit and forbid the creation of certain
indices.

Relates #7513
2019-01-28 09:45:55 +00:00
David Turner a5f578f7ea
Fix duplicate note x-refs in mapping.asciidoc (#37904)
The docs silently accept duplicate note markers (such as `<3>` here) but
formats them in an unexpected way. This change removes this duplication so that
the rendered documentation looks as intended.
2019-01-28 09:29:07 +00:00
David Roberts f2c0c26d15
[ML] Adjust structure finder for Joda to Java time migration (#37306)
The ML file structure finder has always reported both Joda
and Java time format strings.  This change makes the Java time
format strings the ones that are incorporated into mappings
and ingest pipeline definitions.

The BWC syntax of prepending "8" to these formats is used.
This will need to be removed once Java time format strings
become the default in Elasticsearch.

This commit also removes direct imports of Joda classes in the
structure finder unit tests.  Instead the core Joda BWC class
is used.
2019-01-26 20:19:57 +00:00
Jim Ferenczi cb451edb01
Allow nested fields in the composite aggregation (#37178)
This changes adds the support to handle `nested` fields in the `composite`
aggregation. A `nested` aggregation can be used as parent of a `composite`
aggregation in order to target `nested` fields in the `sources`.

Closes #28611
2019-01-25 14:00:39 +01:00
Jim Ferenczi 787acb14b9
Track total hits up to 10,000 by default (#37466)
This commit changes the default for the `track_total_hits` option of the search request
to `10,000`. This means that by default search requests will accurately track the total hit count
up to `10,000` documents, requests that match more than this value will set the `"total.relation"`
to `"gte"` (e.g. greater than or equals) and the `"total.value"` to `10,000` in the search response.
Scroll queries are not impacted, they will continue to count the total hits accurately.
The default is set back to `true` (accurate hit count) if `rest_total_hits_as_int` is set in the search request.
I choose `10,000` as the default because that's also the number we use to limit pagination. This means that users will be able to know how far they can jump (up to 10,000) even if the total number of hits is not accurate.

Closes #33028
2019-01-25 13:45:39 +01:00
Tim Vernum 03690d12b2
Remove TLS 1.0 as a default SSL protocol (#37512)
The default value for ssl.supported_protocols no longer includes TLSv1
as this is an old protocol with known security issues.
Administrators can enable TLSv1.0 support by configuring the
appropriate `ssl.supported_protocols` setting, for example:

xpack.security.http.ssl.supported_protocols: ["TLSv1.2","TLSv1.1","TLSv1"]

Relates: #36021
2019-01-25 15:46:39 +11:00
Mayya Sharipova a30ce6a00a
Rename feature, feature_vector and feature_query (#37794)
Ranaming as follows:
feature -> rank_feature
feature_vector -> rank_features
feature query -> rank_feature query

Ranaming is done to distinguish from other vector types.

Closes #36723
2019-01-24 19:18:48 -05:00
Peter Dyson 4313c2d4a7
[DOCS] More info on disabling swap (#37248)
* [DOCS] More info on disabling swap, inform that no Elasticsearch restart is required.
2019-01-25 08:39:58 +10:00
Lee Hinman 0f3c542850
Deprecate xpack.watcher.history.cleaner_service.enabled (#37782)
This deprecates the `xpack.watcher.history.cleaner_service.enabled` setting,
since all newly created `.watch-history` indices in 7.0 will use ILM to manage
their retention.

In 8.0 the setting itself and cleanup actions will be removed.

Resolves #32041
2019-01-24 15:31:31 -07:00
Christoph Büscher 967de04257
Uppercasing some docs section title (#37781)
Section titles are mostly uppercase, only a few cases where query DSL parameters
or Java method names are used as the title they should be lowercased.
2019-01-24 22:54:55 +01:00
Lisa Cawley 33cac52b2c
[DOCS] Remove beta tag from metricbeat monitoring (#37791) 2019-01-24 09:14:08 -08:00
niloct ddee1926f9 Update update-by-query.asciidoc (#37555)
Similar fix as #37370.
2019-01-24 16:27:42 +01:00
Marios Trivyzas f707fa9e0a
SQL: Introduce SQL DATE data type (#37693)
* SQL: Introduce SQL DATE data type

Support ANSI SQL's DATE type by introducing a runtime-only
ES SQL date type.

Closes: #37340
2019-01-24 13:41:58 +02:00
Albert Zaharovits b6936e3c1e
Remove index audit output type (#37707)
This commit removes the Index Audit Output type, following its deprecation
in 6.7 by 8765a31d4e6770. It also adds the migration notice (settings notice).

In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the
`xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881
2019-01-24 12:36:10 +02:00
Mayya Sharipova fdb66039d4
Change `rational` to `saturation` in script_score (#37766)
This change of the function name is necessary for conformity
with feature queries.

Closes #37714
2019-01-23 14:28:20 -05:00
Lee Hinman 427bc7f940
Use ILM for Watcher history deletion (#37443)
* Use ILM for Watcher history deletion

This commit adds an index lifecycle policy for the `.watch-history-*` indices.
This policy is automatically used for all new watch history indices.

This does not yet remove the automatic cleanup that the monitoring plugin does
for the .watch-history indices, and it does not touch the
`xpack.watcher.history.cleaner_service.enabled` setting.

Relates to #32041
2019-01-23 10:18:08 -07:00
Christoph Büscher b3f9becf5f
Modify removal_of_types.asciidoc (#37648)
After switching the default behaviour of "include_type_name" to "false" in 7.0,
some parts of the types removal documentation can be adapted as well.
2019-01-23 09:48:00 +01:00
Christoph Büscher 95a6951f78
Use new bulk API endpoint in the docs (#37698)
This change switches to using the typeless bulk API endpoint in the
documentation snippets where possible
2019-01-23 09:46:28 +01:00
Boaz Leskes 52ba407931
Expose sequence number and primary terms in search responses (#37639)
Users may require the sequence number and primary terms to perform optimistic concurrency control operations. Currently, you can get the sequence number via the `docvalues_fields` API but the primary term is not accessible because it is maintained by the `SeqNoFieldMapper` and the infrastructure can't find it. 

This commit adds a dedicated sub fetch phase to return both numbers that is connected to a new `seq_no_primary_term` parameter.
2019-01-23 09:01:58 +01:00
Mayya Sharipova 942fc13af5
Use plain text instead of latexmath
As latexmath is not rendered, using plain text instead

Closes #37718
2019-01-22 16:49:03 -05:00
Julie Tibshirani 992bfd2064
Remove additional references to 'type' from the _bulk documentation. (#37722) 2019-01-22 12:13:01 -08:00
Christoph Büscher 34f2d2ec91
Remove remaining occurances of "include_type_name=true" in docs (#37646) 2019-01-22 15:13:52 +01:00
Daniel Mitterdorfer 757932a975
Document that date math is locale independent
With this commit we add a note to the API conventions documentation that
all date math expressions are resolved independently of any locale. This
behavior might be puzzling to users that try to specify a different
calendar than a Gregorian calendar.

Closes #37330 
Relates #37663
2019-01-22 12:04:29 +01:00
David Kyle 0ae7f8630c
Document ml datafeed Id limitations (#37653) 2019-01-21 14:12:20 +00:00
Albert Zaharovits 5308746270
Remove Watcher Account "unsecure" settings (#36736)
Removes all sensitive settings (passwords, auth tokens, urls, etc...) for
watcher notifications accounts. These settings were deprecated (and
herein removed) in favor of their secure sibling that is set inside the
elasticsearch keystore. For example:
`xpack.notification.email.account.<id>.smtp.password`
is no longer a valid setting, and it is replaced by
`xpack.notification.email.account.<id>.smtp.secure_password`
2019-01-20 12:51:24 +02:00
Benjamin Trent 12cdf1cba4
ML: Add support for single bucket aggs in Datafeeds (#37544)
Single bucket aggs are now supported in datafeed aggregation configurations.
2019-01-18 15:08:53 -06:00
Martijn van Groningen a3030c51e2 [ILM] Add unfollow action (#36970)
This change adds the unfollow action for CCR follower indices.

This is needed for the shrink action in case an index is a follower index.
This will give the follower index the opportunity to fully catch up with
the leader index, pause index following and unfollow the leader index.
After this the shrink action can safely perform the ilm shrink.

The unfollow action needs to be added to the hot phase and acts as
barrier for going to the next phase (warm or delete phases), so that
follower indices are being unfollowed properly before indices are expected
to go in read-only mode. This allows the force merge action to execute
its steps safely.

The unfollow action has three steps:
* `wait-for-indexing-complete` step: waits for the index in question
  to get the `index.lifecycle.indexing_complete` setting be set to `true`
* `wait-for-follow-shard-tasks` step: waits for all the shard follow tasks
  for the index being handled to report that the leader shard global checkpoint
  is equal to the follower shard global checkpoint.
* `pause-follower-index` step: Pauses index following, necessary to unfollow
* `close-follower-index` step: Closes the index, necessary to unfollow
* `unfollow-follower-index` step: Actually unfollows the index using 
  the CCR Unfollow API
* `open-follower-index` step: Reopens the index now that it is a normal index
* `wait-for-yellow` step: Waits for primary shards to be allocated after
  reopening the index to ensure the index is ready for the next step

In the case of the last two steps, if the index in being handled is
a regular index then the steps acts as a no-op.

Relates to #34648

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Gordon Brown <gordon.brown@elastic.co>
2019-01-18 13:05:03 -07:00
Martijn van Groningen 6846666b6b
Add ccr follow info api (#37408)
* Add ccr follow info api

This api returns all follower indices and per follower index
the provided parameters at put follow / resume follow time and
whether index following is paused or active.

Closes #37127

* iter

* [DOCS] Edits the get follower info API

* [DOCS] Fixes link to remote cluster

* [DOCS] Clarifies descriptions for configured parameters
2019-01-18 16:37:21 +01:00
Christoph Büscher 3a96608b3f
Remove more include_type_name and types from docs (#37601) 2019-01-18 14:11:18 +01:00
Christoph Büscher 25aac4f77f
Remove `include_type_name` in asciidoc where possible (#37568)
The "include_type_name" parameter was temporarily introduced in #37285 to facilitate
moving the default parameter setting to "false" in many places in the documentation
code snippets. Most of the places can simply be reverted without causing errors.
In this change I looked for asciidoc files that contained the
"include_type_name=true" addition when creating new indices but didn't look
likey they made use of the "_doc" type for mappings. This is mostly the case
e.g. in the analysis docs where index creating often only contains settings. I
manually corrected the use of types in some places where the docs still used an
explicit type name and not the dummy "_doc" type.
2019-01-18 09:34:11 +01:00
Mike Place 080c073618 Minor docs cleanup (#37595)
Not all terminals are accessed via SSH.
2019-01-18 09:28:48 +01:00
Lisa Cawley 6dcb3af4c8
[DOCS] Adds size limitation to the get datafeeds APIs (#37578) 2019-01-17 10:47:15 -08:00
Lisa Cawley a2d9c464b2
[DOCS] Adds limitation to the get jobs API (#37549) 2019-01-17 08:21:37 -08:00
Jake Landis 587034dfa7
Add set_priority action to ILM (#37397)
This commit adds a set_priority action to the hot, warm, and cold
phases for an ILM policy. This action sets the `index.priority`
on the managed index to allow different priorities between the
hot, warm, and cold recoveries.

This commit also includes the HLRC and documentation changes.

closes #36905
2019-01-17 09:55:36 -06:00
Marios Trivyzas 1686c32ba9
SQL: Rename SQL type DATE to DATETIME (#37395)
* SQL: Rename SQL data type DATE to DATETIME

SQL data type DATE has only the date part (e.g.: 2019-01-14)
without any time information. Previously the SQL type DATE was
referring to the ES DATE which contains also the time part along
with TZ information. To conform with SQL data types the data type
`DATE` is renamed to `DATETIME`, since it includes also the time,
as a new runtime SQL `DATE` data type will be introduced down the road,
which only contains the date part and meets the SQL standard.

Closes: #36440

* Address comments
2019-01-17 10:17:58 +02:00
Przemyslaw Gomulka b6e5ccaf8a
Remove the AbstracLifecycleComponent constructor with Settings (#37523)
Adding the migration guide and removing the deprecated in 6.x
constructor

relates #35560
relates #34488
2019-01-17 09:10:09 +01:00
Jason Tedor 18a3e48a4a
Change file descriptor limit to 65535 (#37537)
Some systems default to a nofile ulimit of 65535. To reduce the pain of
deploying Elasticsearch to such systems, this commit lowers the required
limit from 65536 to 65535.
2019-01-16 17:19:12 -05:00
Marios Trivyzas 8932750efb
SQL: [Docs] Add an ES-SQL column for data types (#37529)
In order to distinguish the ES-SQL type from the standard SQL type
add a new ES-SQL column that will make clear this distingstion,
e.g.: datetime vs TIMSTAMP

Fixes: #37519
2019-01-16 16:16:21 +02:00
Lisa Cawley 17b0aa3f31
[DOCS] Intro for adding nodes (#37202) 2019-01-15 11:11:01 -08:00
Lisa Cawley 03cacdf16e
[DOCS] Adds CCR screenshots (#37439) 2019-01-15 10:53:29 -08:00
Lisa Cawley 44e83f30e2
[DOCS] Edits rollup API description (#37444) 2019-01-15 08:46:36 -08:00
niloct e71542ffd3 Update delete-by-query.asciidoc (#37370)
Tried my best to clarify sentence on `_delete_by_query` docs.
2019-01-15 08:05:24 -07:00
Boaz Leskes f7eb517eb8 Update the Flush API documentation (#33551)
The semantics of the API changed considerably since the documentation was written.

The main change is to remove references to memory reduction (this is related to refresh).
Instead, flush refers to recovery times. I also removed the references to trimming the translog
as the translog may be required for other purposes (operation history for ops based recovery
and complement ongoing file based recoveries).

Closes #32869
2019-01-15 11:45:10 +01:00
Julie Tibshirani 36a3b84fc9
Update the default for include_type_name to false. (#37285)
* Default include_type_name to false for get and put mappings.

* Default include_type_name to false for get field mappings.

* Add a constant for the default include_type_name value.

* Default include_type_name to false for get and put index templates.

* Default include_type_name to false for create index.

* Update create index calls in REST documentation to use include_type_name=true.

* Some minor clean-ups around the get index API.

* In REST tests, use include_type_name=true by default for index creation.

* Make sure to use 'expression == false'.

* Clarify the different IndexTemplateMetaData toXContent methods.

* Fix FullClusterRestartIT#testSnapshotRestore.

* Fix the ml_anomalies_default_mappings test.

* Fix GetFieldMappingsResponseTests and GetIndexTemplateResponseTests.

We make sure to specify include_type_name=true during xContent parsing,
so we continue to test the legacy typed responses. XContent generation
for the typeless responses is currently only covered by REST tests,
but we will be adding unit test coverage for these as we implement
each typeless API in the Java HLRC.

This commit also refactors GetMappingsResponse to follow the same appraoch
as the other mappings-related responses, where we read include_type_name
out of the xContent params, instead of creating a second toXContent method.
This gives better consistency in the response parsing code.

* Fix more REST tests.

* Improve some wording in the create index documentation.

* Add a note about types removal in the create index docs.

* Fix SmokeTestMonitoringWithSecurityIT#testHTTPExporterWithSSL.

* Make sure to mention include_type_name in the REST docs for affected APIs.

* Make sure to use 'expression == false' in FullClusterRestartIT.

* Mention include_type_name in the REST templates docs.
2019-01-14 13:08:01 -08:00
Jay Modi f3edbe2911
Security: remove SSL settings fallback (#36846)
This commit removes the fallback for SSL settings. While this may be
seen as a non user friendly change, the intention behind this change
is to simplify the reasoning needed to understand what is actually
being used for a given SSL configuration. Each configuration now needs
to be explicitly specified as there is no global configuration or
fallback to some other configuration.

Closes #29797
2019-01-14 14:06:22 -07:00
Nhat Nguyen 15aa3764a4
Reduce recovery time with compress or secure transport (#36981)
Today file-chunks are sent sequentially one by one in peer-recovery. This is a
correct choice since the implementation is straightforward and recovery is
network bound in most of the time. However, if the connection is encrypted, we
might not be able to saturate the network pipe because encrypting/decrypting
are cpu bound rather than network-bound.

With this commit, a source node can send multiple (default to 2) file-chunks
without waiting for the acknowledgments from the target.

Below are the benchmark results for PMC and NYC_taxis.

- PMC (20.2 GB)

| Transport | Baseline | chunks=1 | chunks=2 | chunks=3 | chunks=4 |
| ----------| ---------| -------- | -------- | -------- | -------- |
| Plain     | 184s     | 137s     | 106s     | 105s     | 106s     |
| TLS       | 346s     | 294s     | 176s     | 153s     | 117s     |
| Compress  | 1556s    | 1407s    | 1193s    | 1183s    | 1211s    |

- NYC_Taxis (38.6GB)

| Transport | Baseline | chunks=1 | chunks=2 | chunks=3 | chunks=4 |
| ----------| ---------| ---------| ---------| ---------| -------- |
| Plain     | 321s     | 249s     | 191s     |  *       | *        |
| TLS       | 618s     | 539s     | 323s     | 290s     | 213s     |
| Compress  | 2622s    | 2421s    | 2018s    | 2029s    | n/a      |

Relates #33844
2019-01-14 15:14:46 -05:00
Peter Dyson 96cfa000a5
[DOCS] copy_to only works one level deep, not recursively (#37249) 2019-01-13 16:24:34 +10:00
Jason Tedor 03be4dbaca
Introduce retention lease persistence (#37375)
This commit introduces the persistence of retention leases by persisting
them in index commits and recovering them when recovering a shard from
store.
2019-01-12 14:43:19 -08:00
Boaz Leskes 093d4a371c followup to #37284 with additional feedback 2019-01-11 08:07:35 -08:00
Boaz Leskes cae71cddfe
Document Seq No powered optimistic concurrency control (#37284)
Add documentation to describe the new sequence number powered optimistic concurrency control

Relates #36148
Relates #10708
2019-01-11 07:59:15 -08:00
Jake Landis fad6329808
add link to freeze from cold phase (#37332) 2019-01-11 08:58:41 -06:00
Alexander Reelsen bbd093059f
Add whitelist to watcher HttpClient (#36817)
This adds a configurable whitelist to the HTTP client in watcher. By
default every URL is allowed to retain BWC. A dynamically configurable
setting named "xpack.http.whitelist" was added that allows to
configure an array of URLs, which can also contain simple regexes.

Closes #29937
2019-01-11 09:22:47 +01:00
Jake Landis 9fd4d6ba4b
remove `+` references for xpack.monitoring.collection.indices (#37282)
`+` for index name inclusions is no longer supported for 6.x+. This
commit removes references of the `+` from the documenation. System
indices additional example is also included.

fixes #37237
2019-01-10 17:06:44 -06:00
Lee Hinman 04dcb13ac4
Remove "beta" modifier from ILM documentation (#37326) 2019-01-10 13:38:14 -07:00
Lee Hinman a2d63ecdc0
Link ILM management and policy information in ILM API documentation (#37324)
Previously these were only linked in a circuitous way rather than being
available from the top level API documentation and "Put Lifecycle" API docs.
This makes them slightly easier to find for a user.
2019-01-10 12:34:44 -07:00
lcawl 2d5a8ec59d [DOCS] Remove unused screenshots 2019-01-10 11:10:25 -08:00
Andrei Stefan 4a92de214a
SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) (#37254)
* provide overriden `hashCode` and toString methods to account for `DISTINCT`
* change the analyzer for scenarios where `COUNT <field_name>` and `COUNT DISTINCT` have different paths
* defined a new `filter` aggregation encapsulating an `exists` query to filter out null or missing values
2019-01-10 09:51:51 +02:00
Albert Zaharovits 4efbda1bc0
[DOCS] Rolling upgrade with old internal indices (#37184)
Upgrading the Elastic Stack perfectly documents the process to
upgrade ES from 5 to 6 when internal indices are present. However,
the rolling upgrade docs do not mention anything about internal indices.

This adds a warning in the rolling upgrade procedure, highlighting that
internal indices should be upgraded before the rolling upgrade procedure
can be started.
2019-01-09 18:14:22 +02:00
Christoph Büscher c149bb8cc2
Support 'include_type_name' in RestGetIndicesAction (#37149)
This change adds support for the 'include_type_name' parameter for the
indices.get API. This parameter, which defaults to `false` starting in 7.0,
changes the response to not include the indices type names any longer.

If the parameter is set in the request, we additionally emit a deprecation
warning since using the parameter should be only temporarily necessary while
adapting to the new response format and we will remove it with the next major
version.
2019-01-09 14:17:17 +01:00
David Turner d9e2ebca67
Add more detail to recovery bandwidth limit docs (#37156) 2019-01-09 08:18:25 +00:00
Jun Ohtani 38b698d455
[Analysis] Deprecate Standard Html Strip Analyzer in master (#26719)
* [Analysis] Deprecate Standard Html Strip Analyzer

Deprecate only Standard Html Strip Analyzer
If user create index with the analyzer since 7.0, es throws an exception.
If an index was created before 7.0, es issue deprecation log
We will remove it in 8.0

Related #4704
2019-01-09 12:42:00 +09:00
Mayya Sharipova ec32e66088 Deprecate reference to _type in lookup queries (#37016)
Relates to #35190
2019-01-08 18:46:41 -08:00
Andrew Banchich 1d371a427f [Docs] Fix wrong math in overview.asciidoc (#37209) 2019-01-08 10:25:02 +01:00
lcawl 382e4d39ef [DOCS] Cleans up xpackml attributes 2019-01-07 14:33:10 -08:00
Lisa Cawley f307847f29
[DOCS] Adds overview and API ref for cluster voting configurations (#36954) 2019-01-07 09:11:14 -08:00
Josh Soref edb48321ba [DOCS] Various spelling corrections (#37046) 2019-01-07 14:44:12 +01:00
Andrei Stefan 39a072389c
SQL: add sub-selects to the Limitations page (#37012) 2019-01-07 10:08:51 +02:00
Jim Ferenczi e38cf1d0dc
Add the ability to set the number of hits to track accurately (#36357)
In Lucene 8 searches can skip non-competitive hits if the total hit count is not requested.
It is also possible to track the number of hits up to a certain threshold. This is a trade off to speed up searches while still being able to know a lower bound of the total hit count. This change adds the ability to set this threshold directly in the track_total_hits search option. A boolean value (true, false) indicates whether the total hit count should be tracked in the response. When set as an integer this option allows to compute a lower bound of the total hits while preserving the ability to skip non-competitive hits when enough matches have been collected.

Relates #33028
2019-01-04 20:36:49 +01:00
Adam Thomson ac4aecc92d [Docs] Update ingest-node.asciidoc (#37116) 2019-01-04 19:33:06 +01:00
Simon Willnauer 41d7e3a2fe
Expose `search.throttled` on `_cat/indices` (#37073)
Today it's very difficult to see which indices are frozen or rather
throttled via the commonly used monitoring APIs. This change adds
a cell to the `_cat/indices` API to render if an index is `search.throttled`

Relates to #34352
2019-01-04 13:49:40 +01:00
Luca Cavanna bfe6f091da [DOCS] fix typo in 7.0.0-alpha2 release notes 2019-01-04 10:38:34 +01:00
Dimitris Athanasiou 586453fef1
[ML] Remove types from datafeed (#36538)
Closes #34265
2019-01-04 09:43:44 +02:00
lcawley 00997b4f60 [DOCS] Fixes broken links 2019-01-04 17:41:28 +10:00
Peter Dyson 7839cec301
subsequent fix to edit in recent cherry-pick 2019-01-04 17:34:24 +10:00
Peter Dyson 7cc9754d94
fix to edit in recent cherry-pick 2019-01-04 17:26:42 +10:00
Peter Dyson 0ff2707c9f Add Profile API to search speed tuning howto (#29489)
* Add Profile API to search speed tuning howto

Seemed useful to mention the Profile API in the context of tuning for search speed.
2019-01-04 16:49:12 +10:00
Tal Levy eaeccd8401
[ILM] Add Freeze Action (#36910)
This commit adds a new ILM Action for
freezing indices in the cold phase.

Closes #34630.
2019-01-03 15:00:40 -08:00
Igor Motov d6acd8e15f
Docs: add clarification about geohash use in geohashgrid agg (#36901)
Adds an example on translating geohashes returned by geohashgrid 
agg as bucket keys into geo bounding box filters in elasticsearch as well
as 3rd party applications.

Closes #36413
2019-01-03 15:40:48 -05:00
Lisa Cawley 366b6b5e11
[DOCS] Adds cluster setting to enable/disable config migration (#36948) 2019-01-03 09:26:57 -08:00
Marios Trivyzas 33137907cf
SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg (#36933)
Enhance error message for the case that the 2nd argument of PERCENTILE
and PERCENTILE_RANK is not a foldable, as it doesn't make sense to have
a dynamic value coming from a field.

Fixes: #36903
2019-01-03 13:55:09 +02:00
Christoph Büscher 046f86f274
Deprecate use of type in reindex request body (#36823)
Types can be used both in the source and dest section of the body which will
be translated to search and index requests respectively. Adding a deprecation warning
for those cases and removing examples using more than one type in reindex since
support for this is going to be removed.
2019-01-03 10:29:14 +01:00
David Turner a40f0545e6
Make ILM examples larger (#37039)
There are a handful of examples in the ILM documentation that could result in
rolling over indices more quickly than we might normally recommend,
contributing to over-sharding in cases where the examples are copied without
modification. This change makes some numbers bigger to try and avoid this.
2019-01-03 08:28:49 +00:00
Daniel Mitterdorfer 75f3443c62
Rename setting to enable mmap
With this commit we rename `node.store.allow_mmapfs` to
`node.store.allow_mmap`. Previously this setting has controlled whether
`mmapfs` could be used as a store type. With the introduction of
`hybridfs` which also relies on memory-mapping,
`node.store.allow_mmapfs` also applies to `hybridfs` and thus we rename
it in order to convey that it is actually used to allow memory-mapping
but not a specific store type.

Relates #36668
Relates #37070
2019-01-03 07:10:34 +01:00
Luca Cavanna 42ea644903
Remove single shard optimization when suggesting shard_size (#37041)
When executing terms aggregations we set the shard_size, meaning the
number of buckets to collect on each shard, to a value that's higher than
the number of requested buckets, to guarantee some basic level of
precision. We have an optimization in place so that we leave shard_size
set to size whenever we are searching against a single shard, in which
case maximum precision is guaranteed by definition.

Such optimization requires us access to the total number of shards that
the search is executing against. In the context of cross-cluster search,
once we will introduce multiple reduction steps (one per cluster) each
cluster will only know the number of local shards, which is problematic
as we should only optimize if we are searching against a single shard in a
single cluster. It could be that we are searching against one shard per cluster
in which case the current code would optimize number of terms causing
a loss of precision.

While discussing how to address the CCS scenario, we decided that we do
not want to introduce further complexity caused by this single shard
optimization, as it benefits only a minority of cases, especially when
the benefits are not so great.

This commit removes the single shard optimization, meaning that we will
always have heuristic enabled on how many number of buckets to collect
on the shards, even when searching against a single shard.

This will cause more buckets to be collected when searching against a single
shard compared to before. If that becomes a problem for some users, they
can work around that by setting the shard_size equal to the size.

Relates to #32125
2019-01-02 17:45:49 +01:00
Abdullah DURSUN 8a02bacf76 Fix typo in multi-search.asciidoc (#37060) 2019-01-02 10:32:42 +01:00
Daniel Mitterdorfer f0052b1a7a
Add hybridfs store type
With this commit we introduce a new store type `hybridfs` that is a
hybrid between `mmapfs` and `niofs`. This store type chooses different
strategies to read Lucene files based on the read access pattern (random
or linear) in order to optimize performance.

This store type has been available in earlier versions of Elasticsearch
as `default_fs`. We have chosen a different name now in order to convey
the intent of the store type instead of tying it to the fact whether it
is the default choice.

Relates #36668
2019-01-02 10:10:32 +01:00
Peter Lyons 794303af73 [Docs] Fix URL path to aliases (#36852)
This API URL contains /aliases/aliases and that seems to be a duplicate.
2018-12-31 17:58:49 +01:00
Andrei Stefan 9c283aed0a
SQL: Move the SQL ODBC client applications pages from the ODBC section into the general ES SQL Client Applications page (#37004) 2018-12-31 14:07:39 +02:00
Andrei Stefan 0b1cb5300e
Move ODBC documentation to `elasticsearch` repository (#36955) 2018-12-27 11:55:20 +02:00