Commit Graph

6473 Commits

Author SHA1 Message Date
Christophe Bismuth acdf9666d5 Add `minimum_should_match` section to the query_string docs
Closes #34142
2018-11-30 16:10:13 +01:00
Jim Ferenczi 08b9e31373 [TEST] fix link in get-watch documentation 2018-11-30 14:40:09 +01:00
patrykk21 bb2cf7e6be [Docs] Clarify search_after behavior
Closes #34232
2018-11-30 14:30:23 +01:00
Jim Ferenczi 5e6460acb3 [TEST] fix typo in get-watch documentation 2018-11-30 12:53:04 +01:00
Jim Ferenczi 5c7b2c5f9b
HLRC: Add get watch API (#35531)
This changes adds the support for the get watch API in the high level rest client.
2018-11-30 11:02:46 +01:00
Peter Dyson 63a9571c4c
Explain nondeterministic template merging (#35405) 2018-11-30 14:56:32 +10:00
Gordon Brown d567f89719
Recommend index templates when using ILM Rollover (#35922)
Because rollover does not propogate the ILM policy in use, add a note to
the docs recommending the use of index templates when using rollover.
2018-11-29 14:09:51 -07:00
Zachary Tong 61c2db5ebb Revert "Deprecate X-Pack centric rollup endpoints (#35962)"
This reverts commit b84f1f6a3a.
2018-11-29 12:58:23 -05:00
Toby McLaughlin 8006068873 [DOCS] Remove "platinum" references for Docker TLS (#35890) 2018-11-29 09:33:08 -08:00
Jason Tedor 9fa9e1419f
Remove X-Pack centric graph endpoints (#36010)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-29 07:09:37 -05:00
Alan Woodward a646f85a99
Ensure TokenFilters only produce single tokens when parsing synonyms (#34331)
A number of tokenfilters can produce multiple tokens at the same position.  This
is a problem when using token chains to parse synonym files, as the SynonymMap
requires that there are no stacked tokens in its input.

This commit ensures that when used to parse synonyms, these tokenfilters either produce
a single version of their input token, or that they throw an error when mappings are 
generated.  In indexes created in elasticsearch 6.x deprecation warnings are emitted in place 
of the error. 

* asciifolding and cjk_bigram produce only the folded or bigrammed token
* decompounders, synonyms and keyword_repeat are skipped
* n-grams, word-delimiter-filter, multiplexer, fingerprint and phonetic throw errors

Fixes #34298
2018-11-29 10:35:38 +00:00
Ryan Ernst fbdfec4305
Docs: Add note about oss repositories for deb/rpm (#35973)
This commit adds a note about configring the yum/apt repositories for
oss only packages.

closes #35960
2018-11-29 00:22:10 -08:00
Martijn van Groningen 6d01170ca1
[HLRC] Added support for CCR Delete Auto Follow Pattern API (#35981)
This change also adds documentation for the Delete Auto Follow Pattern API.

Relates to #33824
2018-11-29 08:06:31 +01:00
Ignacio Vera 93ed8b7d61
HLRC: Add delete user action (#35294)
* HLRC: Add delete user action

It adds delete user action to the high level rest client.

Relates #29827
2018-11-29 07:52:56 +01:00
Alpar Torok e0a678f0c4
Remove version.qualified from MainResponse (#35412)
The fully qualified version will be returned as `version.number`
2018-11-29 08:41:39 +02:00
Lisa Cawley 906421cdad
[DOCS] Adds higher-level monitoring configuration page (#35926) 2018-11-28 12:36:29 -08:00
Gordon Brown c26af3b0a2
Deprecate X-Pack centric Migration endpoints (#35976)
This commit is part of our plan to deprecate and remove the use of
_xpack in the REST API routes.
2018-11-28 13:19:33 -07:00
Lisa Cawley a72430e1ef
[DOCS] Adds TLS warning to rolling upgrades (#35841) 2018-11-28 09:38:58 -08:00
Simon Willnauer daef498cd9 [DOCS] Fix HLRC reference to unfreeze_index 2018-11-28 16:55:55 +01:00
Simon Willnauer 89e4ac8fcc
Add high-level REST client API for `_freeze` and `_unfreeze` (#35723)
This change adds support for `_freeze` and `_unfreeze` to the HLRC

Relates to #34352
2018-11-28 15:42:12 +01:00
Jason Tedor a3186e4a32
Deprecate X-Pack centric license endpoints (#35959)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-28 08:24:35 -05:00
Jason Tedor 78ac12d952
Remove deprecated Graph endpoints (#35956)
We had some endpoints in Graph that are deprecated for removal in
7.0.0. This commit removes these deprecated endpoints.
2018-11-28 08:24:15 -05:00
Andrei Stefan aabff7318e
SQL: DATABASE() and USER() system functions (#35946) 2018-11-28 15:20:04 +02:00
Peter Dyson 1f25a0bd31 [Docs] Add example for updating meta field (#35893) 2018-11-28 12:04:57 +01:00
Christoph Büscher 33865211db
[Docs] Emphazise suggest behaviour with missing query part (#35393)
Add a short extra sentence that explains that a missing query part in a search
request containing a "suggest" section will mean only suggestions are returned.

Closes #31640
2018-11-28 12:01:27 +01:00
Vladimir Dolzhenko f0c5f0c099 [HLRC] XPack ML info action (#35777)
Relates to #29827
2018-11-28 10:58:20 +00:00
Costin Leau 95346926ef
SQL: Lock down JDBC driver (#35798)
Move classes under the same package to avoid internal classes being
 exposed to the outside. Remove public visibility outside 3 classes:
 EsDriver, EsDataSource and EsTypes.
The driver only has one package, namely org.elasticsearch.xpack.sql.jdbc
Use Es prefix for classes to ease name conflict and indicate their
 destination

Fix #35437
2018-11-28 12:48:26 +02:00
Jason Tedor c42d9d91c9
Deprecate X-Pack centric SQL endpoints (#35964)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-27 22:16:21 -05:00
Michael Basnight 19ed17195f
Docs: DRY up indices docs (#35971)
This commit DRYs up the indices folder as well as fixing a few minor
mishaps that were in the docs.
2018-11-27 19:40:49 -06:00
Jason Tedor b84f1f6a3a
Deprecate X-Pack centric rollup endpoints (#35962)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-27 20:34:17 -05:00
Tal Levy a0b319a123 [DOCS] fix HLRC ILM doc misreferenced tag 2018-11-27 14:28:25 -08:00
Ioannis Kakavas 580b5baf21
Add realm information for Authenticate API (#35648)
- Add the authentication realm and lookup realm name and type in the response for the _authenticate API
- The authentication realm is set as the lookup realm too (instead of setting the lookup realm to null or empty ) when no lookup realm is used.
2018-11-27 23:35:42 +02:00
Tal Levy fe603e9163
[ILM] add HLRC docs to remove-policy-from-index (#35759)
This primarily introduces documentation for the HLRC
remove-policy-from-index action.
2018-11-27 13:28:58 -08:00
Zachary Tong 48fa251812
[Rollup] Add more diagnostic stats to job (#35471)
* [Rollup] Add more diagnostic stats to job

To help debug future performance issues, this adds the
 min/max/avg/count/total latencies (in milliseconds) for search
and bulk phase.  This latency is the total service time including
transfer between nodes, not just the `took` time.

It also adds the count of search/bulk failures encountered during
runtime.  This information is also in the log, but a runtime counter
will help expose problems faster

* review cleanup

* Remove dead ParseFields
2018-11-27 15:46:10 -05:00
Jeff Hajewski 49087f16f5 Adds deprecation logging to ScriptDocValues#getValues. (#34279)
`ScriptDocValues#getValues` was added for backwards compatibility but no
longer needed. Scripts using the syntax `doc['foo'].values` when
`doc['foo']` is a list should be using `doc['foo']` instead.

Closes #22919
2018-11-27 14:30:13 -05:00
Simon Willnauer ad1f0dccd4
Validate metdata on `_msearch` (#35938)
MultiSearchRequests issues through `_msearch` now validate all keys
in the metadata section. Previously unknown keys were ignored
while now an exception is thrown.

Closes #35869
2018-11-27 17:08:24 +01:00
Yogesh Gaikwad 31fdb76973
[DOCS] Document authorization_realms for Kerberos realm (#35927)
This commit adds documentation for authorization_realms
setting for the Kerberos realm and also corrects a typo in
existing documentation.

Co-authored-by: @A-Hall
2018-11-27 19:09:17 +11:00
Martijn van Groningen df1e02d0d5
[HLRC] Added support for CCR Put Auto Follow Pattern API (#35780)
This change also adds documentation for the Put Auto Follow Pattern API.

Relates to #33824
2018-11-27 08:53:22 +01:00
Tim Vernum 3435fc4613
HLRC: Add ability to put user with a password hash (#35844)
Update PutUserRequest to support password_hash (see: #35242)

This also updates the documentation to bring it in line with our more
recent approach to HLRC docs.
2018-11-27 15:07:24 +11:00
Gordon Brown 119835decd
Always enforce cluster-wide shard limit (#34892)
This removes the option to run a cluster without enforcing the
cluster-wide shard limit, making strict enforcement the default and only
behavior.  The limit can still be adjusted as desired using the cluster
settings API.
2018-11-26 17:05:12 -07:00
Christoph Büscher 4d525e3e33
[Docs] Align callouts in search.asciidoc (#35897) 2018-11-27 00:05:15 +01:00
Gordon Brown 5c11b8612a
Add HLRC docs for Explain Lifecycle (#35803)
Adds HLRC documentation for the Explain Lifecycle API.
2018-11-26 15:25:56 -07:00
Marios Trivyzas 3f7cae3f0d
SQL: Implement GREATEST and LEAST functions (#35879)
Add GREATEST(expr1, expr2, ... exprN) and LEAST(expr1, expr2, exprN)
functions which are in the family of CONDITIONAL functions.

Implementation follows PostgreSQL behaviour, so the functions return
`NULL` when all of their arguments evaluate to `NULL`.

Renamed `CoalescePipe` and `CoalesceProcessor` to `ConditionalPipe` and
`ConditionalProcessor` respectively, to be able to reuse them for
`Greatest` and `Least` evaluations. To achieve that `ConditionalOperation`
has been added to differentiate between the functionalities at execution
time.

Closes: #35878
2018-11-26 18:21:36 +01:00
Christophe Bismuth b95a4db6e6 Throw a parsing exception when boost is set in span_or query (#28390) (#34112) 2018-11-26 12:15:59 -05:00
Jake Landis 4b99a663c1
ingest: fix broken doc link 2018-11-26 10:34:42 -06:00
Ed Savage 13e11966ca
[HLRC][ML] Add delete expired data API (#35906)
Relates to #29827
2018-11-26 16:15:54 +00:00
Jake Landis 7f7b31723e
ingest: extended `if` documentation (#35044)
part of #33188
2018-11-26 09:35:45 -06:00
Marios Trivyzas b078e2970c
SQL: Implement null safe equality operator `<=>` (#35873)
This operator handles nulls in different way than the normal `=`.
If one of the operants is `null` and the other not it returns `false`.
If both operants are `null` it returns `true`. Therefore in contrary to
`=`, which returns `null` if at least one of the operants is `null`, this one
never returns `null` as a result.

Closes: #35871
2018-11-26 14:02:02 +01:00
Ioannis Kakavas acb65dd004 [DOCS] Fix missing include for get-roles 2018-11-26 11:45:39 +02:00
Ioannis Kakavas 8daa854f90
[HLRC] Add support for get roles API (#35787)
This commits adds support for the Get Roles API to the HLRC

Relates: #29827
2018-11-26 11:25:07 +02:00
Andrei Stefan 00e6fec718
SQL: XPack FeatureSet functionality (#35725)
* Introduced "client.id" parameter for REST requests
* Bug that made the Verifier run twice, fixed in the Analyzer
* Single node IT and unit testing
2018-11-26 07:11:43 +02:00
Tal Levy a0f8a22d53
[TEST] ILM skip checking _ilm/explain response (#35830)
relates #35562

Closes #35824.
2018-11-25 17:19:17 -08:00
Marios Trivyzas 410f570d5f
SQL: Implement NULLIF(expr1, expr2) function (#35826)
NULLIF returns null if the 2 expressions are equal or the
expr1 otherwise.

Closes: #35818
2018-11-23 22:19:27 +01:00
William Desportes a204d1cdff [Docs] Fix typo in datehistogram-aggregation.asciidoc (#35855) 2018-11-23 15:16:53 +01:00
David Roberts 3c059ee057
[HLRC][ML] Add ML find file structure API (#35833)
Relates to #29827
2018-11-23 06:58:05 +00:00
Martijn van Groningen 813e053f62
Fixed response classes in hlrc docs 2018-11-23 07:52:45 +01:00
Stéphane Campinas fb80d298de [Docs] Correct template example description #35829 2018-11-23 05:29:20 +01:00
David Roberts 6f46584380
[ML] Add docs for ML info endpoint (#35783)
This endpoint was not previously documented as it was not
particularly useful to end users.  However, since the HLRC
will support the endpoint we need some documentation to
link to.

The purpose of the endpoint is to provide defaults and
limits used by ML.  These are needed to fully understand
configurations that have missing values because the missing
value means the default should be used.

Relates #35777
2018-11-22 16:23:31 +00:00
Martijn van Groningen 96a741f2c2
[TEST] escape brackets
Relates to #35496
2018-11-22 14:58:42 +01:00
Mayya Sharipova b6014d971c
Forbid negative scores in functon_score query (#35709)
* Forbid negative scores in functon_score query

- Throw an exception when scores are negative in field_value_factor
function
- Throw an exception when scores are negative in script_score
function

Relates to #33309
2018-11-22 06:08:48 -05:00
Marios Trivyzas 92acf47c16
SQL: Implement NVL(expr1, expr2) (#35794)
Add NVL as alias to IFNULL as they have the same
behaviour. Add basic tests and docs.

Closes: #35782
2018-11-22 11:41:00 +01:00
Alan Woodward be8097f9ce
Improve docs for index_prefixes option (#35778)
This commit moves the documentation and examples for the `index_prefixes`
option on text fields to its own file, to bring it in line with other mapping 
parameters, and expands a bit on both.
2018-11-22 09:20:46 +00:00
Lisa Cawley 78209188bc
[DOCS] Adds more authorization info for CCR APIs (#35606) 2018-11-21 15:01:59 -08:00
Benjamin Trent 90a8e4b259
HLRC: ML Delete event from Calendar (#35760)
* HLRC: Delete event from calendar

* adjusting tests

* adjusting code to make it more readable
2018-11-21 16:22:04 -06:00
Marios Trivyzas d95d885bae
SQL: Implement ISNULL(expr1, expr2) (#35793)
Add ISNULL as an alias of IFNULL as they have the
same behaviour. Add basic test and docs.

Closes: #35781
2018-11-21 23:15:10 +01:00
Marios Trivyzas e179bd393d
SQL: Implement IFNULL variant of COALESCE (#35762)
IFNULL is a MySQL variant (also used in other DBs) which
takes only 2 arguments and returns the first one that is not null.

Closes: #35749
2018-11-21 17:07:07 +01:00
Alan Woodward 26cc8ff8c3
Add pointer to the index-phrases option in shingle filter docs (#35771)
We should be discouraging the use of shingle filters and instead pointing users to the
index-phrases parameter on text fields.
2018-11-21 15:27:11 +00:00
Ioannis Kakavas 25f83ae08c
[HLRC] Add support for get application privileges API (#35556)
This commits adds support for the Get Application Privileges
API to the HLRC

Relates: #29827
2018-11-21 16:38:17 +02:00
Martijn van Groningen 9b2ab064cf
[HLRC] Added support for CCR Resume Follow API (#35638)
This change also adds documentation for the Resume Follow API

Relates to #33824
2018-11-21 10:52:03 +01:00
Dimitrios Liappis 1cb578b435
[DOCS] Update sysctl instructions for Docker on Mac (#35755)
Recent Docker for Mac releases[1] have a different path to the tty for
accessing the console of the xhyve vm, required for altering the
`vm.max_map_count` sysctl.

Update instructions on how to enter the xhyve vm for altering the
`vm.max_map_count` sysctl setting on Docker for Mac.

Closes #34817 

[1]
https://forums.docker.com/t/is-it-possible-to-ssh-to-the-xhyve-machine/17426/13
2018-11-21 11:32:29 +02:00
Ed Savage 4f857c4f8d
[HLRC][ML] Add ML revert model snapshot API (#35750)
Relates to #29827
2018-11-21 09:10:37 +00:00
Alan Woodward f6a43b5939
Add a prebuilt ICU Analyzer (#34958)
The ICU plugin provides the building blocks of an analysis chain, but doesn't actually have a prebuilt analyzer. It would be a better for users if there was a simple analyzer that they could use out of the box, and also something we can point to from the CJK Analyzer docs as a superior alternative.

Relates to #34285
2018-11-21 09:00:48 +00:00
Martijn van Groningen a6647a20a9
[HLRC] Added support for CCR Unfollow API (#35693)
This change also adds documentation for the Unfollow API

Relates to #33824
2018-11-21 07:48:29 +01:00
lcawl fa6d8a5004 [DOCS] Fixes navtitle for CCR API 2018-11-20 17:31:01 -08:00
Marios Trivyzas b1818dbdce
SQL: [docs] Add documentation for COALESCE (#35740)
Follows: #35253
2018-11-21 01:43:05 +01:00
Lisa Cawley 04a087fe2f
[DOCS] Adds authorization info for CCR APIs (#35557) 2018-11-20 16:23:01 -08:00
Benjamin Trent 84db1e42c0
HLRC: ML Get Calendar Events (#35747)
* HLRC: ML Get Calendar Events

* Addressing PR comments
2018-11-20 16:40:31 -06:00
Mayya Sharipova 643bb20137
Add a new query type - ScriptScoreQuery (#34533)
* Add a new query type - ScriptScoreQuery

script_score query uses script to calculate document scores.
Added as a substitute for function_score with an intentation
to deprecate function_scoreq query.

```http
GET /_search
{
    "query": {
        "script_score" : {
            "query": {
                "match": { "message": "elasticsearch" }
            },
            "script" : {
              "source": "Math.log(2 + doc['likes'].value)"
            },
            "min_score" : 2
        }
    }
}
```

Add several functions to painless to be used inside script_score:

double rational(double, double)
double sigmoid(double, double, double)
double randomNotReproducible() 
double randomReproducible(String, int) 

double decayGeoLinear(String, String, String, double, GeoPoint)
double decayGeoExp(String, String, String, double, GeoPoint)
double decayGeoGauss(String, String, String, double, GeoPoint)

double decayNumericLinear(String, String, String, double, double)
double decayNumericExp(String, String, String, double, double)
double decayNumericGauss(String, String, String, double, double)

double decayDateLinear(String, String, String, double, JodaCompatibleZonedDateTime)
double decayDateExp(String, String, String, double, JodaCompatibleZonedDateTime)
double decayDateGauss(String, String, String, double, JodaCompatibleZonedDateTime)

Date functions only works on dates in  the default format and default time zone
2018-11-20 16:10:06 -05:00
Gordon Brown 17780ce07e
Add HLRC docs for Delete Lifecycle Policy (#35664)
Adds documenatation for the Delete Lifecycle Policy API to the HLRC
documentation.
2018-11-20 11:32:41 -07:00
Tal Levy d061b3999a
[ILM] HLRC-ILM Retry Lifecycle Policy docs (#35715)
this adds documentation for the retry method in the
high-level-ilm-rest-client.

this PR also renames retryLifecycleStep to retryLifecyclePolicy in the index-lifecycle-client
2018-11-20 07:05:27 -08:00
Benjamin Trent 7657e6d274
HLRC ML Add Event To Calendar API (#35704)
* HLRC: ML Adding Post event to calendar api

* Fixing tests and serialization

* removing unused import
2018-11-20 08:15:21 -06:00
Benjamin Trent d707838c02
HLRC: ML Delete job from calendar (#35713) 2018-11-20 07:43:34 -06:00
Ed Savage 844483a99a
[HLRC][ML] Add ML update model snapshot API (#35537) (#35694)
Relates to #29827
2018-11-20 12:18:29 +00:00
Takuro Wada 7b2d547e8e [Docs] Delete inappropriate backtick (#35722) 2018-11-20 10:08:32 +01:00
Simon Willnauer 29ef442841
Add a `_freeze` / `_unfreeze` API (#35592)
This commit adds a rest endpoint for freezing and unfreezing an index.
Among other cleanups mainly fixing an issue accessing package private APIs
from a plugin that got caught by integration tests this change also adds
documentation for frozen indices.
Note: frozen indices are marked as `beta` and available as a basic feature.

Relates to #34352
2018-11-20 08:03:24 +01:00
Jason Tedor 220c175484
Provide links to auto-follow docs (#35674)
This commit adds a link to the auto-follow docs from the CCR getting
started docs and the relevant CCR APIs.
2018-11-19 20:40:58 -05:00
Gordon Brown e012ac4649
Add HLRC docs for ILM Start/Stop/Status (#35672)
Adds HLRC documentation for Start ILM, Stop ILM, and ILM Status APIs, as
these APIs are tightly linked to each other.
2018-11-19 17:17:01 -07:00
Tal Levy 90c0d444cf
[ILM] DOCS: clarify interaction between rollover and min_age (#35663)
This adds an example to drive home the semantics of `min_age` and explain
how actions from one phase must complete before `min_age` is even tested

closes #34020.
2018-11-19 13:26:20 -08:00
Benjamin Trent 214bc96738
HLRC: ML Add Job to Calendar API (#35666) 2018-11-19 11:41:49 -06:00
Mayya Sharipova aaeb47d1f7
HLRC for _mtermvectors (#35266)
relates to #27205
2018-11-19 12:24:31 -05:00
Itamar Syn-Hershko 156b3cae15 [Docs] Fix filter context in script-query.asciidoc (#35677)
The docs say script queries are typically used in a filter context but
the example uses a boolean must clause.
2018-11-19 16:30:33 +01:00
Benjamin Trent bc7dea4480
ML: changing automatic check_window calculation (#35643)
* ML: changing automatic check_window calculation

* adding docs on how we calculate the default
2018-11-19 08:03:34 -06:00
Peter Dyson 6d8af9731d [Docs] Warn about searching across all fields wt. `query_string` (#35570)
Warn about potential performance impact when a large number of fields
is used with query string query and no default field.
2018-11-19 13:21:59 +01:00
Ignacio Vera ae6a33237f
HLRC: Add ML delete filter action (#35382)
* HLRC: Add ML delete filter action

It adds delete ML filter action to the high level rest client.

Relates #29827
2018-11-19 11:25:35 +01:00
Gordon Brown bb51cdb6de
Add HLRC docs for Get Lifecycle Policy (#35612)
Adds docs for the Get Lifecycle Policy API to the HLRC documentation.
2018-11-16 18:16:30 -07:00
Gordon Brown cce9648f9d
Align RolloverStep's name with other step names (#35655)
RolloverStep previously had a name of "attempt_rollover", which was
inconsistent with all other step names due it its use of an underscore
instead of a dash.
2018-11-16 17:42:48 -07:00
Julie Tibshirani c6a0904e0e
Deprecate types in count and msearch. (#35421)
* Deprecate types in count requests.
* Move RestCountAction to the 'search' package.
* Deprecate types in multi search requests.
* Add tests for types deprecation in the _search endpoint.
2018-11-16 13:04:43 -08:00
debadair fc63ea5357
[DOCS] Add beta warning to ILM pages. (#35571) 2018-11-16 10:49:55 -08:00
Julie Tibshirani 40ba4de5e6
Deprecate types in validate query requests. (#35575) 2018-11-16 08:59:04 -08:00
Martijn van Groningen 797cffbb68
[TEST] Replace fields in response with actual values
Relates to #35496
2018-11-16 10:18:14 +01:00
Przemyslaw Gomulka c6e2634bf8
Docs: HLRC: refactor bulk, migrate and reindex apis (#35413)
fix typos and refactor to DRY up documentation for bulk, reindex and migration apis
relates #35345
2018-11-16 08:58:13 +01:00
Tim Vernum 87a8b99724
HLRC: Add "_has_privileges" API to Security Client (#35479)
This adds the "hasPrivileges()" method to SecurityClient, including
request, response & async variant of the method.

Also includes API documentation.
2018-11-16 13:52:06 +11:00
Jun Ohtani 0cdfc4cd0a
[Doc] Add clarification to boolean query (#32575)
It isn't very clear how boosting query works.
Add explanation of positive/negative query.
2018-11-16 11:45:32 +09:00
Gordon Brown 3883e9bf4c
Split RolloverStep into Wait and Action steps (#35524)
RolloverAction will now periodically check the rollover conditions using
the Rollover API with the dry_run option as an AsyncWaitStep, then run
the rollover itself by calling the Rollover API with no conditions,
which will always roll over, as an AsyncActionStep. This will resolve
race condition issues in policies using RolloverAction.
2018-11-15 17:11:31 -07:00
Mayya Sharipova a8833b097b
Painless Context Doc: Add min should match example (#35423) 2018-11-15 15:10:49 -05:00
Mayya Sharipova 501c03e529
Painless Context Doc: Add filter context example (#35305) 2018-11-15 14:33:34 -05:00
Benjamin Trent f7ada9b29b
Add delayed datacheck to the datafeed job runner (#35387)
* ML: Adding missing datacheck to datafeedjob

* Adding client side and docs

* Making adjustments to validations

* Making values default to on, having more sensible limits

* Intermittent commit, still need to figure out interval

* Adjusting delayed data check interval

* updating docs

* Making parameter Boolean, so it is nullable

* bumping bwc to 7 before backport

* changing to version current

* moving delayed data check config its own object

* Separation of duties for delayed data detection

* fixing checkstyles

* fixing checkstyles

* Adjusting default behavior so that null windows are allowed

* Mentioning the default value

* Fixing comments, syncing up validations
2018-11-15 13:32:45 -06:00
debadair 19001e71cc
[DOCS] Updated release state for 7.0.0-alpha1 (#35596)
* [DOCS] Updated release state for 7.0.0-alpha1

* Update Versions.asciidoc
2018-11-15 08:22:13 -08:00
Ed Savage 2d948a001e
[HLRC][ML] Add ML delete model snapshot API (#35537)
Relates to #29827
2018-11-15 14:57:17 +00:00
Christoph Büscher bc5b1afb55
HLRC: Add parameters to stopRollupJob API (#35545)
With #34811 the API for stopping rollup jobs got two new url parameters
"wait_for_completion" and "timeout". This change adds these to the HLRC APIs as
well.

Relates to #34811
2018-11-15 10:50:48 +01:00
Clinton Gormley cb8bdeae68 Fixed bad link in ingest-geo-point 2018-11-15 09:20:04 +01:00
Tal Levy dc1821c707
explain geo_point mapping in geoip-processor (#29114)
simple docs change to add missing mapping explanation. Users may not be aware this is
a prerequisite for doing geo-queries on this enriched data.
2018-11-14 20:05:45 -08:00
Tal Levy 95a09ab2d6
[ILM] TEST: skip checking _ilm/explain response b/c flaky timing (#35562)
The docs are not resilient to timing issues where the ILM metadata is not set on newly 
created indices, so we shouldn't be so strict on the returned response
2018-11-14 16:45:06 -08:00
Deb Adair 91a38b8fe0 [DOCS] Fixed bad cross doc link. 2018-11-14 16:09:18 -08:00
debadair 38d6bb27f5
[DOCS] Added link to standalone ODBC driver doc. (#35526) 2018-11-14 15:23:28 -08:00
Tal Levy f70d4470f7
[ILM] getting started documentation (#35379) 2018-11-14 13:58:08 -08:00
Tal Levy 30a713cc3f
[ILM] add docs explaining high-level concepts (#35322)
add docs explaining high-level concepts
2018-11-14 09:23:50 -08:00
Benjamin Trent 803eccec11
HLRC: Adding ML Update Filter API (#35522)
* HLRC: Adding ml get filters api

* HLRC: Adding ML Update Filter API
2018-11-14 11:13:11 -06:00
Thiago Souza 98e075227b [Doc] Correct jdbc url example (#35513)
Adds a missing `?` to separate the url parameters.
2018-11-14 17:30:23 +01:00
Tanguy Leroux 5b7446bb5f
Add Delete Privileges API to HLRC (#35454)
This commit adds the Delete Privileges API to the high level REST
client.

Related to #29827
2018-11-14 14:04:30 +01:00
Ed Savage e7b7d52a6a
[HLRC][ML] Add ML get model snapshots API (#35487)
Relates #29827
2018-11-14 13:03:04 +00:00
Jim Ferenczi 72504c2512
Do not recommend to use the _id field in search_after docs (#35370)
The documentation of `search_after` recommends to use the `_id`
field as a tiebreaker for the sort without warning against
the additional memory required. This change changes the recommandation
to use a copy of the `_id` field with doc_values enabled.
2018-11-14 10:50:31 +01:00
Christoph Büscher 603d1a470f
Add stop rollup job support to HL REST Client (#34702)
This change adds support for stoping a rollup job to the High Level REST Client.

Relates to #29827
2018-11-14 05:40:42 +01:00
Zachary Tong c346a0f027
[Rollup] Add `wait_for_completion` option to StopRollupJob API (#34811)
This adds a `wait_for_completion` flag which allows the user to block 
the Stop API until the task has actually moved to a stopped state, 
instead of returning immediately.  If the flag is set, a `timeout` parameter
can be specified to determine how long (at max) to block the API
call.  If unspecified, the timeout is 30s.

If the timeout is exceeded before the job moves to STOPPED, a
timeout exception is thrown.  Note: this is just signifying that the API
call itself timed out.  The job will remain in STOPPING and evenutally
flip over to STOPPED in the background.

If the user asks the API to block, we move over the the generic
threadpool so that we don't hold up a networking thread.
2018-11-13 16:37:17 -05:00
Benjamin Trent b9eb5f7b63
HLRC: Adding ml get filters api (#35502)
* HLRC: Adding ml get filters api

* refactoring setId name
2018-11-13 14:53:32 -06:00
Colin Goodheart-Smithe c934fb087a
Adds HLRC docs for put lifecycle policy (#35457)
* Adds HLRC docs for put lifecycle policy

* Adds link to docs in client javadocs

* Fixes checkstyle

* Make the documentation use the right ack response
2018-11-13 20:40:53 +00:00
Lee Hinman 6cb2aef943
[ILM] Add documentation for error handling in ILM (#35426)
* [ILM] Add documentation for error handling in ILM

This adds some initial documentation for error handling and retrying failed
steps for index lifecycle management
2018-11-13 12:19:22 -07:00
lcawl 2ac4ef7a69 [DOCS] Fixes broken link 2018-11-13 10:04:26 -08:00
Jason Tedor 8b2f22ee07
Introduce CCR overview (#35436)
This commit introduces a basic overview for cross-cluster replication to
the docs.

Co-authored-by: "lcawl <lcawley@elastic.co>"
2018-11-13 12:15:37 -05:00
Luca Cavanna 0b7d18d733
HLRC: add support for get license basic/trial status API (#33176)
Relates to #29827
2018-11-13 15:52:59 +01:00
Igor Motov a76ac5729d
Rest HL client: Add watcher stats API (#35185)
Relates to #29827
2018-11-13 04:47:35 -10:00
Jason Tedor 4b5fbada9f
Fix reference to get follower stats API
This commit fixes a reference to the CCR get follower stats API in the
CCR getting started docs.
2018-11-13 09:14:26 -05:00
Przemyslaw Gomulka fd4cd80496
HLRC: migration api - upgrade (#34898)
Implement high level client for migration upgrade API. It should wrap
RestHighLevelClient and expose high level IndexUpgradeRequest (new),
IndexTaskResponse for submissions with wait_for_completion=false and
BulkByScrollResponse (already used) objects.

refers: #29827
2018-11-13 15:01:53 +01:00
Martijn van Groningen e81671dd45
[HLRC] Added support for CCR Put Follow API (#35409)
This change also adds documentation for the Put Follow API and
adds a CCR HLRC integration test.
2018-11-13 13:22:49 +01:00
Jason Tedor cbb16d5ffc
Remove extraneous spaces from CCR getting started
This commit removes some extraneous spaces that were committed to the
cross-cluster replication getting started docs.
2018-11-13 06:48:56 -05:00
Jason Tedor 92fef40dbf
Introduce CCR getting started guide (#35434)
This commit introduces a basic getting started guide for cross-cluster
replication to the docs.

Co-authored-by: "lcawl <lcawley@elastic.co>"
2018-11-13 06:45:00 -05:00
Lisa Cawley 9aeaceac4b
[DOCS] Clarify results_index_name description (#35463) 2018-11-12 13:08:57 -08:00
Zachary Tong 2da239fb5e
[HLRC] Add GetRollupIndexCaps API (#35102)
Also refactors the caps response tests a bit to share the same
abstract class to reduce duplication of test code
2018-11-12 15:05:03 -05:00
Jason Tedor 326b238852
Add docs for CCR stats API (#35439)
This commit adds API docs for the CCR stats endpoint.
2018-11-12 10:28:35 -05:00
Peter Dyson c8e685eb34 Clarify S3 repository storage class parameter (#35400)
Today it is unclear that the `storage_class` parameter to an S3 repository only
affects new objects and does not rewrite any existing objects. This commit
clarifies this point.
2018-11-12 12:04:33 +00:00
Jason Tedor eb8f346204
Fix beta label on CCR index page
This commit fixes the location of the beta label on the CCR index page.
2018-11-10 15:45:38 -05:00
Jason Tedor d84ff3765d
Mark CCR as beta in the docs
This commit marks CCR as beta in the docs. This beta label is applied to
all CCR pages.
2018-11-10 14:38:22 -05:00
Albert Zaharovits f2d7c94949
DOCS Add audit ignore settings to reference page (#35274)
Adds the logfile audit ignore policy settings synopsis
to the Auditing Reference page.
2018-11-09 16:58:10 +02:00
Tim Vernum b0acd709d7
DOCS: Add SecurityExtension breaking change (#35358)
We changed the way realm settings are defined, and this affects custom
realms in SecurityExtensions. This change adds those details to the
breaking changes docs.

Relates: #30241
2018-11-09 16:57:49 +11:00
Armin Braun 02b4e28534
#31608 Add S3 Setting to Force Path Type Access (#34721)
* SNAPSHOTS: Use Path Style Access in S3

* Use path style access pattern to fix #31608
* closes #31608
2018-11-09 05:07:26 +01:00
debadair b485e69a5f
ILM API ref edits (#35112)
* [DOCS] ILM API Ref edits

* [DOCS] Fixed endpoint for DELETE policy.

* [DOCS] Removed comparison to setting index.lifecycle.name to null.

* [DOCS] Fixed xrefs to explain API.
2018-11-08 15:26:27 -08:00
Lisa Cawley c88b2d7e5a
[DOCS] Cleans up monitoring configuration steps (#35111) 2018-11-08 13:53:09 -08:00
Mayya Sharipova 7e144b318a
Painless Context Doc: Add field context example (#35130)
* Painless Context Doc: Add field context example

relates to #34829
2018-11-08 14:54:20 -05:00
Jason Tedor 730ec1ddfb
Add more detailed OS name on Linux (#35352)
Today our OS information returned in node stats only returns a
high-level name of the OS (e.g., "Linux"). Yet, for some uses this is
too high-level and knowing at a finer level of granularity the
underlying OS can be useful. This commit extracts the pretty name on
Linux from /etc/os-release. This pretty name usually includes the Linux
vendor and the Linux vendor version number (e.g., Fedora 28).
2018-11-08 12:16:58 -05:00
Christoph Büscher 113af7996c
Make limit on number of expanded fields configurable (#35284)
Currently we introduced a hard limit of 1024 to the number of fields a query can
be expanded to in #26541. Instead of using a hard limit, we should make this
configurable. This change removes the hard limit check and uses the existing
`max_clause_count` setting instead.

Closes #34778
2018-11-08 17:04:40 +01:00
Zachary Tong 55c6481b6b
[Docs] Add painless context details for bucket_selector (#35162)
Adds docs for the bucket_selector context, an example
and corresponding doc test
2018-11-08 11:02:30 -05:00
Jim Ferenczi d96202a282 [DOCS] Fix missing callouts 2018-11-08 15:40:01 +01:00
Dominik Stadler d351422215 Add parent-aggregation to parent-join module (#34210)
Add `parent` aggregation, a special single bucket aggregation that joins children documents to their parent.
2018-11-08 14:13:00 +01:00
Albert Zaharovits 025a0c82e5
Remove deprecated audit settings (#35205)
Removes `.prefix` deprecated settings for the logfile
auditing and also documents it in the migrate asciidoc.
2018-11-08 14:06:47 +02:00
Russ Cam 848847d8c7 [Docs] Section header preceded by blank line (#34340) 2018-11-08 12:44:13 +01:00
Jeff Hajewski d00b23c8b1 Fixes fast vector highlighter docs per issue 24318. (#34190)
The `fvh` highlighter does not support span queries. This fix updates
the docs to add a warning stating the lack of span query support for
`fvh`.
2018-11-08 11:09:03 +01:00
Jason Tedor a6073f5130
Add docs on JNA temp directory not being noexec (#35355)
If the underlying mount point for the JNA temporary directory is mounted
noexec on Linux, then the JVM will not be able to map the native code in
as executable. This will prevent JNA from executing and will prevent
Elasticsearch from being able to execute some functions that rely on
native code (e.g., memory locking, and installing system call
filters). We do not want to get into the business of catching exceptions
and parsing messages towards this because these exception messages can
change on us. We also do not want to jump through a lot of hoops to
check the underlying mount point for noexec. Instead, we will rely on
documentation to address this problem. This commit adds to the important
system configuration section of the docs that the JNA temporary
directory is not on a mount point with the noexec mount option.
2018-11-07 22:25:37 -05:00
Jason Tedor 4f4fc3b8f8
Replicate index settings to followers (#35089)
This commit uses the index settings version so that a follower can
replicate index settings changes as needed from the leader.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2018-11-07 21:20:51 -05:00
Alpar Torok 5ae03195d3
Make version field names more meaningful (#35334)
* Consolidate the name of the qualified build version

* Field name in response should not be redundant
2018-11-07 18:36:02 +02:00
Martijn van Groningen 021f80517f
[CCR] Added HLRC support for pause follow API (#35216)
* Moved `AcknowledgedResponse` to core package
* Made AcknowledgedResponse not abstract and provided a default parser,
so that in cases when the field name is not overwritten then there
is no need for a subclass.

Relates to #33824
2018-11-07 17:08:35 +01:00
Chris Cho e572a21c4b [Docs] Improve Convert Processor description (#35280)
Sometimes users are confused about whether they can use the Convert Processor
for changing an existing fields type to other types even if the existing one is already
ingested. This confusion is from the first line of description. Changing this and also
adding a some detail to the code snippet.
2018-11-07 17:01:35 +01:00
Alpar Torok 8a85b2eada
Remove build qualifier from server's Version (#35172)
With this change, `Version` no longer carries information about the qualifier,
we still need a way to show the "display version" that does have both
qualifier and snapshot. This is now stored  by the build and red from `META-INF`.
2018-11-07 14:01:05 +02:00
Julie Tibshirani ec53288fc0
Remove include_type_name from the relevant APIs. (#35192)
We've decided that the bulk, delete, get, index, update, and search APIs should not
contain this request parameter, and we will instead accept both typed and typeless calls.
2018-11-06 14:33:48 -08:00
Jay Modi 6f6b265166
HLRC: add support for the clear realm cache API (#35163)
This change adds support for clearing the cache of a realm. The realms
cache may contain a stale set of credentials or incorrect role
assignment, which can be corrected by clearing the cache of the entire
realm or just that of a specific user.

Relates #29827
2018-11-06 13:12:24 -07:00
Zachary Tong 065264d57c
[Docs] Add painless context details for bucket_script (#35142)
Fleshes out the details of the bucket_script context, adds an example
and corresponding doc test
2018-11-06 12:18:29 -05:00
Gytis Šk 3ee37b425b Docs: Add section about range query for range type (#35222)
This makes their interaction more discoverable.
2018-11-06 10:49:12 -05:00
Tal Levy a85b4f42ca
[ILM] change remove-policy-from-index http method from DELETE to POST (#35268)
The remove-ilm-from-index API was using the DELETE http method
to signify that something is being removed. Although, metadata
about ILM for the index is being deleted, no entity/resource
is being deleted during this operation. POST is more in line with
what this API is actually doing, it is modifying the metadata for
an index. As part of this change, `remove` is also appended to the path 
to be more explicit about its actions.
2018-11-06 07:46:25 -08:00
Jeff Soloshy 14c8a483d5 [Docs] Minor formatting and wording fixes (#35278) 2018-11-06 07:52:13 +01:00
Tim Vernum 7cd27d1a96 [DOCS] Fix missing float in breaking changes 2018-11-06 16:59:11 +11:00
Tim Vernum 3776de5f20
HLRC: Add InvalidateToken security API (#35114)
This change adds the Invalidate Token API
(DELETE /_xpack/security/oauth2/token) to the Elasticsearch
High Level Rest Client.

Relates: #29827
2018-11-06 15:26:12 +11:00
Tim Vernum 574ec6686e
Include realm type in Security Realm setting keys (#30241)
This moves all Realm settings to an Affix definition.
However, because different realm types define different settings
(potentially conflicting settings) this requires that the realm type
become part of the setting key.

Thus, we now need to define realm settings as:

    xpack.security.authc.realms:
      file.file1:
        order: 0

      native.native1:
        order: 1

- This is a breaking change to realm config
- This is also a breaking change to custom security realms (SecurityExtension)
2018-11-06 14:56:50 +11:00
lcawl 8280596f16 [DOCS] Fixes formatting error in auditing settings 2018-11-05 15:30:35 -08:00
Alexander Zhukov 842809ef37 Doc: Drop an extra 'a' in snapshots.asciidoc (#35251) 2018-11-05 13:31:35 -05:00
Nik Everett 59a43180a5 Reindex: Clean up docs around multi-index
We have an example in `reindex`'s docs about copying from many indices
at once. It doesn't work at the moment because we only allow a single
type per index. We didn't notice it in the docs tests because those
tests didn't copy any documents. This change:
1. Adds documents to the docs tests to fully exercise the snippet.
2. Fixes the example by moving all copied documents to the same type.
3. Moves the note about id collisions and expands on it because it is
even more likely than before.

Closes #35150
2018-11-05 12:52:11 -05:00
Jake Landis c2766b65cf
ingest: raise visibility of ingest plugin documentation (#35048)
* move the set security user processor to the main documentation
* link to plugin processors

part of #33188
2018-11-05 11:44:10 -06:00
Lee Hinman 3473217563
Remove Joda usage from ILM (#35220)
This commit removes the Joda time usage from ILM and the HLRC components of ILM.
It also fixes an issue where using the `?human=true` flag could have caused the
parser not to work. These millisecond fields now follow the standard we use
elsewhere in the code, with additional fields added iff the `human` flag is
specified.

This is a breaking change for ILM, but since ILM has not yet been released, no
compatibility shim is needed.
2018-11-05 08:17:15 -07:00
Benjamin Trent a4442dacd7
HLRC: Add ML API PUT filter (#35175) 2018-11-05 08:56:53 -06:00
Mayya Sharipova a2c22ad7ae
Small corrections to HLRC doc for _termvectors (#35221)
Relates to #33447
2018-11-04 18:29:07 -05:00
Jack Conradson 44f08717ba
[Scripting] Make Max Script Length Setting Dynamic (#35184)
This changes the current script.max_size_in_bytes to be dynamic so it can be 
set through the cluster settings API. This setting is also applied to inline scripts 
in the compile method of ScriptService to prevent excessively long inline 
scripts from being compiled. The script length limit is removed from Painless as 
this is no longer necessary with the protection in compile.
2018-11-02 16:07:54 -07:00
Lisa Cawley 3c36ba1f5e
[DOCS] Move CCR overview pages (#34918) 2018-11-02 14:23:13 -07:00
Gordon Brown 9b3d581339 [DOCS] Remove link to nonexistent ILM API 2018-11-02 09:38:40 -06:00
Milan Mrdjen 34677b9c83 HLRC: Add document _count API (#34267)
Add `count()` api method, `CountRequest` and `CountResponse` classes to HLRC. Code in server module is unchanged.

Relates to #27205
2018-11-02 08:21:19 -05:00
Colin Goodheart-Smithe 8f592bda27
Removes link to deleted page 2018-11-02 12:27:51 +00:00
Daniel Mitterdorfer ccbe80c3a0
Introduce durability of circuit breaking exception
With this commit we differentiate between permanent circuit breaking
exceptions (which require intervention from an operator and should not
be automatically retried) and transient ones (which may heal themselves
eventually and should be retried). Furthermore, the parent circuit
breaker will categorize a circuit breaking exception as either transient
or permanent based on the categorization of memory usage of its child
circuit breakers.

Closes #31986
Relates #34460
2018-11-02 13:12:44 +01:00
Colin Goodheart-Smithe fc6e1f7f3f
Merge branch 'master' into index-lifecycle 2018-11-02 10:56:35 +00:00
lcawl f97c9b6af2 [DOCS] Fixes location of beta markup 2018-11-01 14:27:49 -07:00
Julie Tibshirani fda173d7aa
Add a note around using separate indices for percolator queries and documents. (#35109) 2018-11-01 12:41:07 -07:00
lcawl 781130c7ab [DOCS] Fixes formatting of dynamic settings links 2018-11-01 11:52:41 -07:00
Tal Levy c6c01425bb Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-11-01 11:38:42 -07:00
Tal Levy c3cf7dd305 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-11-01 10:13:02 -07:00
Jake Landis 77fab62ebe
ingest: add common options to each processor's documentation (#35091)
* adds `if`, `on_failure`, `tag`, and `ignore_failure` to table for each processor

part of #33188

* added ingore_failure

* fix whitespace noise
2018-11-01 11:08:04 -05:00
Itamar Syn-Hershko d46f3528c0 [Docs] Updating URL for Openstack Swift plugin (#34136)
The repository plugin for Openstack Swift was developed originally by Wikimedia foundation but is now retired. 
This changes the link to the repo where the actively maintained fork lives now.
2018-11-01 11:57:55 +01:00
Ke Li 14f540e8e6 Deprecate unicodeSetFilter in favour of unicode_set_filter (#29215) 2018-11-01 10:06:51 +00:00
Alexandru Rusanescu f3e150b0ea [Docs] Update query_cache.asciidoc (#33340)
Add note about non-visibility of cache content.
2018-11-01 10:22:36 +01:00
lipsill d181d1bab1 Remove deprecated url parameters `_source_include` and `_source_exclude` (#35097)
Removes `_source_include` and `_source_exclude` url parameters. 
These parameters have been deprecated in #33475.

Closes #22792
2018-10-31 17:11:59 -04:00
Qian Hong f27251dfcb [Docs] Improve Redindex/Update/Delete By Query. (#35127)
Fix a copy paste error and improve formatting in
Redindex/Update/Delete By Query and fix formula in Update By Query.
2018-10-31 14:04:48 -04:00
Tal Levy d5d28420b6 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-31 10:47:07 -07:00
Luca Cavanna ef5181c678
Allow to enable pings for specific remote clusters (#34753)
When we connect to remote clusters, there may be a few more routers/firewalls in-between compared to when we connect to nodes in the same cluster. We've experienced cases where firewalls drop connections completely and keep-alives seem not to be enough, or they are not properly configured. With this commit we allow to enable application-level pings specifically from CCS nodes to the selected remote nodes through the new setting `cluster.remote.${clusterAlias}.transport.ping_schedule`.  The new setting is similar `transport.ping_schedule` but it does not affect intra-cluster communication, pings are only sent to specific remote cluster when specifically enabled, as they are disabled by default.

Relates to #34405
2018-10-31 17:32:53 +01:00
Lisa Cawley 72cb885abc
[DOCS] Updates monitoring diagram (#35096) 2018-10-31 09:18:14 -07:00
Mayya Sharipova 0b8465f2c0
Update type to "seat" (#35086)
otherwise this throws an exception: java.lang.IllegalArgumentException: Rejecting mapping update to [seats] as the final mapping would have more than 1 type: [seat, _doc]

May be, later for the types removal, we can modify `seats.json` to have a type `_doc` instead of `seat`
2018-10-31 12:06:48 -04:00
Sue Gallagher 1ce3c92a2d [DOCS] Add info on calendar vs fixed interval. (#31638)
Extensive edit to add additional information on the difference between calendar intervals and fixed-length intervals.
2018-10-31 10:16:36 -04:00
Albert Zaharovits bdd8460906
HLRest: add security authenticate API (#33552)
This adds the security `_authenticate` API to the HLREST client.
It is unlike some of the other APIs because the request does not
have a body.
The commit also creates the `User` entity. It is important
to note that the `User` entity does not have the `enabled`
flag. The `enabled` flag is part of the response, alongside
the `User` entity.
Moreover this adds the `SecurityIT` test class 
(extending `ESRestHighLevelClientTestCase`).

Relates #29827
2018-10-31 14:49:33 +02:00
debadair e540abd551
[DOCS] Fixed edit links for ILM APIs and added the APIs to the REST ref (#35070)
* [DOCS] Fixed edit links for ILM APIs and added the APIs to the REST API section.

* [DOCS] Fixed link to ILM APIs.
2018-10-30 18:27:43 -07:00
Tal Levy 5141084048
rename CRUD api REST path prefix _ilm to _ilm/policy (#35056)
This PR renames the CRUD APIS for ILM

GET _ilm/<policy>, _ilm -> _ilm/policy/<policy>, _ilm/policy
PUT _ilm/<policy> -> _ilm/policy/<policy>
DELETE _ilm/<policy> -> _ilm/policy/<policy>

closes #34929.
2018-10-30 16:19:05 -07:00
Tal Levy f5323cd86e
rename ILM Phase "minimum_age" to "min_age" (#35058)
closes #34927.
2018-10-30 12:54:02 -07:00
Nik Everett 0f9a7b3344 Docs: Fix build after moving SQL QA
I moved SQL's QA in #34496 to make it easier to run all of SQL tests in
one go but forgot to update the docs build. This fixes that.
2018-10-30 15:47:25 -04:00
Nik Everett 2b2e208c4c
Docs: Remove range notation from random score docs (#35093)
The `random_score` function produces values between 0 (inclusive) and 1
(exclusive) and documented it with fancy methematical range notation. It
is so fancy I thought it was a typo. This changes the documentation to
use words.

Relates to #35084
2018-10-30 14:12:59 -04:00
Zachary Tong f9dd33a0b9
[Rollup] Proactively resolve index patterns in RollupSearch endoint (#34930)
This changes the RollupSearch endpoint to proactively resolve index
patterns.  If the index pattern(s) match more than one rollup index,
an exception is throw as before.  But if the pattern only matches one
rollup index, execution is allowed to continue (unlike before where
it would assume all patterns were for raw data).

This also allows the search endpoint to resolve aliases that point to
a rollup index.

Also tweaks the documentation to make this clear.

Closes #34828
2018-10-30 13:50:50 -04:00
Julie Tibshirani 70da490f34
Remove some documentation that only makes sense with multiple types. (#35066)
* Remove a tip about ignore_above that only makes sense with multiple types.
* Remove a line from the percolator documentation that refers to multiple types.
2018-10-30 10:19:12 -07:00
Tal Levy 18c72e86c5 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-30 08:09:57 -07:00
Andy Bristol b8280ea7cc
median absolute deviation agg (#34482)
This commit adds a new single value metric aggregation that calculates
the statistic called median absolute deviation, which is a measure of
variability that works on more types of data than standard deviation

Our calculation of MAD is approximated using t-digests. In the collect
phase, we collect each value visited into a t-digest. In the reduce
phase, we merge all value t-digests, then create a t-digest of
deviations using the first t-digest's median and centroids
2018-10-30 07:22:52 -07:00
Nik Everett cc4dd98872 Docs: Fix build for java-api
The java-api docs had some broken references. Mostly because it is
difficult to build them so they are difficult to check.

Relates to #34528
2018-10-30 09:14:07 -04:00
Alan Woodward cf24231b1b
Add missing full stop in Bootstrap checks doc (#35078)
Fixes #35075
2018-10-30 11:43:43 +00:00
markwalkom 5e0c9b7c1a Update reindex.asciidoc (#35062)
Highlighted that `_source` needs to exist in the source index.
2018-10-30 10:25:05 +01:00
Przemyslaw Gomulka 995bf0ee66
Bulk Api support for global parameters (#34528)
Bulk Request in High level rest client should be consistent with what is
possible in Rest API, therefore should support global parameters. Global
parameters are passed in URL in Rest API.

Some parameters are mandatory - index, type - and would fail validation
if not provided before before the bulk is executed.
Optional parameters - routing, pipeline.

The usage of these should be consistent across sync/async execution,
bulk processor and BulkRequestBuilder

closes #26026
2018-10-30 09:08:12 +01:00
Ken MacInnis d67c88fa12 [Docs] Update update-settings.asciidoc (#35073)
Add explanation for ?include_defaults parameter and link to
Cluster Get Settings reference
2018-10-30 08:58:32 +01:00
Gordon Brown 794d4fa879
Label required scripts in Scripted Metric Agg docs (#35051)
When combine_script and reduce_script were made into required
parameters for Scripted Metric aggregations in #33452, the docs were
not updated to reflect that. This marks those parameters as required
in the documentation.
2018-10-29 15:13:14 -06:00
Tal Levy c9e4d26a53 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-29 14:03:55 -07:00
Nik Everett 3051e37e45 Docs: Document refresh for reindex and friends
Reindex and friends don't support `refresh=wait_for` and this documents
that.
2018-10-29 13:58:55 -04:00
lipsill 6df1c9e818 Deprecate `_source_include` and `_source_exclude` url parameters (#33475)
Deprecates `_source_include` and `_source_exclude` url parameters
in favor of `_source_inclues` and `_source_excludes` because those
are consistent with the rest of Elasticsearch's APIs.

Relates to #22792
2018-10-29 12:06:38 -04:00
Nik Everett 6fd7ea71f1
HLRC: DRY up remaining CRUD docs (#34925)
This further applies the pattern set in #34125 to reduce copy-and-paste
in the multi-document CRUD portion of the High Level REST Client docs.
It also adds line wraps to snippets that are too wide to fit into the box
when rendered in the docs, following up on the work started in #34163.
2018-10-29 10:32:17 -04:00
Jason Tedor 26f5c509af
Fix CCR API specification (#34963)
This commit fixes two issues with the CCR API specification:
 - remove the CCR stats endpoint, it is not currently implemented
 - fix the documentation links
2018-10-29 09:37:13 -04:00
Christoph Büscher 389910f11d
[Docs] Add migration note about expanded fields limit (#34920)
Adds a note to warn users about the limit introduced in #26541.
2018-10-29 10:23:07 +01:00
David Roberts c455be7bc2
[ML] Rename the json file structure to ndjson (#34901)
The file structure finder endpoint can find the NDJSON
(newline-delimited JSON) file format, but called it
`json`.  This change renames the `format` for this file
structure to `ndjson`, which is more precise and will
hopefully avoid confusion.
2018-10-29 10:06:12 +01:00
Martijn van Groningen f13d529448
Fixed ccr docs broken links. 2018-10-29 09:29:28 +01:00
Christoph Büscher 3a464bdee8
Add start rollup job support to HL REST Client (#34623)
This change adds support for starting a rollup job to High Level REST Client.

Relates to #29827
2018-10-29 09:28:11 +01:00
Martijn van Groningen 1801518527
[CCR] Refactor stats APIs (#34912)
* Changed the auto follow stats to also include follow stats.
* Renamed the auto follow stats api to stats api and changed its url path
  from `/_ccr/auto_follow/stats` `/_ccr/stats`.
* Removed `/_ccr/stats` url path for the follow stats api, which makes
  the index parameter a required parameter.
* Fixed docs.
2018-10-29 07:45:27 +01:00
Tim Vernum 9c27b407f0
HLRC: Add security Create Token API (#34791)
This adds the Create Token API (POST /_xpack/security/oauth2/token)
to the High Level Rest Client.

Relates: #29827
2018-10-29 17:17:56 +11:00
Yogesh Gaikwad dc5bfe3a00
[DOCS] Fix typo in get role mappings API (#34952) 2018-10-29 11:15:42 +11:00
Yogesh Gaikwad a5ee134c40
[HLRC] Add support for get role mappings API (#34637)
This commit adds support for get role mappings API
in HLRC.
2018-10-29 10:12:13 +11:00
debadair 544c220942
[DOCS] Added attributes for HLRC javadoc packages. (#34934) 2018-10-26 15:32:15 -07:00
Benjamin Trent 052dfa5646
HLRC: Adding Update datafeed API (#34882)
* HLRC: Adding Update datafeed API

* Addressing unused import

* Adjusting docs and fixing minor comments

* fixing comment
2018-10-26 16:44:12 -05:00
Tal Levy d8322ca069 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-26 12:46:21 -07:00
Tal Levy e737ea7d4a
remove old doc placeholder and migrate ilm docs to top-level (#34615)
we are restructuring the docs, this migrates ILM docs outside of the x-pack doc structure.
2018-10-26 12:19:52 -07:00
Boaz Leskes a086c665a3 HLREST: Add Clear Roles Cache API (#34187)
Adds support for the Clear Roles Cache API to the High Level Rest
Client. As part of this a helper class, NodesResponseHeader, has been
added that enables parsing the nodes header from responses that are
node requests.

Relates to #29827
2018-10-26 12:16:44 -06:00
Jason Tedor fdfdbe486d
Introduce cross-cluster replication API docs (#34726)
This commit is our first introduction to cross-cluster replication
docs. In this commit, we introduce the cross-cluster replication API
docs. We also add skelton docs for additional content that will be added
in a series of follow-up commits.
2018-10-26 11:23:35 -04:00
markharwood 3181083781
HLRC - add support for source exists API (#34519)
HLRC - add support for source exists API
API re-uses the GetRequest object (following the precedent set by the plain “exists” api).

Relates to #27205
2018-10-26 14:21:35 +01:00
Albert Zaharovits 3f1fec1813
[Docs] audit logfile structured format (#34584)
Documents the new structured logfile format for auditing
that was introduced by #31931. Most changes herein
are for 6.x . In 7.0 the deprecated format is gone and a
follow-up PR is in order.
2018-10-26 15:19:35 +03:00
Tal Levy 810cd46a30 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-25 14:35:33 -07:00
Senthil e27a0d5b0d Docs: Align prose with snippet (#34839)
Modified example text to include multiple index patterns used in the code
2018-10-25 16:51:18 -04:00
Stéphane Campinas 27c4d63340 document the search context is freed if the scroll is not extended (#34739)
The `fetchPhaseShouldFreeContext` returns true when there is a scroll context but the scroll parameter is null, thus freeing the search context.

183c32d4c3/server/src/main/java/org/elasticsearch/search/SearchService.java (L491)
2018-10-25 16:49:08 -04:00
Lisa Cawley 312df5546c
[DOCS] Updates Elasticsearch monitoring tasks (#34339) 2018-10-25 10:32:50 -07:00
Lee Hinman 3e7042832a Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-25 11:00:36 -06:00
Christoph Büscher c0c6a28e86
[Docs] Add `indices.query.bool.max_clause_count` setting (#34779)
This change adds a section about the global search setting
`indices.query.bool.max_clause_count` that limits the number of boolean clauses
allowed in a Lucene BooleanQuery.

Closes #19858
2018-10-25 17:59:59 +02:00
Jason Tedor 153157e56d
Enable ingest attachment docs tests on JDK 11 (#34770)
These were disabled because ingest attachement was not playing well with
JDK 11. We have addressed that by upgrading the Tika dependency, yet
forgot to reenable these. This commit enables these tests again.
2018-10-24 23:17:45 -04:00
Mayya Sharipova d7afd7c123 Reduce the number of callouts to 15 in docs
relates to #33447
2018-10-24 17:57:35 -04:00
Mark Tozzi d94406a68a
HLRC: Deactivate Watch API (#34192)
Relates to #29827
2018-10-24 17:01:22 -04:00
Mayya Sharipova 3211760e2c Correct term-vectors.asciidoc
relates to #33447
2018-10-24 16:35:20 -04:00
Benjamin Trent cd27b0b996
Revert "Rollup add default metrics to histo groups (#34534)" (#34815)
This reverts commit 4236358f5d.
2018-10-24 14:25:10 -05:00
Mayya Sharipova bf4d90a5dc
HLRC API for _termvectors (#33447)
* HLRC API for _termvectors

relates to #27205
2018-10-24 14:27:22 -04:00
Gordon Brown da20dfd81c
Add cluster-wide shard limit warnings (#34021)
In a future major version, we will be introducing a soft limit on the
number of shards in a cluster based on the number of nodes in the
cluster. This limit will be configurable, and checked on operations
which create or open shards and issue a warning if the operation would
take the cluster over the limit.

There is an option to enable strict enforcement of the limit, which
turns the warnings into errors.  In a future release, the option will be
removed and strict enforcement will be the default (and only) behavior.
2018-10-23 16:35:10 -06:00
Armin Braun f79bdec58a INGEST: Document Pipeline Processor (#33418)
* Added documentation for Pipeline Processor
* Relates #33188
2018-10-23 15:36:57 -05:00
Zachary Tong 299d044bfc
Collapse pipeline aggs into single package (#34658)
- Restrict visibility of Aggregators and Factories
- Move PipelineAggregatorBuilders up a level so it is consistent with
AggregatorBuilders
- Checkstyle line length fixes for a few classes
- Minor odds/ends (swapping to method references, formatting, etc)
2018-10-23 16:01:01 -04:00
Jake Landis 067d135bfd
ingest: document index.default_pipeline (#34500) 2018-10-23 13:49:25 -05:00
Jake Landis a8e1ee34ca
ingest: document fields that support templating (#34536)
This change also updates many of the examples to use ecs as the example.
Some additional minor improvements are also included.

Part of #33188
2018-10-23 13:28:44 -05:00
Jake Landis c447fc258a
ingest: documentation for the drop processor (#34570) 2018-10-23 12:30:23 -05:00
Tal Levy 62ac2fa5ec Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-23 09:43:46 -07:00
Zachary Tong 4dbf498721
[Rollup] Job deletion should be invoked on the allocated task (#34574)
We should delete a job by directly talking to the allocated 
task and telling it to shutdown. Today we shut down a job 
via the persistent task framework. This is not ideal because, 
while the job has been removed from the persistent task 
CS, the allocated task continues to live until it gets the 
shutdown message.

This means a user can delete a job, immediately delete 
the rollup index, and then see new documents appear in
 the just-deleted index. This happens because the indexer
 in the allocated task is still running and indexes a few 
more documents before getting the shutdown command.

In this PR, the transport action is changed to a TransportTasksAction, 
and we invoke onCancelled() directly on the matching job. 
The race condition still exists after this PR (albeit less likely), 
but this was a precursor to fixing the issue and a self-contained
chunk of code. A second PR will followup to fix the race itself.
2018-10-23 12:23:22 -04:00
Albert Zaharovits 46b49b01c4
[DOCS] .Security index is never auto created (#34589) 2018-10-23 18:51:59 +03:00
Armin Braun f0f732908e
INGEST: Document Processor Conditional (#33388)
* INGEST: Document Processor Conditional

Relates #33188
2018-10-23 17:37:30 +02:00
Costin Leau ca6808e55d
SQL: Support pattern against compatible indices (#34718)
Extend querying support on multiple indices from being strictly
identical to being just compatible.
Use FieldCapabilities API (extended through #33803) for mapping merging.

Close #31837 #31611
2018-10-23 17:07:51 +03:00
Przemyslaw Gomulka f91cc13c18
A small typo in migration-assistance doc (#34704)
typo - missing word "be"
2018-10-23 15:28:25 +02:00
Marios Trivyzas 4a8386f271
SQL: Implement IN(value1, value2, ...) expression. (#34581)
Implement the functionality to translate the
`field IN (value1, value2,...)` expressions to proper Lucene queries
or painless script or local processors depending on the use case.

The `IN` expression can be used in SELECT, WHERE and HAVING clauses.

Closes: #32955
2018-10-23 14:28:23 +02:00
Frederik Bosch 183c32d4c3 [DOCS] Force Merge: clarify execution and storage requirements (#33882) 2018-10-23 12:25:04 +02:00
Marios Trivyzas e9e140790a
SQL: Implement `CONVERT`, an alternative to `CAST` (#34660)
`CONVERT` works exactly like cast with slightly different syntax:
`CONVERT(<value>, <data_type)` as opposed to `CAST(<value> AS <data_type>)`

Moreover it support format of the MS-SQL data types `SQL_<type>`,
e.g.: `SQL_INTEGER`

Closes: #34513
2018-10-23 11:21:15 +02:00
Tal Levy 67bfdb16ad Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-22 13:09:37 -07:00
Julie Tibshirani f854330e06
Make sure to use the type _doc in the REST documentation. (#34662)
* Replace custom type names with _doc in REST examples.
* Avoid using two mapping types in the percolator docs.
* Rename doc -> _doc in the main repository README.
* Also replace some custom type names in the HLRC docs.
2018-10-22 11:54:04 -07:00
Zachary Tong d981746142
[Docs] clarification about cardinality accuracy (#34616)
Adds a bit more clarification about how accuracy is dependent
on the dataset in question.

Closes #18231
2018-10-22 13:15:45 -04:00
Julie Tibshirani fbb9ac34f9
Deprecate type exists requests. (#34663) 2018-10-22 08:46:11 -07:00
Christoph Büscher e869f9a78c [Docs] Update synonym-tokenfilter.asciidoc (#34706)
Remove ugly double-dot.
2018-10-22 17:18:29 +02:00
Jason Tedor bf5f0af491
Use trial license in docs tests (#34673)
This commit switches to using a trial license in the docs tests that run
on the default distribution. This is needed so that docs tests can be
executed against non-basic features.
2018-10-21 06:23:42 -04:00
Ryan Ernst 222652dfce
Scripting: Convert script fields to use script context (#34164)
This commit removes the use of SearchScript for script fields and adds
a new FieldScript.
2018-10-20 16:33:49 -07:00
Jason Tedor aebb855f38
Add missing word to remote clusters docs
This commit adds a missing word in the initial section of the remote
clusters docs.
2018-10-20 12:11:41 -04:00
Jason Tedor aca8c705ae
Fix name of remote clusters reference
This commit fixes the name of the reference to the remote clusters
module on the modules page by adding a missing character.
2018-10-20 11:18:27 -04:00
Jason Tedor ab5a0b15f6
Fix formatting of remotes clusters module section
This commit fixes the formatting of the remote clusters module section
on the modules page. We add a missing newline and wrap the columns at
80.
2018-10-20 09:04:48 -04:00
Jason Tedor 91dbb22950
Fix reference to remote clusters docs
This commit fixes a reference to the remote clusters docs from the
modules page.
2018-10-20 09:02:00 -04:00
Jason Tedor b2dc58450e
Separate remote clusters docs from CCS (#34612)
With remote clusters taking on a larger role, we have make the
infrastructure more generic than being tied to cross-cluster search
(CCS). We want to refer to the remote clusters configuration in the
cross-cluster replication (CCR) docs. Yet, these docs are still tied to
CCS. This commit extracts the remote clusters docs from CCS (with some
wording changes to make them more general) so that we can refer to them
in the CCR docs.
2018-10-20 08:33:59 -04:00
Yannick Welsch 9fb2f7cc20
HLRC: Delete role API (#34620)
Adds the "Delete role" API to the high-level REST client.
2018-10-20 12:11:36 +02:00
Jack Conradson 842bf55497 [Docs] Fixes a bug in the rest-client watcher docs related to
cross-link issue.
2018-10-19 11:52:15 -07:00
Igor Motov 94bde37bcf
Geo: Don't flip longitude of envelopes crossing dateline (#34535)
When a envelope that crosses the dateline is specified as a part of
geo_shape query is parsed it shouldn't have its left and right points
flipped.

Fixes #34418
2018-10-19 13:53:54 -04:00
Julie Tibshirani 67652b5355
Remove references to multiple types in the search documentation. (#34625) 2018-10-19 09:47:34 -07:00
Jack Conradson 1100942455
[HLRC] Add Start/Stop Watch Service APIs. (#34317) 2018-10-19 09:42:57 -07:00
Costin Leau 52104aac27
SQL: Introduce support for NULL values (#34573)
Make SQL aware of missing and/or unmapped fields treating them as NULL
Make _all_ functions and operators null-safe aware, including when used
in filtering or sorting contexts
Add missing and null-safe doc value extractor
Modify dataset to have null fields spread around (in groups of 10)
Enforce missing last and unmapped_type inside sorting
Consolidate Predicate templating and declaration
Add support for Like/RLike in scripting
Generalize NULLS LAST/FIRST
Introduce early schema declaration for CSV spec tests: to keep the doc
snippets in place (introduce schema:: prefix for declaration)
upfront.

Fix #32079
2018-10-19 16:44:33 +03:00
Colin Goodheart-Smithe 84ef91529c
Merge branch 'master' into index-lifecycle 2018-10-19 13:24:04 +01:00
Benjamin Trent 4236358f5d
Rollup add default metrics to histo groups (#34534)
* Rollup: Adding default metrics for histo group timefield (#34379)

* Rollup: Adding default histo metrics and tests

* fixing failing client side test with new default values

* Adding HLRC docs for default values

* Addressing PR comments

* Removing value_count default agg

* Updating docs for rollups

* Minor interval change
2018-10-19 07:23:25 -05:00
Jim Ferenczi 7b49beb9b0
Fix threshold frequency computation in Suggesters (#34312)
The `term` and `phrase` suggesters have different options to filter candidates
based on their frequencies. The `popular` mode for instance filters candidate
terms that occur in less docs than the original term. However when we compute this threshold
we use the total term frequency of a term instead of the document frequency. This is not inline
with the actual filtering which is always based on the document frequency. This change fixes
this discrepancy and clarifies the meaning of the different frequencies in use in the suggesters.
It also ensures that the threshold doesn't overflow the maximum allowed value (Integer.MAX_VALUE).

Closes #34282
2018-10-19 13:33:19 +02:00
markharwood fe623acf66
Docs - removed experimental/beta markers from adjacency matrix aggregation (#34599) 2018-10-19 09:33:59 +01:00
markharwood 2a413abb0b
Docs - remove experimental marker from significant_text aggregation (#34598) 2018-10-19 09:32:02 +01:00
Yogesh Gaikwad 39a6163316
[HLRC] Add support for Delete role mapping API (#34531)
Building on expression dsl and create role mapping API, this
commit adds support for delete role mapping API to high level client.
2018-10-19 13:46:26 +11:00
Christoph Büscher 7cc3ac0fc4
[Docs] Add _cache/clear parameter examples (#34614)
Add example for selectively clearing just the request, query or fielddata cache
and for selectively clearing the cache for specific fields.

Closes #34287
2018-10-19 00:44:37 +02:00
Zachary Tong 45546e71c2
Add GetRollupCaps API to high level rest client (#32880)
Adds GetRollupCaps API to the HLRC, and tweaks some of the
Caps objects to be immutable.  Also various style tweaks
2018-10-18 17:12:38 -04:00
Benjamin Trent fa973007b8
[DOCS] Adds new lazy ml node setting (#34600)
* Adding new xpack.ml.max_lazy_ml_nodes setting to docs

* Fixing docs, making it clearer what the setting does

* Adding note about external process need
2018-10-18 16:11:36 -05:00
Nik Everett 2c0f67318a
Docs: Reenable rollup docs tests (#34564)
We'd disabled them because we didn't have a way to clean up after each
test. I implemented #34342 which adds the clean ups so now we can
re-enable the tests.

In the `setup` sections we have to use `raw` requests instead of
`x-pack` requests because we don't have the json config for x-pack.

Closes #33319
2018-10-18 15:24:02 -04:00
Ryan Ernst d445785f1a
Scripting: Convert domainSplit function for ML to whitelist (#34426)
This commit moves the definition of domainSplit into java and exposes it
as a painless whitelist extension. The method also no longer needs
params, and version which ignores params is added and deprecated.
2018-10-17 15:54:21 -07:00
Colin Goodheart-Smithe 90f7cec7a5
Merge branch 'master' into index-lifecycle 2018-10-17 18:22:23 +01:00
Andy Bristol 18aa1c1381
add start trial API to HLRC (#33406)
Introduces client-specific request and response classes that do not
depend on the server

The `type` parameter is named `licenseType` in the response class to be
more descriptive. The parts that make up the acknowledged-required
response are given slightly different names than their server-response
types to be consistent with the naming in the put license API

Tests do not cover all cases because the integ test cluster starts up
with a trial license - this will be addressed in a future commit
2018-10-17 08:02:04 -07:00
Nik Everett 030e8c8fe3 Docs: Tweak upgrade docs
Tweak the upgrade instructions for moving from pre-6.3-with-x-pack to
post-6.3-default distribution. Specifically, you have to remove the
x-pack plugin before upgrading because 6.4 doesn't understand how to
remove it.

Relates to #34307
2018-10-17 08:17:12 -04:00
Tal Levy fbe8dc014c Merge branch 'master' into index-lifecycle 2018-10-16 13:58:53 -07:00
lipsill 9b5eaafc24 HLRC: Documentation examples cleanup (#34009)
* Replace deprecated field `code` with `source` for stored scripts (#25127)
* Replace examples using the deprecated endpoint `{index}/{type}/_search`
with `{index}/_search` (#29468)
* Use a system property to avoid deprecation warnings after the Update
Scripts have been moved to their own context (#32096)
2018-10-16 10:46:14 -04:00
Vladimir Dolzhenko 230ad5339b
HLRC: Add support for XPack Post Start Basic Licence API (#33606)
Relates to #29827
2018-10-16 15:21:30 +02:00
Paul Sanwald 936faba6c4
Add delete rollup job support to HL REST Client (#34066)
Add support for delete rollup job to HL REST Client.
2018-10-16 09:02:29 -04:00
Jim Ferenczi 544de13d8e
Disallow negative query boost (#34486)
This change disallows negative query boosts. Negative scores are not allowed in Lucene 8 so
it is easier to just disallow negative boosts entirely. We should also deprecate negative boosts
in 6x in order to ensure that users are aware when they'll upgrade to ES 7.

Relates #33309
2018-10-16 11:31:53 +01:00
Yogesh Gaikwad a4c302067e
HLRC: Create/Update role mapping API (#34171)
We added support for role mapper expression DSL in #33745,
that allows us to build the role mapper expression used in the
role mapping (as rules for determining user roles based on what
the boolean expression resolves to).

This change now adds support for create/update role mapping
API to the high-level rest client.
2018-10-16 03:05:46 +01:00
Nikolay Vasiliev f5641e61a2 Docs: improve formatting of Query String Query doc page (#34432)
Merge two tables.
2018-10-15 15:30:48 -04:00
Ioannis Kakavas 55eaf7a3ff
HLRC: Get SSL Certificates API (#34135)
This change adds support for the get SSL certificate API to 
the high level rest client.
2018-10-15 17:20:34 +01:00
Colin Goodheart-Smithe 0b42eda0e3
Merge branch 'master' into index-lifecycle 2018-10-15 16:03:37 +01:00
David Roberts 21c759af0e
[ML] Add an ingest pipeline definition to structure finder (#34350)
The ingest pipeline that is produced is very simple.  It
contains a grok processor if the format is semi-structured
text, a date processor if the format contains a timestamp,
and a remove processor if required to remove the interim
timestamp field parsed out of semi-structured text.

Eventually the UI should offer the option to customize the
pipeline with additional processors to perform other data
preparation steps before ingesting data to an index.
2018-10-12 07:56:35 +01:00
Andrei Stefan d7a94fb6aa
SQL: Functions enhancements (OCTET_LENGTH function, order functions alphabetically, RANDOM function docs) (#34101)
* New OCTET_LENGTH function
* Changed the way the FunctionRegistry stores functions, considering the alphabetic ordering by name
* Added documentation for the RANDOM function
2018-10-09 00:20:18 +03:00
Jim Ferenczi 36557469f6
[DOCS] Removes beta label from composite aggregation (#34329) 2018-10-05 19:46:20 +02:00
Jim Ferenczi 6e28c8f1c4
[DOCS] Remove experimental label from term_set query (#34328) 2018-10-05 19:45:23 +02:00
Jim Ferenczi cfe8eab455
[DOCS] Removes beta label from index sorting (#34327) 2018-10-05 19:44:25 +02:00
Benjamin Trent 6f5daadb3c
Feature/hlrc ml docs cleanup (#34316)
* HLRC: ML Add preview datafeed api

* Changing deprecation handling for parser

* Removing some duplication in docs, will address other APIs in another PR

* HLRC: ML Cleanup docs

* updating get datafeed stats docs
2018-10-05 08:45:46 -07:00
Nik Everett a9fa5f2b33
Docs: DRY up CRUD docs (#34203)
This further applies the pattern set in #34125 to reduce copy-and-paste
in the single document CRUD portion of the High Level REST Client docs.
It also adds line wraps to snippets that are too wide to fit into the box
when rendered in the docs, following up on the work started in #34163.
2018-10-05 11:41:03 -04:00
amoreauCoveo e95dc5474f Minor corrections in geo-queries.asciidoc (#34314) 2018-10-05 17:12:18 +02:00
Jim Ferenczi a9daa5cb90
[DOCS] Remove beta label from normalizers (#34326) 2018-10-05 15:42:00 +02:00
eray daf88335d7 Add max_children limit to nested sort (#33587)
Add an option to `nested` sort to limit the number of children to visit when picking the sort value
of the root document. 

Closes #33592
2018-10-05 12:02:47 +02:00
Ioannis Kakavas f420eebc73 [DOCS] Fix typo and add [float] 2018-10-05 10:07:50 +03:00
Tim Vernum 63dbd1dce0
Allow User/Password realms to disable authc (#34033)
The "lookupUser" method on a realm facilitates the "run-as" and
"authorization_realms" features.
This commit allows a realm to be used for "lookup only", in which
case the "authenticate" method (and associated token methods) are
disabled.
It does this through the introduction of a new
"authentication.enabled" setting, which defaults to true.
2018-10-05 12:10:42 +10:00
Tim Vernum 6608992523
Enable security automaton caching (#34028)
Building automatons can be costly. For the most part we cache things
that use automatons so the cost is limited.
However:
- We don't (currently) do that everywhere (e.g. we don't cache role
  mappings)
- It is sometimes necessary to clear some of those caches which can
  cause significant CPU overhead and processing delays.

This commit introduces a new cache in the Automatons class to avoid
unnecesarily recomputing automatons.
2018-10-05 12:09:36 +10:00
Dimitris Athanasiou 4dacfa95d2
[ML] Allow asynchronous job deletion (#34058)
This changes the delete job API by adding
the choice to delete a job asynchronously.
The commit adds a `wait_for_completion` parameter
to the delete job request. When set to `false`,
the action returns immediately and the response
contains the task id.

This also changes the handling of subsequent
delete requests for a job that is already being
deleted. It now uses the task framework to check
if the job is being deleted instead of the cluster
state. This is a beneficial for it is going to also
be working once the job configs are moved out of the
cluster state and into an index. Also, force delete
requests that are waiting for the job to be deleted
will not proceed with the deletion if the first task
fails. This will prevent overloading the cluster. Instead,
the failure is communicated better via notifications
so that the user may retry.

Finally, this makes the `deleting` property of the job
visible (also it was renamed from `deleted`). This allows
a client to render a deleting job differently.

Closes #32836
2018-10-05 02:41:28 +03:00
Benjamin Trent 026488bcbf
HLRC: ML Adding get datafeed stats API (#34271)
* HLRC: ML Adding get datafeed stats API

* addressing PR comments

* fixing field exclusion filter

* removing unnecessary whitespace
2018-10-04 14:43:06 -07:00
Julie Tibshirani a6c5b6807f
Small fixes to the HLRC watcher documentation. (#34306)
* Make sure to use the new docs style for 'ack watch'.
* Replace 'document' with 'watcher' in page names.
2018-10-04 23:30:20 +02:00
Nik Everett 09aaed4fe4
Tasks: Document that status is not semvered (#34270)
The `status` part of the tasks API reflects the internal status of a
running task. In general, we do not make backwards breaking changes to
the `status` but because it is internal we reserve the right to do so. I
suspect we will very rarely excercise that right but it is important
that we have it so we're not boxed into any particular implementation
for a request.

In some sense this is policy making by documentation change. In another
it is clarification of the way we've always thought of this field.

I also reflect the documentation change into the Javadoc in a few
places. There I acknowledge Kibana's "special relationship" with
Elasticsearch. Kibana parses `_reindex`'s `status` field and, because
we're friends with those folks, we should talk to them before we make
backwards breaking changes to it. We *want* to be friends with everyone
but there is only so much time in the day and we don't *want* to make
backwards breaking fields to `status` at all anyway. So we hope that
breaking changes documentation should be enough for other folks.

Relates to #34245.
2018-10-04 14:42:37 -04:00
Benjamin Trent 2dd058d607
HLRC: ML Add preview datafeed api (#34284)
* HLRC: ML Add preview datafeed api

* Changing deprecation handling for parser

* Removing some duplication in docs, will address other APIs in another PR
2018-10-04 11:28:44 -07:00
Nik Everett dc2cf28fde
Docs: Allow skipping response assertions (#34240)
We generate tests from our documentation, including assertions about the
responses returned by a particular API. But sometimes we *can't* assert
that the response is correct because of some defficiency in our tooling.
Previously we marked the response `// NOTCONSOLE` to skip it, but this
is kind of odd because `// NOTCONSOLE` is really to mark snippets that
are json but aren't requests or responses. This introduces a new
construct to skip response assertions:
```
// TESTRESPONSE[skip:reason we skipped this]
```
2018-10-04 08:03:38 -04:00
Ignacio Vera 3ccb7af56a
HLRC: Add activate watch action (#33988)
* HLRC: Add activate watcher action

Adds activate watch action to the high level rest client.

Relates #29827
2018-10-04 12:49:52 +02:00
Ed Savage 577261ee57
[ML] Label anomalies with multi_bucket_impact (#34233)
* [ML] Label anomalies with  multi_bucket_impact

Add the multi_bucket_impact field to record results.
2018-10-04 09:08:21 +01:00
Ioannis Kakavas 74993fe151
Document smtp.ssl.trust configuration option (#34275)
This adds documentation for `smtp.ssl.trust` that was reintroduced
in #31684 in 6.3.2.

Resolves #32936
2018-10-04 10:53:55 +03:00
Ioannis Kakavas 2c82b80b85
Support PKCS#11 tokens as keystores and truststores (#34063)
This enables Elasticsearch to use the JVM-wide configured
PKCS#11 token as a keystore or a truststore for its TLS configuration.
The JVM is assumed to be configured accordingly with the appropriate
Security Provider implementation that supports PKCS#11 tokens.
For the PKCS#11 token to be used as a keystore or a truststore for an
SSLConfiguration, the .keystore.type or .truststore.type must be
explicitly set to pkcs11 in the configuration.
The fact that the PKCS#11 token configuration is JVM wide implies that
there is only one available keystore and truststore that can be used by TLS
configurations in Elasticsearch.
The PIN for the PKCS#11 token can be set as a truststore parameter in
Elasticsearch or as a JVM parameter ( -Djavax.net.ssl.trustStorePassword).

The basic goal of enabling PKCS#11 token support is to allow PKCS#11-NSS in
FIPS mode to be used as a FIPS 140-2 enabled Security Provider.
2018-10-04 10:51:58 +03:00
Kazuhiro Sera d45fe43a68 Fix a variety of typos and misspelled words (#32792) 2018-10-03 18:11:38 +01:00
Mayya Sharipova 8f10c771e6 Add migration info for missing values in script
Relates to #30975
2018-10-03 11:56:18 -04:00
Lee Hinman 90c55f5e36 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-03 09:11:28 -06:00
albendz f09190c14d Require combine and reduce scripts in scripted metrics aggregation (#33452)
* Make text message not required in constructor for slack

* Remove unnecessary comments in test file

* Throw exception when reduce or combine is not provided; update tests

* Update integration tests for scripted metrics to always include reduce and combine

* Remove some old changes from previous branches

* Rearrange script presence checks to be earlier in build

* Change null check order in script builder for aggregated metrics; correct test scripts in IT

* Add breaking change details to PR
2018-10-03 15:22:01 +01:00
Vladimir Dolzhenko a7f62ee902
[GCE Discovery] Automatically set project-id and zone (#33721)
Fetch default values for project-id and zone from metadata server

Closes #13618
2018-10-03 11:37:36 +02:00
Julie Tibshirani c6fcb60071
Add support for 'ack watch' to the HLRC. (#33962) 2018-10-03 02:03:03 -07:00
Christoph Büscher 6fb9c63ed6 [Docs] Fix broken link for HLRC rethrottle 2018-10-02 23:14:37 +02:00
Wilder Pereira 5af6ae564d Change "REST Verb" to "HTTP Verb" (#34195) 2018-10-02 17:09:54 -04:00
Christoph Büscher a1c441f78a
HLRC: Add throttling for update & delete-by-query (#33951)
This change adds throttling to the update-by-query and delete-by-query cases
similar to throttling for reindex. This mostly means additional methods on the
client class itself, since the request hits the same RestHandler, just with
slightly different endpoints, and also the return values are similar.
2018-10-02 21:44:15 +02:00
Gordon Brown fb907706ec Merge branch 'master' into index-lifecycle 2018-10-02 13:43:46 -06:00
Gordon Brown dd3fe92673
[DOCS] Note that User Cluster Metadata is not private (#34156)
As user-defined cluster metadata is accessible to anyone with access to
get the cluster settings, stored in the logs, and likely to be tracked
by monitoring solutions, it is useful to clarify in the documentation
that it should not be used to store secret information.
2018-10-02 13:36:13 -06:00
Andriy 6b714c9e1e [Docs] Updated link to kafka-elasticsearch-consumer project (#34234) 2018-10-02 17:46:38 +02:00
Lisa Cawley a4cf4ca585
[DOCS] Clarifies examples in reindex and task APIs (#33143) 2018-10-02 08:37:45 -07:00
Serge Populov 13af5d5d7f Docs: Fix typo in field name in aggregations (#34223) 2018-10-02 10:54:29 -04:00
Nik Everett f904c41506
HLRC: Add get rollup job (#33921)
Adds support for the get rollup job to the High Level REST Client. I had
to do three interesting and unexpected things:
1. I ported the rollup state wiping code into the high level client
tests. I'll move this into the test framework in a followup and remove
the x-pack version.
2. The `timeout` in the rollup config was serialized using the
`toString` representation of `TimeValue` which produces fractional time
values which are more human readable but aren't supported by parsing. So
I switched it to `getStringRep`.
3. Refactor the xcontent round trip testing utilities so we can test
parsing of classes that don't implements `ToXContent`.
2018-10-02 09:11:29 -04:00
Ioannis Kakavas 1d049cadbe Fix HLRC docs 2018-10-02 13:23:44 +03:00
Ioannis Kakavas 300896d401
HLRC: add change password API support (#33509)
This change adds support for the change password APIs to the high
level rest client.

Relates #29827
2018-10-02 12:14:25 +03:00
Marios Trivyzas a8a2a83d45
SQL: Fix grammar for `*` in arithm expressions (#34176)
Previously, parsing an arithmetic expression with `*` and no spaces,
e.g.: `2*i` threw a parsing exception as the grammar rule for
tableIdentifier was clashing with the rule for arithmetic operator `*`.

This issue comes already in the lexer and the left part of the
expression (in our example `2*`) was recognised as a
TABLE_IDENTIFIER token.

The solution adopted is to allow the `*` wildcard in the table name
only if it's surrounded with double quotes, e.g.: `"my*index"`

Closes: #33957
2018-10-02 11:47:42 +03:00
Lee Hinman 2d9cb21490 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-01 14:10:09 -06:00
Nik Everett ad3218b4ab
Checkstyle: Catch wide snippets (#34163)
We use wrap code in `// tag` and `//end` to include it in our docs. Our
current docs style wraps code snippets in a box that is only wide enough
for 76 characters and adds a horizontal scroll bar for wider snippets
which makes the snippet much harder to read. This adds a checkstyle check
that looks for java code that is included in the docs and is wider than
that 76 characters so all snippets fit into the box. It solves many of
the failures that this catches but suppresses many more. I will clean
those up in a follow up change.
2018-10-01 15:22:15 -04:00
Vladimir Dolzhenko 84111e9607 fix broken doc due to `elasticsearch-translog` removal 2018-10-01 17:54:32 +02:00
David Roberts a1d2ded98d
[ML] Fix unit test deadlock problem (#34174)
This change fixes a potential deadlock problem in the unit
test introduced in #34117.

It also removes a piece of debug code and corrects a docs
formatting problem that were both added in that same PR.
2018-10-01 15:35:37 +01:00
Vladimir Dolzhenko 2e2ae19b97
drop elasticsearch-translog for 7.0 (#33373)
#32281 adds elasticsearch-shard to provide bwc version of elasticsearch-translog for 6.x; have to remove elasticsearch-translog for 7.0

Relates to #31389
2018-10-01 16:21:14 +02:00
Nik Everett 8082b4ad4a
Docs: DRY up HLRC index docs (#34157)
Use tricks pioneered in #34125 to shorted asciidoc for the index API's
docs. Also slightly shorten the supported-api docs file.
2018-10-01 09:59:36 -04:00
Hendrik Muhs fb85dd8c0c
[Docs] remove double words (#34138)
removes double wording and typo
2018-09-29 15:18:00 +02:00
Lee Hinman 6ea396a476 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-28 15:40:12 -06:00
Nik Everett 47bcb56a89
Docs: Cut down on high level rest client copy-and-paste-ness (#34125)
* Anchors

* includes

* get settings

* more

* More
2018-09-28 14:48:11 -04:00
Lisa Cawley 68b42f7d89
[DOCS] Moves reserved keywords under SQL section (#34124) 2018-09-28 10:17:41 -07:00
lcawl 57052f617a [DOCS] Fixes callout in ML API 2018-09-28 10:06:02 -07:00
Lisa Cawley f143e39a71
[DOCS] Adds new dynamic machine learning settings (#34094) 2018-09-28 09:41:14 -07:00
David Roberts f709c2f694
[ML] Add a timeout option to file structure finder (#34117)
This can be used to restrict the amount of CPU a single
structure finder request can use.

The timeout is not implemented precisely, so requests
may run for slightly longer than the timeout before
aborting.

The default is 25 seconds, which is a little below
Kibana's default timeout of 30 seconds for calls to
Elasticsearch APIs.
2018-09-28 17:32:35 +01:00
Lee Hinman a26cc1a242 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-27 11:00:37 -06:00
Lisa Cawley 37be3e713c
[DOCS] Synchronize location of Breaking Changes (#33588) 2018-09-27 08:41:38 -07:00
Lisa Cawley 949e4e9d1a
[DOCS] Synchronizes captialization in top-level titles (#33605) 2018-09-27 08:36:18 -07:00
Lisa Cawley e4c3fd6d96
[DOCS] Moves graph to docs folder (#33472) 2018-09-26 14:59:31 -07:00
Christoph Büscher 7bf216f4a1
[Docs] Add reason to use Settings API over config file (#32405)
Extending the warning to prefer the `setttings` API over changes in the
configuration file by giving reasons why this might be dangerous.
2018-09-26 11:59:36 +02:00
Nikolay Vasiliev 16956a1a05 [DOCS] Clarify 'type' parameter meaning for custom analyzer (#34012)
This pull request improves the docs on the meaning of type parameter on the custom 
analyzer doc page. 

Closes #33456
2018-09-25 15:32:27 +02:00
David Roberts dfe5af0411
[ML] Return both Joda and Java formats from structure finder (#33900)
Previously the timestamp_formats field in the response
from the find_file_structure endpoint contained Joda
timestamp formats.  This change makes that clear by
renaming the field to joda_timestamp_formats, and also
adds a java_timestamp_formats field containing the
equivalent Java time format strings.
2018-09-25 12:52:51 +01:00
Daniel Mitterdorfer 02fb5aa4ec
Remove leftover doc about format being updatable
With this commit we remove a leftover in the docs about the `format`
field being updatable. This is not true since we removed support for
updates in #25285.

Closes #33986
Relates #25285
Relates #34006
2018-09-25 10:13:23 +02:00
Julie Tibshirani e6cb9d642b Fix a broken reference to types removal in the Java API docs. 2018-09-24 18:06:35 -07:00
Andrei Stefan 0fae6d39f5
SQL: functions docs update (#34000)
* Changed the format of the String functions documentation page.
* Adopted the same format for Math functions, but completely changed the examples.
* Added missing documentation for Math functions.
2018-09-25 02:42:18 +03:00
Julie Tibshirani 704d3e4c24
Add a deprecation warning in the type query documentation. (#34017) 2018-09-24 16:30:38 -07:00
debadair 1c579646d3
[DOCS] Removed Homebrew installation section (#34011) 2018-09-24 10:29:12 -07:00
Lee Hinman 243e863f6e Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-24 10:33:51 -06:00
David Roberts b89551c452
[ML] Display integers without .0 in file structure field stats (#33947)
Previously numeric values in the field_stats created by the
find_file_structure endpoint were always output with a
decimal point.  This looked unfriendly and unnatural for
fields that clearly store integer values.  This change
converts integer values to type Integer before output in
the file structure field stats.
2018-09-22 15:48:59 +01:00
Benjamin Trent ea3f3e4b61
HLRC: ML Stop datafeed API (#33946)
* HLRC: ML stop data feed api
2018-09-21 12:13:35 -07:00
Benjamin Trent bf0a0f74da
HLRC: ML start data feed API (#33898)
* HLRC: ML start data feed API
2018-09-21 05:56:38 -07:00
Nik Everett cac93949fe
API: Drop deprecated methods from Retry (#33925)
We deprecated the `Retry.withBackoff` flavors with `Settings` in 6.5
because they were no longer needed. This drops them form 7.0.
2018-09-21 07:55:50 -04:00
Jan Jíša 822b067a3e Docs: Corrected typo in how to (#33910)
max_context_length -> max_content_length
2018-09-20 16:13:46 -04:00
Christoph Büscher 77145bb477
HLRC: Add support for reindex rethrottling (#33832)
This change adds support for rethrottling reindex requests to the
RestHighLevelClient.
2018-09-20 18:56:12 +02:00
Andrei Stefan 6fb7e49b22
SQL: TRUNCATE and ROUND functions (#33779)
* Added TRUNCATE function, modified ROUND to accept two parameters instead of one. Made the second parameter optional for both functions.
* Added documentation for both functions.
2018-09-20 15:29:53 +03:00
David Roberts f5a2ffc3f6
[DOCS][ML] Document the ML find_file_structure endpoint (#33723)
Relates #33471
Relates #33630
2018-09-20 12:17:09 +01:00
Nik Everett 26c4f1fb6c
Core: Default node.name to the hostname (#33677)
Changes the default of the `node.name` setting to the hostname of the
machine on which Elasticsearch is running. Previously it was the first 8
characters of the node id. This had the advantage of producing a unique
name even when the node name isn't configured but the disadvantage of
being unrecognizable and not being available until fairly late in the
startup process. Of particular interest is that it isn't available until
after logging is configured. This forces us to use a volatile read
whenever we add the node name to the log.

Using the hostname is available immediately on startup and is generally
recognizable but has the disadvantage of not being unique when run on
machines that don't set their hostname or when multiple elasticsearch
processes are run on the same host. I believe that, taken together, it
is better to default to the hostname.

1. Running multiple copies of Elasticsearch on the same node is a fairly
advanced feature. We do it all the as part of the elasticsearch build
for testing but we make sure to set the node name then.
2. That the node.name defaults to some flavor of "localhost" on an
unconfigured box feels like it isn't going to come up too much in
production. I expect most production deployments to at least set the
hostname.

As a bonus, production deployments need no longer set the node name in
most cases. At least in my experience most folks set it to the hostname
anyway.
2018-09-19 15:21:29 -04:00
Lee Hinman 81e9150c7a Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-19 09:43:26 -06:00
Alan Woodward 5107949402
Allow TokenFilterFactories to rewrite themselves against their preceding chain (#33702)
We currently special-case SynonymFilterFactory and SynonymGraphFilterFactory, which need to 
know their predecessors in the analysis chain in order to correctly analyze their synonym lists. This
special-casing doesn't work with Referring filter factories, such as the Multiplexer or Conditional
filters. We also have a number of filters (eg the Multiplexer) that will break synonyms when they
appear before them in a chain, because they produce multiple tokens at the same position.

This commit adds two methods to the TokenFilterFactory interface.

* `getChainAwareTokenFilterFactory()` allows a filter factory to rewrite itself against its preceding
  filter chain, or to resolve references to other filters. It replaces `ReferringFilterFactory` and
  `CustomAnalyzerProvider.checkAndApplySynonymFilter`, and by default returns `this`.
* `getSynonymFilter()` defines whether or not a filter should be applied when building a synonym
  list `Analyzer`. By default it returns `true`.

Fixes #33609
2018-09-19 15:52:14 +01:00
David Kyle 68c0a29578
HLRC: Delete ML calendar (#33775) 2018-09-19 10:20:21 +01:00
Christoph Büscher 6f3b3338ba
[Docs] Clarify accessing Date methods in painless (#33560)
The documentation currently tells users to use `doc['event_date'].value.getMillis` to access
milliseconds in a date. It turns out the way it works is `doc['event_date'].value.millis`. This 
change corrects this and gives a hint at how other date related methods work.
2018-09-19 10:34:53 +02:00
Vladimir Dolzhenko a3e8b831ee
add elasticsearch-shard tool (#32281)
Relates #31389
2018-09-19 10:28:22 +02:00
David Turner c9765d5fb9
Emphasize that filesystem-level backups don't work (#33102)
It is not obvious that a filesystem-level backup may capture an inconsistent
set of files that may fail on restore, or (worse) succeed having silently
discarded some data. This change spells the out, and reorganises the first page
or so of the snapshot/restore docs to make this warning fit more nicely.
2018-09-19 08:36:03 +01:00
Lee Hinman c87cff22b4 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-18 13:57:41 -06:00
Tim Heckel 3928921a1d [DOCS] Update scroll.asciidoc (#32530) 2018-09-18 17:00:22 +02:00
Abdon Pijpelink 32ee6148d2 [DOCS] Clarify scoring for multi_match phrase type (#32672)
The original statement "Runs a match_phrase query on each field and combines the _score from each field." for the phrase type is a but misleading. The phrase type behaves like the best_fields type and does not combine the scores of each fields.
2018-09-18 16:57:33 +02:00
Dan Tennery-Spalding 3596512e6a [DOCS] Corrected several grammar errors (#33781) 2018-09-18 16:46:22 +02:00
David Turner 421f58e172
Remove discovery-file plugin (#33257)
In #33241 we moved the file-based discovery functionality to core
Elasticsearch, but preserved the `discovery-file` plugin, and support for the
existing location of the `unicast_hosts.txt` file, for BWC reasons. This commit
completes the removal of this plugin.
2018-09-18 12:01:16 +01:00
David Kyle 9543992d8e
HLRC: Get ML calendars (#33760) 2018-09-18 11:51:11 +01:00
markharwood 2fa09f062e
New plugin - Annotated_text field type (#30364)
New plugin for annotated_text field type.
Largely a copy of `text` field type but adds ability to include markdown-like syntax in the text.
The “AnnotatedText” class parses text+markup and converts into plain text and AnnotationTokens.
The annotation token values are injected unchanged alongside the regular text tokens to provide a
form of additional indexed overlay useful in positional searches and highlighting.
Annotated_text fields do not support fielddata as we want to phase this out.
Also includes a new "annotated" highlighter type that retains annotations and merges in search
hits as additional annotation markup.

Closes #29467
2018-09-18 10:25:27 +01:00
Armin Braun 87cedef3cf
NETWORKING:Def CName in Http Publish Addr to True (#33631)
* Follow up to #32806 setting the setting to true for 7.x
2018-09-18 10:29:02 +02:00
Shaunak Kashyap 2aba52de8f
Implement xpack.monitoring.elasticsearch.collection.enabled setting (#33474)
* Implement xpack.monitoring.elasticsearch.collection.enabled setting

* Fixing line lengths

* Updating constructor calls in test

* Removing unused import

* Fixing line lengths in test classes

* Make monitoringService.isElasticsearchCollectionEnabled() return true for tests

* Remove wrong expectation

* Adding unit tests for new flag to be false

* Fixing line wrapping/indentation for better readability

* Adding docs

* Fixing logic in ClusterStatsCollector::shouldCollect

* Rebasing with master and resolving conflicts

* Simplifying implementation by gating scheduling

* Doc fixes / improvements

* Making methods package private

* Fixing wording

* Fixing method access
2018-09-17 18:33:43 -07:00
ben5556 012b9c7539 Corrected aggregation name to match the example (#33786) 2018-09-17 18:24:43 -07:00
Or Bin a5bad4d92c Docs: Fixed a grammatical mistake: 'a HTTP ...' -> 'an HTTP ...' (#33744)
Fixed a grammatical mistake: 'a HTTP ...' -> 'an HTTP ...'

Closes #33728
2018-09-17 15:35:54 -04:00
Lee Hinman 7ff11b4ae1 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-17 10:41:10 -06:00
Tanguy Leroux b2413d2068 [Docs] Fix broken external links in HLRC Rollup documentation
Another attempt. Introduced in #33521
2018-09-17 13:37:39 +02:00
Tanguy Leroux baa69a5ed5 [Docs] Fix broken links in HLRC Rollup documentation
Introduced in #33521
2018-09-17 13:02:11 +02:00
Tanguy Leroux e77835c6f5
Add create rollup job api to high level rest client (#33521)
This commit adds the Create Rollup Job API to the high level REST
client. It supersedes #32703 and adds dedicated request/response
objects so that it does not depend on server side components.

Related #29827
2018-09-17 09:10:23 +02:00
Ryan Ernst 3046656ab1
Scripting: Rework joda time backcompat (#33486)
This commit switches the joda time backcompat in scripting to use
augmentation over ZonedDateTime. The augmentation methods provide
compatibility with the missing methods between joda's DateTime and
java's ZonedDateTime. Due to getDayOfWeek returning an enum in the java
API, ZonedDateTime is wrapped so that the method can return int like the
joda time does. The java time api version is renamed to
getDayOfWeekEnum, which will be kept through 7.x for compatibility while
users switch back to getDayOfWeek once joda compatibility is removed.
2018-09-16 19:18:00 -07:00
Dimitris Athanasiou db40315afb
[HLRC][ML] Add ML get datafeed API to HLRC (#33715)
Relates #29827
2018-09-16 11:54:55 +01:00
Lee Hinman e6cbaa5a78 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-14 16:27:37 -06:00
Lisa Cawley 9706584836
[DOCS] Moves security reference to docs folder (#33643) 2018-09-14 13:09:47 -07:00
David Kyle b04faa059b
HLRC: ML PUT Calendar (#33362) 2018-09-14 15:00:18 +01:00
Jay Modi 3914a980f7
Security: remove wrapping in put user response (#33512)
This change removes the wrapping of the created field in the put user
response. The created field was added as a top level field in #32332,
while also still being wrapped within the `user` object of the
response. Since the value is available in both formats in 6.x, we can
remove the wrapped version for 7.0.
2018-09-13 14:40:36 -06:00
Costin Leau 32a22ca00e DOC: improved wording in SQL client app section 2018-09-13 22:07:23 +03:00
Lisa Cawley c3a817957d
[DOCS] Moves securing-communications to docs (#33640) 2018-09-13 10:42:26 -07:00
Dimitris Athanasiou 9600819cef
[HLRC][ML] Add ML delete datafeed API to HLRC (#33667)
Relates #29827
2018-09-13 17:13:36 +01:00
Costin Leau a192785fc8 DOC: Add SQL section on client applications
Add setup instructions for a number of GUI SQL applications
2018-09-13 15:44:52 +03:00
Colin Goodheart-Smithe 8e59de3eb2
Merge branch 'master' into index-lifecycle 2018-09-13 09:46:14 +01:00
Jason Tedor c023f67c5d
Add migration note for remote cluster settings (#33632)
The remote cluster settings search.remote.* have been renamed to
cluster.remote.* and are automatically upgraded in the cluster state on
gateway recovery, and on put. This commit adds a note to the migration
docs for these changes.
2018-09-12 13:37:11 -04:00
Simon Willnauer c783488e97
Add `_source`-only snapshot repository (#32844)
This change adds a `_source` only snapshot repository that allows to wrap
any existing repository as a _backend_ to snapshot only the `_source` part
including live docs markers. Snapshots taken with the `source` repository
won't include any indices,  doc-values or points. The snapshot will be reduced in size and
functionality such that it requires full re-indexing after it's successfully restored.

The restore process will copy the `_source` data locally starts a special shard and engine
to allow `match_all` scrolls and searches. Any other query, or get call will fail with and unsupported operation exception.  The restored index is also marked as read-only.

This feature aims mainly for disaster recovery use-cases where snapshot size is
a concern or where time to restore is less of an issue.

**NOTE**: The snapshot produced by this repository is still a valid lucene index. This change doesn't allow for any longer retention policies which is out of scope for this change.
2018-09-12 17:47:10 +02:00
Christoph Büscher fe478c23b7
[Docs] Fix heading in composite-aggregation.asciidoc (#33627)
The heading for the "Missing buckets" should be on the same level 
as the the "Order" section.
2018-09-12 16:56:03 +02:00
Dimitris Athanasiou 2eb2313b60
[HLRC][ML] Add ML put datafeed API to HLRC (#33603)
This also changes both `DatafeedConfig` and `DatafeedUpdate`
to store the query and aggs as a bytes reference. This allows
the client to remove its dependency to the named objects
registry of the search module.

Relates #29827
2018-09-12 14:52:36 +01:00
Joel Green 0b567c0eeb [Docs] Update match-query.asciidoc (#33610) 2018-09-12 14:35:27 +02:00
Jim Ferenczi 4561c5ee83
Clarify context suggestions filtering and boosting (#33601)
This change clarifies the documentation of the context completion suggester
regarding filtering and boosting with contexts.
Unlike the suggester v1, filtering on multiple contexts
works as a disjunction, a suggestion matches if it contains at least one of the provided
context values and boosting selects the maximum score among the matching contexts.
This commit also adapts an old test that was written for the v1 suggester and commented out
for version 2 because the behavior changed.
2018-09-12 08:47:32 +02:00
Armin Braun 94cdf0ceba
NETWORKING: http.publish_host Should Contain CNAME (#32806)
* NETWORKING: http.publish_host Should Contain CNAME

* Closes #22029
2018-09-12 06:15:36 +02:00
Benjamin Trent 27e07ec859
HLRC: ML Delete Forecast API (#33526)
* HLRC: ML Delete Forecast API
2018-09-11 16:32:52 -07:00
Lisa Cawley cbc6fa0ecb
[DOCS] Adds missing built-in user information (#33585) 2018-09-11 07:56:26 -07:00
Ed Savage 2f3b542d57
HLRC: Add ML get categories API (#33465)
HLRC: Adding the ML 'get categories' API
2018-09-11 12:48:14 +01:00
Alan Woodward f598297f55
Add predicate_token_filter (#33431)
This allows users to filter out tokens from a TokenStream using painless scripts, 
instead of having to write specialised Java code and packaging it up into a plugin.

The commit also refactors the AnalysisPredicateScript.Token class so that it wraps
and makes read-only an AttributeSource.
2018-09-11 09:16:39 +01:00
Colin Goodheart-Smithe cdc4f57a77
Merge branch 'master' into index-lifecycle 2018-09-10 21:30:44 +01:00
Tanguy Leroux 079d130d8c
[Test] Remove duplicate method in TestShardRouting (#32815) 2018-09-10 18:29:00 +02:00
S.Y. Wang 9073dbefd6 HLRC: Add put stored script support to high-level rest client (#31323)
Relates to #27205
2018-09-09 13:47:47 +02:00
lcawl 6b780e9926 [DOCS] Fixing formatting issues in breaking changes 2018-09-07 16:53:36 -07:00
Jay Modi 9d16a7b7f0
HLRC: add enable and disable user API support (#33481)
This change adds support for enable and disable user APIs to the high
level rest client. There is a common request base class for both
requests with specific requests that simplify the use of these APIs.

The response for these APIs is simply an empty object so a new response
class has been created for cases where we expect an empty response to
be returned.

Finally, the put user documentation has been moved to the proper
location that is not within an x-pack sub directory and the document
tags no longer contain x-pack.

See #29827
2018-09-07 11:51:37 -06:00
lcawl 944868908c [DOCS] Fixes formatting error 2018-09-07 10:26:44 -07:00
Benjamin Trent 4d233107f8
HLRC: ML Forecast Job (#33506)
* HLRC: ML Forecast job
2018-09-07 11:16:58 -05:00
Benjamin Trent 9230a48722
HLRC: ML Post Data (#33443)
* HLRC: ML Post data
2018-09-07 07:04:27 -05:00
Colin Goodheart-Smithe 017ffe5d12
Merge branch 'master' into index-lifecycle 2018-09-07 10:59:10 +01:00
Jim Ferenczi 79cd6385fe
Collapse package structure for metrics aggs (#33463)
This change collapses all metrics aggregations classes into a single package `org.elasticsearch.aggregations.metrics`.
It also restricts the visibility of some classes (aggregators and factories) that should not be used outside of the package.

Relates #22868
2018-09-07 10:58:06 +02:00
Lisa Cawley 7441c0376e
[DOCS] Adds delete forecast API (#33401) 2018-09-06 09:20:42 -07:00
Costin Leau 443f9caddd DOC: Enhance SQL Functions documentation
Split function section into multiple chapters
Add String functions
Add (small) section on Conversion/Cast functions
Add missing aggregation functions
Enable documentation testing (was disabled by accident). While at it,
fix failing tests
Improve spec tests to allow multi-line queries (useful for docs)
Add ability to ignore a spec test (name should end with -Ignore)
2018-09-06 18:09:53 +03:00
Jim Ferenczi 7ad71f906a
Upgrade to a Lucene 8 snapshot (#33310)
The main benefit of the upgrade for users is the search optimization for top scored documents when the total hit count is not needed. However this optimization is not activated in this change, there is another issue opened to discuss how it should be integrated smoothly.
Some comments about the change:
* Tests that can produce negative scores have been adapted but we need to forbid them completely: #33309

Closes #32899
2018-09-06 14:42:06 +02:00
Benjamin Trent 9b6bbc0182
HLRC: ML Update Job (#33392)
* HLRC: ML Update Job
2018-09-06 07:18:09 -05:00
Colin Goodheart-Smithe b1257d873b
Merge branch 'master' into index-lifecycle 2018-09-06 08:17:40 +01:00
Jason Tedor d71ced1b00
Generalize search.remote settings to cluster.remote (#33413)
With features like CCR building on the CCS infrastructure, the settings
prefix search.remote makes less sense as the namespace for these remote
cluster settings than does a more general namespace like
cluster.remote. This commit replaces these settings with cluster.remote
with a fallback to the deprecated settings search.remote.
2018-09-05 20:43:44 -04:00
Tal Levy b5f7fb6882 Merge branch 'master' into index-lifecycle 2018-09-05 12:56:58 -07:00
Jay Modi ea52277a1e
HLRest: add put user API (#32332)
This commit adds a security client to the high level rest client, which
includes an implementation for the put user api. As part of these
changes, a new request and response class have been added that are
specific to the high level rest client. One change here is that the response
was previously wrapped inside a user object. The plan is to remove this
wrapping and this PR adds an unwrapped response outside of the user
object so we can remove the user object later on.

See #29827
2018-09-05 10:56:30 -06:00
Christoph Büscher eafc2a5470
Don't count metadata fields towards index.mapping.total_fields.limit (#33386)
The maximum number of fields per index is limited to 1000 by default by the
`index.mapping.total_fields.limit` setting to prevent accidental mapping
explosions due to too many fields. Currently all metadata fields also count
towards this limit, which can lead to some confusion when using lower limits.
It is not obvious for users that they cannot actually add as many fields as
are specified by the limit in this case.

This change takes the number of metadata fields out of the field count that we
check against the field limit. It also adds tests that check that we can add
fields up to the specified limit, but throw an exception for any additional field added.

Closes #24096
2018-09-05 18:27:21 +02:00
Dimitris Athanasiou 24776b2b80
HLRC: Add ML get influencers API (#33389)
Relates #29827
2018-09-05 15:06:55 +01:00
Alan Woodward 636442700c
Add conditional token filter to elasticsearch (#31958)
This allows tokenfilters to be applied selectively, depending on the status of the current token in the tokenstream.  The filter takes a scripted predicate, and only applies its subfilter when the predicate returns true.
2018-09-05 14:52:43 +01:00
Paul Sanwald c303006e6b
Add interval response parameter to AutoDateInterval histogram (#33254)
Adds the interval used to the aggregation response.
2018-09-05 07:35:59 -04:00
Colin Goodheart-Smithe f00a28a909
Merge branch 'master' into index-lifecycle 2018-09-05 09:48:48 +01:00
Gordon Brown cfd3fa72ed
Add user-defined cluster metadata (#33325)
Adds a place for users to store cluster-wide data they wish to associate
with the cluster via the Cluster Settings API. This is strictly for
user-defined data, Elasticsearch makes no other other use of these
settings.
2018-09-04 16:14:18 -06:00
Lisa Cawley f3f8d9b833
[DOCS] Moves monitoring pages to docs folder (#33324) 2018-09-04 10:02:13 -07:00
lcawl c5109a54ee [DOCS] Revert fix for broken link 2018-09-04 09:26:28 -07:00
Costin Leau 43f80fa82b DOCS: Fix anchor and example typos 2018-09-04 19:06:44 +03:00
lcawl 303ae25a6a [DOCS] Fixes broken link 2018-09-04 09:05:30 -07:00
Costin Leau 17c7f99343
SQL: Show/desc commands now support table ids (#33363)
Extend SHOW TABLES, DESCRIBE and SHOW COLUMNS to support table
identifiers not just SQL LIKE pattern.
This allows both Elasticsearch-style multi-index patterns and SQL LIKE.
To disambiguate between the two (as the " vs ' can be easy to miss),
the grammar now requires LIKE keyword as a prefix for all LIKE-like
patterns.

Also added some docs comparing the two types of patterns.

Fix #33294
2018-09-04 16:54:10 +03:00
Sohaib Iftikhar 761e8c461f HLRC: Add delete by query API (#32782)
Adds the delete-by-query API to the High Level REST Client.
2018-09-04 08:56:26 -04:00
Dimitris Athanasiou 1457b07a06
[ML] The sort field on get records should default to the record_score (#33358)
This is not changing the behaviour as when the sort field was set
to `influencer_score` the secondary sort would be used and that
was using the `record_score` at the highest priority.
2018-09-04 11:38:24 +01:00
Colin Goodheart-Smithe 92ab442aee
Merge branch 'master' into index-lifecycle 2018-09-04 10:34:49 +01:00
Nikolay Vasiliev d9f394b099 [DOCS] fix a couple of typos (#33356) 2018-09-04 10:07:11 +02:00
Christoph Büscher 79db16f9bb
[Docs] Add search timeout caveats (#33354)
Global search timeouts and timeouts specified in the search request body use the
same internal mechanism as search cancellation. Therefore the same caveats
apply, mostly around the responsiveness of the timeout which gets only checked
by a running search on segment boundaries by default.

Closes #31263
2018-09-03 20:56:05 +02:00
Dimitris Athanasiou 54fe7fb5a1
HLRC: Add ML get overall buckets API (#33297)
Relates #29827
2018-09-03 13:34:36 +01:00
Christoph Büscher 978d1ed257
[Docs] Improve tuning for speed advice (#33315)
This change merges two sections in the "Tune for search speed" documentation
that recommend mapping numeric identifiers as keywords. Both sections contain
mostly the same advice, so they can be merged.

Closes #32733
2018-09-03 11:09:30 +02:00
Colin Goodheart-Smithe e2c1beb1be
Merge branch 'master' into index-lifecycle 2018-09-03 10:01:16 +01:00
Jim Ferenczi 713c07e14d
Add early termination support to BucketCollector (#33279)
This commit adds the support to early terminate the collection of a leaf
in the aggregation framework. This change introduces a MultiBucketCollector which
handles CollectionTerminatedException exactly like the Lucene MultiCollector.
Any aggregator can now throw a CollectionTerminatedException without stopping
the collection of a sibling aggregator. This is useful for aggregators that
can infer their result without visiting all documents (e.g.: a min/max aggregation on a match_all query).
2018-09-03 09:34:35 +02:00
Jason Tedor ea4eef8641
Merge branch 'master' into ccr
* master:
  HLREST: add update by query API (#32760)
2018-09-02 16:07:50 -04:00
Sohaib Iftikhar 389bf67275 HLREST: add update by query API (#32760)
Adds update by query to the high level rest client.
2018-09-02 15:15:00 -04:00
Nhat Nguyen 3197a6bbdd Merge branch 'master' into ccr
* master:
  HLRC: ML Flush job (#33187)
  HLRC: Adding ML Job stats (#33183)
  LLREST: Drop deprecated methods (#33223)
  Mute testSyncerOnClosingShard
  [DOCS] Moves machine learning APIs to docs folder (#31118)
2018-09-02 09:30:51 -04:00
Benjamin Trent 6770a456b8
HLRC: ML Flush job (#33187)
* HLRC: ML Flush job

* Fixing package, paths, and test

* Addressing comments
2018-09-01 16:01:23 -05:00
Benjamin Trent 19b14fa5ed
HLRC: Adding ML Job stats (#33183)
* HLRC: Adding pojos for get job stats

HLRC: Adding pojos for job stats request

* HLRC: Adding job stats pojos

* HLRC: ML job stats

* Minor syntax changes and adding license headers

* minor comment change

* Moving to client package, minor changes

* Addressing PR comments

* removing bad sleep

* addressing minor comment around test methods

* adding toplevel random fields for tests

* addressing minor review comments
2018-09-01 13:32:18 -05:00
Nik Everett f28cddf951
LLREST: Drop deprecated methods (#33223)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. In a
long series of PRs I've changed all of the old style requests. This
drops the deprecated methods and will be released with 7.0.
2018-09-01 11:11:25 -04:00
Lisa Cawley b7a63f7e7d
[DOCS] Moves machine learning APIs to docs folder (#31118) 2018-08-31 16:49:24 -07:00
Nhat Nguyen b93507608a Merge branch 'master' into ccr
* master:
  Mute test watcher usage stats output
  [Rollup] Fix FullClusterRestart test
  Adjust soft-deletes version after backport into 6.5
  completely drop `index.shard.check_on_startup: fix` for 7.0 (#33194)
  Fix AwaitsFix issue number
  Mute SmokeTestWatcherWithSecurityIT testsi
  drop `index.shard.check_on_startup: fix` (#32279)
  tracked at
  [DOCS] Moves ml folder from x-pack/docs to docs (#33248)
  [DOCS] Move rollup APIs to docs (#31450)
  [DOCS] Rename X-Pack Commands section (#33005)
  TEST: Disable soft-deletes in ParentChildTestCase
  Fixes SecurityIntegTestCase so it always adds at least one alias (#33296)
  Fix pom for build-tools (#33300)
  Lazy evaluate java9home (#33301)
  SQL: test coverage for JdbcResultSet (#32813)
  Work around to be able to generate eclipse projects (#33295)
  Highlight that index_phrases only works if no slop is used (#33303)
  Different handling for security specific errors in the CLI. Fix for https://github.com/elastic/elasticsearch/issues/33230 (#33255)
  [ML] Refactor delimited file structure detection (#33233)
  SQL: Support multi-index format as table identifier (#33278)
  MINOR: Remove Dead Code from PathTrie (#33280)
  Enable forbiddenapis server java9 (#33245)
2018-08-31 19:03:04 -04:00
Vladimir Dolzhenko 00b272af32 completely drop `index.shard.check_on_startup: fix` for 7.0 (#33194)
Relates to #32279
2018-08-31 22:08:28 +02:00
Vladimir Dolzhenko 3d82a30fad
drop `index.shard.check_on_startup: fix` (#32279)
drop `index.shard.check_on_startup: fix`

Relates #31389
2018-08-31 21:29:06 +02:00
Lisa Cawley 874ebcb6d4
[DOCS] Moves ml folder from x-pack/docs to docs (#33248) 2018-08-31 11:56:26 -07:00
Lisa Cawley cdeadfc585
[DOCS] Move rollup APIs to docs (#31450) 2018-08-31 10:50:43 -07:00
Lisa Cawley 08d0527e25
[DOCS] Rename X-Pack Commands section (#33005) 2018-08-31 10:11:58 -07:00
Colin Goodheart-Smithe 3eef74d5d5
Merge branch 'master' into index-lifecycle 2018-08-31 14:45:22 +01:00
Pablo Musa a88f8789a0
Highlight that index_phrases only works if no slop is used (#33303)
Highlight that `index_phrases` only works if no slop is used at query time.
2018-08-31 14:48:55 +02:00
Andrei Stefan 20e1b5e2fe
Different handling for security specific errors in the CLI. Fix for https://github.com/elastic/elasticsearch/issues/33230 (#33255) 2018-08-31 13:08:32 +03:00
Colin Goodheart-Smithe ada3e710f6
Renames XPackField.INDEX _LIFCYCLE value to "ilm" (#33270)
This brings the name in line with everywhere else and means that name
seen on the feature usage and `GET _xpack` APIs will match the plugin
name.

This change also removes `IndexLifcycle.NAME` since this was only used
to name the scheduler job and that can be done using
`XPackField.INDEX_LIFECYCLE` instead
2018-08-31 08:29:44 +01:00
Nhat Nguyen 5330067033 Merge branch 'master' into ccr
* master:
  Integrates soft-deletes into Elasticsearch (#33222)
  Revert "Integrates soft-deletes into Elasticsearch (#33222)"
  Add support for "authorization_realms" (#33262)
2018-08-31 00:07:21 -04:00
Tim Vernum 273c82d7c9
Add support for "authorization_realms" (#33262)
Authorization Realms allow an authenticating realm to delegate the task
of constructing a User object (with name, roles, etc) to one or more
other realms.

E.g. A client could authenticate using PKI, but then delegate to an LDAP
realm. The LDAP realm performs a "lookup" by principal, and then does
regular role-mapping from the discovered user.

This commit includes:
- authorization_realm support in the pki, ldap, saml & kerberos realms
- docs for authorization_realms
- checks that there are no "authorization chains"
   (whereby "realm-a" delegates to "realm-b", but "realm-b" delegates to "realm-c")

Authorization realms is a platinum feature.
2018-08-31 13:25:27 +10:00
Nhat Nguyen d3f32273eb Merge branch 'master' into ccr 2018-08-30 23:22:58 -04:00
Tal Levy 13a0d822d0 Merge branch 'master' into index-lifecycle 2018-08-30 15:04:17 -07:00
David Turner 51cbc61135 Fix docs build after #33241
Recently-merged PR #33241 broke the docs build, and this fixes it.
2018-08-30 09:38:23 +01:00
Albert Zaharovits c31c51dc80
[DOC] Repository GCS ADC not supported (#33238)
Make it clear that automatic default credentials (ADC)
is not supported for the repository-gcs plugin.
"Service Account" method is the only alternative
to authn requests to Google Cloud Storage.
2018-08-30 10:32:08 +03:00
David Turner 47859e56ac
Move file-based discovery to core (#33241)
Today we support a static list of seed hosts in core Elasticsearch, and allow a
dynamic list of seed hosts to be provided via a file using the `discovery-file`
plugin. In fact the ability to provide a dynamic list of seed hosts is
increasingly useful, so this change moves this functionality to core
Elasticsearch to avoid the need for a plugin.

Furthermore, in order to start up nodes in integration tests we currently
assign a known port to each node before startup, which unfortunately sometimes
fails if another process grabs the selected port in the meantime. By moving the
`discovery-file` functionality into the core product we can use it to avoid
this race.

This change also moves the expected path to the file from
`$ES_PATH_CONF/discovery-file/unicast_hosts.txt` to
`$ES_PATH_CONF/unicast_hosts.txt`. An example of this file is not included in
distributions.

For BWC purposes the plugin still exists, but does nothing more than create the
example file in the old location, and issue a warning when it is used. We also
continue to support the old location for the file, but warn about its
deprecation.

Relates #29244
Closes #33030
2018-08-30 06:43:04 +01:00
Nhat Nguyen 5632e31c74 Merge branch 'master' into ccr
* master:
  Painless: Add Bindings (#33042)
  Update version after client credentials backport
  Fix forbidden apis on FIPS (#33202)
  Remote 6.x transport BWC Layer for `_shrink` (#33236)
  Test fix - Graph HLRC tests needed another field adding to randomisation exception list
  HLRC: Add ML Get Records API (#33085)
  [ML] Fix character set finder bug with unencodable charsets (#33234)
  TESTS: Fix overly long lines (#33240)
  Test fix - Graph HLRC test was missing field name to be excluded from randomisation logic
  Remove unsupported group_shard_failures parameter (#33208)
  Update BucketUtils#suggestShardSideQueueSize signature (#33210)
  Parse PEM Key files leniantly (#33173)
  INGEST: Add Pipeline Processor (#32473)
  Core: Add java time xcontent serializers (#33120)
  Consider multi release jars when running third party audit (#33206)
  Update MSI documentation (#31950)
  HLRC: create base timed request class (#33216)
  [DOCS] Fixes command page titles
  HLRC: Move ML protocol classes into client ml package (#33203)
  Scroll queries asking for rescore are considered invalid (#32918)
  Painless: Fix Semicolon Regression (#33212)
  ingest: minor - update test to include dissect (#33211)
  Switch remaining LLREST usage to new style Requests (#33171)
  HLREST: add reindex API (#32679)
2018-08-29 12:30:24 -04:00
Gordon Brown 454ce99b01 Merge branch 'master' into index-lifecycle 2018-08-29 08:28:23 -06:00
Dimitris Athanasiou a5b34c75b0
HLRC: Add ML Get Records API (#33085)
Relates #29827
2018-08-29 15:03:58 +01:00
Stuart Cam 2dc4a5bb56 Update MSI documentation (#31950) 2018-08-29 15:37:47 +10:00
lcawl 1cbde721dc [DOCS] Fixes command page titles 2018-08-28 14:34:39 -07:00
Mark Tozzi 84b61d0738
Scroll queries asking for rescore are considered invalid (#32918)
This PR changes our behavior from silently ignoring rescore in a scroll query to instead report to the user that such a query is invalid.

Closes #31775
2018-08-28 15:48:23 -04:00
Nhat Nguyen c42dc77896 Merge branch 'master' into ccr
* master:
  [Rollup] Better error message when trying to set non-rollup index (#32965)
  HLRC: Use Optional in validation logic (#33104)
  Remove unused User class from protocol (#33137)
  ingest: Introduce the dissect processor (#32884)
  [Docs] Add link to es-kotlin-wrapper-client (#32618)
  [Docs] Remove repeating words (#33087)
  Minor spelling and grammar fix (#32931)
  Remove support for deprecated params._agg/_aggs for scripted metric aggregations (#32979)
  Watcher: Simplify finding next date in cron schedule (#33015)
  Run Third party audit with forbidden APIs CLI  (part3/3) (#33052)
  Fix plugin build test on Windows (#33078)
  HLRC+MINOR: Remove Unused Private Method (#33165)
  Remove old unused test script files (#32970)
  Build analysis-icu client JAR (#33184)
  Ensure to generate identical NoOp for the same failure (#33141)
  ShardSearchFailure#readFrom to set index and shardId (#33161)
2018-08-28 13:56:38 -04:00
Sohaib Iftikhar 7f5e29ddb2 HLREST: add reindex API (#32679)
Adds the reindex API to the high level REST client.
2018-08-28 13:02:23 -04:00
Jake Landis 79b507dbf5
ingest: Introduce the dissect processor (#32884)
* ingest: Introduce the dissect processor

The ingest node dissect processor is an alternative to Grok
to split a string based on a pattern. Dissect differs from
Grok such that regular expressions are not used to split the
string.

Dissect can be used to parse a source text field with a
simpler pattern, and is often faster the Grok for basic string
parsing. This processor uses the dissect library which
does most of the work.
2018-08-28 07:11:20 -07:00
Jilles van Gurp d7e4a49355 [Docs] Add link to es-kotlin-wrapper-client (#32618)
ES Kotlin Wrapper client is a library that wraps the official Highlevel Elasticsearch HTTP client for Java.
2018-08-28 14:25:55 +02:00
lipsill b7c0d2830a [Docs] Remove repeating words (#33087) 2018-08-28 13:16:43 +02:00
Dominik 525cda0331 Minor spelling and grammar fix (#32931) 2018-08-28 12:50:19 +02:00
Jonathan Little 9d92a87ae6 Remove support for deprecated params._agg/_aggs for scripted metric aggregations (#32979) 2018-08-28 09:27:43 +01:00