Commit Graph

5712 Commits

Author SHA1 Message Date
Tanguy Leroux e77835c6f5
Add create rollup job api to high level rest client (#33521)
This commit adds the Create Rollup Job API to the high level REST
client. It supersedes #32703 and adds dedicated request/response
objects so that it does not depend on server side components.

Related #29827
2018-09-17 09:10:23 +02:00
Ryan Ernst 3046656ab1
Scripting: Rework joda time backcompat (#33486)
This commit switches the joda time backcompat in scripting to use
augmentation over ZonedDateTime. The augmentation methods provide
compatibility with the missing methods between joda's DateTime and
java's ZonedDateTime. Due to getDayOfWeek returning an enum in the java
API, ZonedDateTime is wrapped so that the method can return int like the
joda time does. The java time api version is renamed to
getDayOfWeekEnum, which will be kept through 7.x for compatibility while
users switch back to getDayOfWeek once joda compatibility is removed.
2018-09-16 19:18:00 -07:00
Dimitris Athanasiou db40315afb
[HLRC][ML] Add ML get datafeed API to HLRC (#33715)
Relates #29827
2018-09-16 11:54:55 +01:00
Lisa Cawley 9706584836
[DOCS] Moves security reference to docs folder (#33643) 2018-09-14 13:09:47 -07:00
David Kyle b04faa059b
HLRC: ML PUT Calendar (#33362) 2018-09-14 15:00:18 +01:00
Jay Modi 3914a980f7
Security: remove wrapping in put user response (#33512)
This change removes the wrapping of the created field in the put user
response. The created field was added as a top level field in #32332,
while also still being wrapped within the `user` object of the
response. Since the value is available in both formats in 6.x, we can
remove the wrapped version for 7.0.
2018-09-13 14:40:36 -06:00
Costin Leau 32a22ca00e DOC: improved wording in SQL client app section 2018-09-13 22:07:23 +03:00
Lisa Cawley c3a817957d
[DOCS] Moves securing-communications to docs (#33640) 2018-09-13 10:42:26 -07:00
Dimitris Athanasiou 9600819cef
[HLRC][ML] Add ML delete datafeed API to HLRC (#33667)
Relates #29827
2018-09-13 17:13:36 +01:00
Costin Leau a192785fc8 DOC: Add SQL section on client applications
Add setup instructions for a number of GUI SQL applications
2018-09-13 15:44:52 +03:00
Jason Tedor c023f67c5d
Add migration note for remote cluster settings (#33632)
The remote cluster settings search.remote.* have been renamed to
cluster.remote.* and are automatically upgraded in the cluster state on
gateway recovery, and on put. This commit adds a note to the migration
docs for these changes.
2018-09-12 13:37:11 -04:00
Simon Willnauer c783488e97
Add `_source`-only snapshot repository (#32844)
This change adds a `_source` only snapshot repository that allows to wrap
any existing repository as a _backend_ to snapshot only the `_source` part
including live docs markers. Snapshots taken with the `source` repository
won't include any indices,  doc-values or points. The snapshot will be reduced in size and
functionality such that it requires full re-indexing after it's successfully restored.

The restore process will copy the `_source` data locally starts a special shard and engine
to allow `match_all` scrolls and searches. Any other query, or get call will fail with and unsupported operation exception.  The restored index is also marked as read-only.

This feature aims mainly for disaster recovery use-cases where snapshot size is
a concern or where time to restore is less of an issue.

**NOTE**: The snapshot produced by this repository is still a valid lucene index. This change doesn't allow for any longer retention policies which is out of scope for this change.
2018-09-12 17:47:10 +02:00
Christoph Büscher fe478c23b7
[Docs] Fix heading in composite-aggregation.asciidoc (#33627)
The heading for the "Missing buckets" should be on the same level 
as the the "Order" section.
2018-09-12 16:56:03 +02:00
Dimitris Athanasiou 2eb2313b60
[HLRC][ML] Add ML put datafeed API to HLRC (#33603)
This also changes both `DatafeedConfig` and `DatafeedUpdate`
to store the query and aggs as a bytes reference. This allows
the client to remove its dependency to the named objects
registry of the search module.

Relates #29827
2018-09-12 14:52:36 +01:00
Joel Green 0b567c0eeb [Docs] Update match-query.asciidoc (#33610) 2018-09-12 14:35:27 +02:00
Jim Ferenczi 4561c5ee83
Clarify context suggestions filtering and boosting (#33601)
This change clarifies the documentation of the context completion suggester
regarding filtering and boosting with contexts.
Unlike the suggester v1, filtering on multiple contexts
works as a disjunction, a suggestion matches if it contains at least one of the provided
context values and boosting selects the maximum score among the matching contexts.
This commit also adapts an old test that was written for the v1 suggester and commented out
for version 2 because the behavior changed.
2018-09-12 08:47:32 +02:00
Armin Braun 94cdf0ceba
NETWORKING: http.publish_host Should Contain CNAME (#32806)
* NETWORKING: http.publish_host Should Contain CNAME

* Closes #22029
2018-09-12 06:15:36 +02:00
Benjamin Trent 27e07ec859
HLRC: ML Delete Forecast API (#33526)
* HLRC: ML Delete Forecast API
2018-09-11 16:32:52 -07:00
Lisa Cawley cbc6fa0ecb
[DOCS] Adds missing built-in user information (#33585) 2018-09-11 07:56:26 -07:00
Ed Savage 2f3b542d57
HLRC: Add ML get categories API (#33465)
HLRC: Adding the ML 'get categories' API
2018-09-11 12:48:14 +01:00
Alan Woodward f598297f55
Add predicate_token_filter (#33431)
This allows users to filter out tokens from a TokenStream using painless scripts, 
instead of having to write specialised Java code and packaging it up into a plugin.

The commit also refactors the AnalysisPredicateScript.Token class so that it wraps
and makes read-only an AttributeSource.
2018-09-11 09:16:39 +01:00
Tanguy Leroux 079d130d8c
[Test] Remove duplicate method in TestShardRouting (#32815) 2018-09-10 18:29:00 +02:00
S.Y. Wang 9073dbefd6 HLRC: Add put stored script support to high-level rest client (#31323)
Relates to #27205
2018-09-09 13:47:47 +02:00
lcawl 6b780e9926 [DOCS] Fixing formatting issues in breaking changes 2018-09-07 16:53:36 -07:00
Jay Modi 9d16a7b7f0
HLRC: add enable and disable user API support (#33481)
This change adds support for enable and disable user APIs to the high
level rest client. There is a common request base class for both
requests with specific requests that simplify the use of these APIs.

The response for these APIs is simply an empty object so a new response
class has been created for cases where we expect an empty response to
be returned.

Finally, the put user documentation has been moved to the proper
location that is not within an x-pack sub directory and the document
tags no longer contain x-pack.

See #29827
2018-09-07 11:51:37 -06:00
lcawl 944868908c [DOCS] Fixes formatting error 2018-09-07 10:26:44 -07:00
Benjamin Trent 4d233107f8
HLRC: ML Forecast Job (#33506)
* HLRC: ML Forecast job
2018-09-07 11:16:58 -05:00
Benjamin Trent 9230a48722
HLRC: ML Post Data (#33443)
* HLRC: ML Post data
2018-09-07 07:04:27 -05:00
Jim Ferenczi 79cd6385fe
Collapse package structure for metrics aggs (#33463)
This change collapses all metrics aggregations classes into a single package `org.elasticsearch.aggregations.metrics`.
It also restricts the visibility of some classes (aggregators and factories) that should not be used outside of the package.

Relates #22868
2018-09-07 10:58:06 +02:00
Lisa Cawley 7441c0376e
[DOCS] Adds delete forecast API (#33401) 2018-09-06 09:20:42 -07:00
Costin Leau 443f9caddd DOC: Enhance SQL Functions documentation
Split function section into multiple chapters
Add String functions
Add (small) section on Conversion/Cast functions
Add missing aggregation functions
Enable documentation testing (was disabled by accident). While at it,
fix failing tests
Improve spec tests to allow multi-line queries (useful for docs)
Add ability to ignore a spec test (name should end with -Ignore)
2018-09-06 18:09:53 +03:00
Jim Ferenczi 7ad71f906a
Upgrade to a Lucene 8 snapshot (#33310)
The main benefit of the upgrade for users is the search optimization for top scored documents when the total hit count is not needed. However this optimization is not activated in this change, there is another issue opened to discuss how it should be integrated smoothly.
Some comments about the change:
* Tests that can produce negative scores have been adapted but we need to forbid them completely: #33309

Closes #32899
2018-09-06 14:42:06 +02:00
Benjamin Trent 9b6bbc0182
HLRC: ML Update Job (#33392)
* HLRC: ML Update Job
2018-09-06 07:18:09 -05:00
Jason Tedor d71ced1b00
Generalize search.remote settings to cluster.remote (#33413)
With features like CCR building on the CCS infrastructure, the settings
prefix search.remote makes less sense as the namespace for these remote
cluster settings than does a more general namespace like
cluster.remote. This commit replaces these settings with cluster.remote
with a fallback to the deprecated settings search.remote.
2018-09-05 20:43:44 -04:00
Jay Modi ea52277a1e
HLRest: add put user API (#32332)
This commit adds a security client to the high level rest client, which
includes an implementation for the put user api. As part of these
changes, a new request and response class have been added that are
specific to the high level rest client. One change here is that the response
was previously wrapped inside a user object. The plan is to remove this
wrapping and this PR adds an unwrapped response outside of the user
object so we can remove the user object later on.

See #29827
2018-09-05 10:56:30 -06:00
Christoph Büscher eafc2a5470
Don't count metadata fields towards index.mapping.total_fields.limit (#33386)
The maximum number of fields per index is limited to 1000 by default by the
`index.mapping.total_fields.limit` setting to prevent accidental mapping
explosions due to too many fields. Currently all metadata fields also count
towards this limit, which can lead to some confusion when using lower limits.
It is not obvious for users that they cannot actually add as many fields as
are specified by the limit in this case.

This change takes the number of metadata fields out of the field count that we
check against the field limit. It also adds tests that check that we can add
fields up to the specified limit, but throw an exception for any additional field added.

Closes #24096
2018-09-05 18:27:21 +02:00
Dimitris Athanasiou 24776b2b80
HLRC: Add ML get influencers API (#33389)
Relates #29827
2018-09-05 15:06:55 +01:00
Alan Woodward 636442700c
Add conditional token filter to elasticsearch (#31958)
This allows tokenfilters to be applied selectively, depending on the status of the current token in the tokenstream.  The filter takes a scripted predicate, and only applies its subfilter when the predicate returns true.
2018-09-05 14:52:43 +01:00
Paul Sanwald c303006e6b
Add interval response parameter to AutoDateInterval histogram (#33254)
Adds the interval used to the aggregation response.
2018-09-05 07:35:59 -04:00
Gordon Brown cfd3fa72ed
Add user-defined cluster metadata (#33325)
Adds a place for users to store cluster-wide data they wish to associate
with the cluster via the Cluster Settings API. This is strictly for
user-defined data, Elasticsearch makes no other other use of these
settings.
2018-09-04 16:14:18 -06:00
Lisa Cawley f3f8d9b833
[DOCS] Moves monitoring pages to docs folder (#33324) 2018-09-04 10:02:13 -07:00
lcawl c5109a54ee [DOCS] Revert fix for broken link 2018-09-04 09:26:28 -07:00
Costin Leau 43f80fa82b DOCS: Fix anchor and example typos 2018-09-04 19:06:44 +03:00
lcawl 303ae25a6a [DOCS] Fixes broken link 2018-09-04 09:05:30 -07:00
Costin Leau 17c7f99343
SQL: Show/desc commands now support table ids (#33363)
Extend SHOW TABLES, DESCRIBE and SHOW COLUMNS to support table
identifiers not just SQL LIKE pattern.
This allows both Elasticsearch-style multi-index patterns and SQL LIKE.
To disambiguate between the two (as the " vs ' can be easy to miss),
the grammar now requires LIKE keyword as a prefix for all LIKE-like
patterns.

Also added some docs comparing the two types of patterns.

Fix #33294
2018-09-04 16:54:10 +03:00
Sohaib Iftikhar 761e8c461f HLRC: Add delete by query API (#32782)
Adds the delete-by-query API to the High Level REST Client.
2018-09-04 08:56:26 -04:00
Dimitris Athanasiou 1457b07a06
[ML] The sort field on get records should default to the record_score (#33358)
This is not changing the behaviour as when the sort field was set
to `influencer_score` the secondary sort would be used and that
was using the `record_score` at the highest priority.
2018-09-04 11:38:24 +01:00
Nikolay Vasiliev d9f394b099 [DOCS] fix a couple of typos (#33356) 2018-09-04 10:07:11 +02:00
Christoph Büscher 79db16f9bb
[Docs] Add search timeout caveats (#33354)
Global search timeouts and timeouts specified in the search request body use the
same internal mechanism as search cancellation. Therefore the same caveats
apply, mostly around the responsiveness of the timeout which gets only checked
by a running search on segment boundaries by default.

Closes #31263
2018-09-03 20:56:05 +02:00
Dimitris Athanasiou 54fe7fb5a1
HLRC: Add ML get overall buckets API (#33297)
Relates #29827
2018-09-03 13:34:36 +01:00
Christoph Büscher 978d1ed257
[Docs] Improve tuning for speed advice (#33315)
This change merges two sections in the "Tune for search speed" documentation
that recommend mapping numeric identifiers as keywords. Both sections contain
mostly the same advice, so they can be merged.

Closes #32733
2018-09-03 11:09:30 +02:00
Jim Ferenczi 713c07e14d
Add early termination support to BucketCollector (#33279)
This commit adds the support to early terminate the collection of a leaf
in the aggregation framework. This change introduces a MultiBucketCollector which
handles CollectionTerminatedException exactly like the Lucene MultiCollector.
Any aggregator can now throw a CollectionTerminatedException without stopping
the collection of a sibling aggregator. This is useful for aggregators that
can infer their result without visiting all documents (e.g.: a min/max aggregation on a match_all query).
2018-09-03 09:34:35 +02:00
Jason Tedor ea4eef8641
Merge branch 'master' into ccr
* master:
  HLREST: add update by query API (#32760)
2018-09-02 16:07:50 -04:00
Sohaib Iftikhar 389bf67275 HLREST: add update by query API (#32760)
Adds update by query to the high level rest client.
2018-09-02 15:15:00 -04:00
Nhat Nguyen 3197a6bbdd Merge branch 'master' into ccr
* master:
  HLRC: ML Flush job (#33187)
  HLRC: Adding ML Job stats (#33183)
  LLREST: Drop deprecated methods (#33223)
  Mute testSyncerOnClosingShard
  [DOCS] Moves machine learning APIs to docs folder (#31118)
2018-09-02 09:30:51 -04:00
Benjamin Trent 6770a456b8
HLRC: ML Flush job (#33187)
* HLRC: ML Flush job

* Fixing package, paths, and test

* Addressing comments
2018-09-01 16:01:23 -05:00
Benjamin Trent 19b14fa5ed
HLRC: Adding ML Job stats (#33183)
* HLRC: Adding pojos for get job stats

HLRC: Adding pojos for job stats request

* HLRC: Adding job stats pojos

* HLRC: ML job stats

* Minor syntax changes and adding license headers

* minor comment change

* Moving to client package, minor changes

* Addressing PR comments

* removing bad sleep

* addressing minor comment around test methods

* adding toplevel random fields for tests

* addressing minor review comments
2018-09-01 13:32:18 -05:00
Nik Everett f28cddf951
LLREST: Drop deprecated methods (#33223)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. In a
long series of PRs I've changed all of the old style requests. This
drops the deprecated methods and will be released with 7.0.
2018-09-01 11:11:25 -04:00
Lisa Cawley b7a63f7e7d
[DOCS] Moves machine learning APIs to docs folder (#31118) 2018-08-31 16:49:24 -07:00
Nhat Nguyen b93507608a Merge branch 'master' into ccr
* master:
  Mute test watcher usage stats output
  [Rollup] Fix FullClusterRestart test
  Adjust soft-deletes version after backport into 6.5
  completely drop `index.shard.check_on_startup: fix` for 7.0 (#33194)
  Fix AwaitsFix issue number
  Mute SmokeTestWatcherWithSecurityIT testsi
  drop `index.shard.check_on_startup: fix` (#32279)
  tracked at
  [DOCS] Moves ml folder from x-pack/docs to docs (#33248)
  [DOCS] Move rollup APIs to docs (#31450)
  [DOCS] Rename X-Pack Commands section (#33005)
  TEST: Disable soft-deletes in ParentChildTestCase
  Fixes SecurityIntegTestCase so it always adds at least one alias (#33296)
  Fix pom for build-tools (#33300)
  Lazy evaluate java9home (#33301)
  SQL: test coverage for JdbcResultSet (#32813)
  Work around to be able to generate eclipse projects (#33295)
  Highlight that index_phrases only works if no slop is used (#33303)
  Different handling for security specific errors in the CLI. Fix for https://github.com/elastic/elasticsearch/issues/33230 (#33255)
  [ML] Refactor delimited file structure detection (#33233)
  SQL: Support multi-index format as table identifier (#33278)
  MINOR: Remove Dead Code from PathTrie (#33280)
  Enable forbiddenapis server java9 (#33245)
2018-08-31 19:03:04 -04:00
Vladimir Dolzhenko 00b272af32 completely drop `index.shard.check_on_startup: fix` for 7.0 (#33194)
Relates to #32279
2018-08-31 22:08:28 +02:00
Vladimir Dolzhenko 3d82a30fad
drop `index.shard.check_on_startup: fix` (#32279)
drop `index.shard.check_on_startup: fix`

Relates #31389
2018-08-31 21:29:06 +02:00
Lisa Cawley 874ebcb6d4
[DOCS] Moves ml folder from x-pack/docs to docs (#33248) 2018-08-31 11:56:26 -07:00
Lisa Cawley cdeadfc585
[DOCS] Move rollup APIs to docs (#31450) 2018-08-31 10:50:43 -07:00
Lisa Cawley 08d0527e25
[DOCS] Rename X-Pack Commands section (#33005) 2018-08-31 10:11:58 -07:00
Pablo Musa a88f8789a0
Highlight that index_phrases only works if no slop is used (#33303)
Highlight that `index_phrases` only works if no slop is used at query time.
2018-08-31 14:48:55 +02:00
Andrei Stefan 20e1b5e2fe
Different handling for security specific errors in the CLI. Fix for https://github.com/elastic/elasticsearch/issues/33230 (#33255) 2018-08-31 13:08:32 +03:00
Nhat Nguyen 5330067033 Merge branch 'master' into ccr
* master:
  Integrates soft-deletes into Elasticsearch (#33222)
  Revert "Integrates soft-deletes into Elasticsearch (#33222)"
  Add support for "authorization_realms" (#33262)
2018-08-31 00:07:21 -04:00
Tim Vernum 273c82d7c9
Add support for "authorization_realms" (#33262)
Authorization Realms allow an authenticating realm to delegate the task
of constructing a User object (with name, roles, etc) to one or more
other realms.

E.g. A client could authenticate using PKI, but then delegate to an LDAP
realm. The LDAP realm performs a "lookup" by principal, and then does
regular role-mapping from the discovered user.

This commit includes:
- authorization_realm support in the pki, ldap, saml & kerberos realms
- docs for authorization_realms
- checks that there are no "authorization chains"
   (whereby "realm-a" delegates to "realm-b", but "realm-b" delegates to "realm-c")

Authorization realms is a platinum feature.
2018-08-31 13:25:27 +10:00
Nhat Nguyen d3f32273eb Merge branch 'master' into ccr 2018-08-30 23:22:58 -04:00
David Turner 51cbc61135 Fix docs build after #33241
Recently-merged PR #33241 broke the docs build, and this fixes it.
2018-08-30 09:38:23 +01:00
Albert Zaharovits c31c51dc80
[DOC] Repository GCS ADC not supported (#33238)
Make it clear that automatic default credentials (ADC)
is not supported for the repository-gcs plugin.
"Service Account" method is the only alternative
to authn requests to Google Cloud Storage.
2018-08-30 10:32:08 +03:00
David Turner 47859e56ac
Move file-based discovery to core (#33241)
Today we support a static list of seed hosts in core Elasticsearch, and allow a
dynamic list of seed hosts to be provided via a file using the `discovery-file`
plugin. In fact the ability to provide a dynamic list of seed hosts is
increasingly useful, so this change moves this functionality to core
Elasticsearch to avoid the need for a plugin.

Furthermore, in order to start up nodes in integration tests we currently
assign a known port to each node before startup, which unfortunately sometimes
fails if another process grabs the selected port in the meantime. By moving the
`discovery-file` functionality into the core product we can use it to avoid
this race.

This change also moves the expected path to the file from
`$ES_PATH_CONF/discovery-file/unicast_hosts.txt` to
`$ES_PATH_CONF/unicast_hosts.txt`. An example of this file is not included in
distributions.

For BWC purposes the plugin still exists, but does nothing more than create the
example file in the old location, and issue a warning when it is used. We also
continue to support the old location for the file, but warn about its
deprecation.

Relates #29244
Closes #33030
2018-08-30 06:43:04 +01:00
Nhat Nguyen 5632e31c74 Merge branch 'master' into ccr
* master:
  Painless: Add Bindings (#33042)
  Update version after client credentials backport
  Fix forbidden apis on FIPS (#33202)
  Remote 6.x transport BWC Layer for `_shrink` (#33236)
  Test fix - Graph HLRC tests needed another field adding to randomisation exception list
  HLRC: Add ML Get Records API (#33085)
  [ML] Fix character set finder bug with unencodable charsets (#33234)
  TESTS: Fix overly long lines (#33240)
  Test fix - Graph HLRC test was missing field name to be excluded from randomisation logic
  Remove unsupported group_shard_failures parameter (#33208)
  Update BucketUtils#suggestShardSideQueueSize signature (#33210)
  Parse PEM Key files leniantly (#33173)
  INGEST: Add Pipeline Processor (#32473)
  Core: Add java time xcontent serializers (#33120)
  Consider multi release jars when running third party audit (#33206)
  Update MSI documentation (#31950)
  HLRC: create base timed request class (#33216)
  [DOCS] Fixes command page titles
  HLRC: Move ML protocol classes into client ml package (#33203)
  Scroll queries asking for rescore are considered invalid (#32918)
  Painless: Fix Semicolon Regression (#33212)
  ingest: minor - update test to include dissect (#33211)
  Switch remaining LLREST usage to new style Requests (#33171)
  HLREST: add reindex API (#32679)
2018-08-29 12:30:24 -04:00
Dimitris Athanasiou a5b34c75b0
HLRC: Add ML Get Records API (#33085)
Relates #29827
2018-08-29 15:03:58 +01:00
Stuart Cam 2dc4a5bb56 Update MSI documentation (#31950) 2018-08-29 15:37:47 +10:00
lcawl 1cbde721dc [DOCS] Fixes command page titles 2018-08-28 14:34:39 -07:00
Mark Tozzi 84b61d0738
Scroll queries asking for rescore are considered invalid (#32918)
This PR changes our behavior from silently ignoring rescore in a scroll query to instead report to the user that such a query is invalid.

Closes #31775
2018-08-28 15:48:23 -04:00
Nhat Nguyen c42dc77896 Merge branch 'master' into ccr
* master:
  [Rollup] Better error message when trying to set non-rollup index (#32965)
  HLRC: Use Optional in validation logic (#33104)
  Remove unused User class from protocol (#33137)
  ingest: Introduce the dissect processor (#32884)
  [Docs] Add link to es-kotlin-wrapper-client (#32618)
  [Docs] Remove repeating words (#33087)
  Minor spelling and grammar fix (#32931)
  Remove support for deprecated params._agg/_aggs for scripted metric aggregations (#32979)
  Watcher: Simplify finding next date in cron schedule (#33015)
  Run Third party audit with forbidden APIs CLI  (part3/3) (#33052)
  Fix plugin build test on Windows (#33078)
  HLRC+MINOR: Remove Unused Private Method (#33165)
  Remove old unused test script files (#32970)
  Build analysis-icu client JAR (#33184)
  Ensure to generate identical NoOp for the same failure (#33141)
  ShardSearchFailure#readFrom to set index and shardId (#33161)
2018-08-28 13:56:38 -04:00
Sohaib Iftikhar 7f5e29ddb2 HLREST: add reindex API (#32679)
Adds the reindex API to the high level REST client.
2018-08-28 13:02:23 -04:00
Jake Landis 79b507dbf5
ingest: Introduce the dissect processor (#32884)
* ingest: Introduce the dissect processor

The ingest node dissect processor is an alternative to Grok
to split a string based on a pattern. Dissect differs from
Grok such that regular expressions are not used to split the
string.

Dissect can be used to parse a source text field with a
simpler pattern, and is often faster the Grok for basic string
parsing. This processor uses the dissect library which
does most of the work.
2018-08-28 07:11:20 -07:00
Jilles van Gurp d7e4a49355 [Docs] Add link to es-kotlin-wrapper-client (#32618)
ES Kotlin Wrapper client is a library that wraps the official Highlevel Elasticsearch HTTP client for Java.
2018-08-28 14:25:55 +02:00
lipsill b7c0d2830a [Docs] Remove repeating words (#33087) 2018-08-28 13:16:43 +02:00
Dominik 525cda0331 Minor spelling and grammar fix (#32931) 2018-08-28 12:50:19 +02:00
Jonathan Little 9d92a87ae6 Remove support for deprecated params._agg/_aggs for scripted metric aggregations (#32979) 2018-08-28 09:27:43 +01:00
Jonathan Little 29ba143e2c Remove old unused test script files (#32970) 2018-08-27 16:29:50 -07:00
Nhat Nguyen 5b11df9c35
DOCS: Make ellipsis optional in /cat/thread_pool (#33186)
The fix proposed in #31442 fails with the oss distro because the added
3dots does not match anything with the default oss while a 3dots
expression requires matching at least one thread pool.

This change makes an ellipsis optional so the thread_pool list can match
both the oss distro (without ccr) and default distro (with ccr).

Relates #31442
2018-08-27 18:33:57 -04:00
Jason Tedor 0e5d42ca38
Merge branch 'master' into ccr
* master:
  Adjust BWC version on mapping version
  Token API supports the client_credentials grant (#33106)
  Build: forked compiler max memory matches jvmArgs (#33138)
  Introduce mapping version to index metadata (#33147)
  SQL: Enable aggregations to create a separate bucket for missing values (#32832)
  Fix grammar in contributing docs
  SECURITY: Fix Compile Error in ReservedRealmTests (#33166)
  APM server monitoring (#32515)
  Support only string `format` in date, root object & date range (#28117)
  [Rollup] Move toBuilders() methods out of rollup config objects (#32585)
  Fix forbiddenapis on java 11  (#33116)
  Apply publishing to genreate pom (#33094)
  Have circuit breaker succeed on unknown mem usage
  Do not lose default mapper on metadata updates (#33153)
  Fix a mappings update test (#33146)
  Reload Secure Settings REST specs & docs (#32990)
  Refactor CachingUsernamePassword realm (#32646)
2018-08-27 13:49:59 -04:00
Shaunak Kashyap 1779d3376a
APM server monitoring (#32515)
* Adding new MonitoredSystem for APM server

* Teaching Monitoring template utils about APM server monitoring indices

* Documenting new monitoring index for APM server

* Adding monitoring index template for APM server

* Copy pasta typo

* Removing metrics.libbeat.config section from mapping

* Adding built-in user and role for APM server user

* Actually define the role :)

* Adding missing import

* Removing index template and system ID for apm server

* Shortening line lengths

* Updating expected number of built-in users in integration test

* Removing "system" from role and user names

* Rearranging users to make tests pass
2018-08-27 08:42:40 -04:00
Albert Zaharovits fbe609d589
Reload Secure Settings REST specs & docs (#32990)
This is a minimal REST API spec and docs for the REST handler
for the `_nodes/reload_secure_settings endpoint`.

Relates #29135
2018-08-26 14:49:32 +03:00
Nhat Nguyen 75304f405b Merge branch 'master' into ccr
* master:
  Add proxy support to RemoteClusterConnection (#33062)
  TEST: Skip assertSeqNos for closed shards (#33130)
  TEST: resync operation on replica should acquire shard permit (#33103)
  Switch remaining x-pack tests to new style Requests (#33108)
  Switch remaining tests to new style Requests (#33109)
  Switch remaining ml tests to new style Requests (#33107)
  Build: Line up IDE detection logic
  Security index expands to a single replica (#33131)
  HLRC: request/response homogeneity and JavaDoc improvements (#33133)
  Checkstyle!
  [Test] Fix sporadic failure in MembershipActionTests
  Revert "Do NOT allow termvectors on nested fields (#32728)"
  [Rollup] Move toAggCap() methods out of rollup config objects (#32583)
  Fix race condition in scheduler engine test
2018-08-25 21:41:53 -04:00
Mayya Sharipova 6f1ee76443 Revert "Do NOT allow termvectors on nested fields (#32728)"
This reverts commit fdff8f3db0.
2018-08-24 10:12:16 -04:00
Martijn van Groningen 82592dda5a
Merge remote-tracking branch 'es/master' into ccr
* es/master: (62 commits)
  [DOCS] Add docs for Application Privileges (#32635)
  Add versions 5.6.12 and 6.4.1
  Do NOT allow termvectors on nested fields (#32728)
  [Rollup] Return empty response when aggs are missing (#32796)
  [TEST] Add some ACL yaml tests for Rollup (#33035)
  Move non duplicated actions back into xpack core (#32952)
  Test fix - GraphExploreResponseTests should not randomise array elements Closes #33086
  Use `addIfAbsent` instead of checking if an element is contained
  TESTS: Fix Random Fail in MockTcpTransportTests (#33061)
  HLRC: Fix Compile Error From Missing Throws (#33083)
  [DOCS] Remove reload password from docs cf. #32889
  HLRC: Add ML Get Buckets API (#33056)
  Watcher: Improve error messages for CronEvalTool (#32800)
  Search: Support of wildcard on docvalue_fields (#32980)
  Change query field expansion (#33020)
  INGEST: Cleanup Redundant Put Method (#33034)
  SQL: skip uppercasing/lowercasing function tests for AZ locales as well (#32910)
  Fix the default pom file name (#33063)
  Switch ml basic tests to new style Requests (#32483)
  Switch some watcher tests to new style Requests (#33044)
  ...
2018-08-24 12:22:11 +07:00
Tim Vernum a211d24bda [DOCS] Add docs for Application Privileges (#32635) 2018-08-23 18:04:02 -07:00
Mayya Sharipova fdff8f3db0
Do NOT allow termvectors on nested fields (#32728)
Requesting _termvectors on a nested field or any sub-fields of a nested field
returns empty results.

Closes #21625
2018-08-23 16:46:47 -04:00
Albert Zaharovits 07cce95305 [DOCS] Remove reload password from docs cf. #32889
Reload call `_nodes/reload_secure_settings` is not requiring
an empty password anymore (#32889). Reflect this in docs.
2018-08-23 12:11:39 +03:00
Dimitris Athanasiou 50441f97ae
HLRC: Add ML Get Buckets API (#33056)
Relates #29827
2018-08-23 09:35:06 +01:00
Ignacio Vera d7219c05a2
Search: Support of wildcard on docvalue_fields (#32980)
* Search: Support of wildcard on docvalue_fields

For consistency with stored_fields, docvalue_fields should support the use of wildcards. 
Documentation of doc values fields is updated accordingly.

See also: #26390

Closes #26299
2018-08-23 10:04:00 +02:00
Luca Cavanna 393eec1482
Set maxScore for empty TopDocs to Nan rather than 0 (#32938)
We used to set `maxScore` to `0` within `TopDocs` in situations where there is really no score as the size was set to `0` and scores were not even tracked. In such scenarios, `Float.Nan` is more appropriate, which gets converted to `max_score: null` on the REST layer. That's also more consistent with lucene which set `maxScore` to `Float.Nan` when merging empty `TopDocs` (see `TopDocs#merge`).
2018-08-22 17:23:54 +02:00
Dimitrios Liappis abb4c183f1
Clarify ignore_above behavior with arrays of strings
Currently docs don't explain how `ignore_above` behaves with arrays of
strings.

Clarify how `ignore_above` applies for arrays of strings and
also note that all string(s) will still be visible in the
`_source` field.

Relates #33057
2018-08-22 18:18:30 +03:00
Sergey 43f6f435f5 [DOCS] Update remote-info.asciidoc (#32978) 2018-08-22 11:29:10 +02:00
Simon Willnauer ffb1a5d5b7
Expose `max_concurrent_shard_requests` in `_msearch` (#33016)
Today `_msearch` doesn't allow modifying the `max_concurrent_shard_requests`
per sub search request. This change adds support for setting this parameter on
all sub-search requests in an `_msearch`.

Relates to #31877
2018-08-22 08:45:08 +02:00
Ioannis Kakavas 0a4b55c9c0
[DOCS] Add RequestedAuthnContext Documentation (#32946)
Add documentation for #31238

- Add documentation for the req_authn_context_class_ref setting
- Add a section in SAML Guide regarding the use of SAML
  Authentication Context.
2018-08-22 08:37:50 +03:00
Benjamin Trent e2ea83d217
HLRC: Add ML Get Job (#32960)
* HLRC: Adding GET ML Job info API

* HLRC: Adding GET Job ML API

* Fixing QueryPage license header

* Adding serialization tests, addressing minor issues

* Renaming querypage, changing the dependency on it

* Making things immutable

* Fixing build failure due to method rename
2018-08-21 21:02:28 -05:00
Russ Cam 07b3ff9fe7
Add beta label to MSI on install Elasticsearch page (#28126)
The main installation instructions page for the Windows MSI installer includes a header at the top to indicate that the installer is in beta, but the Installing Elasticsearch page does not. This commit adds the beta label to the MSI entry within the installation options.
2018-08-22 11:26:53 +10:00
debadair 15727ae8ed
[DOCS] Fixed formatting of Example headings. (#33038) 2018-08-21 13:13:07 -07:00
Jason Tedor bdfcc326d7
Enable avoiding mmap bootstrap check (#32421)
The maximum map count boostrap check can be a hindrance to users that do
not own the underlying platform on which they are executing
Elasticsearch. This is because addressing it requires tuning the kernel
and a platform provider might now allow this, especially on shared
infrastructure. However, this bootstrap check is not needed if mmapfs is
not in use. Today we do not have a way for the user to communicate that
they are not going to use mmapfs. This commit therefore adds a setting
that enables the user to disallow mmapfs. When mmapfs is disallowed, the
maximum map count bootstrap check is not enforced. Additionally, we
fallback to a different default index store and prevent the explicit use
of mmapfs for an index.
2018-08-21 11:02:25 -04:00
Ioannis Kakavas 1b583978e9
[DOCS] Add FIPS 140-2 documentation (#32928)
* Add relevant documentation for FIPS 140-2 compliance.
* Introduce `fips_mode` setting.
* Discuss necessary configuration for FIPS 140-2
* Discuss introduced limitations by FIPS 140-2
2018-08-21 16:20:00 +03:00
markharwood 38bdf9ce32
HLRC GraphClient and associated tests (#32366)
GraphClient for the high level REST client and associated tests.
Part of #29827 work
2018-08-21 13:29:18 +01:00
Ioannis Kakavas 65d4f27873
[DOCS] Add configurable password hashing docs (#32849)
* [DOCS] Add configurable password hashing docs

Adds documentation about the newly introduced configuration option
for setting the password hashing algorithm to be used for the users
cache and for storing credentials for the native and file realm.
2018-08-21 12:05:42 +03:00
Lisa Cawley 2feda8aae0
[DOC] Splits role mapping APIs into separate pages (#32797) 2018-08-20 14:30:42 -07:00
Benjamin Trent 3fbaae10af
HLRC: ML Close Job (#32943)
* HLRC: Adding ML Close Job API

HLRC: Adding ML Close Job API

* reconciling request converters

* Adding serialization tests and addressing PR comments

* Changing constructor order
2018-08-20 16:05:56 -05:00
Jason Tedor 853eb1c51c
Merge branch 'master' into ccr
* master:
  Generalize remote license checker (#32971)
  Trim translog when safe commit advanced (#32967)
  Fix an inaccuracy in the dynamic templates documentation. (#32890)
  Logging: Use settings when building daemon threads (#32751)
  All Translog inner closes should happen after tragedy exception is set (#32674)
  HLREST: AwaitsFix ML Test
  Pass DiscoveryNode to initiateChannel (#32958)
  Add mzn and dz to unsupported locales (#32957)
  Use settings from the context in BootstrapChecks (#32908)
  Update docs for node specifications (#30468)
  HLRC: Forbid all Elasticsearch logging infra (#32784)
  Only configure publishing if it's applied externally (#32351)
  Fixes libs:dissect when in eclipse
  Protect ScriptedMetricIT test cases against failures on 0-doc shards (#32959) (#32968)
  [Kerberos] Add documentation for Kerberos realm (#32662)
  Watcher: Properly find next valid date in cron expressions (#32734)
  Fix some small issues in the getting started docs (#30346)
  Set forbidden APIs target compatibility to compiler java version   (#32935)
  Move connection listener to ConnectionManager (#32956)
2018-08-20 15:49:31 -04:00
Julie Tibshirani 815c56b677
Fix an inaccuracy in the dynamic templates documentation. (#32890) 2018-08-20 11:00:11 -07:00
Yu a883e7dffc Update docs for node specifications (#30468)
Expands and clarifies exactly what is and isn't allowed when specifying a
subset of the nodes as targets of a cluster API, and adds missing links to this
from the hot threads and cluster stats API docs.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Yu <yu.liu003@gmail.com>
2018-08-20 14:21:31 +01:00
Yogesh Gaikwad e143cce865
[Kerberos] Add documentation for Kerberos realm (#32662)
This commit adds documentation for configuring Kerberos realm.
Configuring Kerberos realm documentation highlights important
terminology and requirements before creating Kerberos realm.
Most of the documentation is centered around configuration from
Elasticsearch rather than go deep into Kerberos implementation.
Kerberos realm settings are mentioned in the security settings
for Kerberos realm.
2018-08-20 17:23:14 +10:00
Tim Ryan dce72c7985 Fix some small issues in the getting started docs (#30346)
* Modified a reference to real time to match the previous line reference of
  realtime.
* Modified eg to e.g. as it's an abbreviation for the latin exempli gratia
* Added missing pronoun to `_executing_filters` section.
2018-08-20 07:54:03 +01:00
Jason Tedor ac75968c0b
Merge remote-tracking branch 'elastic/master' into ccr
* elastic/master: (46 commits)
  NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS (#32764)
  [DOCS] Splits the users API documentation into multiple pages (#32825)
  [DOCS] Splits the token APIs into separate pages (#32865)
  [DOCS] Creates redirects for role management APIs page
  Bypassing failing test PainlessDomainSplitIT#testHRDSplit (#32966)
  TEST: Mute testRetentionPolicyChangeDuringRecovery
  [DOCS] Fixes more broken links to role management APIs
  [Docs] Tweaks and fixes to rollup docs
  [DOCS] Fixes links to role management APIs
  [ML][TEST] Fix BasicRenormalizationIT after adding multibucket feature
  [DOCS] Splits the roles API documentation into multiple pages (#32794)
  [TEST]  Run pre 6.4 nodes in non-FIPS JVMs (#32901)
  Make Geo Context Mapping Parsing More Strict (#32821)
  [ML] fix updating opened jobs scheduled events (#31651) (#32881)
  Scripted metric aggregations: add deprecation warning and system property to control legacy params (#31597)
  Tests: Fix timezone conversion in DateTimeUnitTests
  Enable FIPS140LicenseBootstrapCheck (#32903)
  Fix InternalAutoDateHistogram reproducible failure (#32723)
  Remove assertion in testDocStats on deletedDocs counter (#32914)
  HLRC: Move ML request converters into their own class (#32906)
  ...
2018-08-18 09:48:55 -04:00
Lisa Cawley 532d552ffd
[DOCS] Splits the users API documentation into multiple pages (#32825) 2018-08-17 23:17:33 -07:00
Lisa Cawley fb1c3990d7
[DOCS] Splits the token APIs into separate pages (#32865) 2018-08-17 22:22:09 -07:00
lcawl 1efee66d16 [DOCS] Creates redirects for role management APIs page 2018-08-17 21:55:18 -07:00
Igor Motov da6b61e8ef
Make Geo Context Mapping Parsing More Strict (#32821)
Currently, if geo context is represented by something other than
geo_point or an object with lat and lon fields, the parsing of it
as a geo context can result in ignoring the context altogether,
returning confusing errors such as number_format_exception or trying
to parse the number specifying as long-encoded hash code. It would also
fail if the geo_point was stored.

This commit makes the mapping parsing more strict and will fail during
mapping update or index creation if the geo context doesn't point to
a geo_point field.

Supersedes #32412

Closes #32202
2018-08-17 08:13:16 -07:00
Jonathan Little a08127c072 Scripted metric aggregations: add deprecation warning and system property to control legacy params (#31597)
* Scripted metric aggregations: add deprecation warning and system property to control legacy params

Scripted metric aggregation params._agg/_aggs are replaced by state/states context variables. By default the old params are still present, and a deprecation warning is emitted when Scripted Metric Aggregations are used. A new system property can be used to disable the legacy params. This functionality will be removed in a future revision.

* Fix minor style issue and docs test failure

* Disable deprecated params._agg/_aggs in tests and revise tests to use state/states instead

* Add integration test covering deprecated scripted metrics aggs params._agg/_aggs access

* Disable deprecated params._agg/_aggs in docs integration tests and revise stored scripts to use state/states instead

* Revert unnecessary migrations doc change

A relevant note should be added in the changes destined for 7.0; this PR is going to be backported to 6.x.

* Replace deprecated _agg param bwc integration test with a couple of unit tests

* Fix compatibility test after merge

* Rename backwards compatibility system property per code review feedback

* Tweak deprecation warning text per review feedback
2018-08-17 13:11:18 +01:00
markwalkom ae38cfbaec [DOCS] Update getting-started.asciidoc (#29518)
Highlighted that you can change shard counts using `_shrink` and `_split`.
2018-08-17 11:09:22 +02:00
JeffSaxeVA efdad7d5fc [DOCS] Add "remove a tag" script logic as an example (#32556)
It took me quite a while of online searching and experimenting to realize the function-call asymmetry in the Add versus Remove from a list, like the "tags" list! I realize we cannot give examples for every single thing the user wants to do in Painless, but this is such a common use case (removing a tag from a single doc, or from a set of docs with Update-By-Query) that I believe it ought to be demonstrated immediately after the "add a tag" example. We have an example of removing an entire document field, but not removing one element of a list (a multi-valued field).

Also, a minor grammar fix: I have added an apostrophe to the word "its" in the accompanying text of the example just above.
2018-08-17 10:56:06 +02:00
Jason Tedor b5a85362e3
Fix docs for fixed filename for heap dump path (#32882)
The docs here incorrectly state that it is okay for a heap dump file to
exist when heap dump path is configured to a fixed filename. This is
incorrect, the JVM will fail to write the heap dump if a heap dump file
already exists at the specified location (see the DumpWriter constructor
DumpWriter::DumpWriter(const char* path) in the JVM source).
2018-08-16 11:32:35 -04:00
Benjamin Trent aedc2c1c49
HLRC: adding machine learning delete job (#32820)
* HLRC: adding machine learning delete job

* Fixing whitespace

* Moving docs and tests around

* Unifying ml asciidoc file naming convention
2018-08-16 07:18:43 -05:00
Hazem Khaled b87f3062b7 [DOCS] Update WordPress plugins links (#32194) 2018-08-16 13:55:40 +02:00
datosh e6bfba1d79 [DOCS] Clarify sentence in network-host.asciidoc (#32429) 2018-08-16 11:38:19 +02:00
markharwood 70d80a3d09
Docs enhancement: added reference to cluster-level setting `search.default_allow_partial_results` (#32810)
Closes #32809
2018-08-16 10:21:37 +01:00
Jason Tedor 4475f88c95
Merge branch 'master' into ccr
* master:
  Fix global checkpoint listeners test
  HLRC: adding machine learning open job (#32860)
  [ML] Add log structure finder functionality (#32788)
  INGEST: Add Configuration Except. Data to Metdata (#32322)
2018-08-15 16:07:28 -04:00
Benjamin Trent f90f06ede1
HLRC: adding machine learning open job (#32860)
* HLRC: adding machine learning open job

HLRC: adding ML open job API call

* Addressing @droberts195 comments and fixing minor style issue
2018-08-15 13:19:41 -05:00
Jason Tedor aa147cca44
Merge remote-tracking branch 'elastic/master' into ccr
* elastic/master:
  Revert "cluster formation DSL - Gradle integration -  part 2 (#32028)" (#32876)
  cluster formation DSL - Gradle integration -  part 2 (#32028)
  Introduce global checkpoint listeners (#32696)
  Move connection profile into connection manager (#32858)
  [ML] Temporarily disabling rolling-upgrade tests
  Use generic AcknowledgedResponse instead of extended classes (#32859)
  [ML] Removing old per-partition normalization code (#32816)
  Use JDK 10 for 6.4 BWC builds (#32866)
  Removed flaky test. Looks like randomisation makes these assertions unreliable.
  [test] mute IndexShardTests.testDocStats
  Introduce the dissect library (#32297)
  Security: remove password hash bootstrap check (#32440)
  Move validation to server for put user requests (#32471)
  [ML] Add high level REST client docs for ML put job endpoint (#32843)
  Test: Fix forbidden uses in test framework (#32824)
  Painless: Change fqn_only to no_import (#32817)
  [test] mute testSearchWithSignificantTermsAgg
  Watcher: Remove unused hipchat render method (#32211)
  Watcher: Remove extraneous auth classes (#32300)
  Watcher: migrate PagerDuty v1 events API to v2 API (#32285)
2018-08-15 12:30:35 -04:00
David Roberts c985f500f4
[ML] Add high level REST client docs for ML put job endpoint (#32843)
Relates #29827
Relates #32726
2018-08-14 19:52:54 +01:00
Nhat Nguyen 6556186d9a Merge branch 'master' into ccr 2018-08-14 12:11:35 -04:00
Yannick Welsch 00b006f033
HLRC: Add Delete License API (#32586)
Relates to #29827
2018-08-14 11:55:54 +02:00
Morten Segelvik 4d20e69b83 [DOCS] XContentBuilder#bytes method removed, using BytesReference.bytes(docBuilder) (#32771) 2018-08-13 11:29:18 +02:00
Luca Cavanna 3e692c3f3d
HLRC: migration get assistance API (#32744)
The request and response classes have been extracted from `IndexUpgradeInfoAction` into top-level classes, and moved to the protocol jar. The `UpgradeActionRequired` enum is also moved.

Relates to #29827
2018-08-13 11:05:27 +02:00
Nik Everett f5ba801c6b Test: Only sniff host metadata for node_selectors (#32750)
Our rest testing framework has support for sniffing the host metadata on
startup and, before this change, it'd sniff that metadata before running
the first test. This prevents running these tests against
elasticsearch installations that won't support sniffing like Elastic
Cloud. This change allows tests to only sniff for metadata when they
encounter a test with a `node_selector`. These selectors are the things
that need the metadata anyway and they are super rare. Tests that use
these won't be able to run against installations that don't support
sniffing but we can just skip them. In the case of Elastic Cloud, these
tests were never going to work against Elastic Cloud anyway.
2018-08-10 13:35:47 -04:00
Jonathan Little 8a2b1a7dca Update scripted metric docs to use `state` variable (#32695)
This change brings the scripted metric agg context docs in line with the new agg state context variable.
2018-08-10 10:21:54 -07:00
Luca Cavanna 40b0a3a014
[DOCS] Adjust high-level client x-pack docs (#32747)
- Expose x-pack usage docs page which was not linked in supported-apis page
- make watcher a top-level dir outside of x-pack directory
- move x-pack info and usage pages to miscellaneous
- add new Watcher category to supported-apis (they were under miscellaneous)
- remove x-pack prefix from watcher docs titles
2018-08-10 14:12:42 +02:00
Luca Cavanna 9561a0a66e
[DOCS] Add missing link to get license API docs (#32746) 2018-08-10 10:55:25 +02:00
Jack Conradson 293c8a2b24
Painless: Add an Ingest Script Processor Example (#32302)
This commit adds two pieces. The first is a small set of documentation providing 
instructions on how to get setup to run context examples. This will require a download 
similar to how Kibana works for some of the examples. The second is an ingest processor 
example using the downloaded data. More examples will follow as ideally one per PR. 
This also adds a set of tests to individually test each script as a unit test.
2018-08-09 14:24:55 -07:00
Armin Braun 79375d35bb
Scripting: Replace Update Context (#32096)
* SCRIPTING: Move Update Scripts to their own context
* Added system property for backwards compatibility of change to `ctx.params`
2018-08-09 14:32:36 +02:00
Andy Bristol 8bfb0f3f8d
serialize suggestion responses as named writeables (#30284)
Suggestion responses were previously serialized as streamables which
made writing suggesters in plugins with custom suggestion response types
impossible. This commit makes them serialized as named writeables and
provides a facility for registering a reader for suggestion responses
when registering a suggester.

This also makes Suggestion responses abstract, requiring a suggester
implementation to provide its own types. Suggesters which do not need
anything additional to what is defined in Suggest.Suggestion should
provide a minimal subclass.

The existing plugin suggester integration tests are removed and
replaced with an equivalent implementation as an example
plugin.
2018-08-07 13:31:00 -07:00
David Roberts 733f84c49b [DOCS] Add missing docs include 2018-08-07 17:49:48 +01:00
David Roberts 2608012422
Add temporary directory cleanup workarounds (#32615)
On some Linux distributions tmpfiles.d cleans files and
directories under /tmp if they haven't been accessed for
10 days.

This can cause problems for ML as ML is currently the only
component that uses the temp directory more than a few
seconds after startup. If you didn't open an ML job for
10 days and then tried to open one then the temp directory
would have been deleted.

This commit prevents the problem occurring in the case of
Elasticsearch being managed by systemd, as systemd private
temp directories are not subject to periodic cleanup (by
default).

Additionally there are now some docs to warn people about
the risk and suggest a manual mitigation for .tar.gz users.
2018-08-07 16:59:56 +01:00
simonzheng ab81078949 [Docs] Correct a small typo (#32655) 2018-08-07 10:34:55 +02:00
DeDe Morton e01e4393a8
[Docs] Light edit to info about docker images (#32376) 2018-08-06 12:00:07 -07:00
Nhat Nguyen 5881322b3f Merge branch 'master' into ccr
* master:
  Cross-cluster search: preserve cluster alias in shard failures (#32608)
  Handle AlreadyClosedException when bumping primary term
  [TEST] Allow to run in FIPS JVM (#32607)
  [Test] Add ckb to the list of unsupported languages (#32611)
  SCRIPTING: Move Aggregation Scripts to their own context (#32068)
  Painless: Use LocalMethod Map For Lookup at Runtime (#32599)
  [TEST] Enhance failure message when bulk updates have failures
  [ML] Add ML result classes to protocol library (#32587)
  Suppress LicensingDocumentationIT.testPutLicense in release builds (#32613)
  [Rollup] Update wire version check after backport
  Suppress Wildfly test in FIPS JVMs (#32543)
  [Rollup] Improve ID scheme for rollup documents (#32558)
  ingest: doc: move Dot Expander Processor doc to correct position (#31743)
  [ML] Add some ML config classes to protocol library (#32502)
  [TEST]Split transport verification mode none tests (#32488)
  Core: Move helper date formatters over to java time (#32504)
  [Rollup] Remove builders from DateHistogramGroupConfig (#32555)
  [TEST} unmutes SearchAsyncActionTests and adds debugging info
  [ML] Add Detector config classes to protocol library (#32495)
  [Rollup] Remove builders from MetricConfig (#32536)
  Tests: Add rolling upgrade tests for watcher (#32428)
  Fix race between replica reset and primary promotion (#32442)
2018-08-06 10:27:18 -04:00
Igor Motov e641fccfe3
Rest HL client: Add get license action (#32438)
Rest HL client: Add get license action

Continues to use String instead of a more complex License class to
hold the license text similarly to put license.

Relates #29827
2018-08-06 07:15:40 -07:00
Armin Braun 0a67cb4133
LOGGING: Upgrade to Log4J 2.11.1 (#32616)
* LOGGING: Upgrade to Log4J 2.11.1
* Upgrade to `2.11.1` to fix memory leaks in slow logger when logging large requests
   * This was caused by a bug in Log4J https://issues.apache.org/jira/browse/LOG4J2-2269 and is fixed in `2.11.1` via https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=9496c0c
* Fixes #32537
* Fixes #27300
2018-08-06 14:56:21 +02:00
Jake Landis 3d4c84f7ca
ingest: doc: move Dot Expander Processor doc to correct position (#31743)
No changes to the content.
2018-08-03 07:21:05 -07:00
Nhat Nguyen 6eeb628d6d Merge branch 'master' into ccr
* master:
  HLRC: Move commercial clients from XPackClient (#32596)
  Add cluster UUID to Cluster Stats API response (#32206)
  Security: move User to protocol project (#32367)
  [TEST] Test for shard failures, add debug to testProfileMatchesRegular
  Minor fix for javadoc (applicable for java 11). (#32573)
  Painless: Move Some Lookup Logic to PainlessLookup (#32565)
  TEST: Avoid merges in testSeqNoAndCheckpoints
  [Rollup] Remove builders from HistoGroupConfig (#32533)
  Mutes failing SQL string function tests due to #32589
  fixed elements in array of produced terms (#32519)
  INGEST: Enable default pipelines (#32286)
  Remove cluster state initial customs (#32501)
  Mutes LicensingDocumentationIT due to #32580
  [ML] Remove multiple_bucket_spans (#32496)
  [ML] Rename JobProvider to JobResultsProvider (#32551)
  Correct minor typo in explain.asciidoc for HLRC
  Build: Add elastic maven to repos used by BuildPlugin (#32549)
  Clarify the error message when a pipeline agg is used in the 'order' parameter. (#32522)
  Revert "[test] turn on host io cache for opensuse (#32053)"
  Enable packaging tests on suse boxes
  [ML] Improve error when no available field exists for rule scope (#32550)
  [ML] Improve error for functions with limited rule condition support (#32548)
  Painless: Clean Up PainlessField (#32525)
  Add @AwaitsFix for #32554
  Remove broken @link in Javadoc
  Scripting: Conditionally use java time api in scripting (#31441)
  [ML] Fix thread leak when waiting for job flush (#32196) (#32541)
  Add AwaitsFix to failing test - see #32546
  Core: Minor size reduction for AbstractComponent (#32509)
  SQL: Added support for string manipulating functions with more than one parameter (#32356)
  [DOCS] Reloadable Secure Settings (#31713)
  Watcher: Reenable HttpSecretsIntegrationTests#testWebhookAction test (#32456)
  [Rollup] Remove builders from TermsGroupConfig (#32507)
  Use hostname instead of IP with SPNEGO test (#32514)
  Switch x-pack rolling restart to new style Requests (#32339)
  NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
  [DOCS] Small fixes in rule configuration page (#32516)
  Painless: Clean up PainlessMethod (#32476)
  Build: Remove shadowing from benchmarks (#32475)
  Docs: Add all JDKs to CONTRIBUTING.md
  Add licensing enforcement for FIPS mode (#32437)
  SQL: Add test for handling of partial results (#32474)
  Mute testFilterCacheStats
  [ML][DOCS] Fix typo applied_to => applies_to
  Scripting: Fix painless compiler loader to know about context classes (#32385)
2018-08-02 23:14:37 -04:00
Shaunak Kashyap 0a83968650
Add cluster UUID to Cluster Stats API response (#32206)
* Make cluster stats response contain cluster UUID

* Updating constructor usage in Monitoring tests

* Adding cluster_uuid field to Cluster Stats API reference doc

* Adding rest api spec test for expecting cluster_uuid in cluster stats response

* Adding missing newline

* Indenting do section properly

* Missed a spot!

* Fixing the test cluster ID
2018-08-02 17:14:19 -07:00
Matthias Sieber a39f6f09f4 fixed elements in array of produced terms (#32519) 2018-08-02 11:12:15 -04:00
Mayya Sharipova 43430fe9b8
Correct minor typo in explain.asciidoc for HLRC 2018-08-01 17:58:14 -04:00
Ryan Ernst 478f6d6cf1
Scripting: Conditionally use java time api in scripting (#31441)
This commit adds a boolean system property, `es.scripting.use_java_time`,
which controls the concrete return type used by doc values within
scripts. The return type of accessing doc values for a date field is
changed to Object, essentially duck typing the type to allow
co-existence during the transition from joda time to java time.
2018-08-01 08:58:49 -07:00
Albert Zaharovits 2d87287c0d
[DOCS] Reloadable Secure Settings (#31713)
Docs on reloadable secure settings for plugins #29135 .
2018-08-01 12:07:23 +03:00
Nhat Nguyen 036cb3f864 Merge branch 'master' into ccr
* master:
  Logging: Make node name consistent in logger (#31588)
  Mute SSLTrustRestrictionsTests on JDK 11
  Increase max chunk size to 256Mb for repo-azure (#32101)
  Docs: Fix README upgrade mention (#32313)
  Changed ReindexRequest to use Writeable.Reader (#32401)
  Mute KerberosAuthenticationIT
  Fix AutoIntervalDateHistogram.testReduce random failures (#32301)
  fix no=>not typo (#32463)
  Mute QueryProfilerIT#testProfileMatchesRegular()
  HLRC: Add delete watch action (#32337)
  High-level client: fix clusterAlias parsing in SearchHit (#32465)
  Fix calculation of orientation of polygons (#27967)
  [Kerberos] Add missing javadocs (#32469)
  [Kerberos] Remove Kerberos bootstrap checks (#32451)
  Make get all app privs requires "*" permission (#32460)
  Switch security to new style Requests (#32290)
  Switch security spi example to new style Requests (#32341)
  Painless: Add PainlessConstructor (#32447)
  update rollover to leverage write-alias semantics (#32216)
  Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819)
  INGEST: Clean up Java8 Stream Usage (#32059)
  Ensure KeyStoreWrapper decryption exceptions are handled (#32464)
2018-07-31 10:56:10 -04:00
Nik Everett 22459576d7
Logging: Make node name consistent in logger (#31588)
First, some background: we have 15 different methods to get a logger in
Elasticsearch but they can be broken down into three broad categories
based on what information is provided when building the logger.

Just a class like:
```
private static final Logger logger = ESLoggerFactory.getLogger(ActionModule.class);
```
or:
```
protected final Logger logger = Loggers.getLogger(getClass());
```

The class and settings:
```
this.logger = Loggers.getLogger(getClass(), settings);
```

Or more information like:
```
Loggers.getLogger("index.store.deletes", settings, shardId)
```

The goal of the "class and settings" variant is to attach the node name
to the logger. Because we don't always have the settings available, we
often use the "just a class" variant and get loggers without node names
attached. There isn't any real consistency here. Some loggers get the
node name because it is convenient and some do not.

This change makes the node name available to all loggers all the time.
Almost. There are some caveats are testing that I'll get to. But in
*production* code the node name is node available to all loggers. This
means we can stop using the "class and settings" variants to fetch
loggers which was the real goal here, but a pleasant side effect is that
the ndoe name is now consitent on every log line and optional by editing
the logging pattern. This is all powered by setting the node name
statically on a logging formatter very early in initialization.

Now to tests: tests can't set the node name statically because
subclasses of `ESIntegTestCase` run many nodes in the same jvm, even in
the same class loader. Also, lots of tests don't run with a real node so
they don't *have* a node name at all. To support multiple nodes in the
same JVM tests suss out the node name from the thread name which works
surprisingly well and easy to test in a nice way. For those threads
that are not part of an `ESIntegTestCase` node we stick whatever useful
information we can get form the thread name in the place of the node
name. This allows us to keep the logger format consistent.
2018-07-31 10:54:24 -04:00
Colm O'Shea 97b379e0d4 fix no=>not typo (#32463)
Found a tiny typo while reading the docs
2018-07-31 13:33:23 +01:00
Yannick Welsch 11f6983d69
HLRC: Add delete watch action (#32337)
Adds the "delete watch" API to the High-Level Rest Client.

Relates #29827
2018-07-31 10:29:22 +02:00
Tal Levy 1e0fcebfe1
update rollover to leverage write-alias semantics (#32216)
Rollover should not swap aliases when `is_write_index` is set to `true`.
Instead, both the new and old indices should have the rollover alias,
with the newly created index as the new write index

Updates Rollover to leverage the ability to preserve aliases and swap which is the write index.

Historically, Rollover would swap which index had the designated alias for writing documents against. This required users to keep a separate read-alias that enabled reading against both rolled over and newly created indices, whiles the write-alias was being re-assigned at every rollover.

With the ability for aliases to designate a write index, Rollover can be a bit more flexible with its use of aliases.

Updates include:

- Rollover validates that the target alias has a write index (the index that is being rolled over). This means that the restriction that aliases only point to one index is no longer necessary.
- Rollover explicitly (and atomically) swaps which index is the write-index by explicitly assigning the existing index to have `is_write_index: false` and have the newly created index have its rollover alias as `is_write_index: true`. This is only done when `is_write_index: true` on the write index. Default behavior of removing the alias from the rolled over index stays when `is_write_index` is not explicitly set

Relevant things that are staying the same:

- Rollover is rejected if there exist any templates that match the newly-created index and configure the rollover-alias
   - I think this existed to prevent the situation where an alias pointed to two indices for a short while. Although this can technically be relaxed, the specific cases that are safe are really particular and difficult to reason, so leaving the broad restriction sounds good
2018-07-30 14:32:55 -07:00
w-bonelli 072c0be8af Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819)
Stating that the Fuzzy Query generates "all possible" matching terms is misleading, given that the query's default behavior is to generate a maximum of 50 matching terms.

(cherry picked from commit 345a0071a2a41fd7f80ae9ef8a39a2cb4991aedd)
2018-07-30 13:19:26 -07:00
Nhat Nguyen 2f756b00f6 Merge branch 'master' into ccr
* master:
  Remove reference to non-existent store type (#32418)
  [TEST] Mute failing FlushIT test
  Fix ordering of bootstrap checks in docs (#32417)
  [TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints
  [TEST] Mute failing testConvertLongHexError
  bump lucene version after backport
  Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
  [Kerberos] Avoid vagrant update on precommit (#32416)
  TESTS: Move netty leak detection to paranoid level (#32354)
  [DOCS] Fixes formatting of scope object in job resource
  Copy missing segment attributes in getSegmentInfo (#32396)
  AbstractQueryTestCase should run without type less often (#28936)
  INGEST: Fix Deprecation Warning in Script Proc. (#32407)
  Switch x-pack/plugin to new style Requests (#32327)
  Docs: Correcting a typo in tophits (#32359)
  Build: Stop double generating buildSrc pom (#32408)
  TEST: Avoid triggering merges in FlushIT
  Fix missing JavaDoc for @throws in several places in KerberosTicketValidator.
  Switch x-pack full restart to new style Requests (#32294)
  Release requests in cors handler (#32364)
  Painless: Clean Up PainlessClass Variables (#32380)
  Docs: Fix callouts in put license HL REST docs (#32363)
  [ML] Consistent pattern for strict/lenient parser names (#32399)
  Update update-settings.asciidoc (#31378)
  Remove some dead code (#31993)
  Introduce index store plugins (#32375)
  Rank-Eval: Reduce scope of an unchecked supression
  Make sure _forcemerge respects `max_num_segments`. (#32291)
  TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests (#32377)
  Only enforce password hashing check if FIPS enabled (#32383)
2018-07-27 16:24:03 -04:00
Jason Tedor 588db621ac
Remove reference to non-existent store type (#32418)
We removed the default_fs store type yet the docs still contain a
reference to them. This commit addresses that by removing this
reference, and changing a reference to this section of the docs to
instead refer to mmapfs.
2018-07-27 11:24:03 -04:00
Jason Tedor 3ac57f0ba3
Fix ordering of bootstrap checks in docs (#32417)
In the section of the bootstrap checks docs for the maximum map count
check, we refer to max size virtual memory check and explicitly call out
the maximum size virtual memory check as being the previous
point. However, this is not correct as the previous point is currently
the max file size check. It does make sense for these two checks to be
proximate to each other in the docs so this commit reorders the checks
so that the maximum size virtual memory check indeed comes before the
maximum map count check. This makes the sense in the maximum map count
check correct.
2018-07-27 10:40:16 -04:00
Jim Ferenczi 53ff06e621
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
2018-07-27 08:28:51 +02:00
Sandeep Kanabar 7ad16ffd84 Docs: Correcting a typo in tophits (#32359) 2018-07-26 13:30:01 -04:00
Igor Motov 3b050194f9
Docs: Fix callouts in put license HL REST docs (#32363) 2018-07-26 09:00:35 -07:00
mzbeck 63a0436764 Update update-settings.asciidoc (#31378) 2018-07-26 17:16:53 +02:00
Nhat Nguyen caefed4d33 Merge branch 'master' into ccr
* master:
  [DOCS] Fix formatting error in Slack action
  Painless: Fix documentation links to use existing refs (#32335)
  Painless: Decouple PainlessLookupBuilder and Whitelists (#32346)
  [DOCS] Adds recommendation for xpack.security.enabled (#32345)
  [TEST] Mute ConvertProcessortTests.testConvertIntHexError
  [TEST] Fix failure due to exception message in java11 (#32321)
  [DOCS] Fixes typo in ML aggregations page
  [DOCS] Adds link from bucket_span property to common time units
  [ML][DOCS] Add documentation for detector rules and filters (#32013)
  Add opaque_id to index audit logging (#32260)
  Add 6.5.0 version to master
  fixes broken build for third-party-tests (#32353)
2018-07-25 20:12:13 -04:00
Jack Conradson 10bfedeb53
Painless: Fix documentation links to use existing refs (#32335)
Uses ref and xpack-ref instead of custom ones in the Painless docs for
long-term maintainability.
2018-07-25 10:59:25 -07:00
Lisa Cawley 8067250fbe
[DOCS] Adds recommendation for xpack.security.enabled (#32345) 2018-07-25 09:51:40 -07:00
Nhat Nguyen ab4deefbe0 Merge branch 'master' into ccr
* master:
  Security: revert to old way of merging automata (#32254)
  Networking: Fix test leaking buffer (#32296)
  Undo a debugging change that snuck in during the field aliases merge.
  Painless: Update More Methods to New Naming Scheme (#32305)
  [TEST] Fix assumeFalse -> assumeTrue in SSLReloadIntegTests
  Ingest: Support integer and long hex values in convert (#32213)
  Introduce fips_mode setting and associated checks (#32326)
  Add V_6_3_3 version constant
  [DOCS] Removed extraneous callout number.
  Rest HL client: Add put license action (#32214)
  Add ERR to ranking evaluation documentation (#32314)
  Introduce Application Privileges with support for Kibana RBAC (#32309)
  Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
  [Kerberos] Add Kerberos authentication support (#32263)
  [ML] Extract persistent task methods from MlMetadata (#32319)
  Add Restore Snapshot High Level REST API
  Register ERR metric with NamedXContentRegistry (#32320)
  fixes broken build for third-party-tests (#32315)
  Allow Integ Tests to run in a FIPS-140 JVM (#31989)
  [DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280)
  awaitsfix testRandomClusterStateUpdates
  [TEST] add version skip to weighted_avg tests
  Consistent encoder names (#29492)
  Add WeightedAvg metric aggregation (#31037)
  Switch monitoring to new style Requests (#32255)
  Rename ranking evaluation `quality_level` to `metric_score` (#32168)
  Fix a test bug around nested aggregations and field aliases. (#32287)
  Add new permission for JDK11 to load JAAS libraries (#32132)
  Silence SSL reload test that fails on JDK 11
  [test] package pre-install java check (#32259)
  specify subdirs of lib, bin, modules in package (#32253)
  Switch x-pack:core to new style Requests (#32252)
  awaitsfix SSLConfigurationReloaderTests
  Painless: Clean up add methods in PainlessLookup (#32258)
  Fail shard if IndexShard#storeStats runs into an IOException (#32241)
  AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated
  Remove unnecessary warning supressions (#32250)
  CCE when re-throwing "shard not available" exception in TransportShardMultiGetAction (#32185)
  Add new fields to monitoring template for Beats state (#32085)
2018-07-24 21:33:50 -04:00
debadair 32eb4d41b6
[DOCS] Removed extraneous callout number. 2018-07-24 11:51:52 -07:00
Igor Motov 29c802f88e
Rest HL client: Add put license action (#32214)
In the HL REST client we replace the License object with a string, because of 
complexity of this class. It is also not really needed on the client side since 
end-users are not interacting with the license besides passing it as a string 
to the server.

Relates #29827
2018-07-24 11:19:07 -07:00
Christoph Büscher c1cc0cef61
Add ERR to ranking evaluation documentation (#32314)
This change adds a section about the Expected Reciprocal Rank metric (ERR) to
the Ranking Evaluation documentation.
2018-07-24 19:58:34 +02:00
Daniel Mitterdorfer 73a38895fd
Add Restore Snapshot High Level REST API
With this commit we add the restore snapshot API to the Java high level
REST client.

Relates #27205
Relates #32155
2018-07-24 16:17:09 +02:00
Russ Cam e2b665c2e6
Consistent encoder names (#29492)
This commit updates encoder names to be consistent within documentation
and align with snake casing convention.
2018-07-24 09:21:43 +10:00
Zachary Tong 6ba144ae31
Add WeightedAvg metric aggregation (#31037)
Adds a new single-value metrics aggregation that computes the weighted 
average of numeric values that are extracted from the aggregated 
documents. These values can be extracted from specific numeric
fields in the documents.

When calculating a regular average, each datapoint has an equal "weight"; it
contributes equally to the final value.  In contrast, weighted averages
scale each datapoint differently.  The amount that each datapoint contributes 
to the final value is extracted from the document, or provided by a script.

As a formula, a weighted average is the `∑(value * weight) / ∑(weight)`

A regular average can be thought of as a weighted average where every value has
an implicit weight of `1`.

Closes #15731
2018-07-23 18:33:15 -04:00
Christoph Büscher fe6bb75eb4
Rename ranking evaluation `quality_level` to `metric_score` (#32168)
The notion of "quality" is an overloaded term in the search ranking evaluation 
context. Its usually used to decribe certain levels of "good" vs. "bad" of a 
seach result with respect to the users information need. We currently report the 
result of the ranking evaluation as `quality_level` which is a bit missleading.
This changes the response parameter name to `metric_score` which fits better.
2018-07-23 22:25:02 +02:00
Martijn van Groningen 8ed3624f47
Merge remote-tracking branch 'es/master' into ccr
* es/master: (23 commits)
  Switch full-cluster-restart to new style Requests (#32140)
  [DOCS] Clarified that you must remove X-Pack plugin when upgrading from pre-6.3. (#32016)
  Remove BouncyCastle dependency from runtime (#32193)
  INGEST: Extend KV Processor (#31789) (#32232)
  INGEST: Make a few Processors callable by Painless (#32170)
  Add region ISO code to GeoIP Ingest plugin (#31669)
  [Tests] Remove QueryStringQueryBuilderTests#toQuery class assertions (#32236)
  Make sure that field aliases count towards the total fields limit. (#32222)
  Switch rolling restart to new style Requests (#32147)
  muting failing test for internal auto date histogram to avoid failure before fix is merged
  MINOR: Remove unused `IndexDynamicSettings` (#32237)
  Fix multi level nested sort (#32204)
  Enhance Parent circuit breaker error message (#32056)
  [ML] Use default request durability for .ml-state index (#32233)
  Remove indices stats timeout from monitoring docs
  Rename ranking evaluation response section (#32166)
  Dependencies: Upgrade to joda time 2.10 (#32160)
  Remove aliases resolution limitations when security is enabled (#31952)
  Ensure that field aliases cannot be used in multi-fields. (#32219)
  TESTS: Check for Netty resource leaks (#31861)
  ...
2018-07-21 09:06:13 +02:00
debadair 5525cb1615
[DOCS] Clarified that you must remove X-Pack plugin when upgrading from pre-6.3. (#32016) 2018-07-20 14:17:48 -07:00
Armin Braun 7aa8a0a927
INGEST: Extend KV Processor (#31789) (#32232)
* INGEST: Extend KV Processor (#31789)

Added more capabilities supported by LS to the KV processor:
* Stripping of brackets and quotes from values (`include_brackets` in corresponding LS filter)
* Adding key prefixes
* Trimming specified chars from keys and values

Refactored the way the filter is configured to avoid conditionals during execution.
Refactored Tests a little to not have to add more redundant getters for new parameters.

Relates #31786
* Add documentation
2018-07-20 22:32:50 +02:00
Nick Peihl ac63408655
Add region ISO code to GeoIP Ingest plugin (#31669) 2018-07-20 11:23:29 -07:00
Daniel Mitterdorfer 6afb661ca5
Remove indices stats timeout from monitoring docs
With this commit we remove the documentation for the setting
`xpack.monitoring.collection.indices.stats.timeout` which has already
been removed in code.

Closes #32133
Relates #32229
2018-07-20 12:55:20 +02:00
Christoph Büscher 5cbd9ad177
Rename ranking evaluation response section (#32166)
Currently the ranking evaluation response contains a 'unknown_docs' section 
for each search use case in the evaluation set. It contains document ids for 
results in the search hits that currently don't have a quality rating.
This change renames it to `unrated_docs`, which better reflects its purpose.
2018-07-20 11:43:46 +02:00
Luca Cavanna 00a6ad0e9e
Remove aliases resolution limitations when security is enabled (#31952)
Resolving wildcards in aliases expression is challenging as we may end
up with no aliases to replace the original expression with, but if we
replace with an empty array that means _all which is quite the opposite.
Now that we support and serialize the original requested aliases,
whenever aliases are replaced we will be able to know what was
initially requested. `MetaData#findAliases` can then be updated to not
return anything in case it gets empty aliases, but the original aliases
were not empty. That means that empty aliases are interpreted as _all
only if they were originally requested that way.

Relates to #31516
2018-07-20 09:23:32 +02:00
Julie Tibshirani 0f0068b91c
Ensure that field aliases cannot be used in multi-fields. (#32219) 2018-07-20 00:18:54 -07:00
Nhat Nguyen d0f3ed5abd Merge branch 'master' into ccr
* master:
  Painless: Simplify Naming in Lookup Package (#32177)
  Handle missing values in painless (#32207)
  add support for write index resolution when creating/updating documents (#31520)
  ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
  Remove indication of future multi-homing support (#32187)
  Rest test - allow for snapshots to take 0 milliseconds
  Make x-pack-core generate a pom file
  Rest HL client: Add put watch action (#32026)
  Build: Remove pom generation for plugin zip files (#32180)
  Fix comments causing errors with Java 11
  Fix rollup on date fields that don't support epoch_millis (#31890)
  Detect and prevent configuration that triggers a Gradle bug (#31912)
  [test] port linux package packaging tests (#31943)
  Revert "Introduce a Hashing Processor (#31087)" (#32178)
  Remove empty @return from JavaDoc
  Adjust SSLDriver behavior for JDK11 changes (#32145)
  [test] use randomized runner in packaging tests (#32109)
  Add support for field aliases. (#32172)
  Painless: Fix caching bug and clean up addPainlessClass. (#32142)
  Call setReferences() on custom referring tokenfilters in _analyze (#32157)
  Fix BwC Tests looking for UUID Pre 6.4 (#32158)
  Improve docs for search preferences (#32159)
  use before instead of onOrBefore
  Add more contexts to painless execute api (#30511)
  Add EC2 credential test for repository-s3 (#31918)
  A replica can be promoted and started in one cluster state update (#32042)
  Fix Java 11 javadoc compile problem
  Fix CP for namingConventions when gradle home has spaces (#31914)
  Fix `range` queries on `_type` field for singe type indices (#31756)
  [DOCS] Update TLS on Docker for 6.3 (#32114)
  ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are
  Remove versionType from translog (#31945)
  Switch distribution to new style Requests (#30595)
  Build: Skip jar tests if jar disabled
  Painless: Add PainlessClassBuilder (#32141)
  Build: Make additional test deps of check (#32015)
  Disable C2 from using AVX-512 on JDK 10 (#32138)
  Build: Move shadow customizations into common code (#32014)
  Painless: Fix Bug with Duplicate PainlessClasses (#32110)
  Remove empty @param from Javadoc
  Re-disable packaging tests on suse boxes
  Docs: Fix missing example script quote (#32010)
  [ML] Wait for aliases in multi-node tests (#32086)
  [ML] Move analyzer dependencies out of categorization config (#32123)
  Ensure to release translog snapshot in primary-replica resync (#32045)
  Handle TokenizerFactory  TODOs (#32063)
  Relax TermVectors API to work with textual fields other than TextFieldType (#31915)
  Updates the build to gradle 4.9 (#32087)
  Mute :qa:mixed-cluster indices.stats/10_index/Index - all’
  Check that client methods match API defined in the REST spec (#31825)
  Enable testing in FIPS140 JVM (#31666)
  Fix put mappings java API documentation (#31955)
  Add exclusion option to `keep_types` token filter (#32012)
  [Test] Modify assert statement for ssl handshake (#32072)
2018-07-19 23:03:01 -04:00
Mayya Sharipova 4c68dfe001
Handle missing values in painless (#32207)
Throw an exception for doc['field'].value
if this document is missing a value for the field.

After deprecation changes have been backported to 6.x,
make this a default behaviour in 7.0

Closes #29286
2018-07-19 17:41:06 -04:00
Vladimir Dolzhenko 7c0fc209bf
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)

Closes #26913
2018-07-19 12:54:38 +02:00
David Turner f6d7854f76
Remove indication of future multi-homing support (#32187)
We do not support intra-cluster connections on multiple interfaces, but the
documentation indicates that we will in future. In fact there is currently no
plan to support this, so the forward-looking documentation is misleading. This
commit

- removes the misleading sentence
- fixes that a transport profile affects outbound connections, not inbound ones
- tidies up some nearby text
2018-07-19 11:33:46 +01:00
Alexander Reelsen 202894b832
Rest HL client: Add put watch action (#32026)
Relates #29827

This implementation behaves like the current transport client, that you basically cannot configure a Watch POJO representation as an argument to the put watch API, but only a bytes reference. You can use the the `WatchSourceBuilder` from the `org.elasticsearch.plugin:x-pack-core` dependency to build watches.

This commit also changes the license type to trial, so that watcher is available in high level rest client tests.

/cc @hub-cap
2018-07-19 10:40:54 +02:00
Julie Tibshirani 15ff3da653
Add support for field aliases. (#32172)
* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)
2018-07-18 09:33:09 -07:00
David Turner 380b45b965
Improve docs for search preferences (#32159)
Today it is unclear what guarantees are offered by the search preference
feature, and we claim a guarantee that is stronger than what we really offer:

> A custom value will be used to guarantee that the same shards will be used
> for the same custom value.

This commit clarifies this documentation.

Forward-port of #32098 to `master`.
2018-07-18 12:58:17 +01:00
Martijn van Groningen 1924f5d07c
Add more contexts to painless execute api (#30511)
This change adds two contexts the execute scripts against:

* SEARCH_SCRIPT: Allows to run scripts in a search script context.
This context is used in `function_score` query's script function,
script fields, script sorting and `terms_set` query.

* FILTER_SCRIPT: Allows to run scripts in a filter script context.
This context is used in the `script` query.

In both contexts a index name needs to be specified and a sample document.
The document is needed to create an in-memory index that the script can
access via the `doc[...]` and other notations. The index name is needed
because a mapping is needed to index the document.

Examples:

```
POST /_scripts/painless/_execute
{
  "script": {
    "source": "doc['field'].value.length()"
  },
  "context" : {
    "search_script": {
      "document": {
        "field": "four"
      },
      "index": "my-index"
    }
  }
}
```

Returns:

```
{
  "result": 4
}
```

POST /_scripts/painless/_execute
{
  "script": {
    "source": "doc['field'].value.length() <= params.max_length",
    "params": {
      "max_length": 4
    }
  },
  "context" : {
    "filter_script": {
      "document": {
        "field": "four"
      },
      "index": "my-index"
    }
  }
}

Returns:

```
{
  "result": true
}
```

Also changed PainlessExecuteAction.TransportAction to use TransportSingleShardAction
instead of HandledAction, because now in case score or filter contexts are used
the request needs to be redirected to a node that has an active IndexService
for the index being referenced (a node with a shard copy for that index).
2018-07-18 12:42:07 +02:00
Christoph Büscher 36165265ce
Fix put mappings java API documentation (#31955)
The current docs of the put-mapping Java API is currently broken. It its current
form, it creates an index and uses the whole mapping definition given as a JSON
string as the type name. Since we didn't check the index created in the
IndicesDocumentationIT so far this went unnoticed.

This change adds test to catch this error to the documentation test, changes the
documentation so it works correctly now and adds an input validation to
PutMappingRequest#buildFromSimplifiedDef() which was used internally to reject
calls where no mapping definition is given.

Closes #31906
2018-07-17 09:09:03 +02:00