Commit Graph

6473 Commits

Author SHA1 Message Date
lcawl 504cfb2fb1 [DOCS] Adds missing anchors for profile API 2018-12-18 15:20:19 -08:00
lcawl a8387592db [DOCS] Fixes broken links to tcp_tranport_profiles 2018-12-18 14:43:58 -08:00
Tim Brooks 47a9a8de49
Update transport docs and settings for changes (#36786)
This is related to #36652. In 7.0 we plan to deprecate a number of
settings that make reference to the concept of a tcp transport. We
mostly just have a single transport type now (based on tcp). Settings
should only reference tcp if they are referring to socket options. This
commit updates the settings in the docs. And removes string usages of
the old settings. Additionally it adds a missing remote compress setting
to the docs.
2018-12-18 13:09:58 -07:00
Lisa Cawley c3c785cc1e
[DOCs] Adds ml-cpp PRs to alpha release notes (#36790) 2018-12-18 10:46:03 -08:00
Tal Levy 32ce1240e4
[ILM][DOCS] Update ILM API authorization docs (#36749)
the `view_index_metadata` privilege. Now the ILM API authorization
documentation mentions the need for these roles in more specific
ways.
2018-12-18 09:53:25 -08:00
David Kyle e294056bbf
[ML] Merge the Jindex master feature branch (#36702)
* [ML] Job and datafeed mappings with index template (#32719)

Index mappings for the configuration documents

* [ML] Job config document CRUD operations (#32738)

* [ML] Datafeed config CRUD operations (#32854)

* [ML] Change JobManager to work with Job config in index  (#33064)

* [ML] Change Datafeed actions to read config from the config index (#33273)

* [ML] Allocate jobs based on JobParams rather than cluster state config (#33994)

* [ML] Return missing job error when .ml-config is does not exist (#34177)

* [ML] Close job in index (#34217)

* [ML] Adjust finalize job action to work with documents (#34226)

* [ML] Job in index: Datafeed node selector (#34218)

* [ML] Job in Index: Stop and preview datafeed (#34605)

* [ML] Delete job document (#34595)

* [ML] Convert job data remover to work with index configs (#34532)

* [ML] Job in index: Get datafeed and job stats from index (#34645)

* [ML] Job in Index: Convert get calendar events to index docs (#34710)

* [ML] Job in index: delete filter action (#34642)

This changes the delete filter action to search
for jobs using the filter to be deleted in the index
rather than the cluster state.

* [ML] Job in Index: Enable integ tests (#34851)

Enables the ml integration tests excluding the rolling upgrade tests and a lot of fixes to
make the tests pass again.

* [ML] Reimplement established model memory (#35500)

This is the 7.0 implementation of a master node service to
keep track of the native process memory requirement of each ML
job with an associated native process.

The new ML memory tracker service works when the whole cluster
is upgraded to at least version 6.6. For mixed version clusters
the old mechanism of established model memory stored on the job
in cluster state was used. This means that the old (and complex)
code to keep established model memory up to date on the job object
has been removed in 7.0.

Forward port of #35263

* [ML] Need to wait for shards to replicate in distributed test (#35541)

Because the cluster was expanded from 1 node to 3 indices would
initially start off with 0 replicas.  If the original node was
killed before auto-expansion to 1 replica was complete then
the test would fail because the indices would be unavailable.

* [ML] DelayedDataCheckConfig index mappings (#35646)

* [ML] JIndex: Restore finalize job action (#35939)

* [ML] Replace Version.CURRENT in streaming functions (#36118)

* [ML] Use 'anomaly-detector' in job config doc name (#36254)

* [ML] Job In Index: Migrate config from the clusterstate (#35834)

Migrate ML configuration from clusterstate to index for closed jobs
only once all nodes are v6.6.0 or higher

* [ML] Check groups against job Ids on update (#36317)

* [ML] Adapt to periodic persistent task refresh (#36633)

* [ML] Adapt to periodic persistent task refresh

If https://github.com/elastic/elasticsearch/pull/36069/files is
merged then the approach for reallocating ML persistent tasks
after refreshing job memory requirements can be simplified.
This change begins the simplification process.

* Remove AwaitsFix and implement TODO

* [ML] Default search size for configs

* Fix TooManyJobsIT.testMultipleNodes

Two problems:

1. Stack overflow during async iteration when lots of
   jobs on same machine
2. Not effectively setting search size in all cases

* Use execute() instead of submit() in MlMemoryTracker

We don't need a Future to wait for completion

* [ML][TEST] Fix NPE in JobManagerTests

* [ML] JIindex: Limit the size of bulk migrations (#36481)

* [ML] Prevent updates and upgrade tests (#36649)

* [FEATURE][ML] Add cluster setting that enables/disables config  migration (#36700)

This commit adds a cluster settings called `xpack.ml.enable_config_migration`.
The setting is `true` by default. When set to `false`, no config migration will
be attempted and non-migrated resources (e.g. jobs, datafeeds) will be able
to be updated normally.

Relates #32905

* [ML] Snapshot ml configs before migrating (#36645)

* [FEATURE][ML] Split in batches and migrate all jobs and datafeeds (#36716)

Relates #32905

* SQL: Fix translation of LIKE/RLIKE keywords (#36672)

* SQL: Fix translation of LIKE/RLIKE keywords

Refactor Like/RLike functions to simplify internals and improve query
 translation when chained or within a script context.

Fix #36039
Fix #36584

* Fixing line length for EnvironmentTests and RecoveryTests (#36657)

Relates #34884

* Add back one line removed by mistake regarding java version check and
COMPAT jvm parameter existence

* Do not resolve addresses in remote connection info (#36671)

The remote connection info API leads to resolving addresses of seed
nodes when invoked. This is problematic because if a hostname fails to
resolve, we would not display any remote connection info. Yet, a
hostname not resolving can happen across remote clusters, especially in
the modern world of cloud services with dynamically chaning
IPs. Instead, the remote connection info API should be providing the
configured seed nodes. This commit changes the remote connection info to
display the configured seed nodes, avoiding a hostname resolution. Note
that care was taken to preserve backwards compatibility with previous
versions that expect the remote connection info to serialize a transport
address instead of a string representing the hostname.

* [Painless] Add boxed type to boxed type casts for method/return (#36571)

This adds implicit boxed type to boxed types casts for non-def types to create asymmetric casting relative to the def type when calling methods or returning values. This means that a user calling a method taking an Integer can call it with a Byte, Short, etc. legally which matches the way def works. This creates consistency in the casting model that did not previously exist.

* SNAPSHOTS: Adjust BwC Versions in Restore Logic (#36718)

* Re-enables bwc tests with adjusted version conditions now that #36397 enables concurrent snapshots in 6.6+

* ingest: fix on_failure with Drop processor (#36686)

This commit allows a document to be dropped when a Drop processor
is used in the on_failure fork of the processor chain.

Fixes #36151

* Initialize startup `CcrRepositories` (#36730)

Currently, the CcrRepositoryManger only listens for settings updates
and installs new repositories. It does not install the repositories that
are in the initial settings. This commit, modifies the manager to
install the initial repositories. Additionally, it modifies the ccr
integration test to configure the remote leader node at startup, instead
of using a settings update.

* [TEST] fix float comparison in RandomObjects#getExpectedParsedValue

This commit fixes a test bug introduced with #36597. This caused some
test failure as stored field values comparisons would not work when CBOR
xcontent type was used.

Closes #29080

* [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)

This commit  exposes lucene's LatLonShape field as the
default type in GeoShapeFieldMapper. To use the new 
indexing approach, simply set "type" : "geo_shape" in 
the mappings without setting any of the strategy, precision, 
tree_levels, or distance_error_pct parameters. Note the 
following when using the new indexing approach:

* geo_shape query does not support querying by 
MULTIPOINT.
* LINESTRING and MULTILINESTRING queries do not 
yet support WITHIN relation.
* CONTAINS relation is not yet supported.
The tree, precision, tree_levels, distance_error_pct, 
and points_only parameters are deprecated.

* TESTS:Debug Log. IndexStatsIT#testFilterCacheStats

* ingest: support default pipelines + bulk upserts (#36618)

This commit adds support to enable bulk upserts to use an index's
default pipeline. Bulk upsert, doc_as_upsert, and script_as_upsert
are all supported.

However, bulk script_as_upsert has slightly surprising behavior since
the pipeline is executed _before_ the script is evaluated. This means
that the pipeline only has access the data found in the upsert field
of the script_as_upsert. The non-bulk script_as_upsert (existing behavior)
runs the pipeline _after_ the script is executed. This commit
does _not_ attempt to consolidate the bulk and non-bulk behavior for
script_as_upsert.

This commit also adds additional testing for the non-bulk behavior,
which remains unchanged with this commit.

fixes #36219

* Fix duplicate phrase in shrink/split error message (#36734)

This commit removes a duplicate "must be a" from the shrink/split error
messages.

* Deprecate types in get_source and exist_source (#36426)

This change adds a new untyped endpoint `{index}/_source/{id}` for both the
GET and the HEAD methods to get the source of a document or check for its
existance. It also adds deprecation warnings to RestGetSourceAction that emit
a warning when the old deprecated "type" parameter is still used. Also updating
documentation and tests where appropriate.

Relates to #35190

* Revert "[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)"

This reverts commit 5bc7822562.

* Enhance Invalidate Token API (#35388)

This change:

- Adds functionality to invalidate all (refresh+access) tokens for all users of a realm
- Adds functionality to invalidate all (refresh+access)tokens for a user in all realms
- Adds functionality to invalidate all (refresh+access) tokens for a user in a specific realm
- Changes the response format for the invalidate token API to contain information about the 
   number of the invalidated tokens and possible errors that were encountered.
- Updates the API Documentation

After back-porting to 6.x, the `created` field will be removed from master as a field in the 
response

Resolves: #35115
Relates: #34556

* Add raw sort values to SearchSortValues transport serialization (#36617)

In order for CCS alternate execution mode (see #32125) to be able to do the final reduction step on the CCS coordinating node, we need to serialize additional info in the transport layer as part of each `SearchHit`. Sort values are already present but they are formatted according to the provided `DocValueFormat` provided. The CCS node needs to be able to reconstruct the lucene `FieldDoc` to include in the `TopFieldDocs` and `CollapseTopFieldDocs` which will feed the `mergeTopDocs` method used to reduce multiple search responses (one per cluster) into one.

This commit adds such information to the `SearchSortValues` and exposes it through a new getter method added to `SearchHit` for retrieval. This info is only serialized at transport and never printed out at REST.

* Watcher: Ensure all internal search requests count hits (#36697)

In previous commits only the stored toXContent version of a search
request was using the old format. However an executed search request was
already disabling hit counts. In 7.0 hit counts will stay enabled by
default to allow for proper migration.

Closes #36177

* [TEST] Ensure shard follow tasks have really stopped.

Relates to #36696

* Ensure MapperService#getAllMetaFields elements order is deterministic (#36739)

MapperService#getAllMetaFields returns an array, which is created out of
an `ObjectHashSet`. Such set does not guarantee deterministic hash
ordering. The array returned by its toArray may be sorted differently
at each run. This caused some repeatability issues in our tests (see #29080)
as we pick random fields from the array of possible metadata fields,
but that won't be repeatable if the input array is sorted differently at
every run. Once setting the tests seed, hppc picks that up and the sorting is
deterministic, but failures don't repeat with the seed that gets printed out
originally (as a seed was not originally set).
See also https://issues.carrot2.org/projects/HPPC/issues/HPPC-173.

With this commit, we simply create a static sorted array that is used for
`getAllMetaFields`. The change is in production code but really affects
only testing as the only production usage of this method was to iterate
through all values when parsing fields in the high-level REST client code.
Anyways, this seems like a good change as returning an array would imply
that it's deterministically sorted.

* Expose Sequence Number based Optimistic Concurrency Control in the rest layer (#36721)

Relates #36148 
Relates #10708

* [ML] Mute MlDistributedFailureIT
2018-12-18 17:45:31 +00:00
Nick Knize ec0dc2c0e9
[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#36751)
* [Geo] Expose BKDBackedGeoShapes as new VECTOR strategy

This commit exposes lucene's LatLonShape field as a new
strategy in GeoShapeFieldMapper. To use the new indexing
approach, strategy should be set to "vector" in the
geo_shape field mapper. If the tree parameter is set
the mapper will throw an IAE. Note the following:

When using vector strategy:

* geo_shape query does not support querying by POINT,
MULTIPOINT, or GEOMETRYCOLLECTION.
* LINESTRING and MULTILINESTRING queries do not support
WITHIN relation.
* CONTAINS relation is not supported.
* The tree, precision, tree_levels, distance_error_pct,
and points_only parameters will not throw an exception
but they have no effect and will be marked as
deprecated..

All other features are supported.

* revert change to PercolatorFieldMapper

* fix ExistsQuery for geo_shape vector strategy

* add deprecation logging for tree, precision, tree_levels, distance_error_pct, and points_only

* initial update to geoshape docs, including mapping migration updates

* initial support for GeoCollection queries

* fix docs and javadoc errors

* clean up geocollection queries

* set deprecated mapping tests to NOTCONSOLE

* fix geo-shape mapper asciidoc mapping and test warnings

* add support for point queries using LatLonShapeBoundingBoxQuery

* update GeoShapeQueryBuilderTests to include POINT queries for VECTOR strategy. Other comment cleanups

* add lucene geometry build testing to ShapeBuilder tests

* remove deprecated prefix tree mapping from geo-shape.asciidoc

* refactor GeoShapeFieldMapper into LegacyGeoShapeFieldMapper and GeoShapeFieldMapper

Both classes derive from BaseGeoShapeFieldMapper that provides shared parameters:
coerce, ignoreMalformed, ignore_z_value, orientation.

* update docs to remove vector strategy

* fix GeometryCollectionBuilder#buildLucene to return the object created by the shape builder

* fix LineLength failure in GeoJsonShapeParserTests

* ShapeMapper refactor changes from PR feedback

* fix typo in geo-shape.asciidoc

* ignore circle test in docs

* update indexing-approach ref to geoshape-indexing-approach

* add warnings check for LegacyGeoShapeFieldMapper to AbstractBuilderTestCase

* fix deprecatedParameters setup

* update indexing approach

* fixing unexpected warnings failures

* move orientation back to field type

* remove if in LegacyGeoShapeFieldMapper#doXContent. Fix GeoShapeFieldMapper to work with double array as a point

* fix indexing-approach link in circle section of geoshape docs

* add strategy to deprecation warnings check

* fix test failures

* fix typo in QueryStringQueryBuilderTests

* fix total hits to totalHits().value

* fix version number

* add version check to BaseGeoShapeFieldMapper

* fix line length!

* revert version check in BaseGeoShapeFieldMapper

* Fix serialization of mappings of legacy shapes.
2018-12-18 09:54:56 -06:00
Christoph Büscher 41feaf137c
[Docs] Fix error in Common Grams Token Filter (#36774)
The first example given is missing the two single-token cases for "is" and "a".
The later usage example is slightly wrong in that custom analyzers should
go under `settings.analysis.analyzer`.
2018-12-18 16:54:06 +01:00
Ioannis Kakavas 78f9af19c6
Invalidate Token API enhancements - HLRC (#36362)
* Adds Invalidate Token API enhancements to HLRC

Relates: #35388
2018-12-18 16:12:43 +02:00
Mayya Sharipova f884b2b1cd
Deprecate types in index API (#36575)
* Deprecate types in index API

- deprecate type-based constructors of IndexRequest
- update tests to use typeless IndexRequest constructors
- no yaml tests as they have been already added in #35790

Relates to #35190
2018-12-18 08:53:49 -05:00
Alan Woodward af57575838
Allow word_delimiter_graph_filter to not adjust internal offsets (#36699)
This commit adds an adjust_offsets parameter to the word_delimiter_graph token filter, defaulting
to true. Most of the time you'd want sub-tokens emitted by this filter to have offsets that are
adjusted to their real position in the token stream; however, some token filters can change the 
length or starting position of a token (eg trim) without changing their offset attributes, and this 
can lead to word_delimiter_graph emitting illegal offsets. Setting adjust_offsets to false in these 
cases will allow indexing again.

Fixes #34741, #33710
2018-12-18 13:20:51 +00:00
Ioannis Kakavas 7b9ca62174
Enhance Invalidate Token API (#35388)
This change:

- Adds functionality to invalidate all (refresh+access) tokens for all users of a realm
- Adds functionality to invalidate all (refresh+access)tokens for a user in all realms
- Adds functionality to invalidate all (refresh+access) tokens for a user in a specific realm
- Changes the response format for the invalidate token API to contain information about the 
   number of the invalidated tokens and possible errors that were encountered.
- Updates the API Documentation

After back-porting to 6.x, the `created` field will be removed from master as a field in the 
response

Resolves: #35115
Relates: #34556
2018-12-18 10:05:50 +02:00
Nicholas Knize 96d279ed83 Revert "[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)"
This reverts commit 5bc7822562.
2018-12-17 20:09:46 -06:00
Christoph Büscher 2f5300e3a6
Deprecate types in get_source and exist_source (#36426)
This change adds a new untyped endpoint `{index}/_source/{id}` for both the
GET and the HEAD methods to get the source of a document or check for its
existance. It also adds deprecation warnings to RestGetSourceAction that emit
a warning when the old deprecated "type" parameter is still used. Also updating
documentation and tests where appropriate.

Relates to #35190
2018-12-18 00:57:42 +01:00
Nick Knize 5bc7822562
[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)
This commit  exposes lucene's LatLonShape field as the
default type in GeoShapeFieldMapper. To use the new 
indexing approach, simply set "type" : "geo_shape" in 
the mappings without setting any of the strategy, precision, 
tree_levels, or distance_error_pct parameters. Note the 
following when using the new indexing approach:

* geo_shape query does not support querying by 
MULTIPOINT.
* LINESTRING and MULTILINESTRING queries do not 
yet support WITHIN relation.
* CONTAINS relation is not yet supported.
The tree, precision, tree_levels, distance_error_pct, 
and points_only parameters are deprecated.
2018-12-17 14:38:14 -06:00
Lisa Cawley 4103d3b9ec
[DOCS] Adds monitoring requirement for ingest node (#36665) 2018-12-17 08:18:11 -08:00
Boaz Leskes e356b8cb95
Add doc's sequence number + primary term to GetResult and use it for updates (#36680)
This commit adds the last sequence number and primary term of the last operation that have
modified a document to `GetResult` and uses it to power the Update API.

Relates #36148 
Relates #10708
2018-12-17 15:22:13 +01:00
Martijn van Groningen a181a25226
[CCR] Add time since last auto follow fetch to auto follow stats (#36542)
For each remote cluster the auto follow coordinator, starts an auto
follower that checks the remote cluster state and determines whether an
index needs to be auto followed. The time since last auto follow is
reported per remote cluster and gives insight whether the auto follow
process is alive.

Relates to #33007
Originates from #35895
2018-12-17 14:14:56 +01:00
João Barbosa 276726aea2 Added keyed response to pipeline percentile aggregations 22302 (#36392)
Closes #22302
2018-12-14 16:22:54 -05:00
Julie Tibshirani ccd1beb9b3
Deprecate types in update requests. (#36181)
The following updates were made:
* Add deprecation warnings to `RestUpdateAction`, plus a test in `RestUpdateActionTests`.
* Deprecate relevant methods on the Java HLRC requests/ responses.
* Add HLRC integration tests for the typed APIs.
* Update documentation (for both the REST API and Java HLRC).
* Fix failing integration tests.

Because of an earlier PR, the REST yml tests were already updated (one version without types, and another legacy version that retains types).
2018-12-14 10:47:27 -08:00
Costin Leau 6ee6bb55e2
SQL: Introduce HISTOGRAM grouping function (#36510)
Introduce Histogram grouping function for bucketing/grouping data based
 on a given range. Both date and numeric histograms are supported using
 the appropriate range declaration (numbers vs intervals).

SELECT HISTOGRAM(number, 50) AS h FROM index GROUP BY h
SELECT HISTOGRAM(date, INTERVAL 1 YEAR) AS h FROM index GROUP BY h

In addition add multiply operator for Intervals
Add docs for intervals and histogram

Fix #36509
2018-12-14 18:20:37 +02:00
Alan Woodward 09bf93dc2a
Add intervals query (#36135)
* Add IntervalQueryBuilder with support for match and combine intervals

* Add relative intervals

* feedback

* YAML test - broekn

* yaml test; begin to add block source

* Add block; make disjunction its own source

* WIP

* Extract IntervalBuilder and add tests for it

* Fix eq/hashcode in Disjunction

* New yaml test

* checkstyle

* license headers

* test fix

* YAML format

* YAML formatting again

* yaml tests; javadoc

* Add OR test -> requires fix from LUCENE-8586

* Add docs

* Re-do API

* Clint's API

* Delete bash script

* doc fixes

* imports

* docs

* test fix

* feedback

* comma

* docs fixes

* Tidy up doc references to old  rule
2018-12-14 15:14:00 +00:00
Costin Leau f502ca53d4
SQL: Introduce NOW/CURRENT_TIMESTAMP function (#36562)
Add CURRENT_TIMESTAMP as keyword as well function alongside NOW() 
 These return the current date/time for the  given query, computed when
 the statement reaches the server. For completeness, CURRENT_TIMESTAMP
 also accepts precision as an optional parameter.

Fix #36534
2018-12-14 15:42:36 +02:00
Jeff Hajewski f1f3b28f5c Delete deprecated getValues from ScriptDocValues (#36183)
* Adds deprecation logging to ScriptDocValues#getValues.

First commit addressing issue #22919.

`ScriptDocValues#getValues` was added for backwards compatibility but no
longer needed. Scripts using the syntax `doc['foo'].values` when
`doc['foo']` is a list should be using `doc['foo']` instead.

* Fixes two build errors in #34279

* Removes unused import in ScriptDocValuesDatesTest
* Removes used of `.values` in example in diversified-sampler-aggregation.asciidoc

* Removes use of .values from painless test.

Part of #34279

* Updates tests to use `doc[foo]` syntax rather than `doc[foo].values`.

* Removes use of `getValues()` and replaces use of `doc[foo].values` with `doc[foo]`.

* Indentation fix.

* Remove unnecessary list construction at previous `getValues()` callsite in ScriptDocValues.GeoPoints.

* Update migration doc and add link to `getValue` in ScriptDocValues javadoc.

* Fix compile

* Fix javadoc issue

* Removes ScriptDocValues#getValues usage from painless whitelist.
2018-12-14 07:56:47 -05:00
Armin Braun c5b3ac5578
SNAPSHOTS: Allow Parallel Restore Operations (#36397)
* Enable parallel restore operations
* Add uuid to restore in progress entries to uniquely identify them
* Adjust restore in progress entries to be a map in cluster state
* Added tests for:
   * Parallel restore from two different snapshots
   * Parallel restore from a single snapshot to different indices to test uuid identifiers are correctly used by `RestoreService` and routing allocator
   * Parallel restore with waiting for completion to test transport actions correctly use uuid identifiers
2018-12-14 11:39:23 +01:00
David Pilato 5fe4c9076c Default ec2 endpoint is ec2.us-east-1.amazonaws.com (#30323)
Today our docs claim that the `discovery-ec2` plugin auto-detects the EC2
endpoint, but this is not the case. This fixes these docs.

Closes #27464.
2018-12-14 09:30:39 +00:00
Jason Tedor 692cff830a
Add guidance on using CCR with Logstash (#36609)
* Add guidance on using CCR with Logstash

This commit adds a note to the documentation regarding how to configure
Logstash indices in the context of being available as leader indices for
cross-cluster replication.

* Oh okay

* idk

* notconsole
2018-12-13 16:14:59 -05:00
Gil Raphaelli 64da98070c format numbered list (#36580) 2018-12-13 12:44:34 -08:00
Ryan Ernst 254d1e8f22
Core: Add backcompat for joda time formats (#36531)
This commit adds deprecation warnings when using format specifiers with
joda data formats that will change with java time. It also adds the "8"
prefix which may be used to force the new java time format parsing.
2018-12-13 12:26:51 -08:00
Nick Knize 4b17055035
HLRC: Add get users action (#36332)
This commit adds get user action to the high level rest client.
2018-12-13 12:24:48 -06:00
Jason Tedor 2afa7faefd
Override the JVM DNS cache policy (#36570)
When a security manager is present, the JVM will cache positive hostname
lookups indefinitely. This can be problematic, especially in the modern
world with cloud services where DNS addresses can change, or
environments using Docker containers where IP addresses could be
considered ephemeral. This behavior impacts cluster discovery,
cross-cluster replication and cross-cluster search, reindex from remote,
snapshot repositories, webhooks in Watcher, external authentication
mechanisms, and the Elastic Stack Monitoring Service. The experience of
watching a DNS lookup change yet not be reflected within Elasticsearch
is a poor experience for users. The reason the JVM has this is guard
against DNS cache posioning attacks. Yet, there is already a defense in
the modern world against such attacks: TLS. With proper certificate
validation, even if a resolver falls prey to a DNS cache poisoning
attack, using TLS would neuter the attack. Therefore we have a policy
with dubious security value that significantly impacts usability. As
such we make the usability/security tradeoff towards usability, since
the security risks are very low. This commit introduces new system
properties that Elasticsearch observes to override the JVM DNS cache
policy.
2018-12-13 10:23:45 -05:00
Daniel Mitterdorfer 4d739d71d7
Document dynamic email settings in Watcher
With this commit we document which of the email notification settings in
Watcher can be set dynamically via the cluster settings API.

Relates #36543
2018-12-13 14:09:50 +01:00
Mayya Sharipova bda03163e7 Make vector fields experimental feature
Relates to #33022
2018-12-13 07:17:52 -05:00
David Roberts 13cb0fb98b
Periodically try to reassign unassigned persistent tasks (#36069)
Previously persistent task assignment was checked in the
following situations:

- Persistent tasks are changed
- A node joins or leaves the cluster
- The routing table is changed
- Custom metadata in the cluster state is changed
- A new master node is elected

However, there could be situations when a persistent
task that could not be assigned to a node could become
assignable due to some other change, such as memory
usage on the nodes.

This change adds a timed recheck of persistent task
assignment to account for such situations.  The timer
is suspended while checks triggered by cluster state
changes are in-flight to avoid adding burden to an
already busy cluster.

Closes #35792
2018-12-13 09:15:27 +00:00
Mayya Sharipova d40037c91e
Deprecate uses of _type as a field name in queries (#36503) 2018-12-12 21:21:53 -05:00
Mayya Sharipova b5d532f9e3
Vector field (#33022)
1. Dense vector

PUT dindex
{
  "mappings": {
    "_doc": {
      "properties": {
        "my_vector": {
          "type": "dense_vector"
        },
        "my_text" : {
          "type" : "keyword"
        }
      }
    }
  }
}

PUT dinex/_doc/1
{
  "my_text" : "text1",
  "my_vector" : [ 0.5, 10, 6 ]
}

2. Sparse vector

PUT sindex
{
  "mappings": {
    "_doc": {
      "properties": {
        "my_vector": {
          "type": "sparse_vector"
        },
        "my_text" : {
          "type" : "keyword"
        }
      }
    }
  }
}

PUT sindex/_doc/1
{
  "my_text" : "text1",
  "my_vector" : {"1": 0.5, "99": -0.5,  "5": 1}
}
2018-12-12 21:20:53 -05:00
Max Penet dabe29e783 [Docs] Adding community clojure client built on java low level rest client (#36484) 2018-12-12 13:57:53 +01:00
Tim Vernum 143f151185
HLRC: Implement get-user-privileges API (#36292)
This adds the _security/user/_privileges API to the High
Level Rest Client.

This also makes some changes to the Java model for the Role APIs
in order to better accommodate the GetPrivileges API
2018-12-12 15:12:49 +11:00
Nik Everett 03daad9812
Re-deprecate xpack rollup endpoints (#36451)
Redeprecates the `/_xpack/rollup` endpoints in favor of `/_rollup`.

When we cleanup the rollup in a cluster containing 6.x nodes we need to
use `/_xpack/rollup` instead of `/_rollup` because the 6.x nodes don't
know about `/_rollup`. In those cases we must ignore the deprecation
warnings that the 7.0 node will return for the end point.

Closes #36044
2018-12-11 19:43:17 -05:00
Andrei Stefan de373060fb
SQL: non ISO 8601 versions of DAY_OF_WEEK and WEEK_OF_YEAR functions (#36358)
* Renamed DAY_OF_WEEK and WEEK_OF_YEAR functions to their ISO version and
added the same functions with different functionality.
* Rewritten the datetime functions documentation to follow the format of the other
functions documentation pages.
2018-12-12 02:29:02 +02:00
Nhat Nguyen 51800de2a8
Enable soft-deletes by default on 7.0.0 or later (#36141)
This change enables soft-deletes by default on ES 7.0.0 or later.

Relates #33222

Co-authored-by: Jason Tedor <jason@tedor.me>
2018-12-11 18:58:49 -05:00
lcawl 55743aac47 [DOCS] Adds placeholder for alpha2 release notes 2018-12-11 14:26:41 -08:00
Mayya Sharipova 2f18325384
Deprecate types in update_by_query and delete_by_query (#36365)
Relates to #35190
2018-12-11 17:09:59 -05:00
Gordon Brown 6481f2e380
Add setting to bypass Rollover action (#36235)
Adds a setting that indicates that an index is done indexing, set by ILM
when the Rollover action completes. This indicates that the Rollover
action should be skipped in any future invocations, as long as the index
is no longer the write index for its alias.

This enables 1) an index with a policy that involves the Rollover action
to have the policy removed and switched to another one without use of
the move-to-step API, and 2) integrations with Beats and CCR.
2018-12-11 08:53:05 -07:00
debadair c9e03e6ead
[DOCS] Reworked the shard allocation filtering info. (#36456)
* [DOCS] Reworked the shard allocation filtering info. Closes #36079

* Added multiple index allocation settings example back.

* Removed extraneous space
2018-12-11 07:44:57 -08:00
Marios Trivyzas 031d0566b0
SQL: [docs] Add `beta[]` markup to all SQL pages (#36471)
Closes: #36437
2018-12-11 13:25:29 +02:00
Yu d01b30acba lower fielddata circuit breaker's default limit (#27162)
* Lower fielddata circuit breaker default limit

Lower fielddata circuit breaker default limit from 60% to 40% as we have
moved to doc_values for most of the cases.

* merge master in

* update tests

* update docs
2018-12-11 11:30:58 +01:00
Yannick Welsch d8e3d97a7d
Add discovery types to cluster stats (#36442)
Adds information about the used discovery types to the cluster stats, similar as we have for the network types.
2018-12-11 10:35:58 +01:00
David Turner bf42cedc13
Clarify the order of responses to a bulk request (#36441)
Bulk requests comprise many individual actions, and the responses for each
action comes back in the same order (see e.g. `DocumentActionsIT#testBulk()`).
However the docs do not seem to explicitly state this vital fact. This commit
addresses that omission.
2018-12-11 09:02:23 +00:00
Julie Tibshirani 87831051dc
Deprecate types in explain requests. (#35611)
The following updates were made:
- Add a new untyped endpoint `{index}/_explain/{id}`.
- Add deprecation warnings to Rest*Action, plus tests in Rest*ActionTests.
- For each REST yml test, make sure there is one version without types, and another legacy version that retains types (called *_with_types.yml).
- Deprecate relevant methods on the Java HLRC requests/ responses.
- Update documentation (for both the REST API and Java HLRC).
2018-12-10 19:45:13 -08:00
Ryan Ernst a0da390df2
Scripting: Switch watcher to use joda bwc time objects (#35966)
This commit converts the watcher execution context to use the joda
compat java time objects. It also again removes the joda methods from
the painless whitelist.
2018-12-10 17:29:25 -08:00
Julie Tibshirani 99f89cd3b4
Deprecate types in get, exists, and multi get. (#35930)
For each API, the following updates were made:
- Add deprecation warnings to `Rest*Action`, plus tests in `Rest*ActionTests`.
- For each REST yml test, make sure there is one version without types, and another legacy version that retains types (called *_with_types.yml).
- Deprecate relevant methods on the Java HLRC requests/ responses.
- Update documentation (for both the REST API and Java HLRC).
2018-12-10 17:22:48 -08:00
Nik Everett de06769677 Docs: Fix link to transport client
The transport client was linking to a page in maven central that didn't
have the transport client in it. This is better link and happens to be
the same link that the low and high level rest client use.
2018-12-10 16:34:54 -05:00
Gordon Brown 58a5ad1f1e
Add Tribe removal to breaking changes list (#36239) 2018-12-10 10:23:58 -07:00
Andy Bristol 10938e5135
[hlrc] add index templates exist API (#36132)
This commit adds support for the index templates exist API, creating
new client-side request types for that API and the get index
templates API. Also adds links in hlrc docs to pages for supported
index template APIs
2018-12-10 08:11:57 -08:00
David Pilato c9c691b0cf Change Lucene Snapshot example for 7.0.0 (#35619)
Update to a more recent version our example.
2018-12-10 12:44:39 +01:00
Albert Zaharovits dad6f1c9fe
[HLRC] Put Role (#36209)
This commit adds support for the put role API in the
java high level rest client.
2018-12-10 09:41:31 +02:00
Christoph Büscher a42502df8b
[Docs] Add description of simple query string flags (#36211)
Closes #34944
2018-12-10 01:00:42 +01:00
Martijn van Groningen f4aac8d3f1
[HLRC] Added support for Follow Stats API (#36253)
This change also adds documentation for the Follow Stats API.

Relates to #33824
2018-12-09 19:59:08 +01:00
Yogesh Gaikwad 32c4f99238
[HLRC] Add support for put privileges API (#35679)
This commit adds support for API to create or update
application privileges in high-level rest client.
2018-12-09 16:03:28 +11:00
Gordon Brown 85bba0c3ae
Simplify deprecation issue levels (#36326)
This commit gets rid of the 'NONE' and 'INFO' severity levels for
deprecation issues.

'NONE' is unused and does not make much sense as a severity level.
'INFO' can be separated into two categories: Either 1) we can
definitively tell there will be a problem with the cluster/node/index
configuration that can be resolved prior to upgrade, in which case
the issue should be a WARNING, or 2) we can't, because any issues would
be at the application level, for which the user should review the
deprecation logs and/or response headers.
2018-12-07 15:45:53 -07:00
Simon Willnauer 2ba023be8a
Explicitly recommend to forceMerge before freezing (#36376)
given the benchmark results on #34352 it's important to recommend
users to `_force_merge` their indices to a single segment before freezing.
2018-12-07 23:38:35 +01:00
David Roberts 9e8cfbb40d
[ML] Deprecate X-Pack centric ML endpoints (#36315)
This commit is part of our plan to deprecate and
ultimately remove the use of _xpack in the REST APIs.

Relates #35958
2018-12-07 20:34:11 +00:00
Wilder Pereira 1d73c1712b Docs: Fix Jackson reference (#36366)
Link is totally broken.
2018-12-07 15:18:22 -05:00
Nik Everett ead2b9e08b
HLRC: Add rollup search (#36334)
Relates to #29827
2018-12-07 14:39:58 -05:00
Nhat Nguyen 32a7d7abd5
HLRC: Add delete template API (#36320)
Relates #27205
2018-12-07 10:46:03 -05:00
Jim Ferenczi a53e8653f2
Add support for inlined user dictionary in Nori (#36123)
Add support for inlined user dictionary in Nori

This change adds a new option called `user_dictionary_rules` to the
Nori a tokenizer`. It can be used to set additional tokenization rules
to the Korean tokenizer directly in the settings (instead of using a file).

Closes #35842
2018-12-07 15:26:08 +01:00
Michael Basnight ca09936cdf
HLRC: Get Deprecation Info API (#36279)
This commit adds the Get Deprecation Info API and associated
documentation.

Relates #29827
2018-12-07 08:22:21 -06:00
David Turner c32e4fb83f
[Zen2] Best-effort cluster formation if unconfigured (#36215)
In real deployments it is important that clusters are properly configured to
avoid accidentally forming multiple independent clusters at cluster
bootstrapping time. However we also expect to be able to unpack Elasticsearch
and start up one or more nodes without any up-front configuration, and have
them do their best to find each other and form a cluster after a few seconds.

This change adds a delayed automatic bootstrapping process to nodes that start
up with no relevant settings set to support the desired out-of-the-box
experience without compromising safety in properly-configured deployments.
2018-12-07 12:47:09 +00:00
Armin Braun e6d190613f
[ZEN2] Use Zen2 in REST Tests (#36300) 2018-12-07 09:15:11 +01:00
Jason Tedor fc85c37efc
Fix typo in migration node for one shard per index 2018-12-06 19:20:10 -05:00
Jason Tedor e8fe624570
Add migration note on the number of shards
This commit adds a migration note regarding the default number of shards
changing from five to one.

Relates #30539
2018-12-06 19:15:45 -05:00
Jason Tedor d1184cfb68
Add docs on replicating APM Server or Beats indices (#36333)
This commit adds a brief note to the documentation on how to manage the
index templates that are used to create APM Server and Beats indices.
2018-12-06 16:07:40 -05:00
Tal Levy e944764be3
[ILM] [DOCS] add general info about steps (#36081)
it is important for users to understand that phases
are compiled into Steps that are exposed via the
Explain Lifecycle API.
2018-12-06 10:41:55 -08:00
Julie Tibshirani 3f3cde41d3
Deprecate types in termvector and mtermvector requests. (#36182)
* Add deprecation warnings to `Rest*TermVectorsAction`, plus tests in `Rest*TermVectorsActionTests`.
* Deprecate relevant methods on the Java HLRC requests/ responses.
* Update documentation (for both the REST API and Java HLRC).
* For each REST yml test, create one version without types, and another legacy version that retains types (called *_with_types.yml).
2018-12-06 10:23:15 -08:00
Lisa Cawley fb697c7442
[DOCS] Add missing anchors (#36288) 2018-12-06 10:14:37 -08:00
Ioannis Kakavas 77e6ef7b20
Fix get certificates HLRC API (#36198)
- GetSslCertificatesRequest need not implement toXContentObject
- getRequest() returns a new Request object
- Add tests for GetSslCertificatesResponse
- Adjust docs to the new format
2018-12-06 12:44:51 +02:00
Christoph Büscher 54f39d9852
[Docs] Add Profile API limitations (#36252)
Adding some of the limitations mentioned in #29275.

Closes #29275
2018-12-06 00:09:26 +01:00
Jim Ferenczi 18866c4c0b
Make hits.total an object in the search response (#35849)
This commit changes the format of the `hits.total` in the search response to be an object with
a `value` and a `relation`. The `value` indicates the number of hits that match the query and the
`relation` indicates whether the number is accurate (in which case the relation is equals to `eq`)
or a lower bound of the total (in which case it is equals to `gte`).
This change also adds a parameter called `rest_total_hits_as_int` that can be used in the
search APIs to opt out from this change (retrieve the total hits as a number in the rest response).
Note that currently all search responses are accurate (`track_total_hits: true`) or they don't contain
`hits.total` (`track_total_hits: true`). We'll add a way to get a lower bound of the total hits in a
follow up (to allow numbers to be passed to `track_total_hits`).

Relates #33028
2018-12-05 19:49:06 +01:00
lcawl c4181dc569 [DOCS] Fixes broken link in execute watch 2018-12-05 09:53:21 -08:00
Alan Woodward 73ceaad03a
Update to lucene-8.0.0-snapshot-c78429a554 (#36212)
Includes:

* A fix for a bug in Intervals.or() (https://issues.apache.org/jira/browse/LUCENE-8586)
* The ability to disable offset mangling in WordDelimiterGraphFilter
        (https://issues.apache.org/jira/browse/LUCENE-8509)
* BM25Similarity no longer multiplies scores by k1 + 1
2018-12-05 12:43:56 +00:00
Alan Woodward d2886e1c81
HLRC: execute watch API (#35868)
This change adds support for the execute watch API in the high level rest client
2018-12-05 12:41:42 +00:00
Martijn van Groningen 786697a4b2
[HLRC] Added support for CCR Stats API (#36213)
This change also adds documentation for the CCR Stats API.

Relates to #33824
2018-12-05 13:14:51 +01:00
Martijn van Groningen b9707c29a1
[CCR] Change get autofollow patterns API response format (#36203)
The current response format is:

```
{
    "pattern1": {
        ...
    },
    "pattern2": {
        ...
    }
}
```

The new format is:

```
{
    "patterns": [
        {
            "name": "pattern1",
            "pattern": {
                ...
            }
        },
        {
            "name": "pattern2",
            "pattern": {
                ...
            }
        }
    ]
}
```

This format is more structured and more friendly for parsing and generating specs.
This is a breaking change, but it is better to do this now while ccr
is still a beta feature than later.

Follow up from #36049
2018-12-05 08:41:27 +01:00
Christoph Büscher e825ad5dce
Add javadocs about expected exceptions to RestHighLevelClient (#36216)
Add a more detailed section about what exceptions to expect from the blocking
calls in this class and removing the mostly redundant mentions of the
IOExceptions from each method javadoc since it doesn't give much details about
the expected exceptions anyway.

Closes #30334
2018-12-05 00:31:52 +01:00
Lisa Cawley e1fb1505f9
[DOCS] Moves security config file info (#36232) 2018-12-04 13:18:54 -08:00
Guido Lena Cota 89fae42833 (Minor) Fix some typos (#36180) 2018-12-04 11:10:30 +01:00
Martijn van Groningen 579be9142e
[HLRC] Added support for CCR Get Auto Follow Pattern apis (#36049)
This change also adds documentation for the Get Auto Follow Pattern API.

Relates to #33824
2018-12-04 07:41:29 +01:00
lcawl 05d52b222f [DOCs] More broken painless links 2018-12-03 12:38:53 -08:00
lcawl e082cda0d6 [DOCS] Fixes peer link 2018-12-03 12:17:26 -08:00
João Barbosa d27aa72b17 Added soft limit to open scroll contexts #25244 (#36009)
This change adds a soft limit to open scroll contexts that can be controlled with the dynamic cluster setting `search.max_open_scroll_context` (defaults to 500).
2018-12-03 19:57:10 +01:00
Alan Woodward 19b936dcce
Fix broken links in painless docs (#36170) 2018-12-03 18:25:18 +00:00
Julie Tibshirani 59ee8b5c69
Remove the deprecated _termvector endpoint. (#36131) 2018-12-03 10:22:42 -08:00
Alan Woodward da2dbcdf38
Improve painless docs for score, similarity, weight and sort (#35629) 2018-12-03 17:34:15 +00:00
Jim Ferenczi 74aca756b8
Remove the distinction between query and filter context in QueryBuilders (#35354)
When building a query Lucene distinguishes two cases, queries that require to produce a score and queries that only need to match. We cloned this mechanism in the QueryBuilders in order to be able to produce different queries based on whether they need to produce a score or not. However the only case in es that require this distinction is the BoolQueryBuilder that sets a different minimum_should_match when a `bool` query is built in a filter context..
This behavior doesn't seem right because it makes the matching of `should` clauses different when the score is not required.

Closes #35293
2018-12-03 11:49:11 +01:00
Julie Tibshirani 0e1ddfd825
Deprecate types in document delete requests. (#36087)
* Make sure to use _doc as a type name in the CRUD HLRC tests.
* Deprecate types in document delete requests.
2018-11-30 15:11:29 -08:00
Gordon Brown 3c4953f4d1
State default shard limit is not a recommendation (#36093)
The new limit on the number of open shards in a cluster may be
interpreted by users as a sizing recommendation, but it is not. This
clarifies in the documentation that this is a safety limit, not a
recommendation.
2018-11-30 13:05:14 -07:00
Gordon Brown d7652963b1
Add note about ILM and Snapshots (#36023)
This commit documents how Index Lifecycle Management
interacts with snapshot/restore, and documents a workaround
for situations in which ILM should not immediately resume
managing an index after it is restored.
2018-11-30 12:06:48 -07:00
Chris Koehnke 465a65aa57
Docs: Fix release-state check for oss repositories (#36120)
To get the newly added oss apt/yum sections to get rendered for
`released` and `prerelease` versions the condition needs to be modified.
2018-11-30 13:17:39 -05:00
Lisa Cawley c24be278e4
[DOCS] Refreshes population job examples (#36101) 2018-11-30 08:55:29 -08:00
Christophe Bismuth acdf9666d5 Add `minimum_should_match` section to the query_string docs
Closes #34142
2018-11-30 16:10:13 +01:00
Jim Ferenczi 08b9e31373 [TEST] fix link in get-watch documentation 2018-11-30 14:40:09 +01:00
patrykk21 bb2cf7e6be [Docs] Clarify search_after behavior
Closes #34232
2018-11-30 14:30:23 +01:00
Jim Ferenczi 5e6460acb3 [TEST] fix typo in get-watch documentation 2018-11-30 12:53:04 +01:00
Jim Ferenczi 5c7b2c5f9b
HLRC: Add get watch API (#35531)
This changes adds the support for the get watch API in the high level rest client.
2018-11-30 11:02:46 +01:00
Peter Dyson 63a9571c4c
Explain nondeterministic template merging (#35405) 2018-11-30 14:56:32 +10:00
Gordon Brown d567f89719
Recommend index templates when using ILM Rollover (#35922)
Because rollover does not propogate the ILM policy in use, add a note to
the docs recommending the use of index templates when using rollover.
2018-11-29 14:09:51 -07:00
Zachary Tong 61c2db5ebb Revert "Deprecate X-Pack centric rollup endpoints (#35962)"
This reverts commit b84f1f6a3a.
2018-11-29 12:58:23 -05:00
Toby McLaughlin 8006068873 [DOCS] Remove "platinum" references for Docker TLS (#35890) 2018-11-29 09:33:08 -08:00
Jason Tedor 9fa9e1419f
Remove X-Pack centric graph endpoints (#36010)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-29 07:09:37 -05:00
Alan Woodward a646f85a99
Ensure TokenFilters only produce single tokens when parsing synonyms (#34331)
A number of tokenfilters can produce multiple tokens at the same position.  This
is a problem when using token chains to parse synonym files, as the SynonymMap
requires that there are no stacked tokens in its input.

This commit ensures that when used to parse synonyms, these tokenfilters either produce
a single version of their input token, or that they throw an error when mappings are 
generated.  In indexes created in elasticsearch 6.x deprecation warnings are emitted in place 
of the error. 

* asciifolding and cjk_bigram produce only the folded or bigrammed token
* decompounders, synonyms and keyword_repeat are skipped
* n-grams, word-delimiter-filter, multiplexer, fingerprint and phonetic throw errors

Fixes #34298
2018-11-29 10:35:38 +00:00
Ryan Ernst fbdfec4305
Docs: Add note about oss repositories for deb/rpm (#35973)
This commit adds a note about configring the yum/apt repositories for
oss only packages.

closes #35960
2018-11-29 00:22:10 -08:00
Martijn van Groningen 6d01170ca1
[HLRC] Added support for CCR Delete Auto Follow Pattern API (#35981)
This change also adds documentation for the Delete Auto Follow Pattern API.

Relates to #33824
2018-11-29 08:06:31 +01:00
Ignacio Vera 93ed8b7d61
HLRC: Add delete user action (#35294)
* HLRC: Add delete user action

It adds delete user action to the high level rest client.

Relates #29827
2018-11-29 07:52:56 +01:00
Alpar Torok e0a678f0c4
Remove version.qualified from MainResponse (#35412)
The fully qualified version will be returned as `version.number`
2018-11-29 08:41:39 +02:00
Lisa Cawley 906421cdad
[DOCS] Adds higher-level monitoring configuration page (#35926) 2018-11-28 12:36:29 -08:00
Gordon Brown c26af3b0a2
Deprecate X-Pack centric Migration endpoints (#35976)
This commit is part of our plan to deprecate and remove the use of
_xpack in the REST API routes.
2018-11-28 13:19:33 -07:00
Lisa Cawley a72430e1ef
[DOCS] Adds TLS warning to rolling upgrades (#35841) 2018-11-28 09:38:58 -08:00
Simon Willnauer daef498cd9 [DOCS] Fix HLRC reference to unfreeze_index 2018-11-28 16:55:55 +01:00
Simon Willnauer 89e4ac8fcc
Add high-level REST client API for `_freeze` and `_unfreeze` (#35723)
This change adds support for `_freeze` and `_unfreeze` to the HLRC

Relates to #34352
2018-11-28 15:42:12 +01:00
Jason Tedor a3186e4a32
Deprecate X-Pack centric license endpoints (#35959)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-28 08:24:35 -05:00
Jason Tedor 78ac12d952
Remove deprecated Graph endpoints (#35956)
We had some endpoints in Graph that are deprecated for removal in
7.0.0. This commit removes these deprecated endpoints.
2018-11-28 08:24:15 -05:00
Andrei Stefan aabff7318e
SQL: DATABASE() and USER() system functions (#35946) 2018-11-28 15:20:04 +02:00
Peter Dyson 1f25a0bd31 [Docs] Add example for updating meta field (#35893) 2018-11-28 12:04:57 +01:00
Christoph Büscher 33865211db
[Docs] Emphazise suggest behaviour with missing query part (#35393)
Add a short extra sentence that explains that a missing query part in a search
request containing a "suggest" section will mean only suggestions are returned.

Closes #31640
2018-11-28 12:01:27 +01:00
Vladimir Dolzhenko f0c5f0c099 [HLRC] XPack ML info action (#35777)
Relates to #29827
2018-11-28 10:58:20 +00:00
Costin Leau 95346926ef
SQL: Lock down JDBC driver (#35798)
Move classes under the same package to avoid internal classes being
 exposed to the outside. Remove public visibility outside 3 classes:
 EsDriver, EsDataSource and EsTypes.
The driver only has one package, namely org.elasticsearch.xpack.sql.jdbc
Use Es prefix for classes to ease name conflict and indicate their
 destination

Fix #35437
2018-11-28 12:48:26 +02:00
Jason Tedor c42d9d91c9
Deprecate X-Pack centric SQL endpoints (#35964)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-27 22:16:21 -05:00
Michael Basnight 19ed17195f
Docs: DRY up indices docs (#35971)
This commit DRYs up the indices folder as well as fixing a few minor
mishaps that were in the docs.
2018-11-27 19:40:49 -06:00
Jason Tedor b84f1f6a3a
Deprecate X-Pack centric rollup endpoints (#35962)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-27 20:34:17 -05:00
Tal Levy a0b319a123 [DOCS] fix HLRC ILM doc misreferenced tag 2018-11-27 14:28:25 -08:00
Ioannis Kakavas 580b5baf21
Add realm information for Authenticate API (#35648)
- Add the authentication realm and lookup realm name and type in the response for the _authenticate API
- The authentication realm is set as the lookup realm too (instead of setting the lookup realm to null or empty ) when no lookup realm is used.
2018-11-27 23:35:42 +02:00
Tal Levy fe603e9163
[ILM] add HLRC docs to remove-policy-from-index (#35759)
This primarily introduces documentation for the HLRC
remove-policy-from-index action.
2018-11-27 13:28:58 -08:00
Zachary Tong 48fa251812
[Rollup] Add more diagnostic stats to job (#35471)
* [Rollup] Add more diagnostic stats to job

To help debug future performance issues, this adds the
 min/max/avg/count/total latencies (in milliseconds) for search
and bulk phase.  This latency is the total service time including
transfer between nodes, not just the `took` time.

It also adds the count of search/bulk failures encountered during
runtime.  This information is also in the log, but a runtime counter
will help expose problems faster

* review cleanup

* Remove dead ParseFields
2018-11-27 15:46:10 -05:00
Jeff Hajewski 49087f16f5 Adds deprecation logging to ScriptDocValues#getValues. (#34279)
`ScriptDocValues#getValues` was added for backwards compatibility but no
longer needed. Scripts using the syntax `doc['foo'].values` when
`doc['foo']` is a list should be using `doc['foo']` instead.

Closes #22919
2018-11-27 14:30:13 -05:00
Simon Willnauer ad1f0dccd4
Validate metdata on `_msearch` (#35938)
MultiSearchRequests issues through `_msearch` now validate all keys
in the metadata section. Previously unknown keys were ignored
while now an exception is thrown.

Closes #35869
2018-11-27 17:08:24 +01:00
Yogesh Gaikwad 31fdb76973
[DOCS] Document authorization_realms for Kerberos realm (#35927)
This commit adds documentation for authorization_realms
setting for the Kerberos realm and also corrects a typo in
existing documentation.

Co-authored-by: @A-Hall
2018-11-27 19:09:17 +11:00
Martijn van Groningen df1e02d0d5
[HLRC] Added support for CCR Put Auto Follow Pattern API (#35780)
This change also adds documentation for the Put Auto Follow Pattern API.

Relates to #33824
2018-11-27 08:53:22 +01:00
Tim Vernum 3435fc4613
HLRC: Add ability to put user with a password hash (#35844)
Update PutUserRequest to support password_hash (see: #35242)

This also updates the documentation to bring it in line with our more
recent approach to HLRC docs.
2018-11-27 15:07:24 +11:00
Gordon Brown 119835decd
Always enforce cluster-wide shard limit (#34892)
This removes the option to run a cluster without enforcing the
cluster-wide shard limit, making strict enforcement the default and only
behavior.  The limit can still be adjusted as desired using the cluster
settings API.
2018-11-26 17:05:12 -07:00
Christoph Büscher 4d525e3e33
[Docs] Align callouts in search.asciidoc (#35897) 2018-11-27 00:05:15 +01:00
Gordon Brown 5c11b8612a
Add HLRC docs for Explain Lifecycle (#35803)
Adds HLRC documentation for the Explain Lifecycle API.
2018-11-26 15:25:56 -07:00
Marios Trivyzas 3f7cae3f0d
SQL: Implement GREATEST and LEAST functions (#35879)
Add GREATEST(expr1, expr2, ... exprN) and LEAST(expr1, expr2, exprN)
functions which are in the family of CONDITIONAL functions.

Implementation follows PostgreSQL behaviour, so the functions return
`NULL` when all of their arguments evaluate to `NULL`.

Renamed `CoalescePipe` and `CoalesceProcessor` to `ConditionalPipe` and
`ConditionalProcessor` respectively, to be able to reuse them for
`Greatest` and `Least` evaluations. To achieve that `ConditionalOperation`
has been added to differentiate between the functionalities at execution
time.

Closes: #35878
2018-11-26 18:21:36 +01:00
Christophe Bismuth b95a4db6e6 Throw a parsing exception when boost is set in span_or query (#28390) (#34112) 2018-11-26 12:15:59 -05:00
Jake Landis 4b99a663c1
ingest: fix broken doc link 2018-11-26 10:34:42 -06:00
Ed Savage 13e11966ca
[HLRC][ML] Add delete expired data API (#35906)
Relates to #29827
2018-11-26 16:15:54 +00:00
Jake Landis 7f7b31723e
ingest: extended `if` documentation (#35044)
part of #33188
2018-11-26 09:35:45 -06:00
Marios Trivyzas b078e2970c
SQL: Implement null safe equality operator `<=>` (#35873)
This operator handles nulls in different way than the normal `=`.
If one of the operants is `null` and the other not it returns `false`.
If both operants are `null` it returns `true`. Therefore in contrary to
`=`, which returns `null` if at least one of the operants is `null`, this one
never returns `null` as a result.

Closes: #35871
2018-11-26 14:02:02 +01:00
Ioannis Kakavas acb65dd004 [DOCS] Fix missing include for get-roles 2018-11-26 11:45:39 +02:00
Ioannis Kakavas 8daa854f90
[HLRC] Add support for get roles API (#35787)
This commits adds support for the Get Roles API to the HLRC

Relates: #29827
2018-11-26 11:25:07 +02:00
Andrei Stefan 00e6fec718
SQL: XPack FeatureSet functionality (#35725)
* Introduced "client.id" parameter for REST requests
* Bug that made the Verifier run twice, fixed in the Analyzer
* Single node IT and unit testing
2018-11-26 07:11:43 +02:00
Tal Levy a0f8a22d53
[TEST] ILM skip checking _ilm/explain response (#35830)
relates #35562

Closes #35824.
2018-11-25 17:19:17 -08:00
Marios Trivyzas 410f570d5f
SQL: Implement NULLIF(expr1, expr2) function (#35826)
NULLIF returns null if the 2 expressions are equal or the
expr1 otherwise.

Closes: #35818
2018-11-23 22:19:27 +01:00
William Desportes a204d1cdff [Docs] Fix typo in datehistogram-aggregation.asciidoc (#35855) 2018-11-23 15:16:53 +01:00
David Roberts 3c059ee057
[HLRC][ML] Add ML find file structure API (#35833)
Relates to #29827
2018-11-23 06:58:05 +00:00
Martijn van Groningen 813e053f62
Fixed response classes in hlrc docs 2018-11-23 07:52:45 +01:00
Stéphane Campinas fb80d298de [Docs] Correct template example description #35829 2018-11-23 05:29:20 +01:00
David Roberts 6f46584380
[ML] Add docs for ML info endpoint (#35783)
This endpoint was not previously documented as it was not
particularly useful to end users.  However, since the HLRC
will support the endpoint we need some documentation to
link to.

The purpose of the endpoint is to provide defaults and
limits used by ML.  These are needed to fully understand
configurations that have missing values because the missing
value means the default should be used.

Relates #35777
2018-11-22 16:23:31 +00:00
Martijn van Groningen 96a741f2c2
[TEST] escape brackets
Relates to #35496
2018-11-22 14:58:42 +01:00
Mayya Sharipova b6014d971c
Forbid negative scores in functon_score query (#35709)
* Forbid negative scores in functon_score query

- Throw an exception when scores are negative in field_value_factor
function
- Throw an exception when scores are negative in script_score
function

Relates to #33309
2018-11-22 06:08:48 -05:00
Marios Trivyzas 92acf47c16
SQL: Implement NVL(expr1, expr2) (#35794)
Add NVL as alias to IFNULL as they have the same
behaviour. Add basic tests and docs.

Closes: #35782
2018-11-22 11:41:00 +01:00
Alan Woodward be8097f9ce
Improve docs for index_prefixes option (#35778)
This commit moves the documentation and examples for the `index_prefixes`
option on text fields to its own file, to bring it in line with other mapping 
parameters, and expands a bit on both.
2018-11-22 09:20:46 +00:00
Lisa Cawley 78209188bc
[DOCS] Adds more authorization info for CCR APIs (#35606) 2018-11-21 15:01:59 -08:00
Benjamin Trent 90a8e4b259
HLRC: ML Delete event from Calendar (#35760)
* HLRC: Delete event from calendar

* adjusting tests

* adjusting code to make it more readable
2018-11-21 16:22:04 -06:00
Marios Trivyzas d95d885bae
SQL: Implement ISNULL(expr1, expr2) (#35793)
Add ISNULL as an alias of IFNULL as they have the
same behaviour. Add basic test and docs.

Closes: #35781
2018-11-21 23:15:10 +01:00
Marios Trivyzas e179bd393d
SQL: Implement IFNULL variant of COALESCE (#35762)
IFNULL is a MySQL variant (also used in other DBs) which
takes only 2 arguments and returns the first one that is not null.

Closes: #35749
2018-11-21 17:07:07 +01:00
Alan Woodward 26cc8ff8c3
Add pointer to the index-phrases option in shingle filter docs (#35771)
We should be discouraging the use of shingle filters and instead pointing users to the
index-phrases parameter on text fields.
2018-11-21 15:27:11 +00:00
Ioannis Kakavas 25f83ae08c
[HLRC] Add support for get application privileges API (#35556)
This commits adds support for the Get Application Privileges
API to the HLRC

Relates: #29827
2018-11-21 16:38:17 +02:00
Martijn van Groningen 9b2ab064cf
[HLRC] Added support for CCR Resume Follow API (#35638)
This change also adds documentation for the Resume Follow API

Relates to #33824
2018-11-21 10:52:03 +01:00
Dimitrios Liappis 1cb578b435
[DOCS] Update sysctl instructions for Docker on Mac (#35755)
Recent Docker for Mac releases[1] have a different path to the tty for
accessing the console of the xhyve vm, required for altering the
`vm.max_map_count` sysctl.

Update instructions on how to enter the xhyve vm for altering the
`vm.max_map_count` sysctl setting on Docker for Mac.

Closes #34817 

[1]
https://forums.docker.com/t/is-it-possible-to-ssh-to-the-xhyve-machine/17426/13
2018-11-21 11:32:29 +02:00
Ed Savage 4f857c4f8d
[HLRC][ML] Add ML revert model snapshot API (#35750)
Relates to #29827
2018-11-21 09:10:37 +00:00
Alan Woodward f6a43b5939
Add a prebuilt ICU Analyzer (#34958)
The ICU plugin provides the building blocks of an analysis chain, but doesn't actually have a prebuilt analyzer. It would be a better for users if there was a simple analyzer that they could use out of the box, and also something we can point to from the CJK Analyzer docs as a superior alternative.

Relates to #34285
2018-11-21 09:00:48 +00:00
Martijn van Groningen a6647a20a9
[HLRC] Added support for CCR Unfollow API (#35693)
This change also adds documentation for the Unfollow API

Relates to #33824
2018-11-21 07:48:29 +01:00
lcawl fa6d8a5004 [DOCS] Fixes navtitle for CCR API 2018-11-20 17:31:01 -08:00
Marios Trivyzas b1818dbdce
SQL: [docs] Add documentation for COALESCE (#35740)
Follows: #35253
2018-11-21 01:43:05 +01:00
Lisa Cawley 04a087fe2f
[DOCS] Adds authorization info for CCR APIs (#35557) 2018-11-20 16:23:01 -08:00
Benjamin Trent 84db1e42c0
HLRC: ML Get Calendar Events (#35747)
* HLRC: ML Get Calendar Events

* Addressing PR comments
2018-11-20 16:40:31 -06:00
Mayya Sharipova 643bb20137
Add a new query type - ScriptScoreQuery (#34533)
* Add a new query type - ScriptScoreQuery

script_score query uses script to calculate document scores.
Added as a substitute for function_score with an intentation
to deprecate function_scoreq query.

```http
GET /_search
{
    "query": {
        "script_score" : {
            "query": {
                "match": { "message": "elasticsearch" }
            },
            "script" : {
              "source": "Math.log(2 + doc['likes'].value)"
            },
            "min_score" : 2
        }
    }
}
```

Add several functions to painless to be used inside script_score:

double rational(double, double)
double sigmoid(double, double, double)
double randomNotReproducible() 
double randomReproducible(String, int) 

double decayGeoLinear(String, String, String, double, GeoPoint)
double decayGeoExp(String, String, String, double, GeoPoint)
double decayGeoGauss(String, String, String, double, GeoPoint)

double decayNumericLinear(String, String, String, double, double)
double decayNumericExp(String, String, String, double, double)
double decayNumericGauss(String, String, String, double, double)

double decayDateLinear(String, String, String, double, JodaCompatibleZonedDateTime)
double decayDateExp(String, String, String, double, JodaCompatibleZonedDateTime)
double decayDateGauss(String, String, String, double, JodaCompatibleZonedDateTime)

Date functions only works on dates in  the default format and default time zone
2018-11-20 16:10:06 -05:00
Gordon Brown 17780ce07e
Add HLRC docs for Delete Lifecycle Policy (#35664)
Adds documenatation for the Delete Lifecycle Policy API to the HLRC
documentation.
2018-11-20 11:32:41 -07:00
Tal Levy d061b3999a
[ILM] HLRC-ILM Retry Lifecycle Policy docs (#35715)
this adds documentation for the retry method in the
high-level-ilm-rest-client.

this PR also renames retryLifecycleStep to retryLifecyclePolicy in the index-lifecycle-client
2018-11-20 07:05:27 -08:00
Benjamin Trent 7657e6d274
HLRC ML Add Event To Calendar API (#35704)
* HLRC: ML Adding Post event to calendar api

* Fixing tests and serialization

* removing unused import
2018-11-20 08:15:21 -06:00
Benjamin Trent d707838c02
HLRC: ML Delete job from calendar (#35713) 2018-11-20 07:43:34 -06:00
Ed Savage 844483a99a
[HLRC][ML] Add ML update model snapshot API (#35537) (#35694)
Relates to #29827
2018-11-20 12:18:29 +00:00
Takuro Wada 7b2d547e8e [Docs] Delete inappropriate backtick (#35722) 2018-11-20 10:08:32 +01:00
Simon Willnauer 29ef442841
Add a `_freeze` / `_unfreeze` API (#35592)
This commit adds a rest endpoint for freezing and unfreezing an index.
Among other cleanups mainly fixing an issue accessing package private APIs
from a plugin that got caught by integration tests this change also adds
documentation for frozen indices.
Note: frozen indices are marked as `beta` and available as a basic feature.

Relates to #34352
2018-11-20 08:03:24 +01:00
Jason Tedor 220c175484
Provide links to auto-follow docs (#35674)
This commit adds a link to the auto-follow docs from the CCR getting
started docs and the relevant CCR APIs.
2018-11-19 20:40:58 -05:00
Gordon Brown e012ac4649
Add HLRC docs for ILM Start/Stop/Status (#35672)
Adds HLRC documentation for Start ILM, Stop ILM, and ILM Status APIs, as
these APIs are tightly linked to each other.
2018-11-19 17:17:01 -07:00
Tal Levy 90c0d444cf
[ILM] DOCS: clarify interaction between rollover and min_age (#35663)
This adds an example to drive home the semantics of `min_age` and explain
how actions from one phase must complete before `min_age` is even tested

closes #34020.
2018-11-19 13:26:20 -08:00
Benjamin Trent 214bc96738
HLRC: ML Add Job to Calendar API (#35666) 2018-11-19 11:41:49 -06:00
Mayya Sharipova aaeb47d1f7
HLRC for _mtermvectors (#35266)
relates to #27205
2018-11-19 12:24:31 -05:00
Itamar Syn-Hershko 156b3cae15 [Docs] Fix filter context in script-query.asciidoc (#35677)
The docs say script queries are typically used in a filter context but
the example uses a boolean must clause.
2018-11-19 16:30:33 +01:00
Benjamin Trent bc7dea4480
ML: changing automatic check_window calculation (#35643)
* ML: changing automatic check_window calculation

* adding docs on how we calculate the default
2018-11-19 08:03:34 -06:00
Peter Dyson 6d8af9731d [Docs] Warn about searching across all fields wt. `query_string` (#35570)
Warn about potential performance impact when a large number of fields
is used with query string query and no default field.
2018-11-19 13:21:59 +01:00
Ignacio Vera ae6a33237f
HLRC: Add ML delete filter action (#35382)
* HLRC: Add ML delete filter action

It adds delete ML filter action to the high level rest client.

Relates #29827
2018-11-19 11:25:35 +01:00
Gordon Brown bb51cdb6de
Add HLRC docs for Get Lifecycle Policy (#35612)
Adds docs for the Get Lifecycle Policy API to the HLRC documentation.
2018-11-16 18:16:30 -07:00
Gordon Brown cce9648f9d
Align RolloverStep's name with other step names (#35655)
RolloverStep previously had a name of "attempt_rollover", which was
inconsistent with all other step names due it its use of an underscore
instead of a dash.
2018-11-16 17:42:48 -07:00
Julie Tibshirani c6a0904e0e
Deprecate types in count and msearch. (#35421)
* Deprecate types in count requests.
* Move RestCountAction to the 'search' package.
* Deprecate types in multi search requests.
* Add tests for types deprecation in the _search endpoint.
2018-11-16 13:04:43 -08:00
debadair fc63ea5357
[DOCS] Add beta warning to ILM pages. (#35571) 2018-11-16 10:49:55 -08:00
Julie Tibshirani 40ba4de5e6
Deprecate types in validate query requests. (#35575) 2018-11-16 08:59:04 -08:00
Martijn van Groningen 797cffbb68
[TEST] Replace fields in response with actual values
Relates to #35496
2018-11-16 10:18:14 +01:00
Przemyslaw Gomulka c6e2634bf8
Docs: HLRC: refactor bulk, migrate and reindex apis (#35413)
fix typos and refactor to DRY up documentation for bulk, reindex and migration apis
relates #35345
2018-11-16 08:58:13 +01:00
Tim Vernum 87a8b99724
HLRC: Add "_has_privileges" API to Security Client (#35479)
This adds the "hasPrivileges()" method to SecurityClient, including
request, response & async variant of the method.

Also includes API documentation.
2018-11-16 13:52:06 +11:00
Jun Ohtani 0cdfc4cd0a
[Doc] Add clarification to boolean query (#32575)
It isn't very clear how boosting query works.
Add explanation of positive/negative query.
2018-11-16 11:45:32 +09:00
Gordon Brown 3883e9bf4c
Split RolloverStep into Wait and Action steps (#35524)
RolloverAction will now periodically check the rollover conditions using
the Rollover API with the dry_run option as an AsyncWaitStep, then run
the rollover itself by calling the Rollover API with no conditions,
which will always roll over, as an AsyncActionStep. This will resolve
race condition issues in policies using RolloverAction.
2018-11-15 17:11:31 -07:00
Mayya Sharipova a8833b097b
Painless Context Doc: Add min should match example (#35423) 2018-11-15 15:10:49 -05:00
Mayya Sharipova 501c03e529
Painless Context Doc: Add filter context example (#35305) 2018-11-15 14:33:34 -05:00
Benjamin Trent f7ada9b29b
Add delayed datacheck to the datafeed job runner (#35387)
* ML: Adding missing datacheck to datafeedjob

* Adding client side and docs

* Making adjustments to validations

* Making values default to on, having more sensible limits

* Intermittent commit, still need to figure out interval

* Adjusting delayed data check interval

* updating docs

* Making parameter Boolean, so it is nullable

* bumping bwc to 7 before backport

* changing to version current

* moving delayed data check config its own object

* Separation of duties for delayed data detection

* fixing checkstyles

* fixing checkstyles

* Adjusting default behavior so that null windows are allowed

* Mentioning the default value

* Fixing comments, syncing up validations
2018-11-15 13:32:45 -06:00
debadair 19001e71cc
[DOCS] Updated release state for 7.0.0-alpha1 (#35596)
* [DOCS] Updated release state for 7.0.0-alpha1

* Update Versions.asciidoc
2018-11-15 08:22:13 -08:00
Ed Savage 2d948a001e
[HLRC][ML] Add ML delete model snapshot API (#35537)
Relates to #29827
2018-11-15 14:57:17 +00:00
Christoph Büscher bc5b1afb55
HLRC: Add parameters to stopRollupJob API (#35545)
With #34811 the API for stopping rollup jobs got two new url parameters
"wait_for_completion" and "timeout". This change adds these to the HLRC APIs as
well.

Relates to #34811
2018-11-15 10:50:48 +01:00
Clinton Gormley cb8bdeae68 Fixed bad link in ingest-geo-point 2018-11-15 09:20:04 +01:00
Tal Levy dc1821c707
explain geo_point mapping in geoip-processor (#29114)
simple docs change to add missing mapping explanation. Users may not be aware this is
a prerequisite for doing geo-queries on this enriched data.
2018-11-14 20:05:45 -08:00
Tal Levy 95a09ab2d6
[ILM] TEST: skip checking _ilm/explain response b/c flaky timing (#35562)
The docs are not resilient to timing issues where the ILM metadata is not set on newly 
created indices, so we shouldn't be so strict on the returned response
2018-11-14 16:45:06 -08:00
Deb Adair 91a38b8fe0 [DOCS] Fixed bad cross doc link. 2018-11-14 16:09:18 -08:00
debadair 38d6bb27f5
[DOCS] Added link to standalone ODBC driver doc. (#35526) 2018-11-14 15:23:28 -08:00
Tal Levy f70d4470f7
[ILM] getting started documentation (#35379) 2018-11-14 13:58:08 -08:00
Tal Levy 30a713cc3f
[ILM] add docs explaining high-level concepts (#35322)
add docs explaining high-level concepts
2018-11-14 09:23:50 -08:00
Benjamin Trent 803eccec11
HLRC: Adding ML Update Filter API (#35522)
* HLRC: Adding ml get filters api

* HLRC: Adding ML Update Filter API
2018-11-14 11:13:11 -06:00
Thiago Souza 98e075227b [Doc] Correct jdbc url example (#35513)
Adds a missing `?` to separate the url parameters.
2018-11-14 17:30:23 +01:00
Tanguy Leroux 5b7446bb5f
Add Delete Privileges API to HLRC (#35454)
This commit adds the Delete Privileges API to the high level REST
client.

Related to #29827
2018-11-14 14:04:30 +01:00
Ed Savage e7b7d52a6a
[HLRC][ML] Add ML get model snapshots API (#35487)
Relates #29827
2018-11-14 13:03:04 +00:00
Jim Ferenczi 72504c2512
Do not recommend to use the _id field in search_after docs (#35370)
The documentation of `search_after` recommends to use the `_id`
field as a tiebreaker for the sort without warning against
the additional memory required. This change changes the recommandation
to use a copy of the `_id` field with doc_values enabled.
2018-11-14 10:50:31 +01:00
Christoph Büscher 603d1a470f
Add stop rollup job support to HL REST Client (#34702)
This change adds support for stoping a rollup job to the High Level REST Client.

Relates to #29827
2018-11-14 05:40:42 +01:00
Zachary Tong c346a0f027
[Rollup] Add `wait_for_completion` option to StopRollupJob API (#34811)
This adds a `wait_for_completion` flag which allows the user to block 
the Stop API until the task has actually moved to a stopped state, 
instead of returning immediately.  If the flag is set, a `timeout` parameter
can be specified to determine how long (at max) to block the API
call.  If unspecified, the timeout is 30s.

If the timeout is exceeded before the job moves to STOPPED, a
timeout exception is thrown.  Note: this is just signifying that the API
call itself timed out.  The job will remain in STOPPING and evenutally
flip over to STOPPED in the background.

If the user asks the API to block, we move over the the generic
threadpool so that we don't hold up a networking thread.
2018-11-13 16:37:17 -05:00
Benjamin Trent b9eb5f7b63
HLRC: Adding ml get filters api (#35502)
* HLRC: Adding ml get filters api

* refactoring setId name
2018-11-13 14:53:32 -06:00
Colin Goodheart-Smithe c934fb087a
Adds HLRC docs for put lifecycle policy (#35457)
* Adds HLRC docs for put lifecycle policy

* Adds link to docs in client javadocs

* Fixes checkstyle

* Make the documentation use the right ack response
2018-11-13 20:40:53 +00:00
Lee Hinman 6cb2aef943
[ILM] Add documentation for error handling in ILM (#35426)
* [ILM] Add documentation for error handling in ILM

This adds some initial documentation for error handling and retrying failed
steps for index lifecycle management
2018-11-13 12:19:22 -07:00
lcawl 2ac4ef7a69 [DOCS] Fixes broken link 2018-11-13 10:04:26 -08:00
Jason Tedor 8b2f22ee07
Introduce CCR overview (#35436)
This commit introduces a basic overview for cross-cluster replication to
the docs.

Co-authored-by: "lcawl <lcawley@elastic.co>"
2018-11-13 12:15:37 -05:00
Luca Cavanna 0b7d18d733
HLRC: add support for get license basic/trial status API (#33176)
Relates to #29827
2018-11-13 15:52:59 +01:00
Igor Motov a76ac5729d
Rest HL client: Add watcher stats API (#35185)
Relates to #29827
2018-11-13 04:47:35 -10:00
Jason Tedor 4b5fbada9f
Fix reference to get follower stats API
This commit fixes a reference to the CCR get follower stats API in the
CCR getting started docs.
2018-11-13 09:14:26 -05:00
Przemyslaw Gomulka fd4cd80496
HLRC: migration api - upgrade (#34898)
Implement high level client for migration upgrade API. It should wrap
RestHighLevelClient and expose high level IndexUpgradeRequest (new),
IndexTaskResponse for submissions with wait_for_completion=false and
BulkByScrollResponse (already used) objects.

refers: #29827
2018-11-13 15:01:53 +01:00
Martijn van Groningen e81671dd45
[HLRC] Added support for CCR Put Follow API (#35409)
This change also adds documentation for the Put Follow API and
adds a CCR HLRC integration test.
2018-11-13 13:22:49 +01:00
Jason Tedor cbb16d5ffc
Remove extraneous spaces from CCR getting started
This commit removes some extraneous spaces that were committed to the
cross-cluster replication getting started docs.
2018-11-13 06:48:56 -05:00
Jason Tedor 92fef40dbf
Introduce CCR getting started guide (#35434)
This commit introduces a basic getting started guide for cross-cluster
replication to the docs.

Co-authored-by: "lcawl <lcawley@elastic.co>"
2018-11-13 06:45:00 -05:00
Lisa Cawley 9aeaceac4b
[DOCS] Clarify results_index_name description (#35463) 2018-11-12 13:08:57 -08:00
Zachary Tong 2da239fb5e
[HLRC] Add GetRollupIndexCaps API (#35102)
Also refactors the caps response tests a bit to share the same
abstract class to reduce duplication of test code
2018-11-12 15:05:03 -05:00
Jason Tedor 326b238852
Add docs for CCR stats API (#35439)
This commit adds API docs for the CCR stats endpoint.
2018-11-12 10:28:35 -05:00
Peter Dyson c8e685eb34 Clarify S3 repository storage class parameter (#35400)
Today it is unclear that the `storage_class` parameter to an S3 repository only
affects new objects and does not rewrite any existing objects. This commit
clarifies this point.
2018-11-12 12:04:33 +00:00
Jason Tedor eb8f346204
Fix beta label on CCR index page
This commit fixes the location of the beta label on the CCR index page.
2018-11-10 15:45:38 -05:00
Jason Tedor d84ff3765d
Mark CCR as beta in the docs
This commit marks CCR as beta in the docs. This beta label is applied to
all CCR pages.
2018-11-10 14:38:22 -05:00
Albert Zaharovits f2d7c94949
DOCS Add audit ignore settings to reference page (#35274)
Adds the logfile audit ignore policy settings synopsis
to the Auditing Reference page.
2018-11-09 16:58:10 +02:00
Tim Vernum b0acd709d7
DOCS: Add SecurityExtension breaking change (#35358)
We changed the way realm settings are defined, and this affects custom
realms in SecurityExtensions. This change adds those details to the
breaking changes docs.

Relates: #30241
2018-11-09 16:57:49 +11:00
Armin Braun 02b4e28534
#31608 Add S3 Setting to Force Path Type Access (#34721)
* SNAPSHOTS: Use Path Style Access in S3

* Use path style access pattern to fix #31608
* closes #31608
2018-11-09 05:07:26 +01:00
debadair b485e69a5f
ILM API ref edits (#35112)
* [DOCS] ILM API Ref edits

* [DOCS] Fixed endpoint for DELETE policy.

* [DOCS] Removed comparison to setting index.lifecycle.name to null.

* [DOCS] Fixed xrefs to explain API.
2018-11-08 15:26:27 -08:00
Lisa Cawley c88b2d7e5a
[DOCS] Cleans up monitoring configuration steps (#35111) 2018-11-08 13:53:09 -08:00
Mayya Sharipova 7e144b318a
Painless Context Doc: Add field context example (#35130)
* Painless Context Doc: Add field context example

relates to #34829
2018-11-08 14:54:20 -05:00
Jason Tedor 730ec1ddfb
Add more detailed OS name on Linux (#35352)
Today our OS information returned in node stats only returns a
high-level name of the OS (e.g., "Linux"). Yet, for some uses this is
too high-level and knowing at a finer level of granularity the
underlying OS can be useful. This commit extracts the pretty name on
Linux from /etc/os-release. This pretty name usually includes the Linux
vendor and the Linux vendor version number (e.g., Fedora 28).
2018-11-08 12:16:58 -05:00
Christoph Büscher 113af7996c
Make limit on number of expanded fields configurable (#35284)
Currently we introduced a hard limit of 1024 to the number of fields a query can
be expanded to in #26541. Instead of using a hard limit, we should make this
configurable. This change removes the hard limit check and uses the existing
`max_clause_count` setting instead.

Closes #34778
2018-11-08 17:04:40 +01:00
Zachary Tong 55c6481b6b
[Docs] Add painless context details for bucket_selector (#35162)
Adds docs for the bucket_selector context, an example
and corresponding doc test
2018-11-08 11:02:30 -05:00
Jim Ferenczi d96202a282 [DOCS] Fix missing callouts 2018-11-08 15:40:01 +01:00
Dominik Stadler d351422215 Add parent-aggregation to parent-join module (#34210)
Add `parent` aggregation, a special single bucket aggregation that joins children documents to their parent.
2018-11-08 14:13:00 +01:00
Albert Zaharovits 025a0c82e5
Remove deprecated audit settings (#35205)
Removes `.prefix` deprecated settings for the logfile
auditing and also documents it in the migrate asciidoc.
2018-11-08 14:06:47 +02:00
Russ Cam 848847d8c7 [Docs] Section header preceded by blank line (#34340) 2018-11-08 12:44:13 +01:00
Jeff Hajewski d00b23c8b1 Fixes fast vector highlighter docs per issue 24318. (#34190)
The `fvh` highlighter does not support span queries. This fix updates
the docs to add a warning stating the lack of span query support for
`fvh`.
2018-11-08 11:09:03 +01:00
Jason Tedor a6073f5130
Add docs on JNA temp directory not being noexec (#35355)
If the underlying mount point for the JNA temporary directory is mounted
noexec on Linux, then the JVM will not be able to map the native code in
as executable. This will prevent JNA from executing and will prevent
Elasticsearch from being able to execute some functions that rely on
native code (e.g., memory locking, and installing system call
filters). We do not want to get into the business of catching exceptions
and parsing messages towards this because these exception messages can
change on us. We also do not want to jump through a lot of hoops to
check the underlying mount point for noexec. Instead, we will rely on
documentation to address this problem. This commit adds to the important
system configuration section of the docs that the JNA temporary
directory is not on a mount point with the noexec mount option.
2018-11-07 22:25:37 -05:00
Jason Tedor 4f4fc3b8f8
Replicate index settings to followers (#35089)
This commit uses the index settings version so that a follower can
replicate index settings changes as needed from the leader.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2018-11-07 21:20:51 -05:00
Alpar Torok 5ae03195d3
Make version field names more meaningful (#35334)
* Consolidate the name of the qualified build version

* Field name in response should not be redundant
2018-11-07 18:36:02 +02:00
Martijn van Groningen 021f80517f
[CCR] Added HLRC support for pause follow API (#35216)
* Moved `AcknowledgedResponse` to core package
* Made AcknowledgedResponse not abstract and provided a default parser,
so that in cases when the field name is not overwritten then there
is no need for a subclass.

Relates to #33824
2018-11-07 17:08:35 +01:00
Chris Cho e572a21c4b [Docs] Improve Convert Processor description (#35280)
Sometimes users are confused about whether they can use the Convert Processor
for changing an existing fields type to other types even if the existing one is already
ingested. This confusion is from the first line of description. Changing this and also
adding a some detail to the code snippet.
2018-11-07 17:01:35 +01:00
Alpar Torok 8a85b2eada
Remove build qualifier from server's Version (#35172)
With this change, `Version` no longer carries information about the qualifier,
we still need a way to show the "display version" that does have both
qualifier and snapshot. This is now stored  by the build and red from `META-INF`.
2018-11-07 14:01:05 +02:00
Julie Tibshirani ec53288fc0
Remove include_type_name from the relevant APIs. (#35192)
We've decided that the bulk, delete, get, index, update, and search APIs should not
contain this request parameter, and we will instead accept both typed and typeless calls.
2018-11-06 14:33:48 -08:00
Jay Modi 6f6b265166
HLRC: add support for the clear realm cache API (#35163)
This change adds support for clearing the cache of a realm. The realms
cache may contain a stale set of credentials or incorrect role
assignment, which can be corrected by clearing the cache of the entire
realm or just that of a specific user.

Relates #29827
2018-11-06 13:12:24 -07:00
Zachary Tong 065264d57c
[Docs] Add painless context details for bucket_script (#35142)
Fleshes out the details of the bucket_script context, adds an example
and corresponding doc test
2018-11-06 12:18:29 -05:00
Gytis Šk 3ee37b425b Docs: Add section about range query for range type (#35222)
This makes their interaction more discoverable.
2018-11-06 10:49:12 -05:00
Tal Levy a85b4f42ca
[ILM] change remove-policy-from-index http method from DELETE to POST (#35268)
The remove-ilm-from-index API was using the DELETE http method
to signify that something is being removed. Although, metadata
about ILM for the index is being deleted, no entity/resource
is being deleted during this operation. POST is more in line with
what this API is actually doing, it is modifying the metadata for
an index. As part of this change, `remove` is also appended to the path 
to be more explicit about its actions.
2018-11-06 07:46:25 -08:00
Jeff Soloshy 14c8a483d5 [Docs] Minor formatting and wording fixes (#35278) 2018-11-06 07:52:13 +01:00
Tim Vernum 7cd27d1a96 [DOCS] Fix missing float in breaking changes 2018-11-06 16:59:11 +11:00
Tim Vernum 3776de5f20
HLRC: Add InvalidateToken security API (#35114)
This change adds the Invalidate Token API
(DELETE /_xpack/security/oauth2/token) to the Elasticsearch
High Level Rest Client.

Relates: #29827
2018-11-06 15:26:12 +11:00
Tim Vernum 574ec6686e
Include realm type in Security Realm setting keys (#30241)
This moves all Realm settings to an Affix definition.
However, because different realm types define different settings
(potentially conflicting settings) this requires that the realm type
become part of the setting key.

Thus, we now need to define realm settings as:

    xpack.security.authc.realms:
      file.file1:
        order: 0

      native.native1:
        order: 1

- This is a breaking change to realm config
- This is also a breaking change to custom security realms (SecurityExtension)
2018-11-06 14:56:50 +11:00
lcawl 8280596f16 [DOCS] Fixes formatting error in auditing settings 2018-11-05 15:30:35 -08:00
Alexander Zhukov 842809ef37 Doc: Drop an extra 'a' in snapshots.asciidoc (#35251) 2018-11-05 13:31:35 -05:00
Nik Everett 59a43180a5 Reindex: Clean up docs around multi-index
We have an example in `reindex`'s docs about copying from many indices
at once. It doesn't work at the moment because we only allow a single
type per index. We didn't notice it in the docs tests because those
tests didn't copy any documents. This change:
1. Adds documents to the docs tests to fully exercise the snippet.
2. Fixes the example by moving all copied documents to the same type.
3. Moves the note about id collisions and expands on it because it is
even more likely than before.

Closes #35150
2018-11-05 12:52:11 -05:00
Jake Landis c2766b65cf
ingest: raise visibility of ingest plugin documentation (#35048)
* move the set security user processor to the main documentation
* link to plugin processors

part of #33188
2018-11-05 11:44:10 -06:00
Lee Hinman 3473217563
Remove Joda usage from ILM (#35220)
This commit removes the Joda time usage from ILM and the HLRC components of ILM.
It also fixes an issue where using the `?human=true` flag could have caused the
parser not to work. These millisecond fields now follow the standard we use
elsewhere in the code, with additional fields added iff the `human` flag is
specified.

This is a breaking change for ILM, but since ILM has not yet been released, no
compatibility shim is needed.
2018-11-05 08:17:15 -07:00
Benjamin Trent a4442dacd7
HLRC: Add ML API PUT filter (#35175) 2018-11-05 08:56:53 -06:00
Mayya Sharipova a2c22ad7ae
Small corrections to HLRC doc for _termvectors (#35221)
Relates to #33447
2018-11-04 18:29:07 -05:00
Jack Conradson 44f08717ba
[Scripting] Make Max Script Length Setting Dynamic (#35184)
This changes the current script.max_size_in_bytes to be dynamic so it can be 
set through the cluster settings API. This setting is also applied to inline scripts 
in the compile method of ScriptService to prevent excessively long inline 
scripts from being compiled. The script length limit is removed from Painless as 
this is no longer necessary with the protection in compile.
2018-11-02 16:07:54 -07:00
Lisa Cawley 3c36ba1f5e
[DOCS] Move CCR overview pages (#34918) 2018-11-02 14:23:13 -07:00
Gordon Brown 9b3d581339 [DOCS] Remove link to nonexistent ILM API 2018-11-02 09:38:40 -06:00
Milan Mrdjen 34677b9c83 HLRC: Add document _count API (#34267)
Add `count()` api method, `CountRequest` and `CountResponse` classes to HLRC. Code in server module is unchanged.

Relates to #27205
2018-11-02 08:21:19 -05:00
Colin Goodheart-Smithe 8f592bda27
Removes link to deleted page 2018-11-02 12:27:51 +00:00
Daniel Mitterdorfer ccbe80c3a0
Introduce durability of circuit breaking exception
With this commit we differentiate between permanent circuit breaking
exceptions (which require intervention from an operator and should not
be automatically retried) and transient ones (which may heal themselves
eventually and should be retried). Furthermore, the parent circuit
breaker will categorize a circuit breaking exception as either transient
or permanent based on the categorization of memory usage of its child
circuit breakers.

Closes #31986
Relates #34460
2018-11-02 13:12:44 +01:00
Colin Goodheart-Smithe fc6e1f7f3f
Merge branch 'master' into index-lifecycle 2018-11-02 10:56:35 +00:00
lcawl f97c9b6af2 [DOCS] Fixes location of beta markup 2018-11-01 14:27:49 -07:00
Julie Tibshirani fda173d7aa
Add a note around using separate indices for percolator queries and documents. (#35109) 2018-11-01 12:41:07 -07:00
lcawl 781130c7ab [DOCS] Fixes formatting of dynamic settings links 2018-11-01 11:52:41 -07:00
Tal Levy c6c01425bb Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-11-01 11:38:42 -07:00
Tal Levy c3cf7dd305 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-11-01 10:13:02 -07:00
Jake Landis 77fab62ebe
ingest: add common options to each processor's documentation (#35091)
* adds `if`, `on_failure`, `tag`, and `ignore_failure` to table for each processor

part of #33188

* added ingore_failure

* fix whitespace noise
2018-11-01 11:08:04 -05:00
Itamar Syn-Hershko d46f3528c0 [Docs] Updating URL for Openstack Swift plugin (#34136)
The repository plugin for Openstack Swift was developed originally by Wikimedia foundation but is now retired. 
This changes the link to the repo where the actively maintained fork lives now.
2018-11-01 11:57:55 +01:00
Ke Li 14f540e8e6 Deprecate unicodeSetFilter in favour of unicode_set_filter (#29215) 2018-11-01 10:06:51 +00:00
Alexandru Rusanescu f3e150b0ea [Docs] Update query_cache.asciidoc (#33340)
Add note about non-visibility of cache content.
2018-11-01 10:22:36 +01:00
lipsill d181d1bab1 Remove deprecated url parameters `_source_include` and `_source_exclude` (#35097)
Removes `_source_include` and `_source_exclude` url parameters. 
These parameters have been deprecated in #33475.

Closes #22792
2018-10-31 17:11:59 -04:00
Qian Hong f27251dfcb [Docs] Improve Redindex/Update/Delete By Query. (#35127)
Fix a copy paste error and improve formatting in
Redindex/Update/Delete By Query and fix formula in Update By Query.
2018-10-31 14:04:48 -04:00
Tal Levy d5d28420b6 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-31 10:47:07 -07:00
Luca Cavanna ef5181c678
Allow to enable pings for specific remote clusters (#34753)
When we connect to remote clusters, there may be a few more routers/firewalls in-between compared to when we connect to nodes in the same cluster. We've experienced cases where firewalls drop connections completely and keep-alives seem not to be enough, or they are not properly configured. With this commit we allow to enable application-level pings specifically from CCS nodes to the selected remote nodes through the new setting `cluster.remote.${clusterAlias}.transport.ping_schedule`.  The new setting is similar `transport.ping_schedule` but it does not affect intra-cluster communication, pings are only sent to specific remote cluster when specifically enabled, as they are disabled by default.

Relates to #34405
2018-10-31 17:32:53 +01:00
Lisa Cawley 72cb885abc
[DOCS] Updates monitoring diagram (#35096) 2018-10-31 09:18:14 -07:00
Mayya Sharipova 0b8465f2c0
Update type to "seat" (#35086)
otherwise this throws an exception: java.lang.IllegalArgumentException: Rejecting mapping update to [seats] as the final mapping would have more than 1 type: [seat, _doc]

May be, later for the types removal, we can modify `seats.json` to have a type `_doc` instead of `seat`
2018-10-31 12:06:48 -04:00
Sue Gallagher 1ce3c92a2d [DOCS] Add info on calendar vs fixed interval. (#31638)
Extensive edit to add additional information on the difference between calendar intervals and fixed-length intervals.
2018-10-31 10:16:36 -04:00
Albert Zaharovits bdd8460906
HLRest: add security authenticate API (#33552)
This adds the security `_authenticate` API to the HLREST client.
It is unlike some of the other APIs because the request does not
have a body.
The commit also creates the `User` entity. It is important
to note that the `User` entity does not have the `enabled`
flag. The `enabled` flag is part of the response, alongside
the `User` entity.
Moreover this adds the `SecurityIT` test class 
(extending `ESRestHighLevelClientTestCase`).

Relates #29827
2018-10-31 14:49:33 +02:00
debadair e540abd551
[DOCS] Fixed edit links for ILM APIs and added the APIs to the REST ref (#35070)
* [DOCS] Fixed edit links for ILM APIs and added the APIs to the REST API section.

* [DOCS] Fixed link to ILM APIs.
2018-10-30 18:27:43 -07:00
Tal Levy 5141084048
rename CRUD api REST path prefix _ilm to _ilm/policy (#35056)
This PR renames the CRUD APIS for ILM

GET _ilm/<policy>, _ilm -> _ilm/policy/<policy>, _ilm/policy
PUT _ilm/<policy> -> _ilm/policy/<policy>
DELETE _ilm/<policy> -> _ilm/policy/<policy>

closes #34929.
2018-10-30 16:19:05 -07:00
Tal Levy f5323cd86e
rename ILM Phase "minimum_age" to "min_age" (#35058)
closes #34927.
2018-10-30 12:54:02 -07:00
Nik Everett 0f9a7b3344 Docs: Fix build after moving SQL QA
I moved SQL's QA in #34496 to make it easier to run all of SQL tests in
one go but forgot to update the docs build. This fixes that.
2018-10-30 15:47:25 -04:00
Nik Everett 2b2e208c4c
Docs: Remove range notation from random score docs (#35093)
The `random_score` function produces values between 0 (inclusive) and 1
(exclusive) and documented it with fancy methematical range notation. It
is so fancy I thought it was a typo. This changes the documentation to
use words.

Relates to #35084
2018-10-30 14:12:59 -04:00
Zachary Tong f9dd33a0b9
[Rollup] Proactively resolve index patterns in RollupSearch endoint (#34930)
This changes the RollupSearch endpoint to proactively resolve index
patterns.  If the index pattern(s) match more than one rollup index,
an exception is throw as before.  But if the pattern only matches one
rollup index, execution is allowed to continue (unlike before where
it would assume all patterns were for raw data).

This also allows the search endpoint to resolve aliases that point to
a rollup index.

Also tweaks the documentation to make this clear.

Closes #34828
2018-10-30 13:50:50 -04:00
Julie Tibshirani 70da490f34
Remove some documentation that only makes sense with multiple types. (#35066)
* Remove a tip about ignore_above that only makes sense with multiple types.
* Remove a line from the percolator documentation that refers to multiple types.
2018-10-30 10:19:12 -07:00
Tal Levy 18c72e86c5 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-30 08:09:57 -07:00
Andy Bristol b8280ea7cc
median absolute deviation agg (#34482)
This commit adds a new single value metric aggregation that calculates
the statistic called median absolute deviation, which is a measure of
variability that works on more types of data than standard deviation

Our calculation of MAD is approximated using t-digests. In the collect
phase, we collect each value visited into a t-digest. In the reduce
phase, we merge all value t-digests, then create a t-digest of
deviations using the first t-digest's median and centroids
2018-10-30 07:22:52 -07:00
Nik Everett cc4dd98872 Docs: Fix build for java-api
The java-api docs had some broken references. Mostly because it is
difficult to build them so they are difficult to check.

Relates to #34528
2018-10-30 09:14:07 -04:00
Alan Woodward cf24231b1b
Add missing full stop in Bootstrap checks doc (#35078)
Fixes #35075
2018-10-30 11:43:43 +00:00
markwalkom 5e0c9b7c1a Update reindex.asciidoc (#35062)
Highlighted that `_source` needs to exist in the source index.
2018-10-30 10:25:05 +01:00
Przemyslaw Gomulka 995bf0ee66
Bulk Api support for global parameters (#34528)
Bulk Request in High level rest client should be consistent with what is
possible in Rest API, therefore should support global parameters. Global
parameters are passed in URL in Rest API.

Some parameters are mandatory - index, type - and would fail validation
if not provided before before the bulk is executed.
Optional parameters - routing, pipeline.

The usage of these should be consistent across sync/async execution,
bulk processor and BulkRequestBuilder

closes #26026
2018-10-30 09:08:12 +01:00
Ken MacInnis d67c88fa12 [Docs] Update update-settings.asciidoc (#35073)
Add explanation for ?include_defaults parameter and link to
Cluster Get Settings reference
2018-10-30 08:58:32 +01:00
Gordon Brown 794d4fa879
Label required scripts in Scripted Metric Agg docs (#35051)
When combine_script and reduce_script were made into required
parameters for Scripted Metric aggregations in #33452, the docs were
not updated to reflect that. This marks those parameters as required
in the documentation.
2018-10-29 15:13:14 -06:00
Tal Levy c9e4d26a53 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-29 14:03:55 -07:00
Nik Everett 3051e37e45 Docs: Document refresh for reindex and friends
Reindex and friends don't support `refresh=wait_for` and this documents
that.
2018-10-29 13:58:55 -04:00
lipsill 6df1c9e818 Deprecate `_source_include` and `_source_exclude` url parameters (#33475)
Deprecates `_source_include` and `_source_exclude` url parameters
in favor of `_source_inclues` and `_source_excludes` because those
are consistent with the rest of Elasticsearch's APIs.

Relates to #22792
2018-10-29 12:06:38 -04:00
Nik Everett 6fd7ea71f1
HLRC: DRY up remaining CRUD docs (#34925)
This further applies the pattern set in #34125 to reduce copy-and-paste
in the multi-document CRUD portion of the High Level REST Client docs.
It also adds line wraps to snippets that are too wide to fit into the box
when rendered in the docs, following up on the work started in #34163.
2018-10-29 10:32:17 -04:00
Jason Tedor 26f5c509af
Fix CCR API specification (#34963)
This commit fixes two issues with the CCR API specification:
 - remove the CCR stats endpoint, it is not currently implemented
 - fix the documentation links
2018-10-29 09:37:13 -04:00
Christoph Büscher 389910f11d
[Docs] Add migration note about expanded fields limit (#34920)
Adds a note to warn users about the limit introduced in #26541.
2018-10-29 10:23:07 +01:00
David Roberts c455be7bc2
[ML] Rename the json file structure to ndjson (#34901)
The file structure finder endpoint can find the NDJSON
(newline-delimited JSON) file format, but called it
`json`.  This change renames the `format` for this file
structure to `ndjson`, which is more precise and will
hopefully avoid confusion.
2018-10-29 10:06:12 +01:00
Martijn van Groningen f13d529448
Fixed ccr docs broken links. 2018-10-29 09:29:28 +01:00
Christoph Büscher 3a464bdee8
Add start rollup job support to HL REST Client (#34623)
This change adds support for starting a rollup job to High Level REST Client.

Relates to #29827
2018-10-29 09:28:11 +01:00
Martijn van Groningen 1801518527
[CCR] Refactor stats APIs (#34912)
* Changed the auto follow stats to also include follow stats.
* Renamed the auto follow stats api to stats api and changed its url path
  from `/_ccr/auto_follow/stats` `/_ccr/stats`.
* Removed `/_ccr/stats` url path for the follow stats api, which makes
  the index parameter a required parameter.
* Fixed docs.
2018-10-29 07:45:27 +01:00
Tim Vernum 9c27b407f0
HLRC: Add security Create Token API (#34791)
This adds the Create Token API (POST /_xpack/security/oauth2/token)
to the High Level Rest Client.

Relates: #29827
2018-10-29 17:17:56 +11:00
Yogesh Gaikwad dc5bfe3a00
[DOCS] Fix typo in get role mappings API (#34952) 2018-10-29 11:15:42 +11:00
Yogesh Gaikwad a5ee134c40
[HLRC] Add support for get role mappings API (#34637)
This commit adds support for get role mappings API
in HLRC.
2018-10-29 10:12:13 +11:00
debadair 544c220942
[DOCS] Added attributes for HLRC javadoc packages. (#34934) 2018-10-26 15:32:15 -07:00
Benjamin Trent 052dfa5646
HLRC: Adding Update datafeed API (#34882)
* HLRC: Adding Update datafeed API

* Addressing unused import

* Adjusting docs and fixing minor comments

* fixing comment
2018-10-26 16:44:12 -05:00
Tal Levy d8322ca069 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-26 12:46:21 -07:00
Tal Levy e737ea7d4a
remove old doc placeholder and migrate ilm docs to top-level (#34615)
we are restructuring the docs, this migrates ILM docs outside of the x-pack doc structure.
2018-10-26 12:19:52 -07:00
Boaz Leskes a086c665a3 HLREST: Add Clear Roles Cache API (#34187)
Adds support for the Clear Roles Cache API to the High Level Rest
Client. As part of this a helper class, NodesResponseHeader, has been
added that enables parsing the nodes header from responses that are
node requests.

Relates to #29827
2018-10-26 12:16:44 -06:00
Jason Tedor fdfdbe486d
Introduce cross-cluster replication API docs (#34726)
This commit is our first introduction to cross-cluster replication
docs. In this commit, we introduce the cross-cluster replication API
docs. We also add skelton docs for additional content that will be added
in a series of follow-up commits.
2018-10-26 11:23:35 -04:00
markharwood 3181083781
HLRC - add support for source exists API (#34519)
HLRC - add support for source exists API
API re-uses the GetRequest object (following the precedent set by the plain “exists” api).

Relates to #27205
2018-10-26 14:21:35 +01:00
Albert Zaharovits 3f1fec1813
[Docs] audit logfile structured format (#34584)
Documents the new structured logfile format for auditing
that was introduced by #31931. Most changes herein
are for 6.x . In 7.0 the deprecated format is gone and a
follow-up PR is in order.
2018-10-26 15:19:35 +03:00
Tal Levy 810cd46a30 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-25 14:35:33 -07:00
Senthil e27a0d5b0d Docs: Align prose with snippet (#34839)
Modified example text to include multiple index patterns used in the code
2018-10-25 16:51:18 -04:00
Stéphane Campinas 27c4d63340 document the search context is freed if the scroll is not extended (#34739)
The `fetchPhaseShouldFreeContext` returns true when there is a scroll context but the scroll parameter is null, thus freeing the search context.

183c32d4c3/server/src/main/java/org/elasticsearch/search/SearchService.java (L491)
2018-10-25 16:49:08 -04:00
Lisa Cawley 312df5546c
[DOCS] Updates Elasticsearch monitoring tasks (#34339) 2018-10-25 10:32:50 -07:00
Lee Hinman 3e7042832a Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-25 11:00:36 -06:00
Christoph Büscher c0c6a28e86
[Docs] Add `indices.query.bool.max_clause_count` setting (#34779)
This change adds a section about the global search setting
`indices.query.bool.max_clause_count` that limits the number of boolean clauses
allowed in a Lucene BooleanQuery.

Closes #19858
2018-10-25 17:59:59 +02:00
Jason Tedor 153157e56d
Enable ingest attachment docs tests on JDK 11 (#34770)
These were disabled because ingest attachement was not playing well with
JDK 11. We have addressed that by upgrading the Tika dependency, yet
forgot to reenable these. This commit enables these tests again.
2018-10-24 23:17:45 -04:00
Mayya Sharipova d7afd7c123 Reduce the number of callouts to 15 in docs
relates to #33447
2018-10-24 17:57:35 -04:00
Mark Tozzi d94406a68a
HLRC: Deactivate Watch API (#34192)
Relates to #29827
2018-10-24 17:01:22 -04:00
Mayya Sharipova 3211760e2c Correct term-vectors.asciidoc
relates to #33447
2018-10-24 16:35:20 -04:00
Benjamin Trent cd27b0b996
Revert "Rollup add default metrics to histo groups (#34534)" (#34815)
This reverts commit 4236358f5d.
2018-10-24 14:25:10 -05:00
Mayya Sharipova bf4d90a5dc
HLRC API for _termvectors (#33447)
* HLRC API for _termvectors

relates to #27205
2018-10-24 14:27:22 -04:00
Gordon Brown da20dfd81c
Add cluster-wide shard limit warnings (#34021)
In a future major version, we will be introducing a soft limit on the
number of shards in a cluster based on the number of nodes in the
cluster. This limit will be configurable, and checked on operations
which create or open shards and issue a warning if the operation would
take the cluster over the limit.

There is an option to enable strict enforcement of the limit, which
turns the warnings into errors.  In a future release, the option will be
removed and strict enforcement will be the default (and only) behavior.
2018-10-23 16:35:10 -06:00
Armin Braun f79bdec58a INGEST: Document Pipeline Processor (#33418)
* Added documentation for Pipeline Processor
* Relates #33188
2018-10-23 15:36:57 -05:00
Zachary Tong 299d044bfc
Collapse pipeline aggs into single package (#34658)
- Restrict visibility of Aggregators and Factories
- Move PipelineAggregatorBuilders up a level so it is consistent with
AggregatorBuilders
- Checkstyle line length fixes for a few classes
- Minor odds/ends (swapping to method references, formatting, etc)
2018-10-23 16:01:01 -04:00
Jake Landis 067d135bfd
ingest: document index.default_pipeline (#34500) 2018-10-23 13:49:25 -05:00
Jake Landis a8e1ee34ca
ingest: document fields that support templating (#34536)
This change also updates many of the examples to use ecs as the example.
Some additional minor improvements are also included.

Part of #33188
2018-10-23 13:28:44 -05:00
Jake Landis c447fc258a
ingest: documentation for the drop processor (#34570) 2018-10-23 12:30:23 -05:00
Tal Levy 62ac2fa5ec Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-23 09:43:46 -07:00
Zachary Tong 4dbf498721
[Rollup] Job deletion should be invoked on the allocated task (#34574)
We should delete a job by directly talking to the allocated 
task and telling it to shutdown. Today we shut down a job 
via the persistent task framework. This is not ideal because, 
while the job has been removed from the persistent task 
CS, the allocated task continues to live until it gets the 
shutdown message.

This means a user can delete a job, immediately delete 
the rollup index, and then see new documents appear in
 the just-deleted index. This happens because the indexer
 in the allocated task is still running and indexes a few 
more documents before getting the shutdown command.

In this PR, the transport action is changed to a TransportTasksAction, 
and we invoke onCancelled() directly on the matching job. 
The race condition still exists after this PR (albeit less likely), 
but this was a precursor to fixing the issue and a self-contained
chunk of code. A second PR will followup to fix the race itself.
2018-10-23 12:23:22 -04:00
Albert Zaharovits 46b49b01c4
[DOCS] .Security index is never auto created (#34589) 2018-10-23 18:51:59 +03:00
Armin Braun f0f732908e
INGEST: Document Processor Conditional (#33388)
* INGEST: Document Processor Conditional

Relates #33188
2018-10-23 17:37:30 +02:00
Costin Leau ca6808e55d
SQL: Support pattern against compatible indices (#34718)
Extend querying support on multiple indices from being strictly
identical to being just compatible.
Use FieldCapabilities API (extended through #33803) for mapping merging.

Close #31837 #31611
2018-10-23 17:07:51 +03:00
Przemyslaw Gomulka f91cc13c18
A small typo in migration-assistance doc (#34704)
typo - missing word "be"
2018-10-23 15:28:25 +02:00
Marios Trivyzas 4a8386f271
SQL: Implement IN(value1, value2, ...) expression. (#34581)
Implement the functionality to translate the
`field IN (value1, value2,...)` expressions to proper Lucene queries
or painless script or local processors depending on the use case.

The `IN` expression can be used in SELECT, WHERE and HAVING clauses.

Closes: #32955
2018-10-23 14:28:23 +02:00
Frederik Bosch 183c32d4c3 [DOCS] Force Merge: clarify execution and storage requirements (#33882) 2018-10-23 12:25:04 +02:00
Marios Trivyzas e9e140790a
SQL: Implement `CONVERT`, an alternative to `CAST` (#34660)
`CONVERT` works exactly like cast with slightly different syntax:
`CONVERT(<value>, <data_type)` as opposed to `CAST(<value> AS <data_type>)`

Moreover it support format of the MS-SQL data types `SQL_<type>`,
e.g.: `SQL_INTEGER`

Closes: #34513
2018-10-23 11:21:15 +02:00
Tal Levy 67bfdb16ad Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-22 13:09:37 -07:00
Julie Tibshirani f854330e06
Make sure to use the type _doc in the REST documentation. (#34662)
* Replace custom type names with _doc in REST examples.
* Avoid using two mapping types in the percolator docs.
* Rename doc -> _doc in the main repository README.
* Also replace some custom type names in the HLRC docs.
2018-10-22 11:54:04 -07:00
Zachary Tong d981746142
[Docs] clarification about cardinality accuracy (#34616)
Adds a bit more clarification about how accuracy is dependent
on the dataset in question.

Closes #18231
2018-10-22 13:15:45 -04:00
Julie Tibshirani fbb9ac34f9
Deprecate type exists requests. (#34663) 2018-10-22 08:46:11 -07:00
Christoph Büscher e869f9a78c [Docs] Update synonym-tokenfilter.asciidoc (#34706)
Remove ugly double-dot.
2018-10-22 17:18:29 +02:00
Jason Tedor bf5f0af491
Use trial license in docs tests (#34673)
This commit switches to using a trial license in the docs tests that run
on the default distribution. This is needed so that docs tests can be
executed against non-basic features.
2018-10-21 06:23:42 -04:00
Ryan Ernst 222652dfce
Scripting: Convert script fields to use script context (#34164)
This commit removes the use of SearchScript for script fields and adds
a new FieldScript.
2018-10-20 16:33:49 -07:00
Jason Tedor aebb855f38
Add missing word to remote clusters docs
This commit adds a missing word in the initial section of the remote
clusters docs.
2018-10-20 12:11:41 -04:00
Jason Tedor aca8c705ae
Fix name of remote clusters reference
This commit fixes the name of the reference to the remote clusters
module on the modules page by adding a missing character.
2018-10-20 11:18:27 -04:00
Jason Tedor ab5a0b15f6
Fix formatting of remotes clusters module section
This commit fixes the formatting of the remote clusters module section
on the modules page. We add a missing newline and wrap the columns at
80.
2018-10-20 09:04:48 -04:00
Jason Tedor 91dbb22950
Fix reference to remote clusters docs
This commit fixes a reference to the remote clusters docs from the
modules page.
2018-10-20 09:02:00 -04:00
Jason Tedor b2dc58450e
Separate remote clusters docs from CCS (#34612)
With remote clusters taking on a larger role, we have make the
infrastructure more generic than being tied to cross-cluster search
(CCS). We want to refer to the remote clusters configuration in the
cross-cluster replication (CCR) docs. Yet, these docs are still tied to
CCS. This commit extracts the remote clusters docs from CCS (with some
wording changes to make them more general) so that we can refer to them
in the CCR docs.
2018-10-20 08:33:59 -04:00
Yannick Welsch 9fb2f7cc20
HLRC: Delete role API (#34620)
Adds the "Delete role" API to the high-level REST client.
2018-10-20 12:11:36 +02:00
Jack Conradson 842bf55497 [Docs] Fixes a bug in the rest-client watcher docs related to
cross-link issue.
2018-10-19 11:52:15 -07:00
Igor Motov 94bde37bcf
Geo: Don't flip longitude of envelopes crossing dateline (#34535)
When a envelope that crosses the dateline is specified as a part of
geo_shape query is parsed it shouldn't have its left and right points
flipped.

Fixes #34418
2018-10-19 13:53:54 -04:00
Julie Tibshirani 67652b5355
Remove references to multiple types in the search documentation. (#34625) 2018-10-19 09:47:34 -07:00
Jack Conradson 1100942455
[HLRC] Add Start/Stop Watch Service APIs. (#34317) 2018-10-19 09:42:57 -07:00
Costin Leau 52104aac27
SQL: Introduce support for NULL values (#34573)
Make SQL aware of missing and/or unmapped fields treating them as NULL
Make _all_ functions and operators null-safe aware, including when used
in filtering or sorting contexts
Add missing and null-safe doc value extractor
Modify dataset to have null fields spread around (in groups of 10)
Enforce missing last and unmapped_type inside sorting
Consolidate Predicate templating and declaration
Add support for Like/RLike in scripting
Generalize NULLS LAST/FIRST
Introduce early schema declaration for CSV spec tests: to keep the doc
snippets in place (introduce schema:: prefix for declaration)
upfront.

Fix #32079
2018-10-19 16:44:33 +03:00
Colin Goodheart-Smithe 84ef91529c
Merge branch 'master' into index-lifecycle 2018-10-19 13:24:04 +01:00
Benjamin Trent 4236358f5d
Rollup add default metrics to histo groups (#34534)
* Rollup: Adding default metrics for histo group timefield (#34379)

* Rollup: Adding default histo metrics and tests

* fixing failing client side test with new default values

* Adding HLRC docs for default values

* Addressing PR comments

* Removing value_count default agg

* Updating docs for rollups

* Minor interval change
2018-10-19 07:23:25 -05:00
Jim Ferenczi 7b49beb9b0
Fix threshold frequency computation in Suggesters (#34312)
The `term` and `phrase` suggesters have different options to filter candidates
based on their frequencies. The `popular` mode for instance filters candidate
terms that occur in less docs than the original term. However when we compute this threshold
we use the total term frequency of a term instead of the document frequency. This is not inline
with the actual filtering which is always based on the document frequency. This change fixes
this discrepancy and clarifies the meaning of the different frequencies in use in the suggesters.
It also ensures that the threshold doesn't overflow the maximum allowed value (Integer.MAX_VALUE).

Closes #34282
2018-10-19 13:33:19 +02:00
markharwood fe623acf66
Docs - removed experimental/beta markers from adjacency matrix aggregation (#34599) 2018-10-19 09:33:59 +01:00
markharwood 2a413abb0b
Docs - remove experimental marker from significant_text aggregation (#34598) 2018-10-19 09:32:02 +01:00
Yogesh Gaikwad 39a6163316
[HLRC] Add support for Delete role mapping API (#34531)
Building on expression dsl and create role mapping API, this
commit adds support for delete role mapping API to high level client.
2018-10-19 13:46:26 +11:00
Christoph Büscher 7cc3ac0fc4
[Docs] Add _cache/clear parameter examples (#34614)
Add example for selectively clearing just the request, query or fielddata cache
and for selectively clearing the cache for specific fields.

Closes #34287
2018-10-19 00:44:37 +02:00
Zachary Tong 45546e71c2
Add GetRollupCaps API to high level rest client (#32880)
Adds GetRollupCaps API to the HLRC, and tweaks some of the
Caps objects to be immutable.  Also various style tweaks
2018-10-18 17:12:38 -04:00
Benjamin Trent fa973007b8
[DOCS] Adds new lazy ml node setting (#34600)
* Adding new xpack.ml.max_lazy_ml_nodes setting to docs

* Fixing docs, making it clearer what the setting does

* Adding note about external process need
2018-10-18 16:11:36 -05:00
Nik Everett 2c0f67318a
Docs: Reenable rollup docs tests (#34564)
We'd disabled them because we didn't have a way to clean up after each
test. I implemented #34342 which adds the clean ups so now we can
re-enable the tests.

In the `setup` sections we have to use `raw` requests instead of
`x-pack` requests because we don't have the json config for x-pack.

Closes #33319
2018-10-18 15:24:02 -04:00
Ryan Ernst d445785f1a
Scripting: Convert domainSplit function for ML to whitelist (#34426)
This commit moves the definition of domainSplit into java and exposes it
as a painless whitelist extension. The method also no longer needs
params, and version which ignores params is added and deprecated.
2018-10-17 15:54:21 -07:00
Colin Goodheart-Smithe 90f7cec7a5
Merge branch 'master' into index-lifecycle 2018-10-17 18:22:23 +01:00
Andy Bristol 18aa1c1381
add start trial API to HLRC (#33406)
Introduces client-specific request and response classes that do not
depend on the server

The `type` parameter is named `licenseType` in the response class to be
more descriptive. The parts that make up the acknowledged-required
response are given slightly different names than their server-response
types to be consistent with the naming in the put license API

Tests do not cover all cases because the integ test cluster starts up
with a trial license - this will be addressed in a future commit
2018-10-17 08:02:04 -07:00
Nik Everett 030e8c8fe3 Docs: Tweak upgrade docs
Tweak the upgrade instructions for moving from pre-6.3-with-x-pack to
post-6.3-default distribution. Specifically, you have to remove the
x-pack plugin before upgrading because 6.4 doesn't understand how to
remove it.

Relates to #34307
2018-10-17 08:17:12 -04:00
Tal Levy fbe8dc014c Merge branch 'master' into index-lifecycle 2018-10-16 13:58:53 -07:00
lipsill 9b5eaafc24 HLRC: Documentation examples cleanup (#34009)
* Replace deprecated field `code` with `source` for stored scripts (#25127)
* Replace examples using the deprecated endpoint `{index}/{type}/_search`
with `{index}/_search` (#29468)
* Use a system property to avoid deprecation warnings after the Update
Scripts have been moved to their own context (#32096)
2018-10-16 10:46:14 -04:00
Vladimir Dolzhenko 230ad5339b
HLRC: Add support for XPack Post Start Basic Licence API (#33606)
Relates to #29827
2018-10-16 15:21:30 +02:00