Commit Graph

5727 Commits

Author SHA1 Message Date
David Turner 452f7f67a0
Defer reroute when starting shards (#44539)
Today we reroute the cluster as part of the process of starting a shard, which
runs at `URGENT` priority. In large clusters, rerouting may take some time to
complete, and this means that a mere trickle of shard-started events can cause
starvation for other, lower-priority, tasks that are pending on the master.

However, it isn't really necessary to perform a reroute when starting a shard,
as long as one occurs eventually. This commit removes the inline reroute from
the process of starting a shard and replaces it with a deferred one that runs
at `NORMAL` priority, avoiding starvation of higher-priority tasks.

Backport of #44433 and #44543.
2019-07-18 14:10:40 +01:00
Igor Motov a19c7977ac Docs: fix WKTToSQL function example (#44377)
Fixes wrong example snippet in WKTToSQL documentation.

Closes #44299
2019-07-17 12:24:28 -04:00
Lisa Cawley 51d8e6e3de [DOCS] Adds sync to data frame transform API (#44254) 2019-07-17 08:56:58 -07:00
Yannick Welsch d98b3e4760
Move frozen indices to x-pack module (#44490)
Backport of #44408 and #44286.
2019-07-17 16:53:10 +02:00
James Rodewig 34725e20fb [DOCS] Move Elasticsearch APIs to REST APIs section. (#44238) (#44372)
Moves the following API sections under the REST APIs navigations:
- API Conventions
- Document APIs
- Search APIs
- Index APIs (previously named Indices APIs)
- cat APIs
- Cluster APIs

Other supporting changes:
- Removes the previous index APIs page under REST APIs. Adds a redirect for the removed page.
- Removes several [partintro] macros so the docs build correctly.
- Changes anchors for pages that become sections of a parent page.
- Adds several redirects for existing pages that become sections of a parent page.

This commit re-applies changes from #44238. Changes from that PR were reverted due to broken links in several repos. This commit adds redirects for those broken links.
2019-07-17 09:18:31 -04:00
Lisa Cawley c18a6402cd [DOCS] Renames data frame APIs section (#44458) 2019-07-16 15:10:46 -07:00
Lisa Cawley 53514b0477 [DOCS] Separates data frame analytics APIs (#44451) 2019-07-16 13:33:23 -07:00
Mayya Sharipova 3220709b0a Add positions info into term_vector doc (#44379) 2019-07-16 16:24:50 -04:00
István Zoltán Szabó 642e9019ff [DOCS] Adds max_page_search_size to data frame transform pivot properties (#44126) 2019-07-16 21:33:19 +02:00
James Rodewig ac07eef86c [DOCS] Remove :edit_url: overrides. (#44445)
These overrides do not work in Asciidoctor and are no longer needed.
2019-07-16 15:04:44 -04:00
Yu 563a78829f Do not allow version in Rest Update API (#43516)
The versioning of Update API doesn't rely on version number anymore (and
rather on sequence number). But in rest api level we ignored the
"version" and "version_type" parameter, so that the server cannot raise
the exception when whey were set.

This PR restores "version" and "version_type" parsing in Update Rest API
so that we can get the appropriate errors.

Relates to #42497
2019-07-16 13:19:07 -04:00
Hendrik Muhs 5ea89a4e7b Remove PR 41491 from release notes (#44417)
PR 41491 hasn't been merged into 7.2 but got closed and superseeded by #41800 (merged via feature branch merge into 7.3)
2019-07-16 10:03:40 -07:00
Lee Hinman fb0461ac76
[7.x] Add Snapshot Lifecycle Management (#44382)
* Add Snapshot Lifecycle Management (#43934)

* Add SnapshotLifecycleService and related CRUD APIs

This commit adds `SnapshotLifecycleService` as a new service under the ilm
plugin. This service handles snapshot lifecycle policies by scheduling based on
the policies defined schedule.

This also includes the get, put, and delete APIs for these policies

Relates to #38461

* Make scheduledJobIds return an immutable set

* Use Object.equals for SnapshotLifecyclePolicy

* Remove unneeded TODO

* Implement ToXContentFragment on SnapshotLifecyclePolicyItem

* Copy contents of the scheduledJobIds

* Handle snapshot lifecycle policy updates and deletions (#40062)

(Note this is a PR against the `snapshot-lifecycle-management` feature branch)

This adds logic to `SnapshotLifecycleService` to handle updates and deletes for
snapshot policies. Policies with incremented versions have the old policy
cancelled and the new one scheduled. Deleted policies have their schedules
cancelled when they are no longer present in the cluster state metadata.

Relates to #38461

* Take a snapshot for the policy when the SLM policy is triggered (#40383)

(This is a PR for the `snapshot-lifecycle-management` branch)

This commit fills in `SnapshotLifecycleTask` to actually perform the
snapshotting when the policy is triggered. Currently there is no handling of the
results (other than logging) as that will be added in subsequent work.

This also adds unit tests and an integration test that schedules a policy and
ensures that a snapshot is correctly taken.

Relates to #38461

* Record most recent snapshot policy success/failure (#40619)

Keeping a record of the results of the successes and failures will aid
troubleshooting of policies and make users more confident that their
snapshots are being taken as expected.

This is the first step toward writing history in a more permanent
fashion.

* Validate snapshot lifecycle policies (#40654)

(This is a PR against the `snapshot-lifecycle-management` branch)

With the commit, we now validate the content of snapshot lifecycle policies when
the policy is being created or updated. This checks for the validity of the id,
name, schedule, and repository. Additionally, cluster state is checked to ensure
that the repository exists prior to the lifecycle being added to the cluster
state.

Part of #38461

* Hook SLM into ILM's start and stop APIs (#40871)

(This pull request is for the `snapshot-lifecycle-management` branch)

This change allows the existing `/_ilm/stop` and `/_ilm/start` APIs to also
manage snapshot lifecycle scheduling. When ILM is stopped all scheduled jobs are
cancelled.

Relates to #38461

* Add tests for SnapshotLifecyclePolicyItem (#40912)

Adds serialization tests for SnapshotLifecyclePolicyItem.

* Fix improper import in build.gradle after master merge

* Add human readable version of modified date for snapshot lifecycle policy (#41035)

* Add human readable version of modified date for snapshot lifecycle policy

This small change changes it from:

```
...
"modified_date": 1554843903242,
...
```

To

```
...
"modified_date" : "2019-04-09T21:05:03.242Z",
"modified_date_millis" : 1554843903242,
...
```

Including the `"modified_date"` field when the `?human` field is used.

Relates to #38461

* Fix test

* Add API to execute SLM policy on demand (#41038)

This commit adds the ability to perform a snapshot on demand for a policy. This
can be useful to take a snapshot immediately prior to performing some sort of
maintenance.

```json
PUT /_ilm/snapshot/<policy>/_execute
```

And it returns the response with the generated snapshot name:

```json
{
  "snapshot_name" : "production-snap-2019.04.09-rfyv3j9qreixkdbnfuw0ug"
}
```

Note that this does not allow waiting for the snapshot, and the snapshot could
still fail. It *does* record this information into the cluster state similar to
a regularly trigged SLM job.

Relates to #38461

* Add next_execution to SLM policy metadata (#41221)

* Add next_execution to SLM policy metadata

This adds the next time a snapshot lifecycle policy will be executed when
retriving a policy's metadata, for example:

```json
GET /_ilm/snapshot?human
{
  "production" : {
    "version" : 1,
    "modified_date" : "2019-04-15T21:16:21.865Z",
    "modified_date_millis" : 1555362981865,
    "policy" : {
      "name" : "<production-snap-{now/d}>",
      "schedule" : "*/30 * * * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "foo-*",
          "important"
        ],
        "ignore_unavailable" : true,
        "include_global_state" : false
      }
    },
    "next_execution" : "2019-04-15T21:16:30.000Z",
    "next_execution_millis" : 1555362990000
  },
  "other" : {
    "version" : 1,
    "modified_date" : "2019-04-15T21:12:19.959Z",
    "modified_date_millis" : 1555362739959,
    "policy" : {
      "name" : "<other-snap-{now/d}>",
      "schedule" : "0 30 2 * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "other"
        ],
        "ignore_unavailable" : false,
        "include_global_state" : true
      }
    },
    "next_execution" : "2019-04-16T02:30:00.000Z",
    "next_execution_millis" : 1555381800000
  }
}
```

Relates to #38461

* Fix and enhance tests

* Figured out how to Cron

* Change SLM endpoint from /_ilm/* to /_slm/* (#41320)

This commit changes the endpoint for snapshot lifecycle management from:

```
GET /_ilm/snapshot/<policy>
```

to:

```
GET /_slm/policy/<policy>
```

It mimics the ILM path only using `slm` instead of `ilm`.

Relates to #38461

* Add initial documentation for SLM (#41510)

* Add initial documentation for SLM

This adds the initial documentation for snapshot lifecycle management.

It also includes the REST spec API json files since they're sort of
documentation.

Relates to #38461

* Add `manage_slm` and `read_slm` roles (#41607)

* Add `manage_slm` and `read_slm` roles

This adds two more built in roles -

`manage_slm` which has permission to perform any of the SLM actions, as well as
stopping, starting, and retrieving the operation status of ILM.

`read_slm` which has permission to retrieve snapshot lifecycle policies as well
as retrieving the operation status of ILM.

Relates to #38461

* Add execute to the test

* Fix ilm -> slm typo in test

* Record SLM history into an index (#41707)

It is useful to have a record of the actions that Snapshot Lifecycle
Management takes, especially for the purposes of alerting when a
snapshot fails or has not been taken successfully for a certain amount of
time.

This adds the infrastructure to record SLM actions into an index that
can be queried at leisure, along with a lifecycle policy so that this
history does not grow without bound.

Additionally,
SLM automatically setting up an index + lifecycle policy leads to
`index_lifecycle` custom metadata in the cluster state, which some of
the ML tests don't know how to deal with due to setting up custom
`NamedXContentRegistry`s.  Watcher would cause the same problem, but it
is already disabled (for the same reason).

* High Level Rest Client support for SLM (#41767)

* High Level Rest Client support for SLM

This commit add HLRC support for SLM.

Relates to #38461

* Fill out documentation tests with tags

* Add more callouts and asciidoc for HLRC

* Update javadoc links to real locations

* Add security test testing SLM cluster privileges (#42678)

* Add security test testing SLM cluster privileges

This adds a test to `PermissionsIT` that uses the `manage_slm` and `read_slm`
cluster privileges.

Relates to #38461

* Don't redefine vars

*  Add Getting Started Guide for SLM  (#42878)

This commit adds a basic Getting Started Guide for SLM.

* Include SLM policy name in Snapshot metadata (#43132)

Keep track of which SLM policy in the metadata field of the Snapshots
taken by SLM. This allows users to more easily understand where the
snapshot came from, and will enable future SLM features such as
retention policies.

* Fix compilation after master merge

* [TEST] Move exception wrapping for devious exception throwing

Fixes an issue where an exception was created from one line and thrown in another.

* Fix SLM for the change to AcknowledgedResponse

* Add Snapshot Lifecycle Management Package Docs (#43535)

* Fix compilation for transport actions now that task is required

* Add a note mentioning the privileges needed for SLM (#43708)

* Add a note mentioning the privileges needed for SLM

This adds a note to the top of the "getting started with SLM"
documentation mentioning that there are two built-in privileges to
assist with creating roles for SLM users and administrators.

Relates to #38461

* Mention that you can create snapshots for indices you can't read

* Fix REST tests for new number of cluster privileges

* Mute testThatNonExistingTemplatesAreAddedImmediately (#43951)

* Fix SnapshotHistoryStoreTests after merge

* Remove overridden newResponse functions that have been removed

* Fix compilation for backport

* Fix get snapshot output parsing in test

* [DOCS] Add redirects for removed autogen anchors (#44380)

* Switch <tt>...</tt> in javadocs for {@code ...}
2019-07-16 07:37:13 -06:00
Lucas Groenendaal aa9dd313cf Fix incorrect node name in docs (#43062)
After starting up elasticsearch the documentation said that their node
name was "6-bjhwl" but in the documentation's output I did not see that
node name. Instead I saw the node name as `localhost.localdomain`
2019-07-16 14:58:42 +02:00
Mark Walkom 4a5215d22a [DOCS] Update id-field.asciidoc (#42482)
Adding a note around the size limit for `_id`
2019-07-16 14:57:33 +02:00
Dan Fey 8a2d23671a [DOCS] Update split-index.asciidoc: fix shards example (#41382)
The max value should be 640 instead of 740 in the shard example:
2019-07-16 14:54:27 +02:00
Tanguy Buchier 078efc9ec4 [DOCS] Clarify refresh_interval new behavior (#43726)
Update indexing-speed.asciidoc to clarify refresh_interval new behavior
2019-07-16 14:53:46 +02:00
magnusram05 096c03945c [Docs] Small update to getting-started.asciidoc (#40393) 2019-07-16 13:40:54 +02:00
Tim Vernum 4b50de2e2e
Document xpack.security.dls.bitset.cache settings (#44400)
Two new settings were introduced in #43669 to control the
behaviour of the Document Level Security BitSet cache.

This change adds documentation for these 2 settings.

Backport of: #44100
2019-07-16 16:22:25 +10:00
Jake Landis c00b082701
add 7.2.1 release notes (#44367) 2019-07-15 15:02:56 -05:00
Adrien Grand 3734356955 Update release notes. 2019-07-15 20:01:23 +02:00
Lisa Cawley e7ea49e32f [DOCS] Removes unnecessary resource definition pages (#44289) 2019-07-15 10:03:53 -07:00
Julie Tibshirani 141d09ee15 Correct a formatting mistake in the _field_caps docs. (#44303)
The 'indices' block that was recently added should appear in the top-level of
the response, as opposed to being nested under 'fields'.
2019-07-15 09:46:02 -07:00
Lisa Cawley 6c7f7d4a10 [DOCS] Adds ml-cpp PRs to release notes (#44354) 2019-07-15 09:22:36 -07:00
David Kyle 2382701547 Wait for pending tasks in docs tests cleanup (#44123)
ML and Data Frame tests should wait for pending tasks
2019-07-15 12:04:27 +01:00
Armin Braun d73e2f9c56
HLRC: Fix '+' Not Correctly Encoded in GET Req. (#33164) (#44324)
* HLRC: Fix '+' Not Correctly Encoded in GET Req.

* Encode `+` correctly as `%2B` in URL paths
* Keep encoding `+` as space in URL parameters
* Closes #33077
2019-07-15 10:21:54 +02:00
Nikita Glashenko d187fcb9de Support WKT point conversion to geo_point type (#44107)
This PR adds support for parsing geo_point values from WKT POINT format.
Also, a few minor bugs in geo_point parsing were fixed.

Closes #41821
2019-07-12 14:31:07 -04:00
Zachary Tong 3fa677ce79 Document that pipeline aggs are not compatible with composite agg (#44180) 2019-07-12 12:35:18 -04:00
Lisa Cawley 8fdcf28fac [DOCS] Reformats API parameter details (#44194) 2019-07-12 08:28:49 -07:00
Lisa Cawley 4d8bf1c3e3 [DOCS] Removes links to ML tutorial (#44251) 2019-07-12 08:28:36 -07:00
James Rodewig 9ff8600d46 Revert "[DOCS] Relocate several APIs to REST APIs section (#44238)"
This reverts commit 6ebd59791afe2e0d55be2989fdbb594972237340.
2019-07-11 17:01:32 -04:00
James Rodewig 62b5b81fd2 [DOCS] Relocate several APIs to REST APIs section (#44238) 2019-07-11 16:24:28 -04:00
John Murphy 8030d8f6dc [DOCS] Add `lowercase` filter to phrase suggester example so searches are case insensitive (#44186) 2019-07-11 15:27:31 -04:00
Mayya Sharipova 32cb47b91c Add l1norm and l2norm distances for vectors (#44116)
Add L1norm - Manhattan distance
Add L2norm - Euclidean distance
relates to #37947
2019-07-11 14:30:02 -04:00
István Zoltán Szabó 2171b6b47f [DOCS] Adds data frame analytics API and evaluate API resource documentation (#43972)
This PR adds the resource documentation of the data frame analytics APIs and the evaluate API to the ML API doc pool.
2019-07-11 18:12:48 +02:00
James Rodewig f01a9eeb34 [DOCS] Rewrite `has_child` query to use new format (#44190) 2019-07-11 09:11:26 -04:00
lcawl 4e6cbc2890 [DOCS] Fixes formatting in data frame analytics API 2019-07-10 18:01:47 -07:00
Lisa Cawley 00b16e332d [DOCS] Reformat rollup APIs to use new API format (#44131) 2019-07-10 15:15:02 -07:00
Lisa Cawley fa36f82277 [DOCS] Minor edits to data frame APIs (#44138) 2019-07-10 14:46:03 -07:00
Lisa Cawley aaf8ba9cb4 [DOCS] Adds frequency option to data frame transform resource (#44177) 2019-07-10 14:45:33 -07:00
James Rodewig 4cbd028960 [DOCS] Correct `ignore_unmapped` parm typo for nested query 2019-07-10 10:10:14 -04:00
Andrei Stefan bb3e5351b5 SQL: double quotes escaping bug fix (#43829)
(cherry picked from commit d589dcad18c3708913e13c757b91c846aeb35bb4)
2019-07-10 16:05:22 +03:00
James Rodewig 1ae0db7053 [DOCS] Rewrite nested query to use new format (#44130) 2019-07-10 08:52:04 -04:00
Albert Zaharovits 018d946bba [DOC] Backup & Restore Security Configuration (#42970)
This commit documents the backup and restore of a cluster's
security configuration.

It is not possible to only backup (or only restore) security
configuration, independent to the rest of the cluster's conf,
so this describes how a full configuration backup&restore
will include security as well. Moreover, it explains how part
of the security conf data resides on the special .security
index and how to backup that using regular data snapshot API.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
Co-Authored-By: Tim Vernum <tim@adjective.org>
2019-07-10 14:53:56 +03:00
Przemysław Witek 44781e415e
[7.x] [ML] Add DatafeedTimingStats to datafeed GetDatafeedStatsAction.Response (#43045) (#44118) 2019-07-10 11:51:44 +02:00
David Roberts cb62d4acdf [ML-DataFrame] Add a frequency option to transform config, default 1m (#44120)
Previously a data frame transform would check whether the
source index was changed every 10 seconds. Sometimes it
may be desirable for the check to be done less frequently.
This commit increases the default to 60 seconds but also
allows the frequency to be overridden by a setting in the
data frame transform config.
2019-07-10 09:59:00 +01:00
Adrien Grand 64ff895a32 Add 7.3 release notes. (#44010) 2019-07-10 09:36:51 +02:00
David Kyle 23d7e309da Mute put job docs test
Relates to #43271
2019-07-09 13:23:31 +01:00
Lisa Cawley 94578a8b47 [DOCS] Defines data frame transform resources (#43996)
Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>
2019-07-08 17:53:00 -07:00
lcawl cd4021274a [DOCS] Enables testing for create job ML API (#44022) 2019-07-08 11:43:18 -07:00