Commit Graph

7618 Commits

Author SHA1 Message Date
James Rodewig 67b07ec386
[DOCS] Remove SQL access settings page (#60078) (#60089)
This page previously documented `xpack.sql.enabled`.

However, in 7.8 and above, `xpack.sql.enabled` is always enabled and
the setting has no effect. There is no reason to maintain this page.
2020-07-22 16:59:21 -04:00
James Rodewig f8976505cb
[DOCS] Correct the default value of `ignore_throttled` param (#60036) (#60086)
Co-authored-by: bellengao <gbl_long@163.com>
2020-07-22 16:53:18 -04:00
James Rodewig 0c9791798d
[7.x] [DOCS] Reformat snippets to use two-space indents (#60080) 2020-07-22 15:57:49 -04:00
Lisa Cawley 9ba017f699
[DOCS] Changes level offset of transform pages (#60066) (#60075) 2020-07-22 11:22:57 -07:00
Tim Brooks ba01540d7e
Implement human readable indexing pressure stats (#60058)
The indexing pressure stats do not currently have human readable
variants. This commit add human readable variants and updates the
documentation.
2020-07-22 12:07:59 -06:00
James Rodewig ed10d7407c
[DOCS] Fix shrink index API prereqs (#59985) (#60067) 2020-07-22 14:06:40 -04:00
Tim Brooks ceb54ed655
Add indexing pressure documentation (#59456)
This commit adds documentation about the new indexing pressure memory
limit setting and exposure of this metrics in node stats.
2020-07-22 10:09:18 -06:00
Adam Locke 0a73225cd8
[DOCS] Adding new page for restore snapshot API (#59937) (#60055)
* Adding new page for restore snapshot API.

* Improving test cases, lots of edits, and streamlining content.

* Incorporating review suggestions and feedback.

* Specify `index alias` vs `alias`

* Change parameter order

* Provide clarity around regular expression

* Add link to SLM parameters

* Split sentences in example

* Adding link to master node page.
2020-07-22 12:08:55 -04:00
Lisa Cawley 46d33b1586
[DOCS] 7.9.0 release notes (#60053) 2020-07-22 08:40:59 -07:00
Emily Li 5f27a95346 Fix grammar mistake in SQL data type docs. (#60028)
Remove an extra 'when'.
2020-07-21 16:15:06 -07:00
James Rodewig 293cb8d48c
[DOCS] Fix typo in thread pools docs (#59944) (#60019)
Fix typo where available processors should be allocated processors.

Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
2020-07-21 17:04:36 -04:00
James Rodewig 401e12dc2b
[DOCS] Fix data stream docs (#59818) (#60010) 2020-07-21 17:04:13 -04:00
James Rodewig 04c68ba740
[DOCS] Update search docs to use `my-index` dataset (#60005) (#60012) 2020-07-21 16:14:44 -04:00
James Rodewig b302b09b85
[DOCS] Reformat snippets to use two-space indents (#59973) (#59994) 2020-07-21 15:49:58 -04:00
David Roberts 606b7ea139 [DOCS] Adds extra ml-cpp PRs to release notes (#59967) 2020-07-21 11:47:36 -07:00
Tim Brooks ed315442ac
Update thread pool docs about WRITE queue size (#59643)
This commit updates the thread pool documentation to reflect the change
in the WRITE thread pool default queue size.
2020-07-21 12:38:03 -06:00
James Rodewig 32d7fa1541
[DOCS] Introduce basic ECS logs test (#59713) (#59997)
Adds a new `my-index-00001` REST test for docs snippets.

This test can serve as a lightweight replacement for
our existing `twitter` REST tests.

The new dataset is:

* Based on Apache logs, which is better aligned with Elastic use cases
* Compliant with ECS
* Similar to the existing `twitter` data set, containing the same field data types
* Lightweight, which should keep existing test runtimes roughly the same

Also updates the search API reference docs to use the new test.
2020-07-21 13:25:53 -04:00
James Rodewig fb40ccf8a4
[DOCS] Mark data stream stats API as stable (#59978) (#59987)
Removes experimental admon from data stream stats API.
Relates to #59860.
2020-07-21 11:22:36 -04:00
malpani 0555fef799 Support ignore_keywords flag for word delimiter graph token filter (#59563)
This commit allows customizing the word delimiter token filters to skip processing 
tokens tagged as keyword through the `ignore_keywords` flag Lucene's 
WordDelimiterGraphFilter already exposes.

Fix for #59491
2020-07-21 16:11:55 +01:00
Howard 466e947b0e
[DOCS] Fix missing punctuation in agg docs (#59823) 2020-07-21 10:19:29 -04:00
Przemysław Witek 283a1f605c
Rename binary_soft_classification evaluation to outlier_detection (#59951) (#59970) 2020-07-21 15:15:04 +02:00
Lisa Cawley fb212269ce
[DOCS] Changes level offset of anomaly detection pages (#59911) (#59940) 2020-07-20 17:04:59 -07:00
Julie Tibshirani 8dc5880c3f Add 'point' to the top-level field type docs. (#59731)
Before it was missing from the list. This PR also renames the 'geo data types'
section to 'spatial data types' and consolidates the geo and cartesian types
into that section.
2020-07-20 16:30:12 -07:00
Lisa Cawley 9633d503d8
[DOCS] Changes level offset for anomaly detection APIs (#59920) (#59928) 2020-07-20 13:10:54 -07:00
Lisa Cawley 8f8d24b3c1
[DOCS] Changes level offset in data frame analytics APIs (#59919) (#59923) 2020-07-20 13:06:29 -07:00
James Rodewig ff8a042580
[DOCS] Reformat agg snippets to use two-space indents (#59912) (#59922) 2020-07-20 15:59:00 -04:00
James Rodewig 24fec52447
[DOCS] Add performance warning for scripts (#59890) (#59913) 2020-07-20 15:05:33 -04:00
Armin Braun e16e565c5e
Fix Snapshot Status API Docs Test (#59902) (#59908)
The clock resolution for this API is our default 200ms. It is unlikely but
possible that a shard snapshot starts and ends on separate clock ticks and that breaks the test.
Just allowing any value here seems fine to me (seems we can't match for integer specifically).
2020-07-20 18:43:40 +02:00
Igor Motov 96a5284484
Add hard_bounds documentation (#59809) (#59883)
Fixes #59774
2020-07-20 10:51:23 -04:00
Nik Everett fe10141108
Document supported scenarios for CCS (#58120) (#59886)
Documents the supported scenarios for CCS.

Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-07-20 10:41:53 -04:00
David Turner b75207a09f Remove sporadic min/max usage estimates from stats (#59755)
Today `GET _nodes/stats/fs` includes `{least,most}_usage_estimate`
fields for some nodes. These fields have rather strange semantics. They
are only reported on the elected master and on nodes that have been the
elected master since they were last restarted; when a node stops being
the elected master these stats remain in place but we stop updating them
so they may become arbitrarily stale.

This means that these statistics are pretty meaningless and impossible
to use correctly. Even if they were kept up to date they're never
reported for data-only nodes anyway, despite the fact that data nodes
are the ones where we care most about disk usage. The information needed
to compute the path with the least/most available space is already
provided in the rest the stats output, so we can treat the inclusion of
these stats as a bug and fix it by simply removing them in this commit.
Since these stats were always optional and mostly omitted (for opaque
reasons) this is not considered a breaking change.
2020-07-20 15:22:04 +01:00
James Rodewig e7c7ed6493
[DOCS] Fix `requests_per_second` reindex param (#59871) (#59876)
Corrects the `requests_per_second` query parameter used in the reindex,
delete by query, and update by query API docs.

The parameter defaults to `-1` (no throttle). `0` is not an allowed value.
2020-07-20 10:08:51 -04:00
James Rodewig 76b2dd23e2
[DOCS] Document data stream stats API (#59435) (#59874) 2020-07-20 09:50:26 -04:00
James Rodewig 32c8df68ba
[DOCS] Fix erroneous data stream ref (#59805) (#59868)
Removes an erroneous data stream reference added in #58513.

While technically possible, we don't encourage using date math to name
data streams.
2020-07-20 09:30:30 -04:00
James Rodewig 82a8d9aa0c
[DOCS] Fix keyword marker docs (#59834) (#59863)
Co-authored-by: Rui Almeida <ruial@outlook.com>
2020-07-20 09:27:42 -04:00
James Rodewig 828aa6f640
[DOCS] EQL: Remove collapsible sections from EQL search docs (#59819) (#59861) 2020-07-20 09:26:32 -04:00
James Rodewig a160daa5d9
[DOCS] Remove collapsible examples (#59820) (#59857)
Snippets are now visible without additional clicks.
2020-07-20 09:14:36 -04:00
Nik Everett 514b2f3414
Clean up a few of vwh's rough edges (#59341) (#59807)
This cleans up a few rough edged in the `variable_width_histogram`,
mostly found by @wwang500:
1. Setting its tuning parameters in an unexpected order could cause the
   request to fail.
2. We checked that the maximum number of buckets was both less than
   50000 and MAX_BUCKETS. This drops the 50000.
3. Fixes a divide by 0 that can occur of the `shard_size` is 1.
4. Fixes a divide by 0 that can occur if the `shard_size * 3` overflows
   a signed int.
5. Requires `shard_size * 3 / 4` to be at least `buckets`. If it is less
   than `buckets` we will very consistently return fewer buckets than
   requested. For the most part we expect folks to leave it at the
   default. If they change it, we expect it to be much bigger than
   `buckets`.
6. Allocate a smaller `mergeMap` in when initially bucketing requests
   that don't use the entire `shard_size * 3 / 4`. Its just a waste.
7. Default `shard_size` to `10 * buckets` rather than `100`. It *looks*
   like that was our intention the whole time. And it feels like it'd
   keep the algorithm humming along more smoothly.
8. Default the `initial_buffer` to `min(10 * shard_size, 50000)` like
   we've documented it rather than `5000`. Like the point above, this
   feels like the right thing to do to keep the algorithm happy.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-17 15:16:09 -04:00
Adam Locke 29ff05cbac
[7.x] [DOCS] Update snapshot/restore docs to align with API changes (#59730) (#59803)
* [DOCS] Updating snapshot/restore pages to align with API changes (#59730)

* Updating snapshot/restore pages to align with API changes.

* Fixing texts in delete snapshot page.

* Removing duplicate code sample and making editorial changes.

* Change "deleted" to "delete"

* Incorporating review feedback and making minor editorial changes.

* Remove titleabbrev

* Add paragraph break

* Remove titleabbrev from restore page

* Remove titleabbrev from create page

* Change "Create" to lowercase

* Change API names to lowercase

* Remove extraneous delimiters

* Change "Delete" to lowercase

* Single-sourcing warning and clarifying warning text.

* Fixing tests and removing erroneous example.
2020-07-17 14:33:18 -04:00
Dan Hermann 48df9b1a0e
Update regex file for es user agent node processor (#59697) (#59794) 2020-07-17 11:04:01 -05:00
Adam Locke 6ccf3548e7
Fix Snapshot Status API Docs Test (#59775) (#59787)
Introduce a fix to tests by snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Co-authored-by: Armin Braun <me@obrown.io>
2020-07-17 11:11:23 -04:00
James Rodewig a672a2a2d4
[DOCS] Move highlighting docs to separate page (#59768) (#59781)
Moves the highlighting docs from the deprecated 'Request Body Search'
chapter to the new subpage of the 'Run a search chapter' section.

No substantive changes were made to the content.
2020-07-17 10:57:00 -04:00
Benjamin Trent b7f30fc929
[7.x] Adding new `require_alias` option to indexing requests (#58917) (#59769)
* Adding new `require_alias` option to indexing requests (#58917)

This commit adds the `require_alias` flag to requests that create new documents.

This flag, when `true` prevents the request from automatically creating an index. Instead, the destination of the request MUST be an alias.

When the flag is not set, or `false`, the behavior defaults to the `action.auto_create_index` settings.

This is useful when an alias is required instead of a concrete index.

closes https://github.com/elastic/elasticsearch/issues/55267
2020-07-17 10:24:58 -04:00
James Rodewig fa2167af0a
[7.x] [DOCS] Update upgrade docs and release highlights for 7.9 (#59674) 2020-07-16 15:58:40 -04:00
James Rodewig da85a40e7e
[DOCS] Reformat `predicate_token_filter` tokenfilter (#57705) (#59714) 2020-07-16 13:35:09 -04:00
lcawl f2b530dbdb [DOCS] Re-adds coming macro in release notes 2020-07-16 09:12:39 -07:00
István Zoltán Szabó 35512a9284 [DOCS] Adds security privilege info to inference bucket aggregation (#59604) 2020-07-16 18:03:19 +02:00
Marios Trivyzas c7efbc1b83
SQL: Implement DATE_PARSE function for parsing strings into DATE values (#57391) (#59699)
Implement DATE_PARSE(<date_str>, <pattern_str>) function
which allows to parse a date string according to the specified
pattern into a date object. The patterns allowed are those of
java.time.format.DateTimeFormatter.

Closes #54962

Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
Co-authored-by: Patrick Jiang(白泽) <dreamlike.sky@foxmail.com>

(cherry picked from commit 647a413d9b21bd3938f1716bb19f8407e1334125)
2020-07-16 17:24:30 +02:00
Adam Locke 305b46c7cd
[DOCS] Adding get snapshot status API docs (#59355) (#59670)
* Adding get snapshot status API docs.

* Adding more fields and a link to the new page.

* Adding missing spaces in TESTRESPONSES

* Adding more parameters and making some edits.

* Marking snapshot as optional

* Marking repository as optional

* Add data type for stats

* Add data type for shard_stats

* Incorporating review feedback.

* Lots of review feedback incorporated.

* Fixing tests to unbreak CI builds.

* Changing indices to index.
2020-07-16 11:21:17 -04:00
Benjamin Trent a28547c4b4
[7.x] [ML] add new `custom` field to trained model processors (#59542) (#59700)
* [ML] add new `custom` field to trained model processors (#59542)

This commit adds the new configurable field `custom`.

`custom` indicates if the preprocessor was submitted by a user or automatically created by the analytics job.

Eventually, this field will be used in calculating feature importance. When `custom` is true, the feature importance for
the processed fields is calculated. When `false` the current behavior is the same (we calculate the importance for the originating field/feature).

This also adds new required methods to the preprocessor interface. If users are to supply their own preprocessors
in the analytics job configuration, we need to know the input and output field names.
2020-07-16 10:57:38 -04:00
István Zoltán Szabó 76fbe0a6d9 [DOCS] Sorts agg and grouping names alphabetically in PUT Transforms API docs. (#59688) 2020-07-16 12:45:29 +02:00
Przemysław Witek df4fea79cb
Add a "verbose" option to the data frame analytics stats endpoint (#59589) (#59621) 2020-07-16 09:51:31 +02:00
lcawl 4ad8bef33b [DOCS] Removes docs PR from release notes 2020-07-15 16:07:43 -07:00
James Rodewig 43481441e9
[DOCS] EQL: Update EQL search response format (#59554) (#59668) 2020-07-15 17:23:48 -04:00
James Rodewig e30af2fc35
[DOCS] Fix syntax and wording in EQL docs (#59623) (#59650) 2020-07-15 14:45:56 -04:00
Adam Locke 776e9507fb
[DOCS] Update similarity.asciidoc (#59400) (#59644)
Community contribution to fix linking issues in the Similarity module docs.

Co-authored-by: Xin Yan <SHU_Yanx@hotmail.com>
2020-07-15 14:12:00 -04:00
James Rodewig ef9b14b07e
[DOCS] Add `write_index_only` param to ds mapping tutorials (#59618) (#59639) 2020-07-15 13:02:01 -04:00
Rory Hunter b8d73a1e7e
Default gateway.auto_import_dangling_indices to false (#59302)
Backport of #58898.

Part of #48366. Now that there is a dedicated API for dangling indices, the auto-import
behaviour can default to off. Also add a note to the breaking changes for 7.9.0.
2020-07-15 17:10:42 +01:00
James Rodewig 8cac702171 [DOCS] Note that EQL timestamp field can also be date_nanos 2020-07-15 09:55:55 -04:00
James Rodewig 4e58f967de
[DOCS] Update ds overview for optional `@timestamp` mapping (#59558) (#59614) 2020-07-15 09:46:55 -04:00
Martijn Laarman a699c89133 [DOCS] Add release notes for 7.8.1 (#59594)
(cherry picked from commit f43a233948f13e487d4d0f4be668687c404a71f4)
2020-07-15 11:42:03 +02:00
Armin Braun ecf97e9415
Remove Outdated Documentation On Snapshots (#59358) (#59585)
* We now have concurrent repository operations so the one at a time limit does not apply any longer
* Initialization was never slow solely due to loading information about all existing snaphots (though this contributed)
but also because two cluster state updates and a few writes to the repository had to happen before initialization could return
   * Repo data necessary for a snapshot create operation is now cached on heap so loading it is effectively instant
   * Snapshot initialization is just a single CS update now
   * Initialization does no writes to the repository whatsoever
* Fixed missing `repository`
2020-07-15 07:49:18 +02:00
James Rodewig e5baacbe2e
[DOCS] Simplify index template snippets for data streams (#59533) (#59553)
Removes the `@timestamp` field mapping from several data stream index
template snippets.

With #59317, the `@timestamp` field defaults to a `date` field data type
for data streams.
2020-07-14 17:28:43 -04:00
James Rodewig be4483034c
[DOCS] Add example of ds index template with date_nanos mapping (#59535) (#59570) 2020-07-14 17:28:31 -04:00
Costin Leau 679619c798 EQL: Improve retrieval of results (#59552)
Instead of retrieving an entire SearchHit, get just a reference and
postpone the document retrieval when assembling the final results.
Remove sort information from results to make them consistent.
Move TumblingWindow under the sequence package.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
(cherry picked from commit bccfbcd81f2f1d3552e95e4a9ee2618fb3059bd9)
2020-07-14 23:53:57 +03:00
Julie Tibshirani 3ccc767003 Expand docs for component template merging. (#59466)
This change clarifies the order in which components are merged. It also adds
information on mapping merging, now that this has been implemented.
2020-07-14 11:08:34 -07:00
James Rodewig f4c46075b4
[DOCS] Add data streams to index template API docs (#59462) (#59549) 2020-07-14 12:51:47 -04:00
Tim Brooks a46e5e0f04
Increase default write queue size (#59464)
This commit increases the default write queue size to 10000. This is to
allow a greater number of pending indexing requests. This work is safe
as we have added additional memory limits. Relates to #59263.
2020-07-14 10:35:25 -06:00
Andrei Dan 7dcdaeae49
Default to @timestamp in composable template datastream definition (#59317) (#59516)
This makes the data_stream timestamp field specification optional when
defining a composable template.
When there isn't one specified it will default to `@timestamp`.

(cherry picked from commit 5609353c5d164e15a636c22019c9c17fa98aac30)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-07-14 12:36:54 +01:00
Andrei Dan 4180333bbc
[7.x] Composable templates: add a default mapping for @timestamp (#59244) (#59510)
This adds a low precendece mapping for the `@timestamp` field with
type `date`.
This will aid with the bootstrapping of data streams as a timestamp
mapping can be omitted when nanos precision is not needed.

(cherry picked from commit 4e72f43d62edfe52a934367ce9809b5efbcdb531)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-07-14 11:29:33 +01:00
debadair 7d20d32a8c
Update node.asciidoc (#59201) (#59479)
TIP block was missing due to the lack of line break prior to the "TIP"

Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
2020-07-13 16:51:14 -07:00
James Rodewig db89764539
[DOCS] Add data streams to rollup APIs (#59423) (#59465) 2020-07-13 16:57:40 -04:00
James Rodewig a1cf955dbd
[DOCS] Clarify that passwords are not preserved for `kibana_system` user (#59449) (#59460) 2020-07-13 16:34:11 -04:00
Lee Hinman bf1a60130d
[7.x] Add telemetery for data streams (#59433) (#59454)
This commit adds data stream info to the `/_xpack` and `/_xpack/usage` APIs. Currently the usage is
pretty minimal, returning only the number of data streams and the number of indices currently
abstracted by a data stream:

```
  ...
  "data_streams" : {
    "available" : true,
    "enabled" : true,
    "data_streams" : 3,
    "indices_count" : 17
  }
  ...
```
2020-07-13 14:30:11 -06:00
Adam Locke aa260636e5
Indicating that the size parameter defaults to 10. (#59438) (#59461) 2020-07-13 16:27:20 -04:00
James Rodewig d293e1ae36
[DOCS] Add data streams to reload search analyzers API (#59422) (#59437) 2020-07-13 12:50:47 -04:00
James Rodewig 0a7664e190
[DOCS] Add data streams to validate query API (#59420) (#59436) 2020-07-13 12:50:34 -04:00
homersimpsons f95658d1f8 [DOCS] MatchQuery: `transpositions` to `fuzzy_transpositions` (#59371) 2020-07-13 12:37:30 -04:00
Christos Soulios 3868bcc7b8
[7.x] Histogram integration on Histogram field type (#59431)
Backports #58930 to 7.x
Implements histogram aggregation over histogram fields as requested in #53285.
2020-07-13 19:36:33 +03:00
Dan Hermann c228532ebd
Update docs for delete data stream API to show that multiple names are supported 2020-07-13 09:11:25 -05:00
James Rodewig 27a87c9d0c
[DOCS] Update snapshot/restore and SLM docs for data streams (#58513) (#59403)
Updates the existing snapshot/restore and SLM docs to make them
aware of data streams.
2020-07-13 09:26:51 -04:00
James Rodewig 2629a95e14
[DOCS] EQL: Document `until` keyword support (#59320) (#59408) 2020-07-13 09:05:47 -04:00
James Rodewig 85101fa487
[DOCS] Add data streams to searchable snapshot API docs (#59325) (#59409) 2020-07-13 09:05:27 -04:00
James Rodewig a357ec59f2
[DOCS] Add data streams to index APIs (#59329) (#59410) 2020-07-13 09:05:03 -04:00
James Rodewig 35a78b88ab
[DOCS] Add data streams to ILM explain API (#59343) (#59411) 2020-07-13 09:04:42 -04:00
James Rodewig 896d0ffd9b
[DOCS] EQL: Prepare docs for release (#59259) (#59407)
Changes:

* Swaps the `dev` admonitions for `experimental` admonitions
* Removes `ifdef` statements preventing the docs from appearing in
  released branches
2020-07-13 09:04:15 -04:00
James Rodewig 9d5c091f7a
[DOCS] Add data streams to EQL search docs (#58611) (#59404) 2020-07-13 09:03:55 -04:00
James Rodewig 39bcc4a1a7
[DOCS] Add ingest pipeline ex to data stream docs (#58343) (#59402) 2020-07-13 09:03:36 -04:00
Kartika Prasad 8ab0c1b4a0 Update indexing-speed.asciidoc (#59347)
typo fix
2020-07-13 12:19:43 +01:00
István Zoltán Szabó cdf6a054c6 [DOCS] Fixes getting time features example in Painless in Transforms (#59379) 2020-07-13 10:57:59 +02:00
David Roberts 2f9d4a1c7a [DOCS] Adds extra ml-cpp PRs to release notes (#59354)
Following the rebuild of 7.8.1 two extra ml-cpp PRs will
now be released in 7.8.1.
2020-07-13 09:36:21 +01:00
James Rodewig 1402f787f8
[DOCS] Add data streams to field caps API docs (#59326) (#59340) 2020-07-09 16:54:33 -04:00
James Rodewig 41345d4dd3
[DOCS] Add data streams to clear cache API docs (#59324) (#59339) 2020-07-09 16:54:04 -04:00
James Rodewig 77e227bf9b
[DOCS] Document custom routing support for data streams (#59323) (#59338) 2020-07-09 16:52:30 -04:00
James Rodewig ef74a68bcc
[DOCS] Document index aliases do not support data streams (#59321) (#59337) 2020-07-09 16:51:58 -04:00
Lisa Cawley 54483394ae
[DOCS] Clarify subscription requirements (#58958) (#59307) 2020-07-09 12:24:45 -07:00
James Rodewig fca722cee1
[DOCS] Add x-pack tag to data stream docs (#59241) (#59299) 2020-07-09 13:12:38 -04:00
Dimitris Athanasiou b2243337d8
[7.x][ML] Data frame analytics max_num_threads setting (#59254) (#59308)
This adds a setting to data frame analytics jobs called
`max_number_threads`. The setting expects a positive integer.
When used the user specifies the max number of threads that may
be used by the analysis. Note that the actual number of threads
used is limited by the number of processors on the node where
the job is assigned. Also, the process may use a couple more threads
for operational functionality that is not the analysis itself.

This setting may also be updated for a stopped job.

More threads may reduce the time it takes to complete the job at the cost
of using more CPU.

Backport of #59254 and #57274
2020-07-09 19:15:46 +03:00
Rory Hunter 5debd09808 Dangling indices documentation (#58751)
Part of #48366. Add documentation for the dangling indices
API added in #58176.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-07-09 14:02:23 +01:00
Andrei Stefan c0e0bca84c
Remove search_after and implicit_join_key_field (#59232) (#59280)
(cherry picked from commit 6ede6c59eff321b9fedad30e19508b9e4f788b54)
2020-07-09 12:34:01 +03:00
Bogdan Pintea acfff7b896
Add sample versions of standard deviation and variance funcs (#59093) (#59274)
* Add sample versions of standard deviation and variance functions (#59093)

* Add STDDEV_SAMP, VAR_SAMP

This commit adds the sampling variations of the standard deviation and
variance agg functions.

(cherry picked from commit 8b29817b49e386215f29cb5b3356d0183fd5d9de)

* Fix: workaround for lack of Map#of() in Java8

Replace Map#of() with a HashMap static init.
2020-07-09 10:17:13 +02:00
Adam Locke 96a06685cf
[7.x] [DOCS] Adding get snapshot api docs (#59238)
* [DOCS] Adding get snapshot API docs (#59098)

* Adding page for get snapshot API.

* Adding values for state and cleaning up some other formatting.

* Adding missing forward slash to GET request.

* Updating values for start_time and end_time in TESTRESPONSE.

* Swap "return" for "retrieve"

* Swap "return" for "retrieve" 2

* Change .snapshot to .response

* Adding response parameters and incorporating edits from review.

* Update response example to include repository info

* Change dash to underscore

* Add data type for snapshot in response

* Incorporating review comments and adding missing response definitions.

* Minor rewording in description.

* Removing multi-snapshot support for 7.x.

* Changing end_time value from build error.

* Removing .response from snippet testing.
2020-07-08 16:40:35 -04:00
James Rodewig d2c5a4c5e9
[7.x] [DOCS] Update get data stream API response (#59197) (#59221)
Updates docs and snippets for changes made to the get data stream API with
PR #59128.
2020-07-08 14:04:14 -04:00
James Rodewig 838f717e5f
[DOCS] Add data streams to security docs (#59084) (#59237) 2020-07-08 12:53:56 -04:00
James Rodewig 93a5eb0688
[DOCS] EQL: Document `size` limit for pipes (#59085) (#59236)
Changes:
* Documents the `size` default as `10`.
* Updates `size` param def to note its relation to pipes.
* Updates the `head` and `tail` pipe docs to modify sequences.
* Documents the `fetch_size` parameter.

Relates to #59014 and #59063
2020-07-08 12:22:57 -04:00
Martijn van Groningen 17bd559253
Fix the timestamp field of a data stream to @timestamp (#59210)
Backport of #59076 to 7.x branch.

The commit makes the following changes:
* The timestamp field of a data stream definition in a composable
  index template can only be set to '@timestamp'.
* Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and
  instead only check that the _timestamp field mapping has been defined on a backing index of a data stream.
* Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method
  to `MetadataIndexTemplateService#collectMappings(...)` method.
* Fixed a bug (#58956) that cases timestamp field validation to be performed
  for each template and instead of the final mappings that is created.
* only apply _timestamp meta field if index is created as part of a data stream or data stream rollover,
this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition.

Relates to #58642
Relates to #53100
Closes #58956
Closes #58583
2020-07-08 17:30:46 +02:00
James Rodewig b27de36b5d
[DOCS] EQL: Document `maxspan` keyword (#58931) (#59223) 2020-07-08 11:04:28 -04:00
James Rodewig 37be56ab97
[DOCS] EQL: Document unsupported var comparison (#58941) (#59224)
ES EQL queries do not support the comparison of a variable, such as
a field value, to another variable.

This adds a related para and example to the EQL syntax docs.
2020-07-08 11:04:05 -04:00
David Kyle b87cef6fe7
Include the ml inference aggregation doc (#59219) (#59226)
Add to the list of pipeline aggregations
2020-07-08 14:35:08 +01:00
Yannick Welsch 0b9eb210b8
Add basic searchable snapshots usage information (#58828) (#59160)
Adds super basic usage information for searchable snapshots, to be extended later.

Backport of #58828
2020-07-08 13:09:29 +02:00
Nhat Nguyen ef5c397c0f
Sending operations concurrently in peer recovery (#58018)
Today, we send operations in phase2 of peer recoveries batch by batch
sequentially. Normally that's okay as we should have a fairly small of
operations in phase 2 due to the file-based threshold. However, if
phase1 takes a lot of time and we are actively indexing, then phase2 can
have a lot of operations to replay.

With this change, we will send multiple batches concurrently (defaults
to 1) to reduce the recovery time.

Backport of #58018
2020-07-07 22:03:31 -04:00
Lisa Cawley 2e71db71b6
[DOCS] Clarifies transform node settings (#59023) (#59192) 2020-07-07 13:54:54 -07:00
James Rodewig 6ed356ffc3
[DOCS] Replace `datatype` with `data type` (#58972) (#59184) 2020-07-07 14:59:35 -04:00
James Rodewig 045b893dd1
[DOCS] Add data streams to shard stores API docs (#59070) (#59181) 2020-07-07 14:57:47 -04:00
James Rodewig 225506f0e4
[DOCS] Add data streams to rank eval API docs (#59069) (#59179) 2020-07-07 14:57:35 -04:00
James Rodewig 1431a5436b
[DOCS] Add data streams to force merge API docs (#58951) (#59178) 2020-07-07 14:57:19 -04:00
Lisa Cawley 233857ef6e [DOCS] Adds ml-cpp PRs to release notes (#59188) 2020-07-07 11:56:40 -07:00
James Rodewig 189d69d826
[DOCS] Clarify atomic change for alias swaps (#59154) (#59164)
Small edit highlighting the fact that atomic cluster state change does not guarantee lack of errors for in-flight requests.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
2020-07-07 13:03:12 -04:00
Nik Everett eb169ae226
Fix lookup support in adjacency matrix (backport of #59099) (#59108)
This request:
```
POST /_search
{
  "aggs": {
    "a": {
      "adjacency_matrix": {
        "filters": {
          "1": {
            "terms": { "t": { "index": "lookup", "id": "1", "path": "t" } }
          }
        }
      }
    }
  }
}
```

Would fail with a 500 error and a message like:
```
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason":"async actions are left after rewrite"
      }
    ]
  }
}
```

This fixes that by moving the query rewrite phase from a synchronous
call on the data nodes into the standard aggregation rewrite phase which
can properly handle the asynchronous actions.
2020-07-07 10:28:20 -04:00
David Turner 8f4f844e6e Add docs for filesystem health checks (#59134)
Documents the feature and settings introduced in #52680.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-07-07 14:14:58 +01:00
James Rodewig 664b546771 [DOCS] Fix anchor syntax 2020-07-07 09:02:33 -04:00
James Rodewig a8220ad51e [DOCS] Fix anchor syntax 2020-07-07 08:57:20 -04:00
James Rodewig d66084dcaf
[DOCS] Update data stream mapping and setting docs (#58874) (#59067) 2020-07-06 11:58:43 -04:00
Adam Locke e3469bb6e2
Removing ESS icon for xpack.security.audit.enabled. (#59078) (#59079) 2020-07-06 11:20:53 -04:00
James Rodewig 31c71914b7
[DOCS] Clean up `Use a data stream` test snippets (#58968) (#58978) 2020-07-06 08:39:04 -04:00
Przemysław Witek f35ad0d4e1
Report peak model memory in ModelSizeStats (#59017) (#59055) 2020-07-06 12:55:12 +02:00
David Kyle c651135562
[ML] Make Inference processor field_map and inference_config optional (#59010)
Relaxes the requirement that the inference ingest processor must has a
field_map and inference_config defined even if they are empty.
2020-07-06 11:35:30 +01:00
Martijn van Groningen f0dd9b4ace
Add data stream timestamp validation via metadata field mapper (#59002)
Backport of #58582 to 7.x branch.

This commit adds a new metadata field mapper that validates,
that a document has exactly a single timestamp value in the data stream timestamp field and
that the timestamp field mapping only has `type`, `meta` or `format` attributes configured.
Other attributes can affect the guarantee that an index with this meta field mapper has a
useable timestamp field.

The MetadataCreateIndexService inserts a data stream timestamp field mapper whenever
a new backing index of a data stream is created.

Relates to #53100
2020-07-06 11:32:33 +02:00
David Turner 2fdd8f3a2c Restores do not cause red health (#59015)
Since 2.0.0 (56a264cf6d) we have documented that restoring a snapshot
typically results in `red` cluster health. However since 5.0.0 (#19516)
this hasn't been true, we report `yellow` health for unassigned
primaries that will be recovered from a snapshot in the future. This
commit adjusts these docs to match today's behaviour.
2020-07-04 11:16:28 +01:00
debadair c8e3128fe4
[DOCS] Combo version of ILM docs. (#57909) (#59029)
* [DOCS] Combo version of ILM docs.

* [DOCS] Moved tutorial from Kibana.

* Adds documentation for index lifecycle policies (#28705)

* [DOCS] Adds documentation for index lifecycle policies

* [DOCS] Updated image for policy options to show all menu items

* Update create-policy.asciidoc

* [DOCS] Incorporated review comments on hot and warm phase

* [DOCS] Additional changes to warm phase

* [DOCS] Removed the word open in the warm phase

* Adds X-Pack icon for ILM (#34178)

* Add ILM tutorial (#59502)

* Add tutorial for ILM with filebeat

* Change screenshots and add additional steps

* Update screenshots, add numbered steps, and other minor edits

* Incorporate feedback: update links, formatting, and minor edits

* Move tip inline with list

* Apply suggestions from code review

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Move TIP inline . . . again

* Put TIP inline

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Updates for navigation redesign (#68709)

* [DOCS] Updates for navigation redesign

* Getting started

* Set up text

* Discover

* Dashboard, Graph, ML, Maps, APM, SIEM, Dev tools

* Dev Tools, Stack Monitoring, Management

* Management

* Final changes

* [DOCS] Updates for navigation redesign

* [DOCS] Updates CCR monitoring screenshots

* updates SIEM screenshot and Cases overview text

* Added Brandon's APM image

* [DOCS] Refines CCR shard screenshot

* Removed merge conflict image file

Co-authored-by: lcawl <lcawley@elastic.co>
Co-authored-by: Ben Skelker <ben.skelker@elastic.co>

* [DOCS] Put API examples in collapsible sections like ML does

* Fix include

* Added tutorial images

* Fixed images

* Add short title for FB tutorial

* Add missing files

* Incorporate review feedback

* review feedback

* Incorporated review feedback

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Melori Arellano <melori@elastic.co>
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Kaarina Tungseth <kaarina.tungseth@elastic.co>
Co-authored-by: Ben Skelker <ben.skelker@elastic.co>

Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Melori Arellano <melori@elastic.co>
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Kaarina Tungseth <kaarina.tungseth@elastic.co>
Co-authored-by: Ben Skelker <ben.skelker@elastic.co>
2020-07-03 13:01:08 -07:00
Benjamin Trent b9d9964d10
[ML] add exponent output aggregator to inference (#58933) (#59016)
* [ML] add exponent output aggregator to inference

* fixing docs

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-03 14:51:00 -04:00
Lisa Cawley 935a49a8d6 [DOCS] Deprecates node.ml (#59024) 2020-07-03 11:10:05 -07:00
Lisa Cawley f9b365db6c [DOCS] Edits ML circuit breaker settings (#59026) 2020-07-03 11:07:46 -07:00
Bogdan Pintea 3d96d91efb
[7.x] SQL: fix handling of escaped chars in JDBC connection string (#58429) (#58977)
SQL: fix handling of escaped chars in JDBC connection string (#58429)

This commit fixes an issue emerging when the connection string URI
contains escaped characters.

The original URI is pre-parsed in order to re-assemble a new URI having
the optional elements filled in with defaults. The new URI has been
using however the unescaped query and fragment parts. So if these
contained any escaped `&` or `=` (such as in the password option value),
the unescaping would reveal them and make them later interfere with the
options parsing.

The commit changes that, so that the new URI be built from the unescaped
"raw" parts of the original URI.

(cherry picked from commit 94eb5a05e79c6e203de548d05b13e00295bd4489)
2020-07-03 17:03:00 +02:00
David Kyle f6a0c2c59d
[7.x] Pipeline Inference Aggregation (#58965)
Adds a pipeline aggregation that loads a model and performs inference on the
input aggregation results.
2020-07-03 09:29:04 +01:00
debadair b769e9143d
[DOCS] Add simulate ref (#58579) (#58987)
* [DOCS] Add simulate ref pages

* Add links & experimental tags

* Fixed simulate index response

* Apply suggestions from code review

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

*Incorporate review feedback.
2020-07-02 19:05:32 -07:00
DeDe Morton 2c43421208 [DOCS] Change Beats links to refactored getting started docs (#58790) 2020-07-02 17:11:25 -07:00
Adam Locke 20d04081ec
[7.x] [DOCS] Add supported ESS settings to ES docs (#57953) (#58981)
* Adding ESS icons to supported ES settings.

* Adding new file for supported ESS settings.

* Adding supported ESS settings for HTTP and disk-based shard allocation.

* Adding more supported settings for ESS.

* Adding descriptions for each Cloud section, plus additional settings.

* Adding new warehouse file for Cloud, plus additional settings.

* Adding node settings for Cloud.

* Adding audit settings for Cloud.

* Resolving merge conflict.

* Adding SAML settings (part 1).

* Adding SAML realm encryption and signing settings.

* Adding SAML SSL settings.

* Adding Kerberos realm settings.

* Adding OpenID Connect Realm settings.

* Adding OpenID Connect SSL settings.

* Resolving leftover Git merge markers.

* Removing Cloud settings page and link to it.

* Add link to mapping source

* Update docs/reference/docs/reindex.asciidoc

* Incorporate edit of HTTP settings

* Remove "cloud" from tag and ID

* Remove "cloud" from tag and update description

* Remove "cloud" from tag and ID

* Change "whitelists" to "specifies"

* Remove "cloud" from end tag

* Removing cloud from IDs and tags.

* Changing link reference to fix build issue.

* Adding index management page for missing settings.

* Removing warehouse file for Cloud and moving settings elsewhere.

* Clarifying true/false usage of http.detailed_errors.enabled.

* Changing underscore to dash in link to fix ci build.
2020-07-02 19:40:45 -04:00
James Rodewig 84444e1a54
[DOCS] Add data streams to flush API docs (#58950) (#58975) 2020-07-02 17:26:49 -04:00
James Rodewig e64fe15c1f
[DOCS] Add data streams to cluster APIs docs (#58945) (#58974)
Makes existing docs for the cluster health and cluster state APIs aware
of data streams.
2020-07-02 17:20:55 -04:00
Przemysław Witek 751e84e4c8
Rename regression evaluation metrics to make the names consistent with loss functions (#58887) (#58927) 2020-07-02 17:35:55 +02:00
James Rodewig 9569375ae7
[DOCS] Add data streams to remove lifecycle policy API (#58777) (#58921) 2020-07-02 09:58:54 -04:00
James Rodewig 04ff24ca8e
[DOCS] Add data streams to deprecation info API docs (#58685) (#58920) 2020-07-02 09:52:33 -04:00
James Rodewig 6436792aac
[DOCS] Fix headings for simple analyzer docs (#58910) (#58918) 2020-07-02 09:52:05 -04:00
James Rodewig ff04212257
[DOCS] Fix snippet tests for resolve API docs (#58908) (#58911) 2020-07-02 09:27:14 -04:00
James Rodewig 57d679ff64
[DOCS] Add data streams to put mapping and update settings API docs (#58849) (#58913) 2020-07-02 09:26:57 -04:00
James Rodewig fc5b2b2be4 [DOCS] Fix `scroll` param typo 2020-07-02 08:48:07 -04:00
debadair 4471cd7527 [DOCS] Fix cannot must typo. (#58884) 2020-07-01 17:46:09 -07:00
Adam Locke 9c77862a23
[7.x] [DOCS] Adding delete snapshot API docs. (#58865) (#58871)
* [DOCS] Adding delete snapshot API docs.

* Adding TESTSETUP snippets and fixing original TEST.

* Removing extraneous TESTSETUP.

* Revising <snapshot> description.

* Removing TEST.

* Streamline delete API description.

* Improve TESTSETUP for snippets.
2020-07-01 15:03:02 -04:00
James Rodewig a966513eae
[DOCS] Remove problematic terms (#58832) (#58851) 2020-07-01 13:47:14 -04:00
Nik Everett 326cce624b Document using stored scripts for ingest (#58783)
This documents using stored scripts for complex conditionals in indest.
2020-07-01 13:36:00 -04:00
James Rodewig 860c94ca60
[DOCS] Add redirects for 404 pages (#58846) (#58853) 2020-07-01 12:02:59 -04:00
Lee Hinman d3d03fc1c6
[7.x] Add default composable templates for new indexing strategy (#57629) (#58757)
Backports the following commits to 7.x:

    Add default composable templates for new indexing strategy (#57629)
2020-07-01 09:32:32 -06:00
James Rodewig 7a8da9daa3
[DOCS] Document open requests for data streams (#58615) (#58621)
Adds an open API example to the data streams docs. Also updates the
existing open API docs to make them aware of data streams.
2020-07-01 11:22:45 -04:00
Dan Hermann 27e95d2162
[DOCS] Resolve index API (#58206) (#58843) 2020-07-01 09:41:15 -05:00
Przemysław Witek 909649dd15
[7.x] Implement pseudo Huber loss (PseudoHuber) evaluation metric for regression analysis (#58734) (#58825) 2020-07-01 14:52:06 +02:00
David Turner 822b7421ce Forbid read-only-allow-delete block in blocks API (#58727)
The read-only-allow-delete block is not really under the user's control
since Elasticsearch adds/removes it automatically. This commit removes
support for it from the new API for adding blocks to indices that was
introduced in #58094.
2020-07-01 13:18:26 +01:00
Yannick Welsch 15c85b29fd
Account for recovery throttling when restoring snapshot (#58658) (#58811)
Restoring from a snapshot (which is a particular form of recovery) does not currently take recovery throttling into account
(i.e. the `indices.recovery.max_bytes_per_sec` setting). While restores are subject to their own throttling (repository
setting `max_restore_bytes_per_sec`), this repository setting does not allow for values to be configured differently on a
per-node basis. As restores are very similar in nature to peer recoveries (streaming bytes to the node), it makes sense to
configure throttling in a single place.

The `max_restore_bytes_per_sec` setting is also changed to default to unlimited now, whereas previously it was set to
`40mb`, which is the current default of `indices.recovery.max_bytes_per_sec`). This means that no behavioral change
will be observed by clusters where the recovery and restore settings were not adapted.

Relates https://github.com/elastic/elasticsearch/issues/57023

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-07-01 12:19:29 +02:00
Russ Cam 7d34fa9b67 Update link to .NET BulkAllObservable 2020-07-01 19:54:30 +10:00
David Turner 3a234d2669
Account for remaining recovery in disk allocator (#58800)
Today the disk-based shard allocator accounts for incoming shards by
subtracting the estimated size of the incoming shard from the free space on the
node. This is an overly conservative estimate if the incoming shard has almost
finished its recovery since in that case it is already consuming most of the
disk space it needs.

This change adds to the shard stats a measure of how much larger each store is
expected to grow, computed from the ongoing recovery, and uses this to account
for the disk usage of incoming shards more accurately.

Backport of #58029 to 7.x

* Picky picky

* Missing type
2020-07-01 10:12:44 +01:00
Nik Everett 40850a780d
Fail variable_width_histogram that collects from many (#58619) (#58780)
Adds an explicit check to `variable_width_histogram` to stop it from
trying to collect from many buckets because it can't. I tried to make it
do so but that is more than an afternoon's project, sadly. So for now we
just disallow it.

Relates to #42035
2020-06-30 18:26:45 -04:00
James Rodewig 3aa08fbcde
[DOCS] Add data streams to API conventions (#58695) (#58785)
Updates the existing API conventions docs to make them aware of data
streams.

Co-authored-by: debadair <debadair@elastic.co>
2020-06-30 17:33:35 -04:00
James Rodewig 046d9eeb41 [DOCS] Make `<target>` defs consistent 2020-06-30 15:55:37 -04:00
James Rodewig 416652cdd8
[DOCS] Clarify request formats for index API (#58768) (#58774) 2020-06-30 15:53:03 -04:00
James Rodewig e90c465640
[DOCS] Add data streams to cat APIs (#58699) (#58773) 2020-06-30 15:52:48 -04:00
James Rodewig 3778ca8c25
[DOCS] Add data streams to count API (#58771) (#58772) 2020-06-30 15:52:34 -04:00
James Rodewig 35830a7c12
[DOCS] Add data streams to get field mapping API docs (#58689) (#58759)
Updates the existing get field mapping API docs to make them aware of
data streams. Relates to #58488.
2020-06-30 13:24:41 -04:00
James Rodewig 874ab36b14
[DOCS] Fix error in stop SLM API docs (#58747) (#58750) 2020-06-30 10:16:43 -04:00
James Rodewig 19190c529c [DOCS] Reword admon for index API and data streams 2020-06-30 09:54:24 -04:00
James Rodewig 770f9f11af [DOCS] Fix xref format in async EQL search docs 2020-06-30 09:37:47 -04:00
James Rodewig e5d5b9f5e8
[DOCS] Suppress searchable snapshots in releases (#58740) (#58742)
Fixes a searchable snapshot reference overlooked in #58652
2020-06-30 09:22:32 -04:00
James Rodewig d8731853a3
[DOCS] EQL: Document `head` and `tail` pipes (#58673) (#58739) 2020-06-30 09:12:54 -04:00
James Rodewig d33764583c
[7.x] [DOCS] Document delete/update by query for data streams (#58679) (#58706) 2020-06-30 08:35:13 -04:00
David Turner ceff00997d Suppress searchable snapshots docs in releases (#58652)
This commit adds conditional logic to the docs to avoid including any
docs on searchable snapshots in released versions.

Rework of #58556 which was reverted.
2020-06-30 13:13:09 +01:00
Przemysław Witek 9ea9b7bd3b
[7.x] Implement MSLE (MeanSquaredLogarithmicError) evaluation metric for regression analysis (#58684) (#58731) 2020-06-30 14:09:11 +02:00
Yannick Welsch b885cbff1a
Add index block api (#58716)
Adds an API for putting an index block in place, which also ensures for write blocks that, once successfully returning to
the user, all shards of the index are properly accounting for the block, for example that all in-flight writes to an index have
been completed after adding the write block.

This API allows coordinating more complex workflows, where it is crucial that an index is no longer receiving writes after
the API completes, useful for example when marking an index as read-only during an upgrade in order to reindex its
documents.
2020-06-30 14:06:52 +02:00
James Rodewig 8341ebc061
[7.x] [DOCS] Reformats the update by query API. (#46199) (#58700)
Co-authored-by: debadair <debadair@elastic.co>
2020-06-29 17:50:32 -04:00
Dan Hermann 84513c7539
Document the prohibition on freezing data stream write indices (#58058) (#58705) 2020-06-29 16:34:36 -05:00
Adam Locke 719f2fb135
[DOCS] [7.x] Adding create index snapshot API docs (#58519) (#58692)
* Adding create index snapshot API page.
* Condense API description.
* Remove parameter from query.
* Add POST method and remove `-name` from the snapshot variable.
* Expand description of `<snapshot>`.
* Add data streams to introduction and expand the overall description.
* Add support for data streams.
* Add support for data streams.
* Add data stream and reference for "point-in-time view".
* Add data streams.
* Change `my_backup` to `my_repository`.
* Add description of boolean options for `wait_for_completion` parameter.
* Change command --> response
* Clarify `indices` parameter description
* Update `ignore-unavailable` parameter description
* Reword example description
* Remove "index" from API name
* Incorporating review comments from James R.
* Adding a much better request + response
* Clarify `include_global_state` description
* Incorporating additional edits.
* Changing my_backup to my_repository in example.
* Update snippet test to avoid failures
* Update TESTRESPONSE snippets
* Remove errant space
* Removing the  parameter per reviewer comments
2020-06-29 16:13:53 -04:00
James Rodewig 735a3f344d
[DOCS] EQL: Remove fields from EQL search response (#58667) (#58669) 2020-06-29 09:34:20 -04:00
István Zoltán Szabó 13aa8b8d9a [DOCS] Updates results_field description in the inference processor docs (#58554) 2020-06-29 13:15:15 +02:00
Przemysław Witek 3f7c45472e
[7.x] Introduce DataFrameAnalyticsConfig update API (#58302) (#58648) 2020-06-29 10:56:11 +02:00
David Turner 8f82ec0b19 Revert "Suppress searchable snapshots docs in releases (#58556)"
This reverts commit f0c0ee691a.
2020-06-29 09:21:58 +01:00
David Turner f0c0ee691a Suppress searchable snapshots docs in releases (#58556)
This commit adds conditional logic to the docs to avoid including any
docs on searchable snapshots in released versions.
2020-06-29 08:34:11 +01:00
Dimitris Athanasiou 1817b896c9
[7.x][ML] Add status and increased estimate to memory usage (#58588) (#58606)
Adds parsing of `status` and `memory_reestimate_bytes`
to data frame analytics `memory_usage`. When the training surpasses
the model memory limit, the status will be set to `hard_limit` and
`memory_reestimate_bytes` can be used to update the job's
limit in order to restart the job.

Backport of #58588
2020-06-28 16:27:26 +03:00
Costin Leau 3c81b91474 EQL: Add Head/Tail pipe support (#58536)
Introduce pipe support, in particular head and tail
(which can also be chained).

(cherry picked from commit 4521ca3367147d4d6531cf0ab975d8d705f400ea)
(cherry picked from commit d6731d659d012c96b19879d13cfc9e1eaf4745a4)
2020-06-27 09:49:14 +03:00
James Rodewig 69d8285a28
[DOCS] Add data streams to multi search API docs (#58610) (#58622)
Makes the existing multi search API docs aware of data streams.
2020-06-26 17:32:56 -04:00
James Rodewig c06c89d3db
[DOCS] Remove `composable index template` refs (#58567) (#58612)
Replaces `composable index template` and `composable template` with
`index template` throughout data stream-related docs.

`Composable index template` is only used to contrast with legacy index
templates.
2020-06-26 11:52:58 -04:00
James Rodewig b37b318d0d
[DOCS] EQL: Remove references to partial async EQL results (#58548) (#58609)
Removes references to partial results from the async EQL search docs.
If an EQL search does not complete during the `wait_for_completion_timeout`
timeout period, it returns no results.
2020-06-26 11:11:55 -04:00
James Rodewig 28717d1e02
[DOCS] Fix analyzer page titles (#58362) (#58603)
Changes the titles for analyzer pages to sentence case.

Also changes the 'Pattern character filter' page title to sentence case.
2020-06-26 10:17:01 -04:00
James Rodewig c613e0915a
[DOCS] EQL: Document search API's `tiebreaker_field` param (#57935) (#58540) 2020-06-26 09:25:24 -04:00
James Rodewig ab29162ab3
[DOCS] Fix tokenizer page titles (#58361) (#58598)
Changes the titles for tokenizer pages to sentence case.

Also moves the 'Path hierarchy tokenizer examples' page within the
'Path hierarchy tokenizer' page and adds a related redirect.
2020-06-26 09:24:41 -04:00
Przemyslaw Gomulka 5149554709
Update format.asciidoc to describe strict_date_optional_time_nanos (#57527) (#58581)
closes #57019
2020-06-26 09:02:08 +02:00
Nik Everett d22a242613 Docs: Mark variable_width_histogram experimental (#58574)
We're tracking this aggregation's experimental-progress in #58573. We'd
like a little time to be able to make backwards incompatible changes to
the aggregation because we're not 100% sure about the request and
response format yet.
2020-06-25 16:54:57 -04:00
Jason Tedor 52ad5842a9
Introduce node.roles setting (#58512)
Today we have individual settings for configuring node roles such as
node.data and node.master. Additionally, roles are pluggable and we have
used this to introduce roles such as node.ml and node.voting_only. As
the number of roles is growing, managing these becomes harder for the
user. For example, to create a master-only node, today a user has to
configure:
 - node.data: false
 - node.ingest: false
 - node.remote_cluster_client: false
 - node.ml: false

at a minimum if they are relying on defaults, but also add:
 - node.master: true
 - node.transform: false
 - node.voting_only: false

If they want to be explicit. This is also challenging in cases where a
user wants to have configure a coordinating-only node which requires
disabling all roles, a list which we are adding to, requiring the user
to keep checking whether a node has acquired any of these roles.

This commit addresses this by adding a list setting node.roles for which
a user has explicit control over the list of roles that a node has. If
the setting is configured, the node has exactly the roles in the list,
and not any additional roles. This means to configure a master-only
node, the setting is merely 'node.roles: [master]', and to configure a
coordinating-only node, the setting is merely: 'node.roles: []'.

With this change we deprecate the existing 'node.*' settings such as
'node.data'.
2020-06-25 14:14:51 -04:00
Igor Motov 20af856abd
[7.x] EQL: Adds an ability to execute an asynchronous EQL search (#58192)
Adds async support to EQL searches

Closes #49638

Co-authored-by: James Rodewig james.rodewig@elastic.co
2020-06-25 14:11:57 -04:00
Nik Everett 03e6d1b535
Add Variable Width Histogram Aggregation (backport of #42035) (#58440)
Implements a new histogram aggregation called `variable_width_histogram` which
dynamically determines bucket intervals based on document groupings. These
groups are determined by running a one-pass clustering algorithm on each shard
and then reducing each shard's clusters using an agglomerative
clustering algorithm.

This PR addresses #9572.

The shard-level clustering is done in one pass to minimize memory overhead. The
algorithm was lightly inspired by
[this paper](https://ieeexplore.ieee.org/abstract/document/1198387). It fetches
a small number of documents to sample the data and determine initial clusters.
Subsequent documents are then placed into one of these clusters, or a new one
if they are an outlier. This algorithm is described in more details in the
aggregation's docs.

At reduce time, a
[hierarchical agglomerative clustering](https://en.wikipedia.org/wiki/Hierarchical_clustering)
algorithm inspired by [this paper](https://arxiv.org/abs/1802.00304)
continually merges the closest buckets from all shards (based on their
centroids) until the target number of buckets is reached.

The final values produced by this aggregation are approximate. Each bucket's
min value is used as its key in the histogram. Furthermore, buckets are merged
based on their centroids and not their bounds. So it is possible that adjacent
buckets will overlap after reduction. Because each bucket's key is its min,
this overlap is not shown in the final histogram. However, when such overlap
occurs, we set the key of the bucket with the larger centroid to the midpoint
between its minimum and the smaller bucket’s maximum:
`min[large] = (min[large] + max[small]) / 2`. This heuristic is expected to
increases the accuracy of the clustering.

Nodes are unable to share centroids during the shard-level clustering phase. In
the future, resolving https://github.com/elastic/elasticsearch/issues/50863
would let us solve this issue.

It doesn’t make sense for this aggregation to support the `min_doc_count`
parameter, since clusters are determined dynamically. The `order` parameter is
not supported here to keep this large PR from becoming too complex.

Co-authored-by: James Dorfman <jamesdorfman@users.noreply.github.com>
2020-06-25 11:40:47 -04:00
James Rodewig c3f4034199
[DOCS] Note that DS timestamp field mapping changes require reindex (#58444) (#58517)
With #58096, data streams now track the timestamp field mapping outside
of the template associated with the stream. This means you can no longer
update the timestamp field mapping using template changes.

This updates the associated data stream docs.
2020-06-24 17:21:26 -04:00
markharwood 837f2643eb
Docs - Added field capabilities breaking change (#58509) 2020-06-24 18:39:01 +01:00
Russ Cam 441bc14d21 [DOCS] Update aliases to indicate array (#58469)
Updates the aliases documentation
to correct the parameter to an array.
2020-06-24 09:41:23 -04:00