Commit Graph

52455 Commits

Author SHA1 Message Date
Dan Hermann 7c43cbca82
[7.x] Ignore matching data streams if include_data_streams is false (#59028) 2020-07-03 14:51:32 -05:00
Benjamin Trent b9d9964d10
[ML] add exponent output aggregator to inference (#58933) (#59016)
* [ML] add exponent output aggregator to inference

* fixing docs

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-03 14:51:00 -04:00
Lisa Cawley 935a49a8d6 [DOCS] Deprecates node.ml (#59024) 2020-07-03 11:10:05 -07:00
Lisa Cawley f9b365db6c [DOCS] Edits ML circuit breaker settings (#59026) 2020-07-03 11:07:46 -07:00
Lisa Cawley 5c19464a2f [DOCS] Clarifies number of file and native realms (#58949) 2020-07-03 11:00:28 -07:00
Dan Hermann c1781bc7e7
[7.x] Add include_data_streams flag for authorization (#59008) 2020-07-03 12:58:39 -05:00
Bogdan Pintea 3d96d91efb
[7.x] SQL: fix handling of escaped chars in JDBC connection string (#58429) (#58977)
SQL: fix handling of escaped chars in JDBC connection string (#58429)

This commit fixes an issue emerging when the connection string URI
contains escaped characters.

The original URI is pre-parsed in order to re-assemble a new URI having
the optional elements filled in with defaults. The new URI has been
using however the unescaped query and fragment parts. So if these
contained any escaped `&` or `=` (such as in the password option value),
the unescaping would reveal them and make them later interfere with the
options parsing.

The commit changes that, so that the new URI be built from the unescaped
"raw" parts of the original URI.

(cherry picked from commit 94eb5a05e79c6e203de548d05b13e00295bd4489)
2020-07-03 17:03:00 +02:00
Luca Cavanna e3fc1638d8 Improve error handling in async search code (#57925)
- The exception that we caught when failing to schedule a thread was incorrect.
- We may have failures when reducing the response before returning it, which were not handled correctly and may have caused get or submit async search task to not be properly unregistered from the task manager
- when the completion listener onFailure method is invoked, the search task has to be unregistered. Not doing so may cause the search task to be stuck in the task manager although it has completed.

Closes #58995
2020-07-03 16:07:26 +02:00
Hendrik Muhs ca3da7af85 [ML] handle broken setup with state alias being an index (#58999)
.ml-state-write is supposed to be an index alias, however by accident it can become an index. If
.ml-state-write is a concrete index instead of an alias ML stops working. This change improves error
handling by setting the job to failed and properly log and audit the problem. The user still has to
manually fix the problem. This change should lead to a quicker resolution of the problem.

fixes #58482
2020-07-03 15:26:59 +02:00
Ignacio Vera 2c2486d3d4
Fix GeoHash grid aggregation circuit breaker tests (#58218) (#59001) 2020-07-03 13:46:35 +02:00
Dan Hermann 5e7746d3bd
[7.x] Mirror privileges over data streams to their backing indices (#58991) 2020-07-03 06:33:38 -05:00
Luca Cavanna 4f86f6fb38 Submit async search to not require read privilege (#58942)
When we execute search against remote indices, the remote indices are authorized on the remote cluster and not on the CCS cluster. When we introduced submit async search we added a check that requires that the user running it has the privilege to execute it on some index. That prevents users from executing async searches against remote indices unless they also have read access on the CCS cluster, which is common when the CCS cluster holds no data.

The solution is to let the submit async search go through as we already do for get and delete async search. Note that the inner search action will still check that the user can access local indices, and remote indices on the remote cluster, like search always does.
2020-07-03 12:18:07 +02:00
David Kyle f6a0c2c59d
[7.x] Pipeline Inference Aggregation (#58965)
Adds a pipeline aggregation that loads a model and performs inference on the
input aggregation results.
2020-07-03 09:29:04 +01:00
Armin Braun d22dd437f1
Fix Two Common Zero Len Array Instantiations (#58944) (#58993)
Two spots I found in which we commonly instatiate a non-trivial number of zero length arrays.
2020-07-03 09:18:14 +02:00
David Turner df555bb470 Add section on reviews to CONTRIBUTING.md (#57046)
The review phase is an important part of contributing to Elasticsearch, but we
do not mention it in our instructions to contributors. This commit adds some
notes on how contributions are reviewed and describes some common reasons for
rejection.

Co-authored-by: Colin Goodheart-Smithe <colings86@users.noreply.github.com>
2020-07-03 08:03:47 +01:00
Tim Vernum 1133c29ce9
Treat roles as a SortedSet (#58988)
The Saml SP document stored the role mapping in a Set, but this made
the order in XContent inconsistent. This switched it to use a TreeSet.

Resolves: #54733
Backport of: #55201
2020-07-03 13:40:58 +10:00
Nhat Nguyen 65645217bc Handle IOException while checking translog corruption
We can hit an IOException while reading a translog header after corrupting it.

Relates #58866
2020-07-02 22:38:05 -04:00
debadair b769e9143d
[DOCS] Add simulate ref (#58579) (#58987)
* [DOCS] Add simulate ref pages

* Add links & experimental tags

* Fixed simulate index response

* Apply suggestions from code review

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

*Incorporate review feedback.
2020-07-02 19:05:32 -07:00
Tim Brooks dc9e364ff2
Count coordinating and primary bytes as write bytes (#58984)
This is a follow-up to #57573. This commit combines coordinating and
primary bytes under the same "write" bucket. Double accounting is
prevented by only accounting the bytes at either the reroute phase or
the primary phase. TransportBulkAction calls execute directly, so the
operations handler is skipped and the bytes are not double accounted.
2020-07-02 19:48:19 -06:00
DeDe Morton 2c43421208 [DOCS] Change Beats links to refactored getting started docs (#58790) 2020-07-02 17:11:25 -07:00
Mark Vieira 8fca312a3a
Mute WriteMemoryLimitsIT.testWriteBytesAreIncremented 2020-07-02 16:58:23 -07:00
Adam Locke 20d04081ec
[7.x] [DOCS] Add supported ESS settings to ES docs (#57953) (#58981)
* Adding ESS icons to supported ES settings.

* Adding new file for supported ESS settings.

* Adding supported ESS settings for HTTP and disk-based shard allocation.

* Adding more supported settings for ESS.

* Adding descriptions for each Cloud section, plus additional settings.

* Adding new warehouse file for Cloud, plus additional settings.

* Adding node settings for Cloud.

* Adding audit settings for Cloud.

* Resolving merge conflict.

* Adding SAML settings (part 1).

* Adding SAML realm encryption and signing settings.

* Adding SAML SSL settings.

* Adding Kerberos realm settings.

* Adding OpenID Connect Realm settings.

* Adding OpenID Connect SSL settings.

* Resolving leftover Git merge markers.

* Removing Cloud settings page and link to it.

* Add link to mapping source

* Update docs/reference/docs/reindex.asciidoc

* Incorporate edit of HTTP settings

* Remove "cloud" from tag and ID

* Remove "cloud" from tag and update description

* Remove "cloud" from tag and ID

* Change "whitelists" to "specifies"

* Remove "cloud" from end tag

* Removing cloud from IDs and tags.

* Changing link reference to fix build issue.

* Adding index management page for missing settings.

* Removing warehouse file for Cloud and moving settings elsewhere.

* Clarifying true/false usage of http.detailed_errors.enabled.

* Changing underscore to dash in link to fix ci build.
2020-07-02 19:40:45 -04:00
Tim Brooks 9d1bf383d0
Add test assertions to ensure write bytes released (#58970)
This is a follow-up to #57573. This commit ensures that the bytes marked
in WriteMemoryLimits are released by any test using an internal test
cluster.
2020-07-02 17:38:23 -06:00
James Rodewig 84444e1a54
[DOCS] Add data streams to flush API docs (#58950) (#58975) 2020-07-02 17:26:49 -04:00
James Rodewig e64fe15c1f
[DOCS] Add data streams to cluster APIs docs (#58945) (#58974)
Makes existing docs for the cluster health and cluster state APIs aware
of data streams.
2020-07-02 17:20:55 -04:00
Benjamin Trent bd9b3b6116
[ML] fix inference ml-stats-write alias creation (#58947) (#58959)
The check for potentially creating the .ml-stats-write alias should verify that the indices actually exist.

closes #58662
2020-07-02 16:16:42 -04:00
Tim Brooks 1ef2cd7f1a
Add memory tracking to queued write operations (#58957)
Currently we do not track the memory consuming by in-process write
operations.

This commit adds a mechanism to track write operation memory usage.
2020-07-02 14:14:57 -06:00
Jim Ferenczi a4e08acdd1 Fix exists query on unmapped field in query_string (#58804)
Since #55785, exists queries rewrite to MatchNoneQueryBuilder when the field is unmapped.
This change also introduced a bug in the `query_string` query, using an unmapped field
like `_exists_:foo` throws an exception if the field is unmapped. This commit avoids the
exception if the query is built outside of an `ExistsQueryBuilder`.

Closes #58737
2020-07-02 21:52:03 +02:00
Tim Brooks 605e24ed7c
Use `getPortRange` in http server tests (#58794)
Currently we are leaving the settings to default port range in the nio
and netty4 http server test. This has recently led to tests failing due
to what appears to be a port conflict with other processes. This commit
modifies these tests to use the test case helper method to generate port
ranges.

Fixes #58433 and #58296.
2020-07-02 13:21:45 -06:00
Rene Groeschke 0a6e6ef4be
Fix task input declaration when using LazyPropertyMap (#58850) (#58934) 2020-07-02 20:52:31 +02:00
Nhat Nguyen be804b765d
Avoid flipping translog header version (#58866)
An old translog header does not have a checksum. If we flip the header 
version of an empty translog to the older version, then we won't detect
that corruption, and translog will be considered clean as before.

Closes #58671
2020-07-02 14:34:19 -04:00
Tal Levy d516959774
Re-enable support for array-valued geo_shape fields. (#58786) (#58943)
A regression in the mapping code led to geo_shape no longer supporting
array-valued fields. This commit fixes this support and adds an integration
test to make sure this problem does not return!
2020-07-02 11:21:55 -07:00
Ryan Ernst d825d4352c
Eagerly compile condition script at processor creation (#58882)
Ingest script processors were changed to eagerly compile their scripts
when the ingest pipeline is saved, but conditional scripts were missed.
This commit adds eager compilation to ingest conditional scripts, which
will help surface errors before runtime, as well as adds tests for each
case we might encounter between inline and stored script compilation
failures.

closes #58864
2020-07-02 11:10:20 -07:00
David Roberts 2c04685b81
[ML] Ensure config index mappings are up-to-date before updating configs (#58938)
We already had code to ensure the config index mappings were
up-to-date before creating a new config.  However, it's also
possible that an update to a config could add the latest
settings that require the latest mappings to index correctly.

This change checks that the latest config index mappings are
in place in the 3 update actions in the same way as the checks
are done in the 3 put actions.

Backport of #58916
2020-07-02 18:55:19 +01:00
Robin Clarke 567720d970 [DOCS] Added caveat about the number of file realms (#58369) 2020-07-02 10:27:36 -07:00
Dan Hermann c988afdc15
Data stream support for migrations deprecations info API 2020-07-02 11:16:22 -05:00
Przemysław Witek 751e84e4c8
Rename regression evaluation metrics to make the names consistent with loss functions (#58887) (#58927) 2020-07-02 17:35:55 +02:00
Tanguy Leroux 6aa669c8bb Fix SearchableSnapshotDirectoryStatsTests (#58912)
Similar to #58847 but in a different tests. The failure never 
reproduced locally but occurs from time to time on CI.
2020-07-02 16:39:26 +02:00
Lee Hinman e32623ef52
[7.x] Add test for component templates updated after cluster restart (#58883) (#58914)
This commit adds an integration test that component templates used to form a composite template can
still be updated after a cluster restart.

In #58643 an issue arose where mappings were causing problems because of the way we unwrap `_doc` in
template mappings. This was also related to the mappings being merged manually rather than using the
`MapperService` to do the merging. #58643 was fixed in 7.9 and master with the #58521 change, since
mappings now are read and digested by the actual mapper service.

This test passes for 7.x and master, and I intend to open a separate PR including this test for
7.8.1 along with a bug fix for #58643. This test is to ensure we don't have any regression in the
future.
2020-07-02 08:23:34 -06:00
Armin Braun 62152852dc
Cleanup Duplication in Snapshot ITs (#58818) (#58915)
Just a few obvious static cleanups of duplication to push back against the ever increasing complexity of these tests.
2020-07-02 16:00:01 +02:00
James Rodewig 9569375ae7
[DOCS] Add data streams to remove lifecycle policy API (#58777) (#58921) 2020-07-02 09:58:54 -04:00
James Rodewig 04ff24ca8e
[DOCS] Add data streams to deprecation info API docs (#58685) (#58920) 2020-07-02 09:52:33 -04:00
James Rodewig 6436792aac
[DOCS] Fix headings for simple analyzer docs (#58910) (#58918) 2020-07-02 09:52:05 -04:00
James Rodewig ff04212257
[DOCS] Fix snippet tests for resolve API docs (#58908) (#58911) 2020-07-02 09:27:14 -04:00
James Rodewig 57d679ff64
[DOCS] Add data streams to put mapping and update settings API docs (#58849) (#58913) 2020-07-02 09:26:57 -04:00
Dan Hermann b78bfa01f6
[7.x] Data stream support for graph explore API 2020-07-02 08:19:03 -05:00
Dan Hermann 40655069e2
Data stream support for delete-by-query 2020-07-02 08:17:24 -05:00
Dan Hermann fba1047ad9
Data stream support for update by query API 2020-07-02 08:16:05 -05:00
Alan Woodward 0cd1dc3143 Percolator keyword fields should not store norms (#58899)
The refactoring in #57666 inadvertently enabled norms on two of the percolator subfields,
leading to an increase in memory usage. This commit disables norms on these fields again.
2020-07-02 13:59:28 +01:00
James Rodewig fc5b2b2be4 [DOCS] Fix `scroll` param typo 2020-07-02 08:48:07 -04:00