Commit Graph

48000 Commits

Author SHA1 Message Date
Martijn van Groningen 363beefbcd
Change HLRC count request to accept a QueryBuilder (#46904)
Currently the CountRequest accepts a search source builder, while the
RestCountAction only accepts a top level query object. This can lead
to confusion if another element (e.g. aggregations) is specified,
because that will be ignored on the server side in RestCountAction.

By deprecating the current setter & constructor that accept a
SearchSourceBuilder and adding replacement that accepts a QueryBuilder
it is clear what the count api can handle from HLRC side.

Follow up from #46829
2019-09-23 08:43:07 +02:00
Nhat Nguyen e8515d1d13 Force flush in FullClusterRestartIT#testRecovery (#46956)
If peer recovery happens after indexing, and indexing flushes some shard
at the end, then the explicit flush in the test will be a noop. Then
replicas will have some uncommitted translog , which is transferred in
peer recovery, although all of these operations are in the commit
already. If that replica becomes primary (after we restarted the
cluster), it will have translog to replay and the test will fail.

Another issue in this test is that synced_flush is not a replication
action, then the global checkpoint on replicas might be not up to date.
We need to either wait for the global checkpoint to be synced or call a
replication action to sync it.

Closes #46712
2019-09-22 19:04:01 -04:00
Armin Braun ee4e6b1382
Add TestLogging for #46701 (#46939) (#46949)
This at a very low rate and with the force merge in place
before checking the cache size it's not clear why the cache
is not of size `0` -> seems something else
must be happening here that is unexpected.
-> add debug logging to this test to find out

Relates #46701
2019-09-21 15:24:58 +02:00
Lisa Cawley 875d864be6
[DOCS] Update data frame transform URLs (#46940) (#46946) 2019-09-20 15:57:43 -07:00
Mark Vieira f51b2592a4
Use composition instead of inheritance for extending Gradle plugins (#46888) 2019-09-20 14:51:36 -07:00
Javier Ruiz a5661ac03a [DOCS] Fix calendar interval typos for date histo agg (#46911) 2019-09-20 15:22:41 -04:00
Lisa Cawley 4c688df0ef [DOCS] Add security section to Elasticsearch book (#46883)
Co-Authored-By: Tim Vernum <tim@adjective.org>
2019-09-20 11:34:37 -07:00
James Rodewig 99130114de
[DOCS] Correct several [source,console-result] snippets (#46930) (#46937) 2019-09-20 12:20:12 -04:00
Alpar Torok 0294b8fc60 Fix SymbolicLinkPreservingTarIT on windows (#46908)
Broken path comparison on windows:
 https://scans.gradle.com/s/7y3zl23orfp7w/tests/nr3ff3qzwqzsw-xfh7lvi5orvq4
With the fix:
 https://gradle-enterprise.elastic.co/s/4dcc2nv3haaqw
2019-09-20 17:32:14 +03:00
Andrei Dan 402fb5e882
Fix flaky test addSnapshotLifecyclePolicy (#46881) (#46912)
* addSnapshotLifecyclePolicy drop version assertion

This drops the assertion on the policy version (which was pinned to 1L)
as we want to execute both put policy apis (sync and async) for
documentation purposes. This will sometimes (depending on the async
call) yield a version of 2L. Waiting for the async call to always
complete could be an option but the test is already rather slow and it's
a bit of an overkill as we're already verifying the policy was created.

(cherry picked from commit af4864c39129bcdbf98d00223f445346a62075e4)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2019-09-20 15:07:54 +01:00
Armin Braun 938648fcff
Remove Duplicate Shard Snapshot State Updates (#46862) (#46906)
We were repeatedly trying to send shard state updates for aborted
snapshots on every cluster state update.
This is simply dead-code since those updates are already safely
sent in the callbacks passed to `SnapshotShardsService#snapshot`.
On master failover, we ensure that the status update is resent
via `SnapshotShardsService#syncShardStatsOnNewMaster`.
=> there is no need for trying to send updates here over and over
and this logic can safely be removed
2019-09-20 14:30:03 +02:00
Hendrik Muhs 4a2cb05162 add message about transform disabled if license is missing (#46901)
adds a message for transform about what happens if no license has been activated
2019-09-20 13:47:40 +02:00
Jason Tedor 97acf353fa
Move pipelines resolved assertion (#46892)
This assertion was added during the development of required
pipelines. In the initial version of that work, the notion of whether or
not a request was forwarded from the coordinating node to an ingest node
was introduced. It was realized later that instead we needed to track
whether or not the pipeline for the request was resolved. When that
change was made, this assertion, while not incorrect, was left behind
and only applied if the coordnating node was forwarding the
request. Instead, the assertion applies whether or not the request is
being forwarded. This commit addresses that by moving the assertion and
renaming some variables.
2019-09-20 07:27:56 -04:00
István Zoltán Szabó fcea154f2e [DOCS] Reformats Field capabilities API (#46866)
* [DOCS] Reformats Field capabilities API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-20 11:28:19 +02:00
István Zoltán Szabó 363075cf1d [DOCS] Reformats explain API (#46857)
* [DOCS] Reformats explain API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-20 11:00:33 +02:00
Hendrik Muhs abe889af75
[7.5][Transform] rename classes in transform plugin (#46867)
rename classes and settings in transform plugin, provide BWC for old settings
2019-09-20 10:43:00 +02:00
Martijn van Groningen d82712417f
[HLRC] Send min_score as query string parameter to the count API (#46829)
Prior to this commit min_score was sent as request body parameter
(via SearchSourceBuilder), which is not possible in the count api.

Similar to #46474
2019-09-20 10:23:45 +02:00
Tanguy Leroux add7148f3b GCS deleteBlobsIgnoringIfNotExists should catch StorageException (#46832)
GoogleCloudStorageBlobStore.deleteBlobsIgnoringIfNotExists() does
not correctly catch StorageException thrown by batch.submit().

In the case a snapshot is deleted through BlobStoreRepository.deleteSnapshot()
a storage exception is not caught (only IOException are) so the deletion is
interrupted and indices cannot be cleaned up. The storage exception bubbles
up to SnapshotService.deleteSnapshotFromRepository() but the listener that
 removes the deletion from the cluster state is not executed, leaving the
deletion in the cluster state.

This bug has been reported in #46772 where batch.submit() threw an
exception in the test testIndicesDeletedFromRepository and following
tests failed because a snapshot deletion was running.

Relates #46772
2019-09-20 10:02:23 +02:00
Colin Goodheart-Smithe 74d1588ec5
Updates release notes for 7.4.0 2019-09-20 08:10:47 +01:00
Julie Tibshirani 6ce4825594
Add vector function changes to 7.4 migration docs. (#46893)
This note was accidentally omitted in the breaking change #46004.
2019-09-19 20:26:52 -07:00
Jason Tedor 2425fd1a50
Removed unused import from RequiredPipelineIT.java
This commit removes an unused import that was left behind after cleaning
up a backport. Sorry.
2019-09-19 16:46:27 -04:00
rikardbakkehaug 6b00e00bc6 [DOCS] Correct date math ex for `gt` and `gte` parms in `range` query docs (#46873) 2019-09-19 16:39:03 -04:00
Jason Tedor bd77626177
Add the ability to require an ingest pipeline (#46847)
This commit adds the ability to require an ingest pipeline on an
index. Today we can have a default pipeline, but that could be
overridden by a request pipeline parameter. This commit introduces a new
index setting index.required_pipeline that acts similarly to
index.default_pipeline, except that it can not be overridden by a
request pipeline parameter. Additionally, a default pipeline and a
request pipeline can not both be set. The required pipeline can be set
to _none to ensure that no pipeline ever runs for index requests on that
index.
2019-09-19 16:37:45 -04:00
James Rodewig 251dbd8522 [DOCS] Remove `lowercase_terms` parm from term suggester docs (#46879) 2019-09-19 15:56:47 -04:00
James Rodewig 8307aa3b71 [DOCS] Update tagged region for index alias desc (#46886) 2019-09-19 15:30:33 -04:00
Armin Braun c922743c5d
Remove Bogus Test: testDeleteOrphanSnapshot (#46835) (#46874)
This test is broken with a very low failure rate after recent changes.
Particularly after #45689 which does not check for duplicate snapshot
uuids during snapshot finalization any more.
The check for duplicate uuids during finalization was removed
conciously since it lead to problems during master failover.
This test fails because it increments the repository state id in an
unexpected manner now, starting from the impossible situation of having
the same snapshot UUID for two different repository state ids.
This situation can't normally be reached, but we manually crafted it here.

This test didn't do anything before though, because the manually crafted
cluster state would simply result in an error during finalization before
and nothing but a normal snapshot delete would be tested.
=> removing this test here, it doesn't test anything.

Closes #46843
2019-09-19 18:52:35 +02:00
Yannick Welsch 9638ca20b0 Allow dropping documents with auto-generated ID (#46773)
When using auto-generated IDs + the ingest drop processor (which looks to be used by filebeat
as well) + coordinating nodes that do not have the ingest processor functionality, this can lead
to a NullPointerException.

The issue is that markCurrentItemAsDropped() is creating an UpdateResponse with no id when
the request contains auto-generated IDs. The response serialization is lenient for our
REST/XContent format (i.e. we will send "id" : null) but the internal transport format (used for
communication between nodes) assumes for this field to be non-null, which means that it can't
be serialized between nodes. Bulk requests with ingest functionality are processed on the
coordinating node if the node has the ingest capability, and only otherwise sent to a different
node. This means that, in order to reproduce this, one needs two nodes, with the coordinating
node not having the ingest functionality.

Closes #46678
2019-09-19 16:46:33 +02:00
Takumasa Ochi 7a3054c5dc Fix typos in `match` in profile API (#46723)
* Replace `matches` with correct `match`
* Use present tense consistently
* Replace `metric` with correct `match`
2019-09-19 16:07:52 +02:00
Armin Braun b32031ee03
Fix snapshot_restore_exception in Docs Tests (#46852) (#46870)
We should only snapshot the index we're going to
restore in the next step. Otherwise, we will
potentially not get the correct response or
fail restoring outright due to internal indices
getting created concurrently when running against
the x-pack distribution.

Closes #46844
2019-09-19 16:05:51 +02:00
Jason Tedor 4a9943a906
Bundle AdoptOpenJDK 13 (#46860)
This commit switches to use AdoptOpenJDK 13 for bundling the JDK,
instead of Oracle OpenJDK 13.
2019-09-19 08:56:17 -04:00
Russ Cam e467dddd4c Use snake casing for document field (#45432)
This commit updates the search example to use snake casing for the document field. This is consistent with other examples
2019-09-19 14:27:00 +02:00
Martijn Laarman f1de8c0193 Document PUT as the prefered HTTP method for index API (#45622)
Follow up from #42346. Since the `methods` array is in order of
preference when calling the index API with an `{id}` we prefer to use
the `PUT` http method.
2019-09-19 14:24:22 +02:00
James Rodewig b73a9604c1
[DOCS] Separate and reformat synced flush API docs (#46634) (#46839) 2019-09-19 08:22:37 -04:00
LHearen e555ef2e86 [DOCS] make bulk page more readable (#46200)
comma and semi-colon required to make it more readable
2019-09-19 14:20:41 +02:00
maidoo 0f2e72b7f8 [DOCS] Term dict memory usage is a 7.3.0 enhancement, not 7.2.0 (#46662)
Closes #46635
2019-09-19 14:15:12 +02:00
Toby McLaughlin 7795e21ea3 [DOCS] Fix typo in GCS snapshot repo doc (#46048) 2019-09-19 14:06:37 +02:00
Armin Braun 6b09c2cdbb
Limit Netty Workers in NativeRealmIntegTestCase (#46816) (#46850)
The fact that this test randomly uses a relatively large number
of nodes and hence Netty worker threads created a problem with
running out of direct memory on CI.
Tests run with 512M heap (and hence 512M direct memory) by default.
On a CI worker with 16 cores, this means Netty will by default set
up 32 transport workers. If we get unlucky and a lot of them
actually do work (and thus instantiate a `CopyBytesSocketChannel`
which costs 1M per thread for the thread-local IO buffer) we
would run out of memory.

This specific failure was only seen with `NativeRealmIntegTests` so I
only added the constraint on the Netty worker count here.
We can add it to other tests (or `SecurityIntegTestCase`) if need be
but for now it doesn't seem necessary so I opted for least impact.

Closes #46803
2019-09-19 13:07:42 +02:00
Armin Braun a087553009
Rearrange BlobStoreRepository to Prepare #46250 (#46824) (#46853)
In #46250 we will have to move to two different delete
paths for BwC. Both paths will share the initial
reads from the repository though so I extracted/separated the
delete logic away from the initial reads to significantly simplify
the diff against #46250 here.
Also, I added some JavaDoc from #46250 here as well which makes the
code a little easier to follow even ignoring #46250 I think.
2019-09-19 13:07:00 +02:00
Luca Cavanna a6c9f8ffae [DOCS] mention automatic cancellation in search docs (#46706)
Add a small note around automatic search cancellation which was missed as part of #43332

Relates to #43332
2019-09-19 09:35:30 +02:00
István Zoltán Szabó 3be51fbdf7 [DOCS] Adds regression analytics resources and examples to the data frame analytics APIs and the evaluation API (#46176)
* [DOCS] Adds regression analytics resources and examples to the data frame analytics APIs.
Co-Authored-By: Benjamin Trent <ben.w.trent@gmail.com>
Co-Authored-By: Tom Veasey <tveasey@users.noreply.github.com>
2019-09-19 09:23:18 +02:00
István Zoltán Szabó 65fffcc9c1 [DOCS]Changes failed transforms limitation. (#46754) 2019-09-19 09:04:41 +02:00
Alpar Torok 86d216cc9c Update .ci/bwcVersions 2019-09-19 09:35:01 +03:00
Lisa Cawley a32fe1618c [DOCS] Updates transform landing page (#46689) 2019-09-18 14:00:43 -07:00
Mark Vieira 507d879b70
Disable bwc distribution caching in master branch (#46686)
This commit disables caching of BWC snapshot distributions in the "trunk" (aka master) branch.
Since the previous major release branches move quickly we rarely get cache hits for these 
tasks, and the artifacts themselves are very large. This means the overhead here is high and
savings basically zero. We conditionally disable task output caching in this scenario in CI to 
avoid excessive build cache overhead as well as causing too much turn in the cache itself which
would lead to lots of cache entry evictions.
2019-09-18 11:52:17 -07:00
Dimitris Athanasiou 02a5e153dc
[7.x][ML] Parse and index data frame analytics state (#46804) (#46820)
This commit reuses the same state processor that is used for autodetect
to parse state output from data frame analytics jobs. We then index the
state document into the state index.

Backport of #46804
2019-09-18 20:37:40 +03:00
James Rodewig b08b3741db [DOCS] Use versions file from elastic/docs (#46719) 2019-09-18 11:18:14 -04:00
James Rodewig d00bad9ab2 [DOCS] Reformat force merge API docs (#46685) 2019-09-18 10:10:51 -04:00
Benjamin Trent 9cf9c64ec2
[7.x] [ML][Transforms] remove `force` flag from _start (#46414) (#46748)
* [ML][Transforms] remove `force` flag from _start (#46414)

* [ML][Transforms] remove `force` flag from _start

* fixing expected error message

* adjusting bwc version
2019-09-18 10:06:05 -04:00
Turaç Kangal d1f47cf00e
[HLRC] Send terminate_after as query string parameter to the count API (#46474)
Prior to this commit terminate_after was sent as request body parameter
(via SearchSourceBuilder), which is not possible in the count api.

Closes #46446
2019-09-18 15:54:43 +02:00
James Rodewig 6863b90272 [DOCS] Remove leveloffset for CCR docs (#46818) 2019-09-18 09:44:43 -04:00